项目初始化
This commit is contained in:
98
.agents/skills/brand-voice/SKILL.md
Normal file
98
.agents/skills/brand-voice/SKILL.md
Normal file
@@ -0,0 +1,98 @@
|
||||
---
|
||||
name: brand-voice
|
||||
description: Build a source-derived writing style profile from real posts, essays, launch notes, docs, or site copy, then reuse that profile across content, outreach, and social workflows. Use when the user wants voice consistency without generic AI writing tropes.
|
||||
metadata:
|
||||
origin: ECC
|
||||
---
|
||||
|
||||
# Brand Voice
|
||||
|
||||
Build a durable voice profile from real source material, then use that profile everywhere instead of re-deriving style from scratch or defaulting to generic AI copy.
|
||||
|
||||
## When to Activate
|
||||
|
||||
- the user wants content or outreach in a specific voice
|
||||
- writing for X, LinkedIn, email, launch posts, threads, or product updates
|
||||
- adapting a known author's tone across channels
|
||||
- the existing content lane needs a reusable style system instead of one-off mimicry
|
||||
|
||||
## Source Priority
|
||||
|
||||
Use the strongest real source set available, in this order:
|
||||
|
||||
1. recent original X posts and threads
|
||||
2. articles, essays, memos, launch notes, or newsletters
|
||||
3. real outbound emails or DMs that worked
|
||||
4. product docs, changelogs, README framing, and site copy
|
||||
|
||||
Do not use generic platform exemplars as source material.
|
||||
|
||||
## Collection Workflow
|
||||
|
||||
1. Gather 5 to 20 representative samples when available.
|
||||
2. Prefer recent material over old material unless the user says the older writing is more canonical.
|
||||
3. Separate "public launch voice" from "private working voice" if the source set clearly splits.
|
||||
4. If live X access is available, use `x-api` to pull recent original posts before drafting.
|
||||
5. If site copy matters, include the current ECC landing page and repo/plugin framing.
|
||||
|
||||
## What to Extract
|
||||
|
||||
- rhythm and sentence length
|
||||
- compression vs explanation
|
||||
- capitalization norms
|
||||
- parenthetical use
|
||||
- question frequency and purpose
|
||||
- how sharply claims are made
|
||||
- how often numbers, mechanisms, or receipts show up
|
||||
- how transitions work
|
||||
- what the author never does
|
||||
|
||||
## Output Contract
|
||||
|
||||
Produce a reusable `VOICE PROFILE` block that downstream skills can consume directly. Use the schema in [references/voice-profile-schema.md](references/voice-profile-schema.md).
|
||||
|
||||
Keep the profile structured and short enough to reuse in session context. The point is not literary criticism. The point is operational reuse.
|
||||
|
||||
## Affaan / ECC Defaults
|
||||
|
||||
If the user wants Affaan / ECC voice and live sources are thin, start here unless newer source material overrides it:
|
||||
|
||||
- direct, compressed, concrete
|
||||
- specifics, mechanisms, receipts, and numbers beat adjectives
|
||||
- parentheticals are for qualification, narrowing, or over-clarification
|
||||
- capitalization is conventional unless there is a real reason to break it
|
||||
- questions are rare and should not be used as bait
|
||||
- tone can be sharp, blunt, skeptical, or dry
|
||||
- transitions should feel earned, not smoothed over
|
||||
|
||||
## Hard Bans
|
||||
|
||||
Delete and rewrite any of these:
|
||||
|
||||
- fake curiosity hooks
|
||||
- "not X, just Y"
|
||||
- "no fluff"
|
||||
- forced lowercase
|
||||
- LinkedIn thought-leader cadence
|
||||
- bait questions
|
||||
- "Excited to share"
|
||||
- generic founder-journey filler
|
||||
- corny parentheticals
|
||||
|
||||
## Persistence Rules
|
||||
|
||||
- Reuse the latest confirmed `VOICE PROFILE` across related tasks in the same session.
|
||||
- If the user asks for a durable artifact, save the profile in the requested workspace location or memory surface.
|
||||
- Do not create repo-tracked files that store personal voice fingerprints unless the user explicitly asks for that.
|
||||
|
||||
## Downstream Use
|
||||
|
||||
Use this skill before or inside:
|
||||
|
||||
- `content-engine`
|
||||
- `crosspost`
|
||||
- `lead-intelligence`
|
||||
- article or launch writing
|
||||
- cold or warm outbound across X, LinkedIn, and email
|
||||
|
||||
If another skill already has a partial voice capture section, this skill is the canonical source of truth.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Voice Profile Schema
|
||||
|
||||
Use this exact structure when building a reusable voice profile:
|
||||
|
||||
```text
|
||||
VOICE PROFILE
|
||||
=============
|
||||
Author:
|
||||
Goal:
|
||||
Confidence:
|
||||
|
||||
Source Set
|
||||
- source 1
|
||||
- source 2
|
||||
- source 3
|
||||
|
||||
Rhythm
|
||||
- short note on sentence length, pacing, and fragmentation
|
||||
|
||||
Compression
|
||||
- how dense or explanatory the writing is
|
||||
|
||||
Capitalization
|
||||
- conventional, mixed, or situational
|
||||
|
||||
Parentheticals
|
||||
- how they are used and how they are not used
|
||||
|
||||
Question Use
|
||||
- rare, frequent, rhetorical, direct, or mostly absent
|
||||
|
||||
Claim Style
|
||||
- how claims are framed, supported, and sharpened
|
||||
|
||||
Preferred Moves
|
||||
- concrete moves the author does use
|
||||
|
||||
Banned Moves
|
||||
- specific patterns the author does not use
|
||||
|
||||
CTA Rules
|
||||
- how, when, or whether to close with asks
|
||||
|
||||
Channel Notes
|
||||
- X:
|
||||
- LinkedIn:
|
||||
- Email:
|
||||
```
|
||||
|
||||
Guidelines:
|
||||
|
||||
- Keep the profile concrete and source-backed.
|
||||
- Use short bullets, not essay paragraphs.
|
||||
- Every banned move should be observable in the source set or explicitly requested by the user.
|
||||
- If the source set conflicts, call out the split instead of averaging it into mush.
|
||||
751
.agents/skills/python-patterns/SKILL.md
Normal file
751
.agents/skills/python-patterns/SKILL.md
Normal file
@@ -0,0 +1,751 @@
|
||||
---
|
||||
name: python-patterns
|
||||
description: Pythonic idioms, PEP 8 standards, type hints, and best practices for building robust, efficient, and maintainable Python applications.
|
||||
metadata:
|
||||
origin: ECC
|
||||
---
|
||||
|
||||
# Python Development Patterns
|
||||
|
||||
Idiomatic Python patterns and best practices for building robust, efficient, and maintainable applications.
|
||||
|
||||
## When to Activate
|
||||
|
||||
- Writing new Python code
|
||||
- Reviewing Python code
|
||||
- Refactoring existing Python code
|
||||
- Designing Python packages/modules
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Readability Counts
|
||||
|
||||
Python prioritizes readability. Code should be obvious and easy to understand.
|
||||
|
||||
```python
|
||||
# Good: Clear and readable
|
||||
def get_active_users(users: list[User]) -> list[User]:
|
||||
"""Return only active users from the provided list."""
|
||||
return [user for user in users if user.is_active]
|
||||
|
||||
|
||||
# Bad: Clever but confusing
|
||||
def get_active_users(u):
|
||||
return [x for x in u if x.a]
|
||||
```
|
||||
|
||||
### 2. Explicit is Better Than Implicit
|
||||
|
||||
Avoid magic; be clear about what your code does.
|
||||
|
||||
```python
|
||||
# Good: Explicit configuration
|
||||
import logging
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
|
||||
# Bad: Hidden side effects
|
||||
import some_module
|
||||
some_module.setup() # What does this do?
|
||||
```
|
||||
|
||||
### 3. EAFP - Easier to Ask Forgiveness Than Permission
|
||||
|
||||
Python prefers exception handling over checking conditions.
|
||||
|
||||
```python
|
||||
# Good: EAFP style
|
||||
def get_value(dictionary: dict, key: str, default_value: Any = None) -> Any:
|
||||
try:
|
||||
return dictionary[key]
|
||||
except KeyError:
|
||||
return default_value
|
||||
|
||||
# Bad: LBYL (Look Before You Leap) style
|
||||
def get_value(dictionary: dict, key: str, default_value: Any = None) -> Any:
|
||||
if key in dictionary:
|
||||
return dictionary[key]
|
||||
else:
|
||||
return default_value
|
||||
```
|
||||
|
||||
## Type Hints
|
||||
|
||||
### Basic Type Annotations
|
||||
|
||||
```python
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
def process_user(
|
||||
user_id: str,
|
||||
data: Dict[str, Any],
|
||||
active: bool = True
|
||||
) -> Optional[User]:
|
||||
"""Process a user and return the updated User or None."""
|
||||
if not active:
|
||||
return None
|
||||
return User(user_id, data)
|
||||
```
|
||||
|
||||
### Modern Type Hints (Python 3.9+)
|
||||
|
||||
```python
|
||||
# Python 3.9+ - Use built-in types
|
||||
def process_items(items: list[str]) -> dict[str, int]:
|
||||
return {item: len(item) for item in items}
|
||||
|
||||
# Python 3.8 and earlier - Use typing module
|
||||
from typing import List, Dict
|
||||
|
||||
def process_items(items: List[str]) -> Dict[str, int]:
|
||||
return {item: len(item) for item in items}
|
||||
```
|
||||
|
||||
### Type Aliases and TypeVar
|
||||
|
||||
```python
|
||||
from typing import TypeVar, Union
|
||||
|
||||
# Type alias for complex types
|
||||
JSON = Union[dict[str, Any], list[Any], str, int, float, bool, None]
|
||||
|
||||
def parse_json(data: str) -> JSON:
|
||||
return json.loads(data)
|
||||
|
||||
# Generic types
|
||||
T = TypeVar('T')
|
||||
|
||||
def first(items: list[T]) -> T | None:
|
||||
"""Return the first item or None if list is empty."""
|
||||
return items[0] if items else None
|
||||
```
|
||||
|
||||
### Protocol-Based Duck Typing
|
||||
|
||||
```python
|
||||
from typing import Protocol
|
||||
|
||||
class Renderable(Protocol):
|
||||
def render(self) -> str:
|
||||
"""Render the object to a string."""
|
||||
|
||||
def render_all(items: list[Renderable]) -> str:
|
||||
"""Render all items that implement the Renderable protocol."""
|
||||
return "\n".join(item.render() for item in items)
|
||||
```
|
||||
|
||||
## Error Handling Patterns
|
||||
|
||||
### Specific Exception Handling
|
||||
|
||||
```python
|
||||
# Good: Catch specific exceptions
|
||||
def load_config(path: str) -> Config:
|
||||
try:
|
||||
with open(path) as f:
|
||||
return Config.from_json(f.read())
|
||||
except FileNotFoundError as e:
|
||||
raise ConfigError(f"Config file not found: {path}") from e
|
||||
except json.JSONDecodeError as e:
|
||||
raise ConfigError(f"Invalid JSON in config: {path}") from e
|
||||
|
||||
# Bad: Bare except
|
||||
def load_config(path: str) -> Config:
|
||||
try:
|
||||
with open(path) as f:
|
||||
return Config.from_json(f.read())
|
||||
except:
|
||||
return None # Silent failure!
|
||||
```
|
||||
|
||||
### Exception Chaining
|
||||
|
||||
```python
|
||||
def process_data(data: str) -> Result:
|
||||
try:
|
||||
parsed = json.loads(data)
|
||||
except json.JSONDecodeError as e:
|
||||
# Chain exceptions to preserve the traceback
|
||||
raise ValueError(f"Failed to parse data: {data}") from e
|
||||
```
|
||||
|
||||
### Custom Exception Hierarchy
|
||||
|
||||
```python
|
||||
class AppError(Exception):
|
||||
"""Base exception for all application errors."""
|
||||
pass
|
||||
|
||||
class ValidationError(AppError):
|
||||
"""Raised when input validation fails."""
|
||||
pass
|
||||
|
||||
class NotFoundError(AppError):
|
||||
"""Raised when a requested resource is not found."""
|
||||
pass
|
||||
|
||||
# Usage
|
||||
def get_user(user_id: str) -> User:
|
||||
user = db.find_user(user_id)
|
||||
if not user:
|
||||
raise NotFoundError(f"User not found: {user_id}")
|
||||
return user
|
||||
```
|
||||
|
||||
## Context Managers
|
||||
|
||||
### Resource Management
|
||||
|
||||
```python
|
||||
# Good: Using context managers
|
||||
def process_file(path: str) -> str:
|
||||
with open(path, 'r') as f:
|
||||
return f.read()
|
||||
|
||||
# Bad: Manual resource management
|
||||
def process_file(path: str) -> str:
|
||||
f = open(path, 'r')
|
||||
try:
|
||||
return f.read()
|
||||
finally:
|
||||
f.close()
|
||||
```
|
||||
|
||||
### Custom Context Managers
|
||||
|
||||
```python
|
||||
from contextlib import contextmanager
|
||||
|
||||
@contextmanager
|
||||
def timer(name: str):
|
||||
"""Context manager to time a block of code."""
|
||||
start = time.perf_counter()
|
||||
yield
|
||||
elapsed = time.perf_counter() - start
|
||||
print(f"{name} took {elapsed:.4f} seconds")
|
||||
|
||||
# Usage
|
||||
with timer("data processing"):
|
||||
process_large_dataset()
|
||||
```
|
||||
|
||||
### Context Manager Classes
|
||||
|
||||
```python
|
||||
class DatabaseTransaction:
|
||||
def __init__(self, connection):
|
||||
self.connection = connection
|
||||
|
||||
def __enter__(self):
|
||||
self.connection.begin_transaction()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
if exc_type is None:
|
||||
self.connection.commit()
|
||||
else:
|
||||
self.connection.rollback()
|
||||
return False # Don't suppress exceptions
|
||||
|
||||
# Usage
|
||||
with DatabaseTransaction(conn):
|
||||
user = conn.create_user(user_data)
|
||||
conn.create_profile(user.id, profile_data)
|
||||
```
|
||||
|
||||
## Comprehensions and Generators
|
||||
|
||||
### List Comprehensions
|
||||
|
||||
```python
|
||||
# Good: List comprehension for simple transformations
|
||||
names = [user.name for user in users if user.is_active]
|
||||
|
||||
# Bad: Manual loop
|
||||
names = []
|
||||
for user in users:
|
||||
if user.is_active:
|
||||
names.append(user.name)
|
||||
|
||||
# Complex comprehensions should be expanded
|
||||
# Bad: Too complex
|
||||
result = [x * 2 for x in items if x > 0 if x % 2 == 0]
|
||||
|
||||
# Good: Use a generator function
|
||||
def filter_and_transform(items: Iterable[int]) -> list[int]:
|
||||
result = []
|
||||
for x in items:
|
||||
if x > 0 and x % 2 == 0:
|
||||
result.append(x * 2)
|
||||
return result
|
||||
```
|
||||
|
||||
### Generator Expressions
|
||||
|
||||
```python
|
||||
# Good: Generator for lazy evaluation
|
||||
total = sum(x * x for x in range(1_000_000))
|
||||
|
||||
# Bad: Creates large intermediate list
|
||||
total = sum([x * x for x in range(1_000_000)])
|
||||
```
|
||||
|
||||
### Generator Functions
|
||||
|
||||
```python
|
||||
def read_large_file(path: str) -> Iterator[str]:
|
||||
"""Read a large file line by line."""
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
yield line.strip()
|
||||
|
||||
# Usage
|
||||
for line in read_large_file("huge.txt"):
|
||||
process(line)
|
||||
```
|
||||
|
||||
## Data Classes and Named Tuples
|
||||
|
||||
### Data Classes
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
|
||||
@dataclass
|
||||
class User:
|
||||
"""User entity with automatic __init__, __repr__, and __eq__."""
|
||||
id: str
|
||||
name: str
|
||||
email: str
|
||||
created_at: datetime = field(default_factory=datetime.now)
|
||||
is_active: bool = True
|
||||
|
||||
# Usage
|
||||
user = User(
|
||||
id="123",
|
||||
name="Alice",
|
||||
email="alice@example.com"
|
||||
)
|
||||
```
|
||||
|
||||
### Data Classes with Validation
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class User:
|
||||
email: str
|
||||
age: int
|
||||
|
||||
def __post_init__(self):
|
||||
# Validate email format
|
||||
if "@" not in self.email:
|
||||
raise ValueError(f"Invalid email: {self.email}")
|
||||
# Validate age range
|
||||
if self.age < 0 or self.age > 150:
|
||||
raise ValueError(f"Invalid age: {self.age}")
|
||||
```
|
||||
|
||||
### Named Tuples
|
||||
|
||||
```python
|
||||
from typing import NamedTuple
|
||||
|
||||
class Point(NamedTuple):
|
||||
"""Immutable 2D point."""
|
||||
x: float
|
||||
y: float
|
||||
|
||||
def distance(self, other: 'Point') -> float:
|
||||
return ((self.x - other.x) ** 2 + (self.y - other.y) ** 2) ** 0.5
|
||||
|
||||
# Usage
|
||||
p1 = Point(0, 0)
|
||||
p2 = Point(3, 4)
|
||||
print(p1.distance(p2)) # 5.0
|
||||
```
|
||||
|
||||
## Decorators
|
||||
|
||||
### Function Decorators
|
||||
|
||||
```python
|
||||
import functools
|
||||
import time
|
||||
|
||||
def timer(func: Callable) -> Callable:
|
||||
"""Decorator to time function execution."""
|
||||
@functools.wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
start = time.perf_counter()
|
||||
result = func(*args, **kwargs)
|
||||
elapsed = time.perf_counter() - start
|
||||
print(f"{func.__name__} took {elapsed:.4f}s")
|
||||
return result
|
||||
return wrapper
|
||||
|
||||
@timer
|
||||
def slow_function():
|
||||
time.sleep(1)
|
||||
|
||||
# slow_function() prints: slow_function took 1.0012s
|
||||
```
|
||||
|
||||
### Parameterized Decorators
|
||||
|
||||
```python
|
||||
def repeat(times: int):
|
||||
"""Decorator to repeat a function multiple times."""
|
||||
def decorator(func: Callable) -> Callable:
|
||||
@functools.wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
results = []
|
||||
for _ in range(times):
|
||||
results.append(func(*args, **kwargs))
|
||||
return results
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
@repeat(times=3)
|
||||
def greet(name: str) -> str:
|
||||
return f"Hello, {name}!"
|
||||
|
||||
# greet("Alice") returns ["Hello, Alice!", "Hello, Alice!", "Hello, Alice!"]
|
||||
```
|
||||
|
||||
### Class-Based Decorators
|
||||
|
||||
```python
|
||||
class CountCalls:
|
||||
"""Decorator that counts how many times a function is called."""
|
||||
def __init__(self, func: Callable):
|
||||
functools.update_wrapper(self, func)
|
||||
self.func = func
|
||||
self.count = 0
|
||||
|
||||
def __call__(self, *args, **kwargs):
|
||||
self.count += 1
|
||||
print(f"{self.func.__name__} has been called {self.count} times")
|
||||
return self.func(*args, **kwargs)
|
||||
|
||||
@CountCalls
|
||||
def process():
|
||||
pass
|
||||
|
||||
# Each call to process() prints the call count
|
||||
```
|
||||
|
||||
## Concurrency Patterns
|
||||
|
||||
### Threading for I/O-Bound Tasks
|
||||
|
||||
```python
|
||||
import concurrent.futures
|
||||
import threading
|
||||
|
||||
def fetch_url(url: str) -> str:
|
||||
"""Fetch a URL (I/O-bound operation)."""
|
||||
import urllib.request
|
||||
with urllib.request.urlopen(url) as response:
|
||||
return response.read().decode()
|
||||
|
||||
def fetch_all_urls(urls: list[str]) -> dict[str, str]:
|
||||
"""Fetch multiple URLs concurrently using threads."""
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
|
||||
future_to_url = {executor.submit(fetch_url, url): url for url in urls}
|
||||
results = {}
|
||||
for future in concurrent.futures.as_completed(future_to_url):
|
||||
url = future_to_url[future]
|
||||
try:
|
||||
results[url] = future.result()
|
||||
except Exception as e:
|
||||
results[url] = f"Error: {e}"
|
||||
return results
|
||||
```
|
||||
|
||||
### Multiprocessing for CPU-Bound Tasks
|
||||
|
||||
```python
|
||||
def process_data(data: list[int]) -> int:
|
||||
"""CPU-intensive computation."""
|
||||
return sum(x ** 2 for x in data)
|
||||
|
||||
def process_all(datasets: list[list[int]]) -> list[int]:
|
||||
"""Process multiple datasets using multiple processes."""
|
||||
with concurrent.futures.ProcessPoolExecutor() as executor:
|
||||
results = list(executor.map(process_data, datasets))
|
||||
return results
|
||||
```
|
||||
|
||||
### Async/Await for Concurrent I/O
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def fetch_async(url: str) -> str:
|
||||
"""Fetch a URL asynchronously."""
|
||||
import aiohttp
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url) as response:
|
||||
return await response.text()
|
||||
|
||||
async def fetch_all(urls: list[str]) -> dict[str, str]:
|
||||
"""Fetch multiple URLs concurrently."""
|
||||
tasks = [fetch_async(url) for url in urls]
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
return dict(zip(urls, results))
|
||||
```
|
||||
|
||||
## Package Organization
|
||||
|
||||
### Standard Project Layout
|
||||
|
||||
```
|
||||
myproject/
|
||||
├── src/
|
||||
│ └── mypackage/
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py
|
||||
│ ├── api/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── routes.py
|
||||
│ ├── models/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── user.py
|
||||
│ └── utils/
|
||||
│ ├── __init__.py
|
||||
│ └── helpers.py
|
||||
├── tests/
|
||||
│ ├── __init__.py
|
||||
│ ├── conftest.py
|
||||
│ ├── test_api.py
|
||||
│ └── test_models.py
|
||||
├── pyproject.toml
|
||||
├── README.md
|
||||
└── .gitignore
|
||||
```
|
||||
|
||||
### Import Conventions
|
||||
|
||||
```python
|
||||
# Good: Import order - stdlib, third-party, local
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from fastapi import FastAPI
|
||||
|
||||
from mypackage.models import User
|
||||
from mypackage.utils import format_name
|
||||
|
||||
# Good: Use isort for automatic import sorting
|
||||
# pip install isort
|
||||
```
|
||||
|
||||
### __init__.py for Package Exports
|
||||
|
||||
```python
|
||||
# mypackage/__init__.py
|
||||
"""mypackage - A sample Python package."""
|
||||
|
||||
__version__ = "1.0.0"
|
||||
|
||||
# Export main classes/functions at package level
|
||||
from mypackage.models import User, Post
|
||||
from mypackage.utils import format_name
|
||||
|
||||
__all__ = ["User", "Post", "format_name"]
|
||||
```
|
||||
|
||||
## Memory and Performance
|
||||
|
||||
### Using __slots__ for Memory Efficiency
|
||||
|
||||
```python
|
||||
# Bad: Regular class uses __dict__ (more memory)
|
||||
class Point:
|
||||
def __init__(self, x: float, y: float):
|
||||
self.x = x
|
||||
self.y = y
|
||||
|
||||
# Good: __slots__ reduces memory usage
|
||||
class Point:
|
||||
__slots__ = ['x', 'y']
|
||||
|
||||
def __init__(self, x: float, y: float):
|
||||
self.x = x
|
||||
self.y = y
|
||||
```
|
||||
|
||||
### Generator for Large Data
|
||||
|
||||
```python
|
||||
# Bad: Returns full list in memory
|
||||
def read_lines(path: str) -> list[str]:
|
||||
with open(path) as f:
|
||||
return [line.strip() for line in f]
|
||||
|
||||
# Good: Yields lines one at a time
|
||||
def read_lines(path: str) -> Iterator[str]:
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
yield line.strip()
|
||||
```
|
||||
|
||||
### Avoid String Concatenation in Loops
|
||||
|
||||
```python
|
||||
# Bad: O(n²) due to string immutability
|
||||
result = ""
|
||||
for item in items:
|
||||
result += str(item)
|
||||
|
||||
# Good: O(n) using join
|
||||
result = "".join(str(item) for item in items)
|
||||
|
||||
# Good: Using StringIO for building
|
||||
from io import StringIO
|
||||
|
||||
buffer = StringIO()
|
||||
for item in items:
|
||||
buffer.write(str(item))
|
||||
result = buffer.getvalue()
|
||||
```
|
||||
|
||||
## Python Tooling Integration
|
||||
|
||||
### Essential Commands
|
||||
|
||||
```bash
|
||||
# Code formatting
|
||||
black .
|
||||
isort .
|
||||
|
||||
# Linting
|
||||
ruff check .
|
||||
pylint mypackage/
|
||||
|
||||
# Type checking
|
||||
mypy .
|
||||
|
||||
# Testing
|
||||
pytest --cov=mypackage --cov-report=html
|
||||
|
||||
# Security scanning
|
||||
bandit -r .
|
||||
|
||||
# Dependency management
|
||||
pip-audit
|
||||
safety check
|
||||
```
|
||||
|
||||
### pyproject.toml Configuration
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "mypackage"
|
||||
version = "1.0.0"
|
||||
requires-python = ">=3.9"
|
||||
dependencies = [
|
||||
"requests>=2.31.0",
|
||||
"pydantic>=2.0.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.4.0",
|
||||
"pytest-cov>=4.1.0",
|
||||
"black>=23.0.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.5.0",
|
||||
]
|
||||
|
||||
[tool.black]
|
||||
line-length = 88
|
||||
target-version = ['py39']
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 88
|
||||
select = ["E", "F", "I", "N", "W"]
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.9"
|
||||
warn_return_any = true
|
||||
warn_unused_configs = true
|
||||
disallow_untyped_defs = true
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
addopts = "--cov=mypackage --cov-report=term-missing"
|
||||
```
|
||||
|
||||
## Quick Reference: Python Idioms
|
||||
|
||||
| Idiom | Description |
|
||||
|-------|-------------|
|
||||
| EAFP | Easier to Ask Forgiveness than Permission |
|
||||
| Context managers | Use `with` for resource management |
|
||||
| List comprehensions | For simple transformations |
|
||||
| Generators | For lazy evaluation and large datasets |
|
||||
| Type hints | Annotate function signatures |
|
||||
| Dataclasses | For data containers with auto-generated methods |
|
||||
| `__slots__` | For memory optimization |
|
||||
| f-strings | For string formatting (Python 3.6+) |
|
||||
| `pathlib.Path` | For path operations (Python 3.4+) |
|
||||
| `enumerate` | For index-element pairs in loops |
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
```python
|
||||
# Bad: Mutable default arguments
|
||||
def append_to(item, items=[]):
|
||||
items.append(item)
|
||||
return items
|
||||
|
||||
# Good: Use None and create new list
|
||||
def append_to(item, items=None):
|
||||
if items is None:
|
||||
items = []
|
||||
items.append(item)
|
||||
return items
|
||||
|
||||
# Bad: Checking type with type()
|
||||
if type(obj) == list:
|
||||
process(obj)
|
||||
|
||||
# Good: Use isinstance
|
||||
if isinstance(obj, list):
|
||||
process(obj)
|
||||
|
||||
# Bad: Comparing to None with ==
|
||||
if value == None:
|
||||
process()
|
||||
|
||||
# Good: Use is
|
||||
if value is None:
|
||||
process()
|
||||
|
||||
# Bad: from module import *
|
||||
from os.path import *
|
||||
|
||||
# Good: Explicit imports
|
||||
from os.path import join, exists
|
||||
|
||||
# Bad: Bare except
|
||||
try:
|
||||
risky_operation()
|
||||
except:
|
||||
pass
|
||||
|
||||
# Good: Specific exception
|
||||
try:
|
||||
risky_operation()
|
||||
except SpecificError as e:
|
||||
logger.error(f"Operation failed: {e}")
|
||||
```
|
||||
|
||||
__Remember__: Python code should be readable, explicit, and follow the principle of least surprise. When in doubt, prioritize clarity over cleverness.
|
||||
817
.agents/skills/python-testing/SKILL.md
Normal file
817
.agents/skills/python-testing/SKILL.md
Normal file
@@ -0,0 +1,817 @@
|
||||
---
|
||||
name: python-testing
|
||||
description: Python testing strategies using pytest, TDD methodology, fixtures, mocking, parametrization, and coverage requirements.
|
||||
metadata:
|
||||
origin: ECC
|
||||
---
|
||||
|
||||
# Python Testing Patterns
|
||||
|
||||
Comprehensive testing strategies for Python applications using pytest, TDD methodology, and best practices.
|
||||
|
||||
## When to Activate
|
||||
|
||||
- Writing new Python code (follow TDD: red, green, refactor)
|
||||
- Designing test suites for Python projects
|
||||
- Reviewing Python test coverage
|
||||
- Setting up testing infrastructure
|
||||
|
||||
## Core Testing Philosophy
|
||||
|
||||
### Test-Driven Development (TDD)
|
||||
|
||||
Always follow the TDD cycle:
|
||||
|
||||
1. **RED**: Write a failing test for the desired behavior
|
||||
2. **GREEN**: Write minimal code to make the test pass
|
||||
3. **REFACTOR**: Improve code while keeping tests green
|
||||
|
||||
```python
|
||||
# Step 1: Write failing test (RED)
|
||||
def test_add_numbers():
|
||||
result = add(2, 3)
|
||||
assert result == 5
|
||||
|
||||
# Step 2: Write minimal implementation (GREEN)
|
||||
def add(a, b):
|
||||
return a + b
|
||||
|
||||
# Step 3: Refactor if needed (REFACTOR)
|
||||
```
|
||||
|
||||
### Coverage Requirements
|
||||
|
||||
- **Target**: 80%+ code coverage
|
||||
- **Critical paths**: 100% coverage required
|
||||
- Use `pytest --cov` to measure coverage
|
||||
|
||||
```bash
|
||||
pytest --cov=mypackage --cov-report=term-missing --cov-report=html
|
||||
```
|
||||
|
||||
## pytest Fundamentals
|
||||
|
||||
### Basic Test Structure
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
def test_addition():
|
||||
"""Test basic addition."""
|
||||
assert 2 + 2 == 4
|
||||
|
||||
def test_string_uppercase():
|
||||
"""Test string uppercasing."""
|
||||
text = "hello"
|
||||
assert text.upper() == "HELLO"
|
||||
|
||||
def test_list_append():
|
||||
"""Test list append."""
|
||||
items = [1, 2, 3]
|
||||
items.append(4)
|
||||
assert 4 in items
|
||||
assert len(items) == 4
|
||||
```
|
||||
|
||||
### Assertions
|
||||
|
||||
```python
|
||||
# Equality
|
||||
assert result == expected
|
||||
|
||||
# Inequality
|
||||
assert result != unexpected
|
||||
|
||||
# Truthiness
|
||||
assert result # Truthy
|
||||
assert not result # Falsy
|
||||
assert result is True # Exactly True
|
||||
assert result is False # Exactly False
|
||||
assert result is None # Exactly None
|
||||
|
||||
# Membership
|
||||
assert item in collection
|
||||
assert item not in collection
|
||||
|
||||
# Comparisons
|
||||
assert result > 0
|
||||
assert 0 <= result <= 100
|
||||
|
||||
# Type checking
|
||||
assert isinstance(result, str)
|
||||
|
||||
# Exception testing (preferred approach)
|
||||
with pytest.raises(ValueError):
|
||||
raise ValueError("error message")
|
||||
|
||||
# Check exception message
|
||||
with pytest.raises(ValueError, match="invalid input"):
|
||||
raise ValueError("invalid input provided")
|
||||
|
||||
# Check exception attributes
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
raise ValueError("error message")
|
||||
assert str(exc_info.value) == "error message"
|
||||
```
|
||||
|
||||
## Fixtures
|
||||
|
||||
### Basic Fixture Usage
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.fixture
|
||||
def sample_data():
|
||||
"""Fixture providing sample data."""
|
||||
return {"name": "Alice", "age": 30}
|
||||
|
||||
def test_sample_data(sample_data):
|
||||
"""Test using the fixture."""
|
||||
assert sample_data["name"] == "Alice"
|
||||
assert sample_data["age"] == 30
|
||||
```
|
||||
|
||||
### Fixture with Setup/Teardown
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def database():
|
||||
"""Fixture with setup and teardown."""
|
||||
# Setup
|
||||
db = Database(":memory:")
|
||||
db.create_tables()
|
||||
db.insert_test_data()
|
||||
|
||||
yield db # Provide to test
|
||||
|
||||
# Teardown
|
||||
db.close()
|
||||
|
||||
def test_database_query(database):
|
||||
"""Test database operations."""
|
||||
result = database.query("SELECT * FROM users")
|
||||
assert len(result) > 0
|
||||
```
|
||||
|
||||
### Fixture Scopes
|
||||
|
||||
```python
|
||||
# Function scope (default) - runs for each test
|
||||
@pytest.fixture
|
||||
def temp_file():
|
||||
with open("temp.txt", "w") as f:
|
||||
yield f
|
||||
os.remove("temp.txt")
|
||||
|
||||
# Module scope - runs once per module
|
||||
@pytest.fixture(scope="module")
|
||||
def module_db():
|
||||
db = Database(":memory:")
|
||||
db.create_tables()
|
||||
yield db
|
||||
db.close()
|
||||
|
||||
# Session scope - runs once per test session
|
||||
@pytest.fixture(scope="session")
|
||||
def shared_resource():
|
||||
resource = ExpensiveResource()
|
||||
yield resource
|
||||
resource.cleanup()
|
||||
```
|
||||
|
||||
### Fixture with Parameters
|
||||
|
||||
```python
|
||||
@pytest.fixture(params=[1, 2, 3])
|
||||
def number(request):
|
||||
"""Parameterized fixture."""
|
||||
return request.param
|
||||
|
||||
def test_numbers(number):
|
||||
"""Test runs 3 times, once for each parameter."""
|
||||
assert number > 0
|
||||
```
|
||||
|
||||
### Using Multiple Fixtures
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def user():
|
||||
return User(id=1, name="Alice")
|
||||
|
||||
@pytest.fixture
|
||||
def admin():
|
||||
return User(id=2, name="Admin", role="admin")
|
||||
|
||||
def test_user_admin_interaction(user, admin):
|
||||
"""Test using multiple fixtures."""
|
||||
assert admin.can_manage(user)
|
||||
```
|
||||
|
||||
### Autouse Fixtures
|
||||
|
||||
```python
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_config():
|
||||
"""Automatically runs before every test."""
|
||||
Config.reset()
|
||||
yield
|
||||
Config.cleanup()
|
||||
|
||||
def test_without_fixture_call():
|
||||
# reset_config runs automatically
|
||||
assert Config.get_setting("debug") is False
|
||||
```
|
||||
|
||||
### Conftest.py for Shared Fixtures
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import pytest
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Shared fixture for all tests."""
|
||||
app = create_app(testing=True)
|
||||
with app.test_client() as client:
|
||||
yield client
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers(client):
|
||||
"""Generate auth headers for API testing."""
|
||||
response = client.post("/api/login", json={
|
||||
"username": "test",
|
||||
"password": "test"
|
||||
})
|
||||
token = response.json["token"]
|
||||
return {"Authorization": f"Bearer {token}"}
|
||||
```
|
||||
|
||||
## Parametrization
|
||||
|
||||
### Basic Parametrization
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize("input,expected", [
|
||||
("hello", "HELLO"),
|
||||
("world", "WORLD"),
|
||||
("PyThOn", "PYTHON"),
|
||||
])
|
||||
def test_uppercase(input, expected):
|
||||
"""Test runs 3 times with different inputs."""
|
||||
assert input.upper() == expected
|
||||
```
|
||||
|
||||
### Multiple Parameters
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize("a,b,expected", [
|
||||
(2, 3, 5),
|
||||
(0, 0, 0),
|
||||
(-1, 1, 0),
|
||||
(100, 200, 300),
|
||||
])
|
||||
def test_add(a, b, expected):
|
||||
"""Test addition with multiple inputs."""
|
||||
assert add(a, b) == expected
|
||||
```
|
||||
|
||||
### Parametrize with IDs
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize("input,expected", [
|
||||
("valid@email.com", True),
|
||||
("invalid", False),
|
||||
("@no-domain.com", False),
|
||||
], ids=["valid-email", "missing-at", "missing-domain"])
|
||||
def test_email_validation(input, expected):
|
||||
"""Test email validation with readable test IDs."""
|
||||
assert is_valid_email(input) is expected
|
||||
```
|
||||
|
||||
### Parametrized Fixtures
|
||||
|
||||
```python
|
||||
@pytest.fixture(params=["sqlite", "postgresql", "mysql"])
|
||||
def db(request):
|
||||
"""Test against multiple database backends."""
|
||||
if request.param == "sqlite":
|
||||
return Database(":memory:")
|
||||
elif request.param == "postgresql":
|
||||
return Database("postgresql://localhost/test")
|
||||
elif request.param == "mysql":
|
||||
return Database("mysql://localhost/test")
|
||||
|
||||
def test_database_operations(db):
|
||||
"""Test runs 3 times, once for each database."""
|
||||
result = db.query("SELECT 1")
|
||||
assert result is not None
|
||||
```
|
||||
|
||||
## Markers and Test Selection
|
||||
|
||||
### Custom Markers
|
||||
|
||||
```python
|
||||
# Mark slow tests
|
||||
@pytest.mark.slow
|
||||
def test_slow_operation():
|
||||
time.sleep(5)
|
||||
|
||||
# Mark integration tests
|
||||
@pytest.mark.integration
|
||||
def test_api_integration():
|
||||
response = requests.get("https://api.example.com")
|
||||
assert response.status_code == 200
|
||||
|
||||
# Mark unit tests
|
||||
@pytest.mark.unit
|
||||
def test_unit_logic():
|
||||
assert calculate(2, 3) == 5
|
||||
```
|
||||
|
||||
### Run Specific Tests
|
||||
|
||||
```bash
|
||||
# Run only fast tests
|
||||
pytest -m "not slow"
|
||||
|
||||
# Run only integration tests
|
||||
pytest -m integration
|
||||
|
||||
# Run integration or slow tests
|
||||
pytest -m "integration or slow"
|
||||
|
||||
# Run tests marked as unit but not slow
|
||||
pytest -m "unit and not slow"
|
||||
```
|
||||
|
||||
### Configure Markers in pytest.ini
|
||||
|
||||
```ini
|
||||
[pytest]
|
||||
markers =
|
||||
slow: marks tests as slow
|
||||
integration: marks tests as integration tests
|
||||
unit: marks tests as unit tests
|
||||
django: marks tests as requiring Django
|
||||
```
|
||||
|
||||
## Mocking and Patching
|
||||
|
||||
### Mocking Functions
|
||||
|
||||
```python
|
||||
from unittest.mock import patch, Mock
|
||||
|
||||
@patch("mypackage.external_api_call")
|
||||
def test_with_mock(api_call_mock):
|
||||
"""Test with mocked external API."""
|
||||
api_call_mock.return_value = {"status": "success"}
|
||||
|
||||
result = my_function()
|
||||
|
||||
api_call_mock.assert_called_once()
|
||||
assert result["status"] == "success"
|
||||
```
|
||||
|
||||
### Mocking Return Values
|
||||
|
||||
```python
|
||||
@patch("mypackage.Database.connect")
|
||||
def test_database_connection(connect_mock):
|
||||
"""Test with mocked database connection."""
|
||||
connect_mock.return_value = MockConnection()
|
||||
|
||||
db = Database()
|
||||
db.connect()
|
||||
|
||||
connect_mock.assert_called_once_with("localhost")
|
||||
```
|
||||
|
||||
### Mocking Exceptions
|
||||
|
||||
```python
|
||||
@patch("mypackage.api_call")
|
||||
def test_api_error_handling(api_call_mock):
|
||||
"""Test error handling with mocked exception."""
|
||||
api_call_mock.side_effect = ConnectionError("Network error")
|
||||
|
||||
with pytest.raises(ConnectionError):
|
||||
api_call()
|
||||
|
||||
api_call_mock.assert_called_once()
|
||||
```
|
||||
|
||||
### Mocking Context Managers
|
||||
|
||||
```python
|
||||
@patch("builtins.open", new_callable=mock_open)
|
||||
def test_file_reading(mock_file):
|
||||
"""Test file reading with mocked open."""
|
||||
mock_file.return_value.read.return_value = "file content"
|
||||
|
||||
result = read_file("test.txt")
|
||||
|
||||
mock_file.assert_called_once_with("test.txt", "r")
|
||||
assert result == "file content"
|
||||
```
|
||||
|
||||
### Using Autospec
|
||||
|
||||
```python
|
||||
@patch("mypackage.DBConnection", autospec=True)
|
||||
def test_autospec(db_mock):
|
||||
"""Test with autospec to catch API misuse."""
|
||||
db = db_mock.return_value
|
||||
db.query("SELECT * FROM users")
|
||||
|
||||
# This would fail if DBConnection doesn't have query method
|
||||
db_mock.assert_called_once()
|
||||
```
|
||||
|
||||
### Mock Class Instances
|
||||
|
||||
```python
|
||||
class TestUserService:
|
||||
@patch("mypackage.UserRepository")
|
||||
def test_create_user(self, repo_mock):
|
||||
"""Test user creation with mocked repository."""
|
||||
repo_mock.return_value.save.return_value = User(id=1, name="Alice")
|
||||
|
||||
service = UserService(repo_mock.return_value)
|
||||
user = service.create_user(name="Alice")
|
||||
|
||||
assert user.name == "Alice"
|
||||
repo_mock.return_value.save.assert_called_once()
|
||||
```
|
||||
|
||||
### Mock Property
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def mock_config():
|
||||
"""Create a mock with a property."""
|
||||
config = Mock()
|
||||
type(config).debug = PropertyMock(return_value=True)
|
||||
type(config).api_key = PropertyMock(return_value="test-key")
|
||||
return config
|
||||
|
||||
def test_with_mock_config(mock_config):
|
||||
"""Test with mocked config properties."""
|
||||
assert mock_config.debug is True
|
||||
assert mock_config.api_key == "test-key"
|
||||
```
|
||||
|
||||
## Testing Async Code
|
||||
|
||||
### Async Tests with pytest-asyncio
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_function():
|
||||
"""Test async function."""
|
||||
result = await async_add(2, 3)
|
||||
assert result == 5
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_with_fixture(async_client):
|
||||
"""Test async with async fixture."""
|
||||
response = await async_client.get("/api/users")
|
||||
assert response.status_code == 200
|
||||
```
|
||||
|
||||
### Async Fixture
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
async def async_client():
|
||||
"""Async fixture providing async test client."""
|
||||
app = create_app()
|
||||
async with app.test_client() as client:
|
||||
yield client
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_api_endpoint(async_client):
|
||||
"""Test using async fixture."""
|
||||
response = await async_client.get("/api/data")
|
||||
assert response.status_code == 200
|
||||
```
|
||||
|
||||
### Mocking Async Functions
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
@patch("mypackage.async_api_call")
|
||||
async def test_async_mock(api_call_mock):
|
||||
"""Test async function with mock."""
|
||||
api_call_mock.return_value = {"status": "ok"}
|
||||
|
||||
result = await my_async_function()
|
||||
|
||||
api_call_mock.assert_awaited_once()
|
||||
assert result["status"] == "ok"
|
||||
```
|
||||
|
||||
## Testing Exceptions
|
||||
|
||||
### Testing Expected Exceptions
|
||||
|
||||
```python
|
||||
def test_divide_by_zero():
|
||||
"""Test that dividing by zero raises ZeroDivisionError."""
|
||||
with pytest.raises(ZeroDivisionError):
|
||||
divide(10, 0)
|
||||
|
||||
def test_custom_exception():
|
||||
"""Test custom exception with message."""
|
||||
with pytest.raises(ValueError, match="invalid input"):
|
||||
validate_input("invalid")
|
||||
```
|
||||
|
||||
### Testing Exception Attributes
|
||||
|
||||
```python
|
||||
def test_exception_with_details():
|
||||
"""Test exception with custom attributes."""
|
||||
with pytest.raises(CustomError) as exc_info:
|
||||
raise CustomError("error", code=400)
|
||||
|
||||
assert exc_info.value.code == 400
|
||||
assert "error" in str(exc_info.value)
|
||||
```
|
||||
|
||||
## Testing Side Effects
|
||||
|
||||
### Testing File Operations
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
def test_file_processing():
|
||||
"""Test file processing with temp file."""
|
||||
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as f:
|
||||
f.write("test content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = process_file(temp_path)
|
||||
assert result == "processed: test content"
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
```
|
||||
|
||||
### Testing with pytest's tmp_path Fixture
|
||||
|
||||
```python
|
||||
def test_with_tmp_path(tmp_path):
|
||||
"""Test using pytest's built-in temp path fixture."""
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("hello world")
|
||||
|
||||
result = process_file(str(test_file))
|
||||
assert result == "hello world"
|
||||
# tmp_path automatically cleaned up
|
||||
```
|
||||
|
||||
### Testing with tmpdir Fixture
|
||||
|
||||
```python
|
||||
def test_with_tmpdir(tmpdir):
|
||||
"""Test using pytest's tmpdir fixture."""
|
||||
test_file = tmpdir.join("test.txt")
|
||||
test_file.write("data")
|
||||
|
||||
result = process_file(str(test_file))
|
||||
assert result == "data"
|
||||
```
|
||||
|
||||
## Test Organization
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── conftest.py # Shared fixtures
|
||||
├── __init__.py
|
||||
├── unit/ # Unit tests
|
||||
│ ├── __init__.py
|
||||
│ ├── test_models.py
|
||||
│ ├── test_utils.py
|
||||
│ └── test_services.py
|
||||
├── integration/ # Integration tests
|
||||
│ ├── __init__.py
|
||||
│ ├── test_api.py
|
||||
│ └── test_database.py
|
||||
└── e2e/ # End-to-end tests
|
||||
├── __init__.py
|
||||
└── test_user_flow.py
|
||||
```
|
||||
|
||||
### Test Classes
|
||||
|
||||
```python
|
||||
class TestUserService:
|
||||
"""Group related tests in a class."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self):
|
||||
"""Setup runs before each test in this class."""
|
||||
self.service = UserService()
|
||||
|
||||
def test_create_user(self):
|
||||
"""Test user creation."""
|
||||
user = self.service.create_user("Alice")
|
||||
assert user.name == "Alice"
|
||||
|
||||
def test_delete_user(self):
|
||||
"""Test user deletion."""
|
||||
user = User(id=1, name="Bob")
|
||||
self.service.delete_user(user)
|
||||
assert not self.service.user_exists(1)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### DO
|
||||
|
||||
- **Follow TDD**: Write tests before code (red-green-refactor)
|
||||
- **Test one thing**: Each test should verify a single behavior
|
||||
- **Use descriptive names**: `test_user_login_with_invalid_credentials_fails`
|
||||
- **Use fixtures**: Eliminate duplication with fixtures
|
||||
- **Mock external dependencies**: Don't depend on external services
|
||||
- **Test edge cases**: Empty inputs, None values, boundary conditions
|
||||
- **Aim for 80%+ coverage**: Focus on critical paths
|
||||
- **Keep tests fast**: Use marks to separate slow tests
|
||||
|
||||
### DON'T
|
||||
|
||||
- **Don't test implementation**: Test behavior, not internals
|
||||
- **Don't use complex conditionals in tests**: Keep tests simple
|
||||
- **Don't ignore test failures**: All tests must pass
|
||||
- **Don't test third-party code**: Trust libraries to work
|
||||
- **Don't share state between tests**: Tests should be independent
|
||||
- **Don't catch exceptions in tests**: Use `pytest.raises`
|
||||
- **Don't use print statements**: Use assertions and pytest output
|
||||
- **Don't write tests that are too brittle**: Avoid over-specific mocks
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Testing API Endpoints (FastAPI/Flask)
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def client():
|
||||
app = create_app(testing=True)
|
||||
return app.test_client()
|
||||
|
||||
def test_get_user(client):
|
||||
response = client.get("/api/users/1")
|
||||
assert response.status_code == 200
|
||||
assert response.json["id"] == 1
|
||||
|
||||
def test_create_user(client):
|
||||
response = client.post("/api/users", json={
|
||||
"name": "Alice",
|
||||
"email": "alice@example.com"
|
||||
})
|
||||
assert response.status_code == 201
|
||||
assert response.json["name"] == "Alice"
|
||||
```
|
||||
|
||||
### Testing Database Operations
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def db_session():
|
||||
"""Create a test database session."""
|
||||
session = Session(bind=engine)
|
||||
session.begin_nested()
|
||||
yield session
|
||||
session.rollback()
|
||||
session.close()
|
||||
|
||||
def test_create_user(db_session):
|
||||
user = User(name="Alice", email="alice@example.com")
|
||||
db_session.add(user)
|
||||
db_session.commit()
|
||||
|
||||
retrieved = db_session.query(User).filter_by(name="Alice").first()
|
||||
assert retrieved.email == "alice@example.com"
|
||||
```
|
||||
|
||||
### Testing Class Methods
|
||||
|
||||
```python
|
||||
class TestCalculator:
|
||||
@pytest.fixture
|
||||
def calculator(self):
|
||||
return Calculator()
|
||||
|
||||
def test_add(self, calculator):
|
||||
assert calculator.add(2, 3) == 5
|
||||
|
||||
def test_divide_by_zero(self, calculator):
|
||||
with pytest.raises(ZeroDivisionError):
|
||||
calculator.divide(10, 0)
|
||||
```
|
||||
|
||||
## pytest Configuration
|
||||
|
||||
### pytest.ini
|
||||
|
||||
```ini
|
||||
[pytest]
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
addopts =
|
||||
--strict-markers
|
||||
--disable-warnings
|
||||
--cov=mypackage
|
||||
--cov-report=term-missing
|
||||
--cov-report=html
|
||||
markers =
|
||||
slow: marks tests as slow
|
||||
integration: marks tests as integration tests
|
||||
unit: marks tests as unit tests
|
||||
```
|
||||
|
||||
### pyproject.toml
|
||||
|
||||
```toml
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"--strict-markers",
|
||||
"--cov=mypackage",
|
||||
"--cov-report=term-missing",
|
||||
"--cov-report=html",
|
||||
]
|
||||
markers = [
|
||||
"slow: marks tests as slow",
|
||||
"integration: marks tests as integration tests",
|
||||
"unit: marks tests as unit tests",
|
||||
]
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run specific file
|
||||
pytest tests/test_utils.py
|
||||
|
||||
# Run specific test
|
||||
pytest tests/test_utils.py::test_function
|
||||
|
||||
# Run with verbose output
|
||||
pytest -v
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=mypackage --cov-report=html
|
||||
|
||||
# Run only fast tests
|
||||
pytest -m "not slow"
|
||||
|
||||
# Run until first failure
|
||||
pytest -x
|
||||
|
||||
# Run and stop on N failures
|
||||
pytest --maxfail=3
|
||||
|
||||
# Run last failed tests
|
||||
pytest --lf
|
||||
|
||||
# Run tests with pattern
|
||||
pytest -k "test_user"
|
||||
|
||||
# Run with debugger on failure
|
||||
pytest --pdb
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Pattern | Usage |
|
||||
|---------|-------|
|
||||
| `pytest.raises()` | Test expected exceptions |
|
||||
| `@pytest.fixture()` | Create reusable test fixtures |
|
||||
| `@pytest.mark.parametrize()` | Run tests with multiple inputs |
|
||||
| `@pytest.mark.slow` | Mark slow tests |
|
||||
| `pytest -m "not slow"` | Skip slow tests |
|
||||
| `@patch()` | Mock functions and classes |
|
||||
| `tmp_path` fixture | Automatic temp directory |
|
||||
| `pytest --cov` | Generate coverage report |
|
||||
| `assert` | Simple and readable assertions |
|
||||
|
||||
**Remember**: Tests are code too. Keep them clean, readable, and maintainable. Good tests catch bugs; great tests prevent them.
|
||||
78
.env.example
Normal file
78
.env.example
Normal file
@@ -0,0 +1,78 @@
|
||||
# 企微群机器人 webhook（早报推送，与 bot API 模式凭证不同）
|
||||
WECOM_WEBHOOK_KEY=your-webhook-key
|
||||
|
||||
# 早报内容
|
||||
DAILY_TRENDING_LIMIT=150
|
||||
DAILY_HOT_LIMIT=150
|
||||
DAILY_GITHUB_TRENDING_LIMIT=10
|
||||
DAILY_GITHUB_EMERGING_LIMIT=10
|
||||
DAILY_GITHUB_TOPIC_LIMIT=10
|
||||
|
||||
# GitHub Trending
|
||||
GITHUB_TRENDING_SINCE=daily
|
||||
# GITHUB_TRENDING_LANGUAGE=python
|
||||
# GITHUB_TRENDING_MODE=scrape
|
||||
# scrape — 爬 github.com/trending，失败时用 token + Search API 补充
|
||||
# api — 仅 Search API（必须 GITHUB_TOKEN）
|
||||
|
||||
# GitHub 新兴 / Topic（Search API，需 GITHUB_TOKEN）
|
||||
# GITHUB_EMERGING_DAYS=14
|
||||
# GITHUB_EMERGING_MIN_STARS=200
|
||||
# GITHUB_TOPIC=llm
|
||||
# GITHUB_TOPIC_PUSHED_DAYS=7
|
||||
# GITHUB_TOPIC_MIN_STARS=50
|
||||
|
||||
# GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxx
|
||||
# GITHUB_API_ENRICH=1
|
||||
|
||||
# 企微短版（各榜 Top N，默认 10）
|
||||
DAILY_WECOM_TRENDING=10
|
||||
DAILY_WECOM_HOT=10
|
||||
DAILY_WECOM_GITHUB_TRENDING=10
|
||||
DAILY_WECOM_GITHUB_EMERGING=10
|
||||
DAILY_WECOM_GITHUB_TOPIC=10
|
||||
DAILY_WECOM_AI_NEWS=10
|
||||
# 企微 Skills 合并前扫描池大小（同 source 合并后仍凑满 Top N）
|
||||
# DAILY_WECOM_SKILL_POOL=200
|
||||
|
||||
# Skills 榜单来源：website=skills.sh 官网（600+ 条）| feed=第三方 feed.json（50 条）
|
||||
# SKILLS_BOARD_SOURCE=website
|
||||
# 企微单条 markdown 上限 4096；超长自动分多条推送（不截断正文）
|
||||
DAILY_WECOM_CHUNK_BYTES=4096
|
||||
# DAILY_WECOM_MAX_PARTS=5
|
||||
|
||||
# 国际 AI 时讯（RSS，见 daily/news/feeds.py）
|
||||
DAILY_AI_NEWS=1
|
||||
# 英文描述 → 简短中文（DAILY_CURSOR_EDITOR=0 时生效）
|
||||
# DAILY_ZH_DESC=1
|
||||
# DAILY_ZH_DESC_BATCH=20
|
||||
# DAILY_LLM_API_KEY=
|
||||
# DAILY_LLM_API_BASE=https://api.openai.com/v1
|
||||
# DAILY_LLM_MODEL=gpt-4o-mini
|
||||
|
||||
# Tier B：Cursor 编辑层（主题 + 速览 + 中文描述，一次 LLM 调用）
|
||||
# 需 CURSOR_API_KEY 或 DAILY_LLM_API_KEY；开启后 DAILY_ZH_DESC 自动跳过
|
||||
# DAILY_CURSOR_EDITOR=1
|
||||
|
||||
# Agent 工作流（推荐：叙事化早报）
|
||||
# DAILY_REPORT_MODE=agent
|
||||
# CURSOR_API_KEY=cursor_...
|
||||
# CURSOR_MODEL=composer-2.5
|
||||
# DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily
|
||||
|
||||
# 新增榜对比（较昨日 Top15，供 Agent 导语/signals；列表展示 Top N）
|
||||
# DAILY_DELTA_COMPARE_DEPTH=15
|
||||
# DAILY_WECOM_NEW_MAX=10
|
||||
# DAILY_DELTA_LOOKBACK_DAYS=7
|
||||
# DAILY_FULL_DESC_LIMIT=0
|
||||
# DAILY_FULL_NEWS_SUMMARY_LIMIT=0
|
||||
DAILY_AI_NEWS_HOURS=72
|
||||
DAILY_AI_NEWS_PER_FEED=3
|
||||
DAILY_AI_NEWS_PER_CATEGORY=5
|
||||
|
||||
# Reddit RSS（403/429 时在 Reddit 偏好设置 → RSS feeds 复制 user / feed 参数）
|
||||
# REDDIT_RSS_USER=your_username
|
||||
# REDDIT_RSS_FEED=your_feed_token
|
||||
|
||||
# 可选：关注仓库 Release
|
||||
# GITHUB_REPOS=vercel-labs/skills,obra/superpowers
|
||||
8
.gitignore
vendored
Normal file
8
.gitignore
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
.env
|
||||
.env.local
|
||||
logs/
|
||||
.cache/
|
||||
bot/.env
|
||||
bot/.venv/
|
||||
bot/.cache/
|
||||
output
|
||||
3
.idea/.gitignore
generated
vendored
Normal file
3
.idea/.gitignore
generated
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
# Default ignored files
|
||||
/shelf/
|
||||
/workspace.xml
|
||||
9
.idea/daily-robots.iml
generated
Normal file
9
.idea/daily-robots.iml
generated
Normal file
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<module type="JAVA_MODULE" version="4">
|
||||
<component name="NewModuleRootManager" inherit-compiler-output="true">
|
||||
<exclude-output />
|
||||
<content url="file://$MODULE_DIR$" />
|
||||
<orderEntry type="inheritedJdk" />
|
||||
<orderEntry type="sourceFolder" forTests="false" />
|
||||
</component>
|
||||
</module>
|
||||
6
.idea/misc.xml
generated
Normal file
6
.idea/misc.xml
generated
Normal file
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="ProjectRootManager" version="2" languageLevel="JDK_21" default="true" project-jdk-name="21" project-jdk-type="JavaSDK">
|
||||
<output url="file://$PROJECT_DIR$/out" />
|
||||
</component>
|
||||
</project>
|
||||
8
.idea/modules.xml
generated
Normal file
8
.idea/modules.xml
generated
Normal file
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="ProjectModuleManager">
|
||||
<modules>
|
||||
<module fileurl="file://$PROJECT_DIR$/.idea/daily-robots.iml" filepath="$PROJECT_DIR$/.idea/daily-robots.iml" />
|
||||
</modules>
|
||||
</component>
|
||||
</project>
|
||||
6
.idea/vcs.xml
generated
Normal file
6
.idea/vcs.xml
generated
Normal file
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="VcsDirectoryMappings">
|
||||
<mapping directory="" vcs="Git" />
|
||||
</component>
|
||||
</project>
|
||||
329
README.md
Normal file
329
README.md
Normal file
@@ -0,0 +1,329 @@
|
||||
# skills-hot-daily
|
||||
|
||||
Skills / GitHub 早报推送 + 企微对话机器人（同一仓库、两套企微接入）。
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
skills-hot-daily/
|
||||
├── README.md
|
||||
├── .env.example # 早报 webhook、GitHub 等
|
||||
├── requirements.txt # 早报 Python 依赖
|
||||
├── run-daily.ps1 # 生成 + 推送一条龙
|
||||
├── send-wecom.ps1 # 仅推送
|
||||
├── daily/ # 早报 Python 包
|
||||
│ ├── __main__.py # python -m daily [generate|push]
|
||||
│ ├── config.py
|
||||
│ ├── generate.py
|
||||
│ ├── report_data.py # JSON 中间层
|
||||
│ ├── cursor_editor.py # Cursor 编辑层
|
||||
│ ├── llm_client.py
|
||||
│ ├── localize.py # 仅中文化（Tier A）
|
||||
│ ├── format_wecom.py
|
||||
│ ├── webhook.py
|
||||
│ ├── news/ # 国际 AI 时讯 RSS
|
||||
│ │ ├── feeds.py
|
||||
│ │ └── fetch.py
|
||||
│ └── github/
|
||||
│ ├── auth.py
|
||||
│ ├── search.py
|
||||
│ └── trending.py
|
||||
├── output/ # YYYY-MM-DD.md / .wecom.md / .data.json / .editorial.json
|
||||
├── skills/daily-editor/ # 早报 Cursor 编辑 Skill
|
||||
│ └── SKILL.md
|
||||
├── logs/
|
||||
├── .cache/
|
||||
└── bot/ # 企微 API 模式对话机器人（独立 venv）
|
||||
├── main.py
|
||||
├── skills_service.py
|
||||
└── scenarios/
|
||||
```
|
||||
|
||||
| 模块 | 配置文件 | 启动方式 |
|
||||
|------|----------|----------|
|
||||
| **早报推送** | 根目录 `.env`（`WECOM_WEBHOOK_KEY` 等） | `.\run-daily.ps1` |
|
||||
| **对话 Bot** | `bot/.env`（`WECOM_BOT_ID` / `SECRET` 等） | `cd bot` → `python main.py` |
|
||||
|
||||
---
|
||||
|
||||
## 一、早报推送
|
||||
|
||||
```powershell
|
||||
cd d:\LY\test\tech\skills-hot-daily
|
||||
pip install -r requirements.txt
|
||||
copy .env.example .env
|
||||
.\run-daily.ps1
|
||||
```
|
||||
|
||||
- 生成：`output/YYYY-MM-DD.md`（完整版）、`output/YYYY-MM-DD.wecom.md`（企微短版）
|
||||
- 仅生成：`.\run-daily.ps1 -SkipPush`
|
||||
- 仅推送：`python -m daily push output\2026-06-25.wecom.md`
|
||||
|
||||
**Webhook 配置**：企微群 → 群机器人 → 添加，将 `key=` 后的值写入项目根 `.env`：
|
||||
|
||||
```env
|
||||
WECOM_WEBHOOK_KEY=your-key
|
||||
```
|
||||
|
||||
**GitHub 数据源**（见 `.env.example`）：
|
||||
|
||||
| 来源 | 说明 |
|
||||
|------|------|
|
||||
| GitHub Trending | `GITHUB_TRENDING_MODE=scrape` 或 `api` |
|
||||
| 新兴 / Topic | Search API，需 `GITHUB_TOKEN` |
|
||||
| Release | 可选 `GITHUB_REPOS=owner/repo` |
|
||||
|
||||
**国际 AI 时讯**（RSS，见 `daily/news/feeds.py`）：
|
||||
|
||||
| 类别 | 覆盖 |
|
||||
|------|------|
|
||||
| 厂商官方 | Anthropic、OpenAI、Google、Meta、Microsoft、Mistral、Cursor 等 |
|
||||
| Agent / LLM 开发者 | LangChain、LlamaIndex、Hugging Face、Copilot 等 |
|
||||
| 综合科技媒体 | The Verge、TechCrunch、Ars、Wired、MIT TR 等 |
|
||||
| Newsletter | Ben's Bites、Latent Space、Simon Willison、TLDR AI 等 |
|
||||
| 研究 / 论文 | arXiv cs.CL/AI/LG、HF Papers |
|
||||
| 社区讨论 | HN、Reddit r/LocalLLaMA / ClaudeAI / ML 等 |
|
||||
|
||||
环境变量：`DAILY_AI_NEWS=1` · `DAILY_AI_NEWS_HOURS=72` · `DAILY_WECOM_AI_NEWS=8`
|
||||
|
||||
### 生成架构（Tier B · Cursor 编辑层）
|
||||
|
||||
早报默认走 **Python 抓取 + 模板渲染**；可选开启 Cursor 做「编辑」：
|
||||
|
||||
```
|
||||
抓取数据 → output/日期.data.json → Cursor 读 Skill 写 editorial → 模板填字 → .md / .wecom.md
|
||||
```
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `output/YYYY-MM-DD.data.json` | 结构化榜单（供 LLM 输入） |
|
||||
| `output/YYYY-MM-DD.editorial.json` | Cursor 输出的主题、速览、中文描述 |
|
||||
| `skills/daily-editor/SKILL.md` | 编辑规范（语气、JSON 格式） |
|
||||
|
||||
```env
|
||||
# 开启 Tier B（需 CURSOR_API_KEY 或 DAILY_LLM_API_KEY）
|
||||
DAILY_CURSOR_EDITOR=1
|
||||
```
|
||||
|
||||
- 开启后：**一次 LLM 调用** 生成 `theme_line` + `highlights` + 全部中文描述
|
||||
- 关闭时（默认）：规则主题 + `DAILY_ZH_DESC` 仅中文化描述
|
||||
- LLM 失败自动回退规则模式，不影响推送
|
||||
|
||||
### 生成架构（Agent 工作流 · 推荐）
|
||||
|
||||
若觉得模板版「榜单堆砌」不友好，可改用 **Agent 三步流水线**：
|
||||
|
||||
```
|
||||
Python 抓取 → Step1 趋势分析 → Step2 叙事写稿 → Python 分条推送企微
|
||||
(.trends.json) (.wecom.md)
|
||||
```
|
||||
|
||||
| 模式 | 环境变量 | 企微版风格 |
|
||||
|------|----------|------------|
|
||||
| `classic`（默认） | — | 分区榜单 + 模板 |
|
||||
| `editor` | `DAILY_CURSOR_EDITOR=1` | 模板 + LLM 中文化 |
|
||||
| `agent` | `DAILY_REPORT_MODE=agent` | **导语 + 信号 + 精选**，全中文叙述 |
|
||||
|
||||
```env
|
||||
DAILY_REPORT_MODE=agent
|
||||
DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily # 早报 LLM 工作目录（与 bot 的 CURSOR_CWD 独立）
|
||||
```
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `output/YYYY-MM-DD.trends.json` | Step1 趋势分析结果 |
|
||||
| `skills/daily-agent/SKILL.md` | Agent 工作流规范 |
|
||||
|
||||
- 完整版 `YYYY-MM-DD.md` 仍为数据表格归档；企微版由 Agent 直接写 Markdown
|
||||
- Agent 失败自动回退 `classic`，不影响 `run-daily.ps1`
|
||||
|
||||
定时推送：Windows 任务计划程序或 `/loop 1d` 执行 `run-daily.ps1`。
|
||||
|
||||
---
|
||||
|
||||
## 二、可对话 Skills 助手（企业微信智能机器人）
|
||||
|
||||
在企微里 @ 机器人即可：
|
||||
- **快查**：`trending 10`、`hot 10`、`搜索 react`（本地 skills 数据，秒回）
|
||||
- **截图预览**：`preview` / `截图`（基于 `.env` 的 `CURSOR_CWD` 启动前端并发图）
|
||||
- **通用任务**：任意自然语言需求，由 **Cursor Agent** 执行并回传结果
|
||||
|
||||
### 1. 创建 API 模式机器人
|
||||
|
||||
1. [企业微信管理后台](https://work.weixin.qq.com/) → **安全与管理** → **管理工具** → **智能机器人** → **创建机器人**
|
||||
2. 选择 **API 模式创建** → **使用长连接**
|
||||
3. 记录 **Bot ID** 和 **Secret**（Secret 只显示一次，请立即保存）
|
||||
4. 设置可见范围，将机器人 **添加到目标群** 或允许成员单聊
|
||||
|
||||
普通成员路径：工作台 → 智能机器人 → 手动创建 → API 模式 → 长连接
|
||||
|
||||
### 2. 启动本地服务
|
||||
|
||||
```powershell
|
||||
cd d:\LY\test\tech\skills-hot-daily\bot
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium
|
||||
copy .env.example .env
|
||||
# 编辑 .env：WECOM_BOT_ID / WECOM_BOT_SECRET / CURSOR_API_KEY
|
||||
python main.py
|
||||
```
|
||||
|
||||
服务需 **常驻运行**（本机、服务器或 Docker）。长连接模式下机器人进程须在线才能收消息。
|
||||
|
||||
### 3. 路由模式（ROUTING_MODE）
|
||||
|
||||
| 模式 | 行为 |
|
||||
|------|------|
|
||||
| `hybrid`（默认） | `trending`/`hot`/`搜索`/`详情` 走本地快查；其余 @ 消息交给 Cursor |
|
||||
| `cursor` | 所有消息都交给 Cursor 执行 |
|
||||
| `skills` | 仅本地 skills 快查（旧行为） |
|
||||
|
||||
**Cursor 任务示例**（群里发送）：
|
||||
|
||||
```
|
||||
@test 总结 trending top10，并推荐 3 个适合前端团队的 skill
|
||||
@test 对比 mattpocock/skills 和 obra/superpowers 各有哪些热门 skill
|
||||
@test 帮我写一段 npx skills add 的安装说明
|
||||
```
|
||||
|
||||
Cursor 在本机 `CURSOR_CWD` 目录下运行，默认 `d:\LY\test\tech`。复杂任务可能需要 1–10 分钟，流式消息会显示「Cursor 正在执行任务…」。
|
||||
|
||||
### 4. 前端截图预览（API 模式发图）
|
||||
|
||||
项目路径读取 `.env` 中的 **`CURSOR_CWD`**。机器人会：
|
||||
|
||||
1. 在 `CURSOR_CWD` 检测 `package.json`，若有 `dev` 脚本则执行 `PREVIEW_DEV_COMMAND`（默认 `npm run dev`）
|
||||
2. 等待 `PREVIEW_PORT`（默认 `5173`）就绪，或用 `PREVIEW_URL` 直接访问
|
||||
3. Playwright 打开页面并截图
|
||||
4. 通过 API 模式 **上传图片 + 回复 image 消息** 到群
|
||||
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `preview` / `截图` / `预览` | 访问 `http://127.0.0.1:5173/` 并截图 |
|
||||
| `preview /login` | 指定路径 |
|
||||
| `preview / 3000` | 指定端口 |
|
||||
| `preview http://127.0.0.1:8080/` | 指定完整 URL |
|
||||
|
||||
**多步网页操作**（登录、点菜单、再截图）见下一节，不再写死在代码里。
|
||||
|
||||
`.env` 可选配置：
|
||||
|
||||
```env
|
||||
CURSOR_CWD=d:\LY\test\tech
|
||||
PREVIEW_PORT=5173
|
||||
PREVIEW_URL=http://127.0.0.1:5173/
|
||||
PREVIEW_DEV_COMMAND=npm run dev
|
||||
PREVIEW_STARTUP_TIMEOUT=120
|
||||
```
|
||||
|
||||
若 `CURSOR_CWD` 下暂无前端项目，可先手动启动 dev server，或设置 `PREVIEW_URL` 指向已运行地址。
|
||||
|
||||
### 4b. 网页操作（Playwright 步骤引擎）
|
||||
|
||||
支持三种方式定义操作流程，**无需改 Python 代码**：
|
||||
|
||||
**1. 自然语言（企微里直接说）**
|
||||
|
||||
```
|
||||
@test 访问登录页，输入账号密码，点击登录后进入主页，点击智能体管理菜单然后截图
|
||||
```
|
||||
|
||||
账号密码从 `.env` 读取（`{{PREVIEW_LOGIN_USER}}` / `{{PREVIEW_LOGIN_PASSWORD}}`），勿在群里发密码。
|
||||
|
||||
**2. 场景文件 YAML**
|
||||
|
||||
`bot/scenarios/xiaobao-agent-manage.yaml` 示例：
|
||||
|
||||
```yaml
|
||||
name: xiaobao-agent-manage
|
||||
steps:
|
||||
- goto: /login
|
||||
- fill:
|
||||
field: 账号
|
||||
value: "{{PREVIEW_LOGIN_USER}}"
|
||||
- fill:
|
||||
field: 密码
|
||||
value: "{{PREVIEW_LOGIN_PASSWORD}}"
|
||||
- click: 登录
|
||||
- wait:
|
||||
url: "**/app/**"
|
||||
- click: 智能体管理
|
||||
- wait: 1500
|
||||
- screenshot
|
||||
```
|
||||
|
||||
触发：`@test browser xiaobao-agent-manage`
|
||||
|
||||
场景搜索路径：`bot/scenarios/`、`CURSOR_CWD/.browser-scenarios/`、环境变量 `BROWSER_SCENARIOS_DIR`。
|
||||
|
||||
**3. 消息内 DSL**
|
||||
|
||||
```
|
||||
browser:
|
||||
goto /login
|
||||
fill 账号 {{PREVIEW_LOGIN_USER}}
|
||||
fill 密码 {{PREVIEW_LOGIN_PASSWORD}}
|
||||
click 登录
|
||||
click 智能体管理
|
||||
screenshot
|
||||
```
|
||||
|
||||
**支持的步骤**：`goto` · `fill` · `click` · `wait` · `screenshot` · `press`
|
||||
|
||||
`.env` 登录与场景配置：
|
||||
|
||||
```env
|
||||
PREVIEW_LOGIN_USER=test_account
|
||||
PREVIEW_LOGIN_PASSWORD=your_password
|
||||
# BROWSER_DEFAULT_SCENARIO=xiaobao-agent-manage
|
||||
```
|
||||
|
||||
### 5. 支持的快查命令
|
||||
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `trending 10` / `趋势 10` | 近期增长榜 Top N（默认 10，最大 30） |
|
||||
| `hot 10` / `实时 10` | 实时热度榜 |
|
||||
| `all 10` / `总榜 10` | 历史总安装榜 |
|
||||
| `搜索 react` / `search tdd` | 关键词搜索 |
|
||||
| `详情 find-skills` | 单个 skill 详情 + 安装命令 |
|
||||
| `preview` / `截图` | 启动 CURSOR_CWD 前端并截图发群 |
|
||||
| `帮助` | 命令列表 |
|
||||
|
||||
自然语言（非显式快查命令）会交给 **Cursor** 处理，例如 `@test 查 trending 并写推荐` 。
|
||||
|
||||
### 6. 本地测试（无需企微凭证）
|
||||
|
||||
```powershell
|
||||
cd d:\LY\test\tech\skills-hot-daily\bot
|
||||
python -c "from skills_service import handle_command; print(handle_command('trending 5'))"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 其他推送方案
|
||||
|
||||
| 方案 | 适用场景 | 复杂度 |
|
||||
|------|----------|--------|
|
||||
| **群机器人 webhook** | 推送到固定群 | 低 |
|
||||
| **应用消息 API** | 推送给指定成员/部门 | 中（需 corp_id、secret、agent_id） |
|
||||
| **邮件 + 企业微信邮箱** | 已有 SMTP | 中 |
|
||||
| **PushPlus / Server酱** | 个人微信中转 | 低（第三方） |
|
||||
|
||||
### 应用消息 API（简要）
|
||||
|
||||
适合「推送给某个人」而非群聊。需在 [企业微信管理后台](https://work.weixin.qq.com/) 创建自建应用，调用：
|
||||
|
||||
`POST https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=TOKEN`
|
||||
|
||||
消息体支持 `text` / `markdown` / `news` 等。需先 `gettoken` 再发消息，并维护 access_token 缓存。
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
- Webhook **不要提交到 Git**，只用环境变量
|
||||
- 企业微信 markdown 为**子集**（不支持完整 GitHub 表格语法时可改为文本列表）
|
||||
- 单条消息约 **4096 字节** 上限，`send-wecom.ps1` 已做截断
|
||||
29
bot/.env.example
Normal file
29
bot/.env.example
Normal file
@@ -0,0 +1,29 @@
|
||||
# 企业微信智能机器人（API 模式 · 长连接）
|
||||
# 管理后台 → 安全与管理 → 管理工具 → 智能机器人 → 创建 → API 模式 → 使用长连接
|
||||
WECOM_BOT_ID=your-bot-id
|
||||
WECOM_BOT_SECRET=your-bot-secret
|
||||
|
||||
# Cursor SDK（@ 机器人后的通用任务由 Cursor 执行）
|
||||
CURSOR_API_KEY=cursor_...
|
||||
CURSOR_CWD=d:\LY\test\tech
|
||||
CURSOR_MODEL=composer-2.5
|
||||
CURSOR_TIMEOUT=600
|
||||
|
||||
# 前端截图预览（基于 CURSOR_CWD）
|
||||
PREVIEW_PORT=5173
|
||||
PREVIEW_URL=http://127.0.0.1:5173/
|
||||
# PREVIEW_DEV_COMMAND=npm run dev
|
||||
# PREVIEW_STARTUP_TIMEOUT=120
|
||||
|
||||
# 登录后截图（账号密码只放 .env，切勿发到企微群）
|
||||
# PREVIEW_LOGIN_USER=your_account_or_phone
|
||||
# PREVIEW_LOGIN_PASSWORD=your_password
|
||||
# PREVIEW_AFTER_LOGIN_URL=/app/dashboard
|
||||
# PREVIEW_AUTO_LOGIN=true
|
||||
|
||||
# 网页操作场景目录（可选，默认 bot/scenarios 与 CURSOR_CWD/.browser-scenarios）
|
||||
# BROWSER_SCENARIOS_DIR=d:\path\to\scenarios
|
||||
# BROWSER_DEFAULT_SCENARIO=xiaobao-agent-manage
|
||||
|
||||
# hybrid=快查走本地 / 其余走 Cursor | cursor=全部 Cursor | skills=仅本地
|
||||
ROUTING_MODE=hybrid
|
||||
4
bot/.gitignore
vendored
Normal file
4
bot/.gitignore
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
.cache/
|
||||
.env
|
||||
.venv/
|
||||
.cache/screenshots/
|
||||
12
bot/bot_types.py
Normal file
12
bot/bot_types.py
Normal file
@@ -0,0 +1,12 @@
|
||||
"""Bot 内部数据结构。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class RouteResult:
|
||||
source: str
|
||||
text: str
|
||||
image_path: str | None = None
|
||||
156
bot/bridge_manager.py
Normal file
156
bot/bridge_manager.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""Windows 兼容的 Cursor SDK bridge 管理。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import codecs
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
|
||||
import env_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
READY_LINE_PREFIX = "cursor-sdk-bridge ready "
|
||||
_bridge_lock = threading.Lock()
|
||||
_bridge_process: subprocess.Popen[bytes] | None = None
|
||||
|
||||
|
||||
def _cursor_cwd() -> str:
|
||||
return env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
|
||||
|
||||
def _parse_discovery_line(line: str) -> Mapping[str, Any] | None:
|
||||
if not line.startswith(READY_LINE_PREFIX):
|
||||
return None
|
||||
payload = line[len(READY_LINE_PREFIX) :].strip()
|
||||
loaded = json.loads(payload)
|
||||
if not isinstance(loaded, dict):
|
||||
raise RuntimeError("Bridge discovery payload must be an object")
|
||||
return loaded
|
||||
|
||||
|
||||
def _read_discovery_polling(process: subprocess.Popen[bytes], timeout: float = 60) -> Mapping[str, Any]:
|
||||
"""不用 selectors，避免 Windows 上 WinError 10038。"""
|
||||
if process.stderr is None:
|
||||
raise RuntimeError("Bridge stderr unavailable")
|
||||
|
||||
fd = process.stderr.fileno()
|
||||
was_blocking = os.get_blocking(fd)
|
||||
os.set_blocking(fd, False)
|
||||
decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
|
||||
pending = ""
|
||||
stderr_lines: list[str] = []
|
||||
deadline = time.monotonic() + timeout
|
||||
|
||||
try:
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
chunk = os.read(fd, 8192)
|
||||
except BlockingIOError:
|
||||
chunk = b""
|
||||
|
||||
if chunk:
|
||||
pending += decoder.decode(chunk)
|
||||
while "\n" in pending:
|
||||
line, pending = pending.split("\n", 1)
|
||||
stderr_lines.append(line)
|
||||
discovery = _parse_discovery_line(line)
|
||||
if discovery is not None:
|
||||
return discovery
|
||||
else:
|
||||
code = process.poll()
|
||||
if code is not None:
|
||||
pending += decoder.decode(b"", final=True)
|
||||
if pending.strip():
|
||||
stderr_lines.append(pending.strip())
|
||||
joined = "\n".join(stderr_lines)[-2000:]
|
||||
raise RuntimeError(
|
||||
f"Bridge 启动失败 exit={code}: {joined or '无 stderr 输出'}"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
finally:
|
||||
os.set_blocking(fd, was_blocking)
|
||||
|
||||
raise RuntimeError("等待 Cursor bridge 就绪超时")
|
||||
|
||||
|
||||
def _auth_token_from_discovery(discovery: Mapping[str, Any]) -> str:
|
||||
token = str(discovery.get("authToken") or "").strip()
|
||||
if token:
|
||||
return token
|
||||
token_file = discovery.get("authTokenFile")
|
||||
if token_file:
|
||||
return Path(str(token_file)).read_text(encoding="utf-8").strip()
|
||||
raise RuntimeError("Bridge discovery 缺少 auth token")
|
||||
|
||||
|
||||
def warm_cursor_bridge(force: bool = False) -> None:
|
||||
"""启动 cursor-sdk-bridge 并写入 CURSOR_SDK_BRIDGE_* 环境变量。"""
|
||||
global _bridge_process
|
||||
|
||||
with _bridge_lock:
|
||||
if (
|
||||
not force
|
||||
and _bridge_process is not None
|
||||
and _bridge_process.poll() is None
|
||||
and os.environ.get("CURSOR_SDK_BRIDGE_URL")
|
||||
and os.environ.get("CURSOR_SDK_BRIDGE_TOKEN")
|
||||
):
|
||||
return
|
||||
|
||||
if _bridge_process is not None and _bridge_process.poll() is None:
|
||||
_bridge_process.terminate()
|
||||
try:
|
||||
_bridge_process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
_bridge_process.kill()
|
||||
|
||||
from cursor_sdk._vendor import resolve_bridge_path
|
||||
|
||||
cwd = _cursor_cwd()
|
||||
argv = [resolve_bridge_path(), "--workspace", cwd]
|
||||
logger.info("启动 Cursor bridge workspace=%s", cwd)
|
||||
|
||||
process = subprocess.Popen(
|
||||
argv,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
try:
|
||||
discovery = _read_discovery_polling(process)
|
||||
except Exception:
|
||||
process.kill()
|
||||
process.wait(timeout=5)
|
||||
raise
|
||||
|
||||
url = str(discovery.get("url") or "").strip()
|
||||
if not url:
|
||||
host = str(discovery.get("host") or "127.0.0.1")
|
||||
port = discovery.get("port")
|
||||
url = f"http://{host}:{port}"
|
||||
|
||||
token = _auth_token_from_discovery(discovery)
|
||||
os.environ["CURSOR_SDK_BRIDGE_URL"] = url
|
||||
os.environ["CURSOR_SDK_BRIDGE_TOKEN"] = token
|
||||
_bridge_process = process
|
||||
logger.info("Cursor bridge 就绪: %s", url)
|
||||
|
||||
|
||||
def shutdown_cursor_bridge() -> None:
|
||||
global _bridge_process
|
||||
with _bridge_lock:
|
||||
if _bridge_process is None:
|
||||
return
|
||||
if _bridge_process.poll() is None:
|
||||
_bridge_process.terminate()
|
||||
try:
|
||||
_bridge_process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
_bridge_process.kill()
|
||||
_bridge_process = None
|
||||
49
bot/browser_env.py
Normal file
49
bot/browser_env.py
Normal file
@@ -0,0 +1,49 @@
|
||||
"""浏览器场景变量替换与 base URL 解析。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import env_config
|
||||
|
||||
_VAR_PATTERN = re.compile(r"\{\{([A-Z0-9_]+)\}\}")
|
||||
|
||||
|
||||
def interpolate(value: str) -> str:
|
||||
def repl(match: re.Match[str]) -> str:
|
||||
key = match.group(1)
|
||||
resolved = env_config.env(key)
|
||||
if resolved is None:
|
||||
raise RuntimeError(f"场景变量未配置：{key}")
|
||||
return resolved
|
||||
|
||||
return _VAR_PATTERN.sub(repl, value)
|
||||
|
||||
|
||||
def default_base_url() -> str:
|
||||
explicit = (env_config.env("PREVIEW_BASE_URL") or "").strip()
|
||||
if explicit:
|
||||
return interpolate(explicit.rstrip("/"))
|
||||
|
||||
preview = (env_config.env("PREVIEW_URL") or "").strip()
|
||||
if preview:
|
||||
parsed = urlparse(preview)
|
||||
scheme = parsed.scheme or "http"
|
||||
host = parsed.hostname or "127.0.0.1"
|
||||
port = parsed.port
|
||||
if port and port not in (80, 443):
|
||||
return f"{scheme}://{host}:{port}"
|
||||
return f"{scheme}://{host}"
|
||||
|
||||
port = env_config.env("PREVIEW_PORT", "5173") or "5173"
|
||||
return f"http://127.0.0.1:{port}"
|
||||
|
||||
|
||||
def resolve_url(base_url: str, target: str) -> str:
|
||||
target = interpolate(target.strip())
|
||||
if target.startswith("http://") or target.startswith("https://"):
|
||||
return target
|
||||
if not target.startswith("/"):
|
||||
target = "/" + target
|
||||
return base_url.rstrip("/") + target
|
||||
269
bot/browser_executor.py
Normal file
269
bot/browser_executor.py
Normal file
@@ -0,0 +1,269 @@
|
||||
"""通用 Playwright 步骤执行器（不写死业务页面）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from browser_env import interpolate, resolve_url
|
||||
from browser_models import BrowserResult, BrowserScenario
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCREENSHOT_DIR = Path(__file__).resolve().parent / ".cache" / "screenshots"
|
||||
|
||||
FIELD_HINTS: dict[str, list[str]] = {
|
||||
"账号": [
|
||||
"#login-username",
|
||||
"input#login-username",
|
||||
"input[autocomplete='username']",
|
||||
"username",
|
||||
"account",
|
||||
"phone",
|
||||
"账号",
|
||||
"手机号",
|
||||
"企业账号",
|
||||
],
|
||||
"密码": [
|
||||
"#login-password input",
|
||||
"#login-password",
|
||||
"input#login-password",
|
||||
"input[type='password']",
|
||||
"password",
|
||||
"密码",
|
||||
],
|
||||
"用户名": ["#login-username", "input#login-username", "username", "account", "账号"],
|
||||
}
|
||||
|
||||
def _step_label(step: dict[str, Any], index: int) -> str:
|
||||
action = step.get("action", "?")
|
||||
target = step.get("target") or step.get("field") or step.get("url") or ""
|
||||
return f"{index + 1}. {action} {target}".strip()
|
||||
|
||||
|
||||
def _resolve_fill_locator(page, field: str, step: dict[str, Any]):
|
||||
if step.get("selector"):
|
||||
return page.locator(interpolate(str(step["selector"])))
|
||||
|
||||
field_key = interpolate(str(field))
|
||||
if step.get("label"):
|
||||
return page.get_by_label(interpolate(str(step["label"])), exact=False)
|
||||
if step.get("placeholder"):
|
||||
return page.get_by_placeholder(interpolate(str(step["placeholder"])), exact=False)
|
||||
|
||||
hints = FIELD_HINTS.get(field_key, [field_key])
|
||||
for hint in hints:
|
||||
if hint.startswith("#") or hint.startswith(".") or hint.startswith("["):
|
||||
locator = page.locator(hint)
|
||||
if locator.count() > 0:
|
||||
return locator.first
|
||||
for getter in (
|
||||
lambda h=hint: page.get_by_label(h, exact=False),
|
||||
lambda h=hint: page.get_by_placeholder(h, exact=False),
|
||||
):
|
||||
locator = getter()
|
||||
if locator.count() > 0:
|
||||
return locator.first
|
||||
|
||||
return page.locator("input, textarea").filter(has_text=field_key).first
|
||||
|
||||
|
||||
def _fill_field(page, field: str, step: dict[str, Any]) -> None:
|
||||
value = interpolate(str(step.get("value", "")))
|
||||
locator = _resolve_fill_locator(page, field, step)
|
||||
locator.click(timeout=10_000)
|
||||
locator.fill("", timeout=5_000)
|
||||
locator.fill(value, timeout=10_000)
|
||||
|
||||
|
||||
def _page_error_text(page) -> str | None:
|
||||
for selector in (
|
||||
".ant-message-error",
|
||||
".ant-form-item-explain-error",
|
||||
".ant-alert-error",
|
||||
):
|
||||
try:
|
||||
locator = page.locator(selector).first
|
||||
if locator.is_visible(timeout=300):
|
||||
text = locator.inner_text(timeout=1_000).strip()
|
||||
if text:
|
||||
return text
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _pathname_matches(pattern: str, pathname: str) -> bool:
|
||||
pattern = pattern.strip()
|
||||
if pattern in {"**/app/**", "**/app/*", "/app/**"}:
|
||||
return pathname.startswith("/app")
|
||||
if pattern.endswith("/**"):
|
||||
prefix = pattern[:-3].rstrip("/")
|
||||
if prefix.startswith("**/"):
|
||||
prefix = prefix[3:]
|
||||
if not prefix.startswith("/"):
|
||||
prefix = "/" + prefix
|
||||
return pathname.startswith(prefix)
|
||||
if "**" in pattern or "*" in pattern:
|
||||
regex = "^" + re.escape(pattern).replace(r"\*\*", ".*").replace(r"\*", "[^/]*") + "$"
|
||||
return re.search(regex, pathname) is not None
|
||||
return pathname == pattern or pathname.startswith(pattern)
|
||||
|
||||
|
||||
def _wait_for_url_pattern(page, pattern: str, timeout: int = 60_000) -> None:
|
||||
"""SPA 路由用 pathname 轮询；glob 模式不依赖 navigation 事件。"""
|
||||
deadline = time.monotonic() + timeout / 1000
|
||||
last_error: str | None = None
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
pathname = page.evaluate("() => window.location.pathname")
|
||||
if _pathname_matches(pattern, pathname):
|
||||
try:
|
||||
page.wait_for_load_state("networkidle", timeout=8_000)
|
||||
except Exception:
|
||||
page.wait_for_timeout(800)
|
||||
return
|
||||
|
||||
err = _page_error_text(page)
|
||||
if err and err != last_error:
|
||||
last_error = err
|
||||
logger.warning("页面提示：%s", err)
|
||||
if "/login" in pathname:
|
||||
raise RuntimeError(f"登录失败：{err}")
|
||||
|
||||
page.wait_for_timeout(400)
|
||||
|
||||
err = _page_error_text(page)
|
||||
hint_parts = [f"当前 URL：`{page.url}`"]
|
||||
if err:
|
||||
hint_parts.append(f"页面错误：{err}")
|
||||
elif last_error:
|
||||
hint_parts.append(f"页面错误：{last_error}")
|
||||
hint_parts.append("请确认 PREVIEW_LOGIN_USER/PASSWORD 正确，且登录 API（内网网关）可达。")
|
||||
raise RuntimeError(f"等待 URL 匹配 `{pattern}` 超时（{timeout}ms）。{' '.join(hint_parts)}")
|
||||
|
||||
|
||||
def _click_target(page, target: str) -> None:
|
||||
target = interpolate(target.strip())
|
||||
if target.lower() in {"登录", "login"}:
|
||||
for selector in ("button.login-submit", "button[type='submit']"):
|
||||
locator = page.locator(selector)
|
||||
if locator.count() > 0:
|
||||
locator.first.click(timeout=10_000)
|
||||
return
|
||||
|
||||
candidates = [
|
||||
page.get_by_role("menuitem", name=target, exact=True),
|
||||
page.get_by_role("button", name=target, exact=True),
|
||||
page.get_by_role("link", name=target, exact=True),
|
||||
page.get_by_text(target, exact=True),
|
||||
page.get_by_text(target, exact=False),
|
||||
]
|
||||
for locator in candidates:
|
||||
if locator.count() > 0:
|
||||
locator.first.click(timeout=10_000)
|
||||
return
|
||||
raise RuntimeError(f"未找到可点击元素：{target}")
|
||||
|
||||
|
||||
def _execute_step(page, base_url: str, step: dict[str, Any]) -> None:
|
||||
action = str(step.get("action", "")).lower()
|
||||
if action == "goto":
|
||||
target = step.get("target") or step.get("url") or "/"
|
||||
url = resolve_url(base_url, str(target))
|
||||
page.goto(url, wait_until="networkidle", timeout=60_000)
|
||||
return
|
||||
|
||||
if action == "fill":
|
||||
field = str(step.get("field") or step.get("target") or "账号")
|
||||
_fill_field(page, field, step)
|
||||
return
|
||||
|
||||
if action == "click":
|
||||
target = step.get("target") or step.get("text")
|
||||
if not target:
|
||||
raise RuntimeError("click 步骤缺少 target")
|
||||
_click_target(page, str(target))
|
||||
page.wait_for_timeout(800)
|
||||
return
|
||||
|
||||
if action == "wait":
|
||||
timeout = int(step.get("timeout") or 60_000)
|
||||
if step.get("url"):
|
||||
_wait_for_url_pattern(page, str(step["url"]), timeout=timeout)
|
||||
return
|
||||
if step.get("selector"):
|
||||
page.locator(interpolate(str(step["selector"]))).wait_for(timeout=30_000)
|
||||
return
|
||||
if step.get("text"):
|
||||
page.get_by_text(interpolate(str(step["text"])), exact=False).wait_for(timeout=30_000)
|
||||
return
|
||||
ms = int(step.get("ms") or 1500)
|
||||
page.wait_for_timeout(ms)
|
||||
return
|
||||
|
||||
if action == "press":
|
||||
key = str(step.get("key") or step.get("target") or "Enter")
|
||||
page.keyboard.press(key)
|
||||
return
|
||||
|
||||
if action == "screenshot":
|
||||
return
|
||||
|
||||
raise RuntimeError(f"未知步骤 action={action}")
|
||||
|
||||
|
||||
def run_browser_scenario_sync(scenario: BrowserScenario) -> BrowserResult:
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
slug = (scenario.name or "browser").replace(" ", "-")
|
||||
output = SCREENSHOT_DIR / f"{slug}-{stamp}.png"
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
step_log: list[str] = []
|
||||
final_url = scenario.base_url
|
||||
|
||||
with sync_playwright() as playwright:
|
||||
browser = playwright.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 720})
|
||||
|
||||
steps = list(scenario.steps)
|
||||
if steps and steps[-1].get("action") != "screenshot" and not any(
|
||||
s.get("action") == "screenshot" for s in steps
|
||||
):
|
||||
steps.append({"action": "screenshot"})
|
||||
|
||||
for index, step in enumerate(steps):
|
||||
label = _step_label(step, index)
|
||||
logger.info("执行步骤 %s", label)
|
||||
action = str(step.get("action", "")).lower()
|
||||
if action == "screenshot":
|
||||
page.wait_for_timeout(int(step.get("ms") or 1500))
|
||||
page.screenshot(path=str(output), full_page=False, type="png")
|
||||
final_url = page.url
|
||||
step_log.append(label + " ✓")
|
||||
continue
|
||||
try:
|
||||
_execute_step(page, scenario.base_url, step)
|
||||
final_url = page.url
|
||||
step_log.append(label + " ✓")
|
||||
except Exception as exc:
|
||||
err = _page_error_text(page)
|
||||
detail = f"（{err}）" if err else ""
|
||||
raise RuntimeError(f"步骤失败：{label} @ {page.url}{detail}") from exc
|
||||
|
||||
browser.close()
|
||||
|
||||
return BrowserResult(
|
||||
scenario_name=scenario.name,
|
||||
base_url=scenario.base_url,
|
||||
final_url=final_url,
|
||||
screenshot_path=output,
|
||||
step_count=len(steps),
|
||||
step_log=step_log,
|
||||
)
|
||||
26
bot/browser_models.py
Normal file
26
bot/browser_models.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""浏览器自动化步骤模型。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class BrowserScenario:
|
||||
name: str | None
|
||||
base_url: str
|
||||
steps: list[dict[str, Any]]
|
||||
source: str = "natural"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BrowserResult:
|
||||
scenario_name: str | None
|
||||
base_url: str
|
||||
final_url: str
|
||||
screenshot_path: Path
|
||||
step_count: int
|
||||
started_dev_server: bool = False
|
||||
step_log: list[str] = field(default_factory=list)
|
||||
307
bot/browser_parser.py
Normal file
307
bot/browser_parser.py
Normal file
@@ -0,0 +1,307 @@
|
||||
"""解析自然语言 / YAML / 场景名 → 浏览器步骤。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
import env_config
|
||||
from browser_env import default_base_url, interpolate
|
||||
from browser_models import BrowserScenario
|
||||
|
||||
SCENARIO_DIRS = [
|
||||
Path(__file__).resolve().parent / "scenarios",
|
||||
Path(__file__).resolve().parent.parent / "scenarios",
|
||||
]
|
||||
|
||||
|
||||
def _project_cwd() -> Path:
|
||||
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return Path(raw).resolve()
|
||||
|
||||
|
||||
def _strip_mention(text: str) -> str:
|
||||
return re.sub(r"@\S+\s*", "", text).strip()
|
||||
|
||||
|
||||
def _scenario_search_dirs() -> list[Path]:
|
||||
dirs = list(SCENARIO_DIRS)
|
||||
dirs.append(_project_cwd() / ".browser-scenarios")
|
||||
custom = (env_config.env("BROWSER_SCENARIOS_DIR") or "").strip()
|
||||
if custom:
|
||||
dirs.append(Path(custom).resolve())
|
||||
return dirs
|
||||
|
||||
|
||||
def is_browser_intent(text: str) -> bool:
|
||||
raw = _strip_mention(text)
|
||||
if not raw:
|
||||
return False
|
||||
if re.match(r"^(browser|网页|网页操作|操作)\b", raw, re.IGNORECASE):
|
||||
return True
|
||||
if re.search(r"```(?:yaml|yml)", raw, re.IGNORECASE):
|
||||
return True
|
||||
if re.search(r"(?m)^browser\s*:", raw, re.IGNORECASE):
|
||||
return True
|
||||
|
||||
if re.match(r"^(preview|截图|预览|截屏)\s", raw, re.IGNORECASE):
|
||||
if not re.search(r"[，,。；;]|然后|输入|点击|填写|访问|打开|登录", raw):
|
||||
return False
|
||||
|
||||
if len(_split_segments(text)) >= 2:
|
||||
return True
|
||||
|
||||
verbs = 0
|
||||
for pattern in (r"访问", r"打开", r"输入", r"填写", r"点击", r"点选", r"选择", r"登录"):
|
||||
if re.search(pattern, raw):
|
||||
verbs += 1
|
||||
return verbs >= 2
|
||||
|
||||
|
||||
def _load_yaml_scenario(path: Path) -> BrowserScenario:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise RuntimeError(f"场景文件格式错误：{path}")
|
||||
base_url = interpolate(str(data.get("base_url") or default_base_url()))
|
||||
steps = data.get("steps")
|
||||
if not isinstance(steps, list) or not steps:
|
||||
raise RuntimeError(f"场景缺少 steps：{path}")
|
||||
return BrowserScenario(
|
||||
name=data.get("name") or path.stem,
|
||||
base_url=base_url,
|
||||
steps=_normalize_steps(steps),
|
||||
source=f"file:{path.name}",
|
||||
)
|
||||
|
||||
|
||||
def _find_scenario_file(name: str) -> Path | None:
|
||||
slug = name.strip().replace(" ", "-")
|
||||
for directory in _scenario_search_dirs():
|
||||
for candidate in (directory / f"{slug}.yaml", directory / f"{slug}.yml"):
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_steps(raw_steps: list[Any]) -> list[dict[str, Any]]:
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for item in raw_steps:
|
||||
if isinstance(item, str):
|
||||
normalized.append({"action": item})
|
||||
continue
|
||||
if not isinstance(item, dict) or not item:
|
||||
raise RuntimeError(f"无效步骤：{item!r}")
|
||||
if "action" in item:
|
||||
normalized.append(dict(item))
|
||||
continue
|
||||
if len(item) == 1:
|
||||
action, payload = next(iter(item.items()))
|
||||
step = {"action": action}
|
||||
if payload is not None:
|
||||
if isinstance(payload, dict):
|
||||
step.update(payload)
|
||||
elif action == "wait" and isinstance(payload, int):
|
||||
step["ms"] = payload
|
||||
elif action == "wait" and isinstance(payload, str) and payload.isdigit():
|
||||
step["ms"] = int(payload)
|
||||
else:
|
||||
step["target"] = payload
|
||||
normalized.append(step)
|
||||
continue
|
||||
raise RuntimeError(f"无效步骤：{item!r}")
|
||||
return normalized
|
||||
|
||||
|
||||
def _parse_inline_dsl(text: str) -> BrowserScenario | None:
|
||||
raw = _strip_mention(text)
|
||||
match = re.search(r"(?ms)^browser\s*:\s*\n(.+)$", raw, re.IGNORECASE)
|
||||
if not match:
|
||||
return None
|
||||
|
||||
steps: list[dict[str, Any]] = []
|
||||
for line in match.group(1).splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
line = re.sub(r"^[-*]\s*", "", line)
|
||||
if not line:
|
||||
continue
|
||||
steps.append(_parse_dsl_line(line))
|
||||
|
||||
if not steps:
|
||||
return None
|
||||
return BrowserScenario(
|
||||
name="inline",
|
||||
base_url=default_base_url(),
|
||||
steps=steps,
|
||||
source="inline-dsl",
|
||||
)
|
||||
|
||||
|
||||
def _parse_dsl_line(line: str) -> dict[str, Any]:
|
||||
parts = line.split(None, 2)
|
||||
action = parts[0].lower()
|
||||
if action == "goto":
|
||||
return {"action": "goto", "target": parts[1] if len(parts) > 1 else "/"}
|
||||
if action == "click":
|
||||
return {"action": "click", "target": " ".join(parts[1:])}
|
||||
if action == "fill":
|
||||
if len(parts) < 3:
|
||||
raise RuntimeError(f"fill 语法：fill 字段 值（{line}）")
|
||||
return {"action": "fill", "field": parts[1], "value": parts[2]}
|
||||
if action == "wait":
|
||||
payload = parts[1] if len(parts) > 1 else "1500"
|
||||
if payload.isdigit():
|
||||
return {"action": "wait", "ms": int(payload)}
|
||||
return {"action": "wait", "url": payload}
|
||||
if action in {"screenshot", "shot"}:
|
||||
return {"action": "screenshot"}
|
||||
raise RuntimeError(f"未知 DSL 步骤：{line}")
|
||||
|
||||
|
||||
def _split_segments(text: str) -> list[str]:
|
||||
raw = _strip_mention(text)
|
||||
raw = re.sub(r"^(browser|网页|网页操作|操作)\s*[:：]?\s*", "", raw, flags=re.IGNORECASE)
|
||||
raw = re.sub(r"然后截图|再截图|最后截图", "截图", raw)
|
||||
chunks = re.split(r"[，,。；;]\s*|\s+然后\s+|\s+接着\s+|\s+并\s*", raw)
|
||||
expanded: list[str] = []
|
||||
for chunk in chunks:
|
||||
chunk = chunk.strip()
|
||||
if not chunk:
|
||||
continue
|
||||
subchunks = re.split(r"\s+然后\s+", chunk)
|
||||
if "后" in chunk and len(subchunks) == 1:
|
||||
subchunks = re.split(r"(?<=[登录页表单])后(?=[进入打开等待点击访问])", chunk)
|
||||
for part in subchunks:
|
||||
part = part.strip()
|
||||
if part:
|
||||
expanded.append(part)
|
||||
return expanded
|
||||
|
||||
|
||||
def _parse_segment(segment: str) -> list[dict[str, Any]]:
|
||||
seg = segment.strip()
|
||||
if not seg or seg.lower() in {"browser", "网页操作"}:
|
||||
return []
|
||||
|
||||
if re.fullmatch(r"截图|截屏", seg, re.IGNORECASE):
|
||||
return [{"action": "screenshot"}]
|
||||
|
||||
match = re.search(r"访问登录页|打开登录页|进入登录页|运行登录页", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "goto", "target": "/login"}]
|
||||
|
||||
match = re.search(r"输入账号密码|填写账号密码|输入账号和密码", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [
|
||||
{"action": "fill", "field": "账号", "value": "{{PREVIEW_LOGIN_USER}}"},
|
||||
{"action": "fill", "field": "密码", "value": "{{PREVIEW_LOGIN_PASSWORD}}"},
|
||||
]
|
||||
|
||||
match = re.search(r"输入账号|填写账号|输入用户名|填写用户名", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "fill", "field": "账号", "value": "{{PREVIEW_LOGIN_USER}}"}]
|
||||
|
||||
match = re.search(r"输入密码|填写密码", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "fill", "field": "密码", "value": "{{PREVIEW_LOGIN_PASSWORD}}"}]
|
||||
|
||||
match = re.search(r"进入主页|进入首页|打开主页|打开首页|等待主页", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "wait", "url": "**/app/**"}]
|
||||
|
||||
match = re.search(r"等待\s*(\d+)\s*秒", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "wait", "ms": int(match.group(1)) * 1000}]
|
||||
|
||||
match = re.search(
|
||||
r"(?:点击|点选|选择)\s*(.+?)(?:菜单|按钮|链接)?$",
|
||||
seg,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if match:
|
||||
target = match.group(1).strip()
|
||||
target = re.sub(r"(然后|再|并)?\s*(截图|截屏).*$", "", target, flags=re.IGNORECASE).strip()
|
||||
target = re.sub(r"(然后|再|之后)$", "", target).strip()
|
||||
target = re.sub(r"(菜单|按钮|链接)$", "", target).strip()
|
||||
if target:
|
||||
return [{"action": "click", "target": target}]
|
||||
|
||||
match = re.search(
|
||||
r"(?:访问|打开|进入)\s*(https?://\S+|/\S+|登录页|主页|首页)",
|
||||
seg,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if match:
|
||||
target = match.group(1)
|
||||
mapping = {"登录页": "/login", "主页": "/app/dashboard", "首页": "/app/dashboard"}
|
||||
return [{"action": "goto", "target": mapping.get(target, target)}]
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def parse_natural_language(text: str) -> BrowserScenario | None:
|
||||
segments = _split_segments(text)
|
||||
steps: list[dict[str, Any]] = []
|
||||
for segment in segments:
|
||||
steps.extend(_parse_segment(segment))
|
||||
|
||||
if not steps:
|
||||
return None
|
||||
if not any(step.get("action") == "screenshot" for step in steps):
|
||||
if re.search(r"截图|截屏", text, re.IGNORECASE):
|
||||
steps.append({"action": "screenshot"})
|
||||
if not steps:
|
||||
return None
|
||||
|
||||
return BrowserScenario(
|
||||
name="natural",
|
||||
base_url=default_base_url(),
|
||||
steps=steps,
|
||||
source="natural-language",
|
||||
)
|
||||
|
||||
|
||||
def parse_browser_request(text: str) -> BrowserScenario | None:
|
||||
if not is_browser_intent(text):
|
||||
return None
|
||||
|
||||
raw = _strip_mention(text)
|
||||
yaml_block = re.search(r"```(?:yaml|yml)\s*\n(.+?)```", raw, re.IGNORECASE | re.DOTALL)
|
||||
if yaml_block:
|
||||
data = yaml.safe_load(yaml_block.group(1))
|
||||
if isinstance(data, dict):
|
||||
base_url = interpolate(str(data.get("base_url") or default_base_url()))
|
||||
steps = data.get("steps") or []
|
||||
return BrowserScenario(
|
||||
name=data.get("name") or "yaml-inline",
|
||||
base_url=base_url,
|
||||
steps=_normalize_steps(steps),
|
||||
source="yaml-inline",
|
||||
)
|
||||
|
||||
inline = _parse_inline_dsl(text)
|
||||
if inline:
|
||||
return inline
|
||||
|
||||
match = re.match(r"^(browser|网页|网页操作|操作)\s+([\w\-./]+)\s*$", raw, re.IGNORECASE)
|
||||
if match:
|
||||
path = _find_scenario_file(match.group(2))
|
||||
if not path:
|
||||
raise RuntimeError(f"未找到场景文件：{match.group(2)}.yaml")
|
||||
return _load_yaml_scenario(path)
|
||||
|
||||
scenario = parse_natural_language(text)
|
||||
if scenario:
|
||||
return scenario
|
||||
|
||||
default_name = (env_config.env("BROWSER_DEFAULT_SCENARIO") or "").strip()
|
||||
if default_name:
|
||||
path = _find_scenario_file(default_name)
|
||||
if path:
|
||||
return _load_yaml_scenario(path)
|
||||
|
||||
return None
|
||||
87
bot/browser_service.py
Normal file
87
bot/browser_service.py
Normal file
@@ -0,0 +1,87 @@
|
||||
"""浏览器自动化服务：解析场景 + 启动 dev server + 执行步骤。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from browser_executor import run_browser_scenario_sync
|
||||
from browser_models import BrowserResult, BrowserScenario
|
||||
from browser_parser import parse_browser_request
|
||||
from preview_service import (
|
||||
_package_dev_script,
|
||||
_preview_port,
|
||||
_project_cwd,
|
||||
_startup_timeout,
|
||||
_wait_for_port,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _ensure_dev_server(base_url: str) -> bool:
|
||||
parsed = urlparse(base_url)
|
||||
host = parsed.hostname or "127.0.0.1"
|
||||
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||
|
||||
if _wait_for_port(host, port, timeout=3):
|
||||
return False
|
||||
|
||||
cwd = _project_cwd()
|
||||
dev_command = _package_dev_script(cwd)
|
||||
if not dev_command:
|
||||
raise RuntimeError(
|
||||
f"无法访问 {base_url}，且未找到可启动的 dev 脚本。"
|
||||
"请先手动启动前端，或设置 PREVIEW_URL。"
|
||||
)
|
||||
|
||||
import subprocess
|
||||
|
||||
logger.info("启动 dev server: %s (cwd=%s)", dev_command, cwd)
|
||||
proc = subprocess.Popen(
|
||||
dev_command,
|
||||
cwd=str(cwd),
|
||||
shell=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
if not _wait_for_port(host, port, timeout=_startup_timeout()):
|
||||
err = ""
|
||||
if proc.stderr:
|
||||
err = proc.stderr.read().decode("utf-8", errors="replace")[-1000:]
|
||||
proc.kill()
|
||||
raise RuntimeError(
|
||||
f"dev server 在 {_startup_timeout()}s 内未就绪 ({base_url})。"
|
||||
f"{(' 日志: ' + err) if err else ''}"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
async def run_browser_automation(text: str) -> BrowserResult:
|
||||
scenario = parse_browser_request(text)
|
||||
if scenario is None:
|
||||
raise RuntimeError("无法解析网页操作步骤")
|
||||
|
||||
started = await asyncio.to_thread(_ensure_dev_server, scenario.base_url)
|
||||
result = await asyncio.to_thread(run_browser_scenario_sync, scenario)
|
||||
result.started_dev_server = started
|
||||
return result
|
||||
|
||||
|
||||
def format_browser_caption(result: BrowserResult) -> str:
|
||||
lines = [
|
||||
"**网页操作完成**",
|
||||
f"> 场景：`{result.scenario_name or '自定义'}`",
|
||||
f"> 起始：`{result.base_url}`",
|
||||
f"> 最终：`{result.final_url}`",
|
||||
f"> 步骤数：{result.step_count}",
|
||||
]
|
||||
if result.started_dev_server:
|
||||
lines.append("> dev server：已自动启动")
|
||||
if result.step_log:
|
||||
lines.append("")
|
||||
lines.append("执行记录：")
|
||||
for item in result.step_log[-8:]:
|
||||
lines.append(f"- {item}")
|
||||
return "\n".join(lines)
|
||||
106
bot/cursor_runner.py
Normal file
106
bot/cursor_runner.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""通过 Cursor SDK 执行用户任务。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from typing import Awaitable, Callable
|
||||
|
||||
import env_config
|
||||
from bridge_manager import warm_cursor_bridge
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_cursor_lock = asyncio.Lock()
|
||||
|
||||
WECHAT_SYSTEM_PREFIX = """你是企业微信群里的 Skills 助手，正在回复群成员的消息。
|
||||
|
||||
要求：
|
||||
- 用简洁的中文回答（除非用户用其他语言提问）
|
||||
- 使用企业微信支持的 Markdown 子集（加粗、链接、列表；避免复杂表格）
|
||||
- 直接给出结论，不要冗长铺垫
|
||||
- 若任务涉及 skills.sh，可说明安装命令 `npx skills add owner/repo/skill-name`
|
||||
- **不要**在回复里写 `[图片]` 占位符；企微无法通过 Markdown 显示图片
|
||||
- 若用户要页面截图，请明确告知其发送：`截图` 或 `preview`（由 bot 自动发图）
|
||||
|
||||
用户任务：
|
||||
"""
|
||||
|
||||
|
||||
def _cursor_settings() -> dict[str, str | int]:
|
||||
timeout_raw = env_config.env("CURSOR_TIMEOUT", "600") or "600"
|
||||
return {
|
||||
"api_key": env_config.env("CURSOR_API_KEY"),
|
||||
"cwd": env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech",
|
||||
"model": env_config.env("CURSOR_MODEL", "composer-2.5") or "composer-2.5",
|
||||
"timeout": int(timeout_raw),
|
||||
}
|
||||
|
||||
|
||||
def strip_mention(text: str) -> str:
|
||||
return re.sub(r"@\S+\s*", "", text).strip()
|
||||
|
||||
|
||||
def _build_prompt(task: str) -> str:
|
||||
return WECHAT_SYSTEM_PREFIX + task.strip()
|
||||
|
||||
|
||||
def execute_cursor_task_sync(task: str) -> str:
|
||||
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
|
||||
|
||||
settings = _cursor_settings()
|
||||
api_key = settings["api_key"]
|
||||
if not api_key:
|
||||
raise RuntimeError(
|
||||
"未配置 CURSOR_API_KEY。请在 bot/.env 中设置，"
|
||||
"密钥见 https://cursor.com/dashboard/integrations"
|
||||
)
|
||||
|
||||
warm_cursor_bridge()
|
||||
|
||||
cwd = str(settings["cwd"])
|
||||
prompt = _build_prompt(task)
|
||||
logger.info("Cursor 执行任务 cwd=%s model=%s", cwd, settings["model"])
|
||||
|
||||
try:
|
||||
result = Agent.prompt(
|
||||
prompt,
|
||||
AgentOptions(
|
||||
api_key=api_key,
|
||||
model=settings["model"],
|
||||
local=LocalAgentOptions(cwd=cwd),
|
||||
),
|
||||
)
|
||||
except CursorAgentError as exc:
|
||||
raise RuntimeError(
|
||||
f"Cursor 启动失败：{exc.message}"
|
||||
+ ("（可重试）" if exc.is_retryable else "")
|
||||
) from exc
|
||||
|
||||
if result.status == "error":
|
||||
detail = result.result or "运行失败，无详细错误"
|
||||
raise RuntimeError(f"Cursor 执行失败：{detail}")
|
||||
|
||||
text = (result.result or "").strip()
|
||||
if not text:
|
||||
return "Cursor 已完成任务，但没有返回文本内容。"
|
||||
return text
|
||||
|
||||
|
||||
async def run_cursor_task(
|
||||
task: str,
|
||||
on_progress: Callable[[str], Awaitable[None]] | None = None,
|
||||
) -> str:
|
||||
timeout = int(_cursor_settings()["timeout"])
|
||||
if on_progress:
|
||||
await on_progress("Cursor 正在执行任务，请稍候…")
|
||||
|
||||
async with _cursor_lock:
|
||||
try:
|
||||
return await asyncio.wait_for(
|
||||
asyncio.to_thread(execute_cursor_task_sync, task),
|
||||
timeout=timeout,
|
||||
)
|
||||
except asyncio.TimeoutError as exc:
|
||||
raise RuntimeError(f"Cursor 执行超时（>{timeout}s）") from exc
|
||||
16
bot/env_config.py
Normal file
16
bot/env_config.py
Normal file
@@ -0,0 +1,16 @@
|
||||
"""加载 bot/.env，供各模块在 import 时统一读取环境变量。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
_BOT_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(_BOT_DIR / ".env")
|
||||
load_dotenv(_BOT_DIR / ".env.local", override=True)
|
||||
|
||||
|
||||
def env(key: str, default: str | None = None) -> str | None:
|
||||
return os.getenv(key, default)
|
||||
58
bot/image_extract.py
Normal file
58
bot/image_extract.py
Normal file
@@ -0,0 +1,58 @@
|
||||
"""从文本/Cursor 回复中解析本地截图路径。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import env_config
|
||||
|
||||
IMAGE_SUFFIXES = (".png", ".jpg", ".jpeg", ".webp")
|
||||
|
||||
|
||||
def _project_cwd() -> Path:
|
||||
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return Path(raw).resolve()
|
||||
|
||||
|
||||
def _resolve_candidate(raw: str, cwd: Path) -> Path | None:
|
||||
cleaned = raw.strip().strip("`\"'[]()")
|
||||
if not cleaned or cleaned.startswith("http"):
|
||||
return None
|
||||
path = Path(cleaned)
|
||||
if not path.is_absolute():
|
||||
path = cwd / path
|
||||
try:
|
||||
resolved = path.resolve()
|
||||
except OSError:
|
||||
return None
|
||||
if resolved.is_file() and resolved.suffix.lower() in IMAGE_SUFFIXES:
|
||||
return resolved
|
||||
return None
|
||||
|
||||
|
||||
def find_image_paths(text: str) -> list[Path]:
|
||||
cwd = _project_cwd()
|
||||
seen: set[Path] = set()
|
||||
found: list[Path] = []
|
||||
|
||||
patterns = [
|
||||
r"(?:保存(?:至|到)|saved\s+to|screenshot\s*[:：])\s*([^\s\n\]]+\.(?:png|jpe?g|webp))",
|
||||
r"([A-Za-z]:\\[^\s\n\]]+\.(?:png|jpe?g|webp))",
|
||||
r"([^\s\n\]]+\.(?:png|jpe?g|webp))",
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
for match in re.finditer(pattern, text, re.IGNORECASE):
|
||||
path = _resolve_candidate(match.group(1), cwd)
|
||||
if path and path not in seen:
|
||||
seen.add(path)
|
||||
found.append(path)
|
||||
|
||||
return found
|
||||
|
||||
|
||||
def strip_fake_image_markdown(text: str) -> str:
|
||||
text = re.sub(r"^\s*\[图片\]\s*$", "", text, flags=re.MULTILINE)
|
||||
text = re.sub(r"\n{3,}", "\n\n", text)
|
||||
return text.strip()
|
||||
156
bot/main.py
Normal file
156
bot/main.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""企业微信智能机器人 · Skills 助手（skills 快查 + 截图预览 + Cursor 执行任务）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
|
||||
import env_config
|
||||
from bridge_manager import shutdown_cursor_bridge, warm_cursor_bridge
|
||||
from router import route_message, routing_mode
|
||||
from skills_service import handle_command, warm_feed_cache
|
||||
from wecom_media import reply_image, upload_image
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
)
|
||||
logger = logging.getLogger("skills-bot")
|
||||
|
||||
BOT_ID = env_config.env("WECOM_BOT_ID") or env_config.env("WECHAT_BOT_ID")
|
||||
BOT_SECRET = env_config.env("WECOM_BOT_SECRET") or env_config.env("WECHAT_BOT_SECRET")
|
||||
|
||||
|
||||
def _require_credentials() -> None:
|
||||
if not BOT_ID or not BOT_SECRET:
|
||||
print(
|
||||
"请设置环境变量 WECOM_BOT_ID 和 WECOM_BOT_SECRET\n"
|
||||
"（企业微信 → 智能机器人 → API 模式 → 长连接）",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def create_client():
|
||||
from aibot import WSClient, WSClientOptions, generate_req_id
|
||||
|
||||
ws_client = WSClient(
|
||||
WSClientOptions(
|
||||
bot_id=BOT_ID,
|
||||
secret=BOT_SECRET,
|
||||
)
|
||||
)
|
||||
|
||||
@ws_client.on("authenticated")
|
||||
def on_authenticated():
|
||||
logger.info("企业微信长连接认证成功，路由模式=%s", routing_mode())
|
||||
cursor_key = env_config.env("CURSOR_API_KEY")
|
||||
if cursor_key:
|
||||
logger.info("CURSOR_API_KEY 已加载（%s…）", cursor_key[:8])
|
||||
try:
|
||||
warm_cursor_bridge()
|
||||
logger.info("Cursor bridge 预启动完成")
|
||||
except Exception as exc:
|
||||
logger.warning("Cursor bridge 预启动失败（Cursor 任务时会重试）: %s", exc)
|
||||
else:
|
||||
logger.warning("CURSOR_API_KEY 未配置，Cursor 任务将失败")
|
||||
try:
|
||||
warm_feed_cache()
|
||||
logger.info("skills 数据预加载完成")
|
||||
except Exception as exc:
|
||||
logger.warning("skills 数据预加载失败: %s", exc)
|
||||
|
||||
@ws_client.on("event.enter_chat")
|
||||
async def on_enter_chat(frame):
|
||||
help_text = handle_command("help")
|
||||
extra = (
|
||||
"\n\n---\n"
|
||||
"**单页截图**：`preview` / `截图` / `预览 [路径或URL]`\n"
|
||||
"**网页操作**：自然语言多步操作，或 `browser 场景名`\n"
|
||||
"例：`访问登录页，输入账号密码，点击登录，点击智能体管理，截图`\n"
|
||||
"场景文件：`bot/scenarios/*.yaml`（可用 `browser xiaobao-agent-manage`）"
|
||||
)
|
||||
await ws_client.reply_welcome(
|
||||
frame,
|
||||
{
|
||||
"msgtype": "markdown",
|
||||
"markdown": {"content": help_text + extra},
|
||||
},
|
||||
)
|
||||
|
||||
@ws_client.on("message.text")
|
||||
async def on_text(frame):
|
||||
body = frame.get("body", {})
|
||||
content = body.get("text", {}).get("content", "")
|
||||
logger.info("收到消息: %s", content)
|
||||
|
||||
stream_id = generate_req_id("stream")
|
||||
last_progress = ""
|
||||
|
||||
async def on_progress(message: str) -> None:
|
||||
nonlocal last_progress
|
||||
if message != last_progress:
|
||||
last_progress = message
|
||||
await ws_client.reply_stream(frame, stream_id, message, False)
|
||||
|
||||
await ws_client.reply_stream(frame, stream_id, "收到，正在处理…", False)
|
||||
|
||||
try:
|
||||
result = await route_message(content, on_progress=on_progress)
|
||||
reply = result.text
|
||||
logger.info(
|
||||
"回复来源: %s, 文本长度=%d, 图片=%s",
|
||||
result.source,
|
||||
len(reply),
|
||||
result.image_path or "-",
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.exception("处理失败")
|
||||
reply = f"处理失败：{exc}"
|
||||
result = None
|
||||
|
||||
if len(reply) > 3800:
|
||||
reply = reply[:3800] + "\n\n> …内容已截断"
|
||||
|
||||
await ws_client.reply_stream(frame, stream_id, reply, True)
|
||||
|
||||
if result and result.image_path:
|
||||
try:
|
||||
media_id = await upload_image(ws_client, result.image_path)
|
||||
await reply_image(ws_client, frame, media_id)
|
||||
logger.info("图片已发送到企微: %s", result.image_path)
|
||||
except Exception as exc:
|
||||
logger.exception("发送图片失败")
|
||||
await ws_client.reply(
|
||||
frame,
|
||||
{
|
||||
"msgtype": "markdown",
|
||||
"markdown": {
|
||||
"content": f"截图文件：`{result.image_path}`\n发图失败：{exc}\n\n请确认 bot 已重启，或发送 `截图` 重试。",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
@ws_client.on("error")
|
||||
def on_error(error):
|
||||
logger.error("连接错误: %s", error)
|
||||
|
||||
@ws_client.on("disconnected")
|
||||
def on_disconnected(reason):
|
||||
logger.warning("连接断开: %s", reason)
|
||||
|
||||
return ws_client
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import atexit
|
||||
|
||||
atexit.register(shutdown_cursor_bridge)
|
||||
_require_credentials()
|
||||
client = create_client()
|
||||
logger.info("启动 Skills 助手，Bot ID=%s…", BOT_ID[:8] if BOT_ID else "?")
|
||||
client.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
255
bot/preview_service.py
Normal file
255
bot/preview_service.py
Normal file
@@ -0,0 +1,255 @@
|
||||
"""在 CURSOR_CWD 启动/访问前端并截图（单页，不含多步操作）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import env_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCREENSHOT_DIR = Path(__file__).resolve().parent / ".cache" / "screenshots"
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewResult:
|
||||
url: str
|
||||
screenshot_path: Path
|
||||
started_dev_server: bool
|
||||
final_url: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewRequest:
|
||||
url: str | None
|
||||
port: int | None
|
||||
|
||||
|
||||
def _project_cwd() -> Path:
|
||||
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return Path(raw).resolve()
|
||||
|
||||
|
||||
def _preview_port() -> int:
|
||||
raw = env_config.env("PREVIEW_PORT", "5173") or "5173"
|
||||
return int(raw)
|
||||
|
||||
|
||||
def _startup_timeout() -> int:
|
||||
raw = env_config.env("PREVIEW_STARTUP_TIMEOUT", "120") or "120"
|
||||
return int(raw)
|
||||
|
||||
|
||||
def _dev_command() -> str:
|
||||
return env_config.env("PREVIEW_DEV_COMMAND", "npm run dev") or "npm run dev"
|
||||
|
||||
|
||||
def parse_preview_command(text: str) -> tuple[str | None, int | None] | None:
|
||||
raw = re.sub(r"@\S+\s*", "", text).strip()
|
||||
if not raw:
|
||||
return None
|
||||
|
||||
m = re.match(
|
||||
r"^(preview|截图|预览|截屏)(?:\s+(https?://\S+|/\S*))?(?:\s+(\d{2,5}))?$",
|
||||
raw,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if not m:
|
||||
return None
|
||||
|
||||
url_part = m.group(2)
|
||||
port_part = m.group(3)
|
||||
port = int(port_part) if port_part else None
|
||||
|
||||
if url_part and url_part.startswith("/"):
|
||||
port = port or _preview_port()
|
||||
return f"http://127.0.0.1:{port}{url_part}", port
|
||||
|
||||
return url_part, port
|
||||
|
||||
|
||||
def resolve_preview_request(text: str) -> PreviewRequest | None:
|
||||
explicit = parse_preview_command(text)
|
||||
if explicit is not None:
|
||||
url_override, port_override = explicit
|
||||
return PreviewRequest(url=url_override, port=port_override)
|
||||
|
||||
if not is_preview_intent(text):
|
||||
return None
|
||||
|
||||
url_override = extract_url_from_text(text)
|
||||
if not url_override:
|
||||
env_url = env_config.env("PREVIEW_URL")
|
||||
url_override = env_url.strip() if env_url else f"http://127.0.0.1:{_preview_port()}/"
|
||||
|
||||
return PreviewRequest(url=url_override, port=None)
|
||||
|
||||
|
||||
_PREVIEW_INTENT = re.compile(
|
||||
r"^(preview|截图|预览|截屏)\b|"
|
||||
r"(页面预览|运行.*(前端|项目|页面)|"
|
||||
r"打开.*(前端|页面|项目)|"
|
||||
r"访问.*(并)?.*(截图|截屏)|"
|
||||
r"启动.*(前端|项目|dev|服务).*(截图|截屏)?)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def is_preview_intent(text: str) -> bool:
|
||||
raw = re.sub(r"@\S+\s*", "", text).strip()
|
||||
if parse_preview_command(text) is not None:
|
||||
return True
|
||||
return bool(_PREVIEW_INTENT.search(raw))
|
||||
|
||||
|
||||
def extract_url_from_text(text: str) -> str | None:
|
||||
raw = re.sub(r"@\S+\s*", "", text)
|
||||
match = re.search(
|
||||
r"(https?://[^\s\]`\"']+|localhost:\d+[/\w\-./]*)",
|
||||
raw,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if not match:
|
||||
return None
|
||||
url = match.group(1).rstrip(".,，。")
|
||||
if url.lower().startswith("localhost"):
|
||||
url = "http://" + url
|
||||
return url
|
||||
|
||||
|
||||
def _capture_screenshot_sync(url: str, output: Path) -> str:
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 720})
|
||||
page.goto(url, wait_until="networkidle", timeout=60_000)
|
||||
page.wait_for_timeout(1500)
|
||||
page.screenshot(path=str(output), full_page=False, type="png")
|
||||
final_url = page.url
|
||||
browser.close()
|
||||
return final_url
|
||||
|
||||
|
||||
def _wait_for_port(host: str, port: int, timeout: int) -> bool:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with socket.create_connection((host, port), timeout=2):
|
||||
return True
|
||||
except OSError:
|
||||
time.sleep(1)
|
||||
return False
|
||||
|
||||
|
||||
def _resolve_target_url(url_override: str | None, port_override: int | None) -> tuple[str, str | None]:
|
||||
if url_override:
|
||||
parsed = urlparse(url_override)
|
||||
if parsed.scheme and parsed.netloc:
|
||||
return url_override, None
|
||||
raise RuntimeError(f"无效 URL：{url_override}")
|
||||
|
||||
env_url = env_config.env("PREVIEW_URL")
|
||||
if env_url:
|
||||
return env_url.strip(), None
|
||||
|
||||
port = port_override or _preview_port()
|
||||
cwd = _project_cwd()
|
||||
dev_script = _package_dev_script(cwd)
|
||||
base = f"http://127.0.0.1:{port}/"
|
||||
return base, dev_script
|
||||
|
||||
|
||||
def _package_dev_script(cwd: Path) -> str | None:
|
||||
pkg = cwd / "package.json"
|
||||
if not pkg.exists():
|
||||
return None
|
||||
try:
|
||||
data = json.loads(pkg.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
scripts = data.get("scripts") or {}
|
||||
for key in ("dev", "preview", "start"):
|
||||
if scripts.get(key):
|
||||
cmd = _dev_command()
|
||||
if key != "dev" and cmd == "npm run dev":
|
||||
return f"npm run {key}"
|
||||
return cmd
|
||||
return None
|
||||
|
||||
|
||||
def _capture_preview_sync(url: str, dev_command: str | None) -> PreviewResult:
|
||||
cwd = _project_cwd()
|
||||
parsed = urlparse(url)
|
||||
host = parsed.hostname or "127.0.0.1"
|
||||
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||
|
||||
dev_proc: subprocess.Popen | None = None
|
||||
started = False
|
||||
|
||||
if dev_command:
|
||||
if _wait_for_port(host, port, timeout=3):
|
||||
logger.info("检测到端口 %s 已监听，跳过启动 dev server", port)
|
||||
else:
|
||||
logger.info("启动 dev server: %s (cwd=%s)", dev_command, cwd)
|
||||
dev_proc = subprocess.Popen(
|
||||
dev_command,
|
||||
cwd=str(cwd),
|
||||
shell=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
started = True
|
||||
if not _wait_for_port(host, port, timeout=_startup_timeout()):
|
||||
err = ""
|
||||
if dev_proc.stderr:
|
||||
err = dev_proc.stderr.read().decode("utf-8", errors="replace")[-1000:]
|
||||
raise RuntimeError(
|
||||
f"dev server 在 {_startup_timeout()}s 内未就绪 ({url})。"
|
||||
f"{(' 日志: ' + err) if err else ''}"
|
||||
)
|
||||
else:
|
||||
if not _wait_for_port(host, port, timeout=5):
|
||||
raise RuntimeError(
|
||||
f"无法访问 {url}。请在 CURSOR_CWD 放置前端项目，"
|
||||
"或先手动启动 dev server，或设置 PREVIEW_URL。"
|
||||
)
|
||||
|
||||
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
output = SCREENSHOT_DIR / f"preview-{stamp}.png"
|
||||
|
||||
try:
|
||||
final_url = _capture_screenshot_sync(url, output)
|
||||
finally:
|
||||
if dev_proc and dev_proc.poll() is None:
|
||||
dev_proc.terminate()
|
||||
try:
|
||||
dev_proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
dev_proc.kill()
|
||||
|
||||
return PreviewResult(
|
||||
url=url,
|
||||
screenshot_path=output,
|
||||
started_dev_server=started,
|
||||
final_url=final_url,
|
||||
)
|
||||
|
||||
|
||||
async def capture_preview(
|
||||
url_override: str | None = None,
|
||||
port_override: int | None = None,
|
||||
) -> PreviewResult:
|
||||
url, dev_command = _resolve_target_url(url_override, port_override)
|
||||
return await asyncio.to_thread(_capture_preview_sync, url, dev_command)
|
||||
7
bot/requirements.txt
Normal file
7
bot/requirements.txt
Normal file
@@ -0,0 +1,7 @@
|
||||
wecom-aibot-python-sdk>=1.0.2
|
||||
python-dotenv>=1.0.0
|
||||
httpx>=0.27.0
|
||||
certifi>=2024.0.0
|
||||
cursor-sdk>=0.1.0
|
||||
playwright>=1.49.0
|
||||
PyYAML>=6.0.0
|
||||
109
bot/router.py
Normal file
109
bot/router.py
Normal file
@@ -0,0 +1,109 @@
|
||||
"""消息路由：skills 快查 / 网页操作 / 截图预览 / Cursor 通用任务。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import env_config
|
||||
from browser_parser import is_browser_intent, parse_browser_request
|
||||
from browser_service import format_browser_caption, run_browser_automation
|
||||
from cursor_runner import run_cursor_task, strip_mention
|
||||
from image_extract import find_image_paths, strip_fake_image_markdown
|
||||
from preview_service import capture_preview, is_preview_intent, resolve_preview_request
|
||||
from skills_service import handle_command, parse_command
|
||||
from bot_types import RouteResult
|
||||
|
||||
|
||||
def routing_mode() -> str:
|
||||
return (env_config.env("ROUTING_MODE", "hybrid") or "hybrid").lower()
|
||||
|
||||
|
||||
def _normalize(text: str) -> str:
|
||||
return re.sub(r"@\S+\s*", "", text).strip().lower()
|
||||
|
||||
|
||||
def is_skills_fast_command(text: str) -> bool:
|
||||
raw = _normalize(text)
|
||||
if not raw:
|
||||
return True
|
||||
if raw in {"help", "帮助", "?", "h"}:
|
||||
return True
|
||||
|
||||
cmd = parse_command(text)
|
||||
if cmd.kind in {"help", "list", "detail"}:
|
||||
return True
|
||||
if cmd.kind == "search" and re.match(r"^(search|搜索|find|查)\s+", raw):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _run_browser(text: str, on_progress=None) -> RouteResult:
|
||||
if parse_browser_request(text) is None:
|
||||
raise RuntimeError("无法解析网页操作步骤")
|
||||
|
||||
if on_progress:
|
||||
await on_progress("正在按步骤执行网页操作…")
|
||||
|
||||
result = await run_browser_automation(text)
|
||||
return RouteResult(
|
||||
source="browser",
|
||||
text=format_browser_caption(result),
|
||||
image_path=str(result.screenshot_path),
|
||||
)
|
||||
|
||||
|
||||
async def _run_preview(text: str, on_progress=None) -> RouteResult:
|
||||
preview_req = resolve_preview_request(text)
|
||||
if preview_req is None:
|
||||
raise RuntimeError("无法解析截图请求")
|
||||
|
||||
if on_progress:
|
||||
await on_progress(f"正在访问并截图：{preview_req.url or '默认地址'}…")
|
||||
|
||||
result = await capture_preview(preview_req.url, preview_req.port)
|
||||
caption = (
|
||||
f"**页面预览**\n"
|
||||
f"> URL：`{result.final_url or result.url}`\n"
|
||||
f"> 项目：`{env_config.env('CURSOR_CWD', '')}`\n"
|
||||
f"> dev server：{'已自动启动' if result.started_dev_server else '使用已有服务'}"
|
||||
)
|
||||
return RouteResult(
|
||||
source="preview",
|
||||
text=caption,
|
||||
image_path=str(result.screenshot_path),
|
||||
)
|
||||
|
||||
|
||||
async def route_message(text: str, on_progress=None) -> RouteResult:
|
||||
task = strip_mention(text)
|
||||
if not task:
|
||||
return RouteResult("skills", handle_command("help"))
|
||||
|
||||
if is_browser_intent(text):
|
||||
return await _run_browser(text, on_progress=on_progress)
|
||||
|
||||
if resolve_preview_request(text) is not None:
|
||||
return await _run_preview(text, on_progress=on_progress)
|
||||
|
||||
mode = routing_mode()
|
||||
if mode == "skills":
|
||||
return RouteResult("skills", handle_command(text))
|
||||
|
||||
if mode == "cursor" or not is_skills_fast_command(text):
|
||||
reply = await run_cursor_task(task, on_progress=on_progress)
|
||||
reply = strip_fake_image_markdown(reply)
|
||||
|
||||
image_path: str | None = None
|
||||
paths = find_image_paths(reply)
|
||||
if paths:
|
||||
image_path = str(paths[0])
|
||||
elif is_preview_intent(text) or is_browser_intent(text):
|
||||
if on_progress:
|
||||
await on_progress("未找到截图文件，改用 Playwright 自动执行…")
|
||||
if is_browser_intent(text):
|
||||
return await _run_browser(text, on_progress=on_progress)
|
||||
return await _run_preview(text, on_progress=on_progress)
|
||||
|
||||
return RouteResult("cursor", reply, image_path=image_path)
|
||||
|
||||
return RouteResult("skills", handle_command(text))
|
||||
17
bot/scenarios/xiaobao-agent-manage.yaml
Normal file
17
bot/scenarios/xiaobao-agent-manage.yaml
Normal file
@@ -0,0 +1,17 @@
|
||||
name: xiaobao-agent-manage
|
||||
description: 登录后打开智能体管理并截图
|
||||
steps:
|
||||
- goto: /login
|
||||
- fill:
|
||||
field: 账号
|
||||
value: "{{PREVIEW_LOGIN_USER}}"
|
||||
- fill:
|
||||
field: 密码
|
||||
value: "{{PREVIEW_LOGIN_PASSWORD}}"
|
||||
- click: 登录
|
||||
- wait:
|
||||
url: "**/app/**"
|
||||
timeout: 60000
|
||||
- click: 智能体管理
|
||||
- wait: 1500
|
||||
- screenshot
|
||||
317
bot/skills_service.py
Normal file
317
bot/skills_service.py
Normal file
@@ -0,0 +1,317 @@
|
||||
"""skills.sh 数据查询与命令解析。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FEED_URLS = [
|
||||
# jsDelivr 在国内通常比 raw.githubusercontent.com 更稳定
|
||||
"https://cdn.jsdelivr.net/gh/NeverSight/skills.sh_feed@main/data/feed.json",
|
||||
"https://raw.githubusercontent.com/NeverSight/skills.sh_feed/main/data/feed.json",
|
||||
]
|
||||
CACHE_TTL_SECONDS = 600
|
||||
CACHE_DIR = Path(__file__).resolve().parent / ".cache"
|
||||
CACHE_FILE = CACHE_DIR / "feed.json"
|
||||
|
||||
_cache: dict[str, Any] = {"data": None, "fetched_at": 0.0}
|
||||
|
||||
Board = Literal["trending", "hot", "all"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Command:
|
||||
kind: Literal["help", "list", "search", "detail"]
|
||||
board: Board = "trending"
|
||||
limit: int = 10
|
||||
query: str = ""
|
||||
|
||||
|
||||
def _fetch_json(url: str) -> dict[str, Any]:
|
||||
headers = {
|
||||
"User-Agent": "skills-hot-bot/1.0",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
with httpx.Client(
|
||||
timeout=httpx.Timeout(20.0, connect=10.0),
|
||||
verify=certifi.where(),
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
resp = client.get(url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
def _load_disk_cache() -> dict[str, Any] | None:
|
||||
if not CACHE_FILE.exists():
|
||||
return None
|
||||
try:
|
||||
return json.loads(CACHE_FILE.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
logger.warning("读取本地缓存失败: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _save_disk_cache(data: dict[str, Any]) -> None:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
|
||||
def load_feed(force: bool = False) -> dict[str, Any]:
|
||||
now = time.time()
|
||||
if not force and _cache["data"] and now - _cache["fetched_at"] < CACHE_TTL_SECONDS:
|
||||
return _cache["data"]
|
||||
|
||||
errors: list[str] = []
|
||||
for url in FEED_URLS:
|
||||
for attempt in range(3):
|
||||
try:
|
||||
data = _fetch_json(url)
|
||||
_cache["data"] = data
|
||||
_cache["fetched_at"] = now
|
||||
_save_disk_cache(data)
|
||||
logger.info("skills 数据已更新: %s", url)
|
||||
return data
|
||||
except Exception as exc:
|
||||
msg = f"{url} (#{attempt + 1}): {exc}"
|
||||
errors.append(msg)
|
||||
logger.debug("拉取失败 %s", msg)
|
||||
time.sleep(0.5 * (attempt + 1))
|
||||
|
||||
stale = _load_disk_cache()
|
||||
if stale:
|
||||
logger.warning("网络不可用，回退到本地缓存")
|
||||
_cache["data"] = stale
|
||||
_cache["fetched_at"] = now
|
||||
return stale
|
||||
|
||||
raise RuntimeError(f"无法获取 skills 数据。最近错误: {errors[-1]}")
|
||||
|
||||
|
||||
def warm_feed_cache() -> None:
|
||||
"""启动时预加载，避免首条消息才触发网络请求。"""
|
||||
load_feed(force=True)
|
||||
|
||||
|
||||
def _normalize_text(text: str) -> str:
|
||||
text = re.sub(r"@\S+\s*", "", text)
|
||||
return text.strip().lower()
|
||||
|
||||
|
||||
def _parse_limit(raw: str | None, default: int = 10) -> int:
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
n = int(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
return max(1, min(n, 30))
|
||||
|
||||
|
||||
def _match_list(raw: str, board: Board, aliases: str) -> Command | None:
|
||||
m = re.match(rf"^({aliases})(?:\s+top)?\s*(\d+)?$", raw)
|
||||
if m:
|
||||
return Command(kind="list", board=board, limit=_parse_limit(m.group(2)))
|
||||
m = re.match(rf"^(查|查询)\s+({aliases})(?:\s+top)?\s*(\d+)?$", raw)
|
||||
if m:
|
||||
return Command(kind="list", board=board, limit=_parse_limit(m.group(3)))
|
||||
return None
|
||||
|
||||
|
||||
def parse_command(text: str) -> Command:
|
||||
raw = _normalize_text(text)
|
||||
if not raw or raw in {"help", "帮助", "?", "h"}:
|
||||
return Command(kind="help")
|
||||
|
||||
for board, aliases in (
|
||||
("trending", "trending|趋势|top"),
|
||||
("hot", "hot|实时|热门"),
|
||||
("all", "all|总榜|alltime|all-time"),
|
||||
):
|
||||
cmd = _match_list(raw, board, aliases)
|
||||
if cmd:
|
||||
return cmd
|
||||
|
||||
m = re.match(r"^(search|搜索|find|查)\s+(.+)$", raw)
|
||||
if m:
|
||||
return Command(kind="search", query=m.group(2).strip(), limit=5)
|
||||
|
||||
m = re.match(r"^(detail|详情|skill|info)\s+(.+)$", raw)
|
||||
if m:
|
||||
return Command(kind="detail", query=m.group(2).strip())
|
||||
|
||||
if raw.startswith("trending") or raw.startswith("趋势"):
|
||||
parts = raw.split(maxsplit=1)
|
||||
return Command(kind="list", board="trending", limit=_parse_limit(parts[1] if len(parts) > 1 else None))
|
||||
|
||||
return Command(kind="search", query=raw, limit=5)
|
||||
|
||||
|
||||
def _format_installs(n: int | float) -> str:
|
||||
if n >= 1_000_000:
|
||||
return f"{n / 1_000_000:.1f}M"
|
||||
if n >= 1_000:
|
||||
return f"{n / 1_000:.1f}K"
|
||||
return str(int(n))
|
||||
|
||||
|
||||
def _board_items(feed: dict[str, Any], board: Board) -> list[dict[str, Any]]:
|
||||
key = {"trending": "topTrending", "hot": "topHot", "all": "topAllTime"}[board]
|
||||
return feed.get(key, [])
|
||||
|
||||
|
||||
def _board_title(board: Board) -> str:
|
||||
return {
|
||||
"trending": "Trending（近期增长）",
|
||||
"hot": "Hot（实时热度）",
|
||||
"all": "All Time（总安装榜）",
|
||||
}[board]
|
||||
|
||||
|
||||
def format_list(board: Board, limit: int) -> str:
|
||||
feed = load_feed()
|
||||
items = _board_items(feed, board)[:limit]
|
||||
updated = feed.get("updatedAt", "未知")[:10]
|
||||
|
||||
lines = [
|
||||
f"**skills.sh {_board_title(board)} Top {limit}**",
|
||||
f"> 数据更新：{updated}",
|
||||
"",
|
||||
]
|
||||
|
||||
for i, item in enumerate(items, 1):
|
||||
title = item.get("title", "?")
|
||||
source = item.get("source", "?")
|
||||
installs = _format_installs(item.get("installs", 0))
|
||||
desc = item.get("description", "")
|
||||
if len(desc) > 80:
|
||||
desc = desc[:77] + "..."
|
||||
link = item.get("link", "")
|
||||
lines.append(f"{i}. **{title}** · {installs}")
|
||||
lines.append(f" `{source}`")
|
||||
if desc:
|
||||
lines.append(f" {desc}")
|
||||
if link:
|
||||
lines.append(f" [查看]({link})")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines).strip()
|
||||
|
||||
|
||||
def format_search(query: str, limit: int) -> str:
|
||||
feed = load_feed()
|
||||
q = query.lower()
|
||||
seen: set[str] = set()
|
||||
matches: list[dict[str, Any]] = []
|
||||
|
||||
for board in ("topTrending", "topHot", "topAllTime"):
|
||||
for item in feed.get(board, []):
|
||||
item_id = item.get("id") or item.get("title", "")
|
||||
if item_id in seen:
|
||||
continue
|
||||
haystack = " ".join(
|
||||
[
|
||||
item.get("title", ""),
|
||||
item.get("source", ""),
|
||||
item.get("description", ""),
|
||||
]
|
||||
).lower()
|
||||
if q in haystack:
|
||||
seen.add(item_id)
|
||||
matches.append(item)
|
||||
if len(matches) >= limit:
|
||||
break
|
||||
if len(matches) >= limit:
|
||||
break
|
||||
|
||||
if not matches:
|
||||
return f"未找到与 **{query}** 相关的 skill。\n\n试试：`trending 10` / `hot 10` / `搜索 react`"
|
||||
|
||||
lines = [f"**搜索「{query}」** 共 {len(matches)} 条", ""]
|
||||
for i, item in enumerate(matches, 1):
|
||||
title = item.get("title", "?")
|
||||
source = item.get("source", "?")
|
||||
installs = _format_installs(item.get("installs", 0))
|
||||
link = item.get("link", "")
|
||||
lines.append(f"{i}. **{title}** · {installs} · `{source}`")
|
||||
if link:
|
||||
lines.append(f" [查看]({link})")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_detail(name: str) -> str:
|
||||
feed = load_feed()
|
||||
q = name.lower().strip()
|
||||
best: dict[str, Any] | None = None
|
||||
|
||||
for board in ("topTrending", "topHot", "topAllTime"):
|
||||
for item in feed.get(board, []):
|
||||
title = (item.get("title") or "").lower()
|
||||
item_id = (item.get("id") or "").lower()
|
||||
if title == q or q in title or q in item_id:
|
||||
if best is None or item.get("installs", 0) > best.get("installs", 0):
|
||||
best = item
|
||||
|
||||
if not best:
|
||||
return f"未找到 skill：**{name}**\n\n试试：`搜索 {name}`"
|
||||
|
||||
desc = best.get("description", "无描述")
|
||||
return "\n".join(
|
||||
[
|
||||
f"**{best.get('title', '?')}**",
|
||||
f"`{best.get('source', '?')}`",
|
||||
f"安装量：**{_format_installs(best.get('installs', 0))}**",
|
||||
"",
|
||||
desc,
|
||||
"",
|
||||
f"[skills.sh 详情]({best.get('link', 'https://skills.sh')})",
|
||||
"",
|
||||
f"安装：`npx skills add {best.get('source', '')}/{best.get('title', '')}`",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def format_help() -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
"**Skills 助手 · 命令帮助**",
|
||||
"",
|
||||
"`trending 10` / `趋势 10` — 近期增长榜",
|
||||
"`hot 10` / `实时 10` — 实时热度榜",
|
||||
"`all 10` / `总榜 10` — 历史总安装榜",
|
||||
"`搜索 react` / `search tdd` — 关键词搜索",
|
||||
"`详情 find-skills` — 查看单个 skill",
|
||||
"`preview` / `截图` / `预览` — 单页截图",
|
||||
"`browser 场景名` — 执行 YAML 场景（见 bot/scenarios/）",
|
||||
"自然语言 — 如：访问登录页，输入账号密码，点击登录，点击智能体管理，截图",
|
||||
"`preview /about 5173` — 指定路径和端口",
|
||||
"",
|
||||
"示例：",
|
||||
"• trending top10",
|
||||
"• 查 grill",
|
||||
"• 详情 remotion-render",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def handle_command(text: str) -> str:
|
||||
cmd = parse_command(text)
|
||||
if cmd.kind == "help":
|
||||
return format_help()
|
||||
if cmd.kind == "list":
|
||||
return format_list(cmd.board, cmd.limit)
|
||||
if cmd.kind == "search":
|
||||
return format_search(cmd.query, cmd.limit)
|
||||
if cmd.kind == "detail":
|
||||
return format_detail(cmd.query)
|
||||
return format_help()
|
||||
96
bot/wecom_media.py
Normal file
96
bot/wecom_media.py
Normal file
@@ -0,0 +1,96 @@
|
||||
"""企业微信 API 模式：上传图片并回复。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from aibot import generate_req_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CHUNK_SIZE = 512 * 1024
|
||||
MAX_IMAGE_BYTES = 9 * 1024 * 1024
|
||||
|
||||
|
||||
def _ensure_image_size(path: Path) -> bytes:
|
||||
data = path.read_bytes()
|
||||
if len(data) > MAX_IMAGE_BYTES:
|
||||
raise RuntimeError(
|
||||
f"截图过大（{len(data) // 1024}KB），请缩小页面或使用 viewport 截图（上限 9MB）"
|
||||
)
|
||||
return data
|
||||
|
||||
|
||||
def _response_body(frame: dict[str, Any]) -> dict[str, Any]:
|
||||
if frame.get("errcode", 0) != 0:
|
||||
raise RuntimeError(
|
||||
f"企微接口错误 errcode={frame.get('errcode')} errmsg={frame.get('errmsg')}"
|
||||
)
|
||||
body = frame.get("body")
|
||||
return body if isinstance(body, dict) else {}
|
||||
|
||||
|
||||
async def upload_image(ws_client: Any, image_path: str | Path) -> str:
|
||||
path = Path(image_path)
|
||||
if not path.exists():
|
||||
raise RuntimeError(f"截图不存在: {path}")
|
||||
|
||||
data = _ensure_image_size(path)
|
||||
md5 = hashlib.md5(data).hexdigest()
|
||||
chunks = [data[i : i + CHUNK_SIZE] for i in range(0, len(data), CHUNK_SIZE)]
|
||||
total_chunks = len(chunks)
|
||||
|
||||
manager = ws_client._ws_manager
|
||||
|
||||
init_frame = await manager.send_reply(
|
||||
generate_req_id("upload_init"),
|
||||
{
|
||||
"type": "image",
|
||||
"filename": path.name,
|
||||
"total_size": len(data),
|
||||
"total_chunks": total_chunks,
|
||||
"md5": md5,
|
||||
},
|
||||
"aibot_upload_media_init",
|
||||
)
|
||||
upload_id = _response_body(init_frame).get("upload_id")
|
||||
if not upload_id:
|
||||
raise RuntimeError("上传初始化失败：未返回 upload_id")
|
||||
|
||||
for index, chunk in enumerate(chunks):
|
||||
chunk_frame = await manager.send_reply(
|
||||
generate_req_id("upload_chunk"),
|
||||
{
|
||||
"upload_id": upload_id,
|
||||
"chunk_index": index,
|
||||
"base64_data": base64.b64encode(chunk).decode("ascii"),
|
||||
},
|
||||
"aibot_upload_media_chunk",
|
||||
)
|
||||
_response_body(chunk_frame)
|
||||
|
||||
finish_frame = await manager.send_reply(
|
||||
generate_req_id("upload_finish"),
|
||||
{"upload_id": upload_id},
|
||||
"aibot_upload_media_finish",
|
||||
)
|
||||
media_id = _response_body(finish_frame).get("media_id")
|
||||
if not media_id:
|
||||
raise RuntimeError("上传完成但未返回 media_id")
|
||||
|
||||
logger.info("图片已上传 media_id=%s…", str(media_id)[:12])
|
||||
return str(media_id)
|
||||
|
||||
|
||||
async def reply_image(ws_client: Any, frame: dict[str, Any], media_id: str) -> None:
|
||||
await ws_client.reply(
|
||||
frame,
|
||||
{
|
||||
"msgtype": "image",
|
||||
"image": {"media_id": media_id},
|
||||
},
|
||||
)
|
||||
6
daily/__init__.py
Normal file
6
daily/__init__.py
Normal file
@@ -0,0 +1,6 @@
|
||||
"""早报生成与企微推送。"""
|
||||
|
||||
from daily.generate import generate_report, main as generate_main
|
||||
from daily.webhook import send_report
|
||||
|
||||
__all__ = ["generate_report", "generate_main", "send_report"]
|
||||
22
daily/__main__.py
Normal file
22
daily/__main__.py
Normal file
@@ -0,0 +1,22 @@
|
||||
"""CLI: python -m daily [generate|push] [report_path]"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
|
||||
from daily.generate import main as generate_main
|
||||
from daily.webhook import main as push_main
|
||||
|
||||
|
||||
def main() -> int:
|
||||
cmd = (sys.argv[1] if len(sys.argv) > 1 else "generate").lower()
|
||||
if cmd in {"generate", "gen", "g"}:
|
||||
return generate_main()
|
||||
if cmd in {"push", "send", "webhook"}:
|
||||
return push_main(sys.argv[2:])
|
||||
print(f"未知命令: {cmd}\n用法: python -m daily [generate|push] [report_path]", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
134
daily/agent_workflow.py
Normal file
134
daily/agent_workflow.py
Normal file
@@ -0,0 +1,134 @@
|
||||
"""Agent 工作流：趋势分析 → 叙事化企微早报。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from daily.config import OUTPUT_DIR, ROOT, env
|
||||
from daily.llm_client import extract_json_object, has_llm_configured, llm_chat
|
||||
from daily.report_data import save_json
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SKILL_DIR = ROOT / "skills" / "daily-agent"
|
||||
_MD_BLOCK = re.compile(r"```(?:markdown|md)?\s*([\s\S]*?)```", re.IGNORECASE)
|
||||
_WECOM_NEW_ENTRY_NOTE = re.compile(r"（新入[^）]*）")
|
||||
|
||||
|
||||
def _strip_new_entry_notes(md: str) -> str:
|
||||
md = _WECOM_NEW_ENTRY_NOTE.sub("", md)
|
||||
return re.sub(r"\*\*—", "** —", md)
|
||||
|
||||
|
||||
def report_mode() -> str:
|
||||
return (env("DAILY_REPORT_MODE") or "classic").strip().lower()
|
||||
|
||||
|
||||
def is_agent_mode() -> bool:
|
||||
if report_mode() != "agent":
|
||||
return False
|
||||
if not has_llm_configured():
|
||||
logger.warning("DAILY_REPORT_MODE=agent 但未配置 LLM，回退 classic")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def trends_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.trends.json"
|
||||
|
||||
|
||||
def _load_skill() -> str:
|
||||
path = _SKILL_DIR / "SKILL.md"
|
||||
if path.exists():
|
||||
return path.read_text(encoding="utf-8").strip()
|
||||
return "你是早报主编 Agent。"
|
||||
|
||||
|
||||
def _extract_markdown(text: str) -> str:
|
||||
text = text.strip()
|
||||
match = _MD_BLOCK.search(text)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
if text.startswith("📰"):
|
||||
return text
|
||||
return text
|
||||
|
||||
|
||||
def analyze_trends(llm_input: dict[str, Any], *, date_str: str) -> dict[str, Any] | None:
|
||||
skill = _load_skill()
|
||||
system = (
|
||||
f"{skill}\n\n"
|
||||
"当前执行 **Step 1：趋势分析**。\n"
|
||||
"只输出 trends JSON（headline, opening, themes, top_picks, signals），不要 Markdown。"
|
||||
)
|
||||
user = json.dumps(llm_input, ensure_ascii=False, indent=2)
|
||||
try:
|
||||
raw = llm_chat(system, user)
|
||||
except Exception as exc:
|
||||
logger.warning("Agent Step1 趋势分析失败：%s", exc)
|
||||
return None
|
||||
if not raw:
|
||||
return None
|
||||
parsed = extract_json_object(raw)
|
||||
if not parsed.get("headline") and not parsed.get("opening"):
|
||||
logger.warning("Agent Step1 JSON 无效")
|
||||
return None
|
||||
save_json(trends_json_path(date_str), parsed)
|
||||
logger.info("Agent Step1 完成：%s", parsed.get("headline", "?"))
|
||||
return parsed
|
||||
|
||||
|
||||
def write_wecom_report(
|
||||
llm_input: dict[str, Any],
|
||||
trends: dict[str, Any],
|
||||
*,
|
||||
date_str: str,
|
||||
time_str: str,
|
||||
updated: str,
|
||||
) -> str | None:
|
||||
skill = _load_skill()
|
||||
system = (
|
||||
f"{skill}\n\n"
|
||||
"当前执行 **Step 2：撰写企微早报**。\n"
|
||||
f"日期={date_str}，时间={time_str}，数据截至={updated}。\n"
|
||||
"只输出企微 Markdown 正文，不要代码块，不要 JSON。"
|
||||
)
|
||||
payload = {"data": llm_input, "trends": trends}
|
||||
user = json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
try:
|
||||
raw = llm_chat(system, user)
|
||||
except Exception as exc:
|
||||
logger.warning("Agent Step2 写稿失败：%s", exc)
|
||||
return None
|
||||
if not raw:
|
||||
return None
|
||||
md = _extract_markdown(raw)
|
||||
if not md.startswith("📰"):
|
||||
md = f"📰 **早报 · {date_str}**\n> ⏱ {time_str} · 数据截至 {updated}\n\n{md}"
|
||||
md = _strip_new_entry_notes(md)
|
||||
logger.info("Agent Step2 完成：%d bytes", len(md.encode("utf-8")))
|
||||
return md
|
||||
|
||||
|
||||
def run_agent_workflow(
|
||||
llm_input: dict[str, Any],
|
||||
*,
|
||||
date_str: str,
|
||||
time_str: str,
|
||||
updated: str,
|
||||
) -> str | None:
|
||||
"""两步 Agent 工作流；成功返回企微 Markdown，失败返回 None。"""
|
||||
trends = analyze_trends(llm_input, date_str=date_str)
|
||||
if not trends:
|
||||
return None
|
||||
return write_wecom_report(
|
||||
llm_input,
|
||||
trends,
|
||||
date_str=date_str,
|
||||
time_str=time_str,
|
||||
updated=updated,
|
||||
)
|
||||
81
daily/config.py
Normal file
81
daily/config.py
Normal file
@@ -0,0 +1,81 @@
|
||||
"""项目路径与环境变量加载。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
BOT_DIR = ROOT / "bot"
|
||||
OUTPUT_DIR = ROOT / "output"
|
||||
LOG_DIR = ROOT / "logs"
|
||||
CACHE_DIR = ROOT / ".cache"
|
||||
SNAPSHOT_FILE = CACHE_DIR / "last-report.json"
|
||||
|
||||
# 企微 webhook markdown.content 硬上限
|
||||
WECOM_MARKDOWN_LIMIT = 4096
|
||||
|
||||
|
||||
def wecom_chunk_bytes() -> int:
|
||||
"""单条企微 markdown 消息字节上限。"""
|
||||
chunk = env_int("DAILY_WECOM_CHUNK_BYTES", -1)
|
||||
if chunk < 0:
|
||||
chunk = env_int("DAILY_WECOM_MAX_BYTES", WECOM_MARKDOWN_LIMIT)
|
||||
return min(chunk, WECOM_MARKDOWN_LIMIT)
|
||||
|
||||
|
||||
def wecom_skill_desc_limit() -> int:
|
||||
"""企微 Skills 榜单条简介建议字数。"""
|
||||
return env_int("DAILY_WECOM_SKILL_DESC_LIMIT", 56)
|
||||
|
||||
|
||||
def full_desc_limit() -> int:
|
||||
"""完整版早报摘要长度；0 表示不截断。"""
|
||||
return env_int("DAILY_FULL_DESC_LIMIT", 0)
|
||||
|
||||
|
||||
def news_summary_limit() -> int:
|
||||
return env_int("DAILY_FULL_NEWS_SUMMARY_LIMIT", 0)
|
||||
|
||||
|
||||
def wecom_max_bytes() -> int:
|
||||
"""兼容旧配置名。"""
|
||||
return wecom_chunk_bytes()
|
||||
|
||||
load_dotenv(ROOT / ".env")
|
||||
load_dotenv(ROOT / ".env.local", override=True)
|
||||
|
||||
|
||||
def ensure_bot_on_path() -> None:
|
||||
bot = str(BOT_DIR)
|
||||
if bot not in sys.path:
|
||||
sys.path.insert(0, bot)
|
||||
|
||||
|
||||
def _clean_env_value(raw: str | None) -> str | None:
|
||||
if raw is None:
|
||||
return None
|
||||
value = raw.strip()
|
||||
if not value:
|
||||
return None
|
||||
# .env 行内注释（未加引号时 python-dotenv 不会自动去掉）
|
||||
if " #" in value:
|
||||
value = value.split(" #", 1)[0].rstrip()
|
||||
return value or None
|
||||
|
||||
|
||||
def env(key: str, default: str | None = None) -> str | None:
|
||||
return _clean_env_value(os.getenv(key, default))
|
||||
|
||||
|
||||
def env_int(key: str, default: int) -> int:
|
||||
raw = env(key)
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
144
daily/cursor_editor.py
Normal file
144
daily/cursor_editor.py
Normal file
@@ -0,0 +1,144 @@
|
||||
"""Cursor 编辑层：JSON 数据 → 主题 / 速览 / 中文描述。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from daily.config import ROOT, env
|
||||
from daily.llm_client import extract_json_object, has_llm_configured, llm_chat
|
||||
from daily.text_utils import clip_text
|
||||
from daily.report_data import editorial_json_path, save_json, skill_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SKILL_DIR = ROOT / "skills" / "daily-editor"
|
||||
|
||||
|
||||
def is_enabled() -> bool:
|
||||
raw = (env("DAILY_CURSOR_EDITOR") or "").strip().lower()
|
||||
if raw in {"1", "true", "yes", "on"}:
|
||||
return has_llm_configured()
|
||||
if raw in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _load_skill_prompt() -> str:
|
||||
skill_path = _SKILL_DIR / "SKILL.md"
|
||||
if skill_path.exists():
|
||||
return skill_path.read_text(encoding="utf-8").strip()
|
||||
return "你是技术早报编辑。根据输入 JSON 输出编辑结果 JSON。"
|
||||
|
||||
|
||||
def _build_system_prompt() -> str:
|
||||
skill = _load_skill_prompt()
|
||||
return (
|
||||
f"{skill}\n\n"
|
||||
"再次强调：只输出 JSON 对象，包含 theme_line、highlights（3条）、descriptions。"
|
||||
)
|
||||
|
||||
|
||||
def run_editorial(llm_input: dict[str, Any], *, date_str: str) -> dict[str, Any] | None:
|
||||
"""调用 LLM 生成 editorial；失败返回 None。"""
|
||||
if not is_enabled():
|
||||
return None
|
||||
system = _build_system_prompt()
|
||||
user = json.dumps(llm_input, ensure_ascii=False, indent=2)
|
||||
try:
|
||||
raw = llm_chat(system, user)
|
||||
except Exception as exc:
|
||||
logger.warning("Cursor 编辑失败，回退规则模式：%s", exc)
|
||||
return None
|
||||
if not raw:
|
||||
logger.warning("Cursor 编辑无响应，回退规则模式")
|
||||
return None
|
||||
parsed = extract_json_object(raw)
|
||||
if not parsed.get("theme_line") and not parsed.get("descriptions"):
|
||||
logger.warning("Cursor 编辑 JSON 无效，回退规则模式")
|
||||
return None
|
||||
editorial = _normalize_editorial(parsed)
|
||||
save_json(editorial_json_path(date_str), editorial)
|
||||
return editorial
|
||||
|
||||
|
||||
def _normalize_editorial(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
theme = str(raw.get("theme_line") or "").strip()
|
||||
highlights_raw = raw.get("highlights") or []
|
||||
highlights: list[str] = []
|
||||
if isinstance(highlights_raw, list):
|
||||
for item in highlights_raw:
|
||||
if isinstance(item, str) and item.strip():
|
||||
highlights.append(item.strip())
|
||||
descriptions_raw = raw.get("descriptions") or {}
|
||||
descriptions: dict[str, str] = {}
|
||||
if isinstance(descriptions_raw, dict):
|
||||
for key, value in descriptions_raw.items():
|
||||
if isinstance(value, str) and value.strip():
|
||||
limit = 40 if str(key).startswith("github:") else 36
|
||||
descriptions[str(key)] = clip_text(value, limit)
|
||||
return {
|
||||
"theme_line": theme,
|
||||
"highlights": highlights[:3],
|
||||
"descriptions": descriptions,
|
||||
}
|
||||
|
||||
|
||||
def theme_line_from_editorial(editorial: dict[str, Any]) -> str:
|
||||
theme = editorial.get("theme_line", "")
|
||||
if not theme:
|
||||
return ""
|
||||
if "今日主题" in theme:
|
||||
return theme if theme.startswith("**") else f"**{theme}**"
|
||||
return f"**今日主题**：{theme}"
|
||||
|
||||
|
||||
def apply_descriptions(
|
||||
*,
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
github_topic: list[dict[str, Any]],
|
||||
ai_news: dict[str, Any],
|
||||
descriptions: dict[str, str],
|
||||
) -> None:
|
||||
if not descriptions:
|
||||
return
|
||||
|
||||
for item in trending + hot:
|
||||
key = f"skill:{skill_id(item)}"
|
||||
if key in descriptions:
|
||||
item["description"] = descriptions[key]
|
||||
|
||||
for repo_list in (github_trending, github_emerging, github_topic):
|
||||
for item in repo_list:
|
||||
key = f"github:{item.get('repo', '')}"
|
||||
if key in descriptions:
|
||||
item["description"] = descriptions[key]
|
||||
|
||||
if not ai_news.get("enabled"):
|
||||
return
|
||||
|
||||
def _apply_news_item(item: dict[str, Any]) -> None:
|
||||
key = f"news:{item.get('link', '')}"
|
||||
if key in descriptions:
|
||||
item["summary"] = descriptions[key]
|
||||
|
||||
for cat in ai_news.get("categories") or []:
|
||||
for item in cat.get("items") or []:
|
||||
_apply_news_item(item)
|
||||
for item in ai_news.get("flat") or []:
|
||||
_apply_news_item(item)
|
||||
|
||||
|
||||
def load_cached_editorial(date_str: str) -> dict[str, Any] | None:
|
||||
path = editorial_json_path(date_str)
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
return _normalize_editorial(json.loads(path.read_text(encoding="utf-8")))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
313
daily/delta.py
Normal file
313
daily/delta.py
Normal file
@@ -0,0 +1,313 @@
|
||||
"""榜单异动：对比昨日 Top N，仅识别「新入榜」条目（分榜、限条）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
from daily.config import OUTPUT_DIR, env_int
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
KeyFn = Callable[[dict[str, Any]], str]
|
||||
|
||||
|
||||
def compare_depth() -> int:
|
||||
return env_int("DAILY_DELTA_COMPARE_DEPTH", 15)
|
||||
|
||||
|
||||
def wecom_new_limit() -> int:
|
||||
return env_int("DAILY_WECOM_NEW_MAX", 10)
|
||||
|
||||
|
||||
def skill_id(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
|
||||
|
||||
|
||||
def _data_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.data.json"
|
||||
|
||||
|
||||
def _load_data_json_file(path: Path) -> dict[str, Any]:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _key_set(items: list[dict[str, Any]], key_fn: KeyFn, *, depth: int) -> set[str]:
|
||||
return {key_fn(item) for item in items[:depth] if key_fn(item)}
|
||||
|
||||
|
||||
def find_previous_data(date_str: str) -> tuple[str, dict[str, Any]] | None:
|
||||
"""查找最近一份早于 date_str 的 data.json。"""
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
return None
|
||||
lookback = env_int("DAILY_DELTA_LOOKBACK_DAYS", 7)
|
||||
for days in range(1, lookback + 1):
|
||||
prev_date = (dt - timedelta(days=days)).strftime("%Y-%m-%d")
|
||||
path = _data_json_path(prev_date)
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
payload = _load_data_json_file(path)
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("读取异动基准 %s 失败：%s", path, exc)
|
||||
continue
|
||||
data = payload.get("data")
|
||||
if isinstance(data, dict) and data.get("date"):
|
||||
return prev_date, data
|
||||
return None
|
||||
|
||||
|
||||
def _prev_board_items(prev_data: dict[str, Any], board: str, depth: int) -> list[dict[str, Any]]:
|
||||
baseline = prev_data.get("movement_baseline") or {}
|
||||
if board in baseline and isinstance(baseline[board], list):
|
||||
return baseline[board][:depth]
|
||||
legacy = {
|
||||
"skills_trending": "skills_trending",
|
||||
"skills_hot": "skills_hot",
|
||||
"github_trending": "github_trending",
|
||||
"github_emerging": "github_emerging",
|
||||
"github_topic": "github_topic",
|
||||
}
|
||||
if board == "github_topic":
|
||||
topic = prev_data.get("github_topic") or {}
|
||||
repos = topic.get("repos") if isinstance(topic, dict) else []
|
||||
return (repos or [])[:depth]
|
||||
field = legacy.get(board, board)
|
||||
items = prev_data.get(field) or []
|
||||
return items[:depth] if isinstance(items, list) else []
|
||||
|
||||
|
||||
def _format_new_note(board: str, rank: int, *, topic_name: str = "llm") -> str:
|
||||
labels = {
|
||||
"trending": "Skills Trending",
|
||||
"hot": "Skills Hot",
|
||||
"github_trending": "GitHub Trending",
|
||||
"github_emerging": "GitHub 新兴",
|
||||
"github_topic": f"Topic `{topic_name}`",
|
||||
}
|
||||
return f"新入 {labels.get(board, board)} #{rank}"
|
||||
|
||||
|
||||
def _build_skill_board_moves(
|
||||
*,
|
||||
board: str,
|
||||
items: list[dict[str, Any]],
|
||||
prev_data: dict[str, Any] | None,
|
||||
depth: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
board_key = f"skills_{board}"
|
||||
prev_ids = (
|
||||
_key_set(_prev_board_items(prev_data, board_key, depth), skill_id, depth=depth)
|
||||
if prev_data
|
||||
else set()
|
||||
)
|
||||
moves: list[dict[str, Any]] = []
|
||||
for rank, item in enumerate(items[:depth], 1):
|
||||
sid = skill_id(item)
|
||||
if not sid or not prev_data or sid in prev_ids:
|
||||
continue
|
||||
moves.append(
|
||||
{
|
||||
"kind": "skill",
|
||||
"board": board,
|
||||
"id": sid,
|
||||
"title": item.get("title", ""),
|
||||
"source": item.get("source", ""),
|
||||
"installs": int(item.get("installs") or 0),
|
||||
"link": item.get("link", ""),
|
||||
"description": item.get("description", ""),
|
||||
"rank": rank,
|
||||
"is_new": True,
|
||||
"note": _format_new_note(board, rank),
|
||||
}
|
||||
)
|
||||
return moves
|
||||
|
||||
|
||||
def _repo_key(item: dict[str, Any]) -> str:
|
||||
return str(item.get("repo") or "")
|
||||
|
||||
|
||||
def _build_github_board_moves(
|
||||
*,
|
||||
board: str,
|
||||
items: list[dict[str, Any]],
|
||||
prev_data: dict[str, Any] | None,
|
||||
depth: int,
|
||||
topic_name: str = "llm",
|
||||
) -> list[dict[str, Any]]:
|
||||
prev_ids = (
|
||||
_key_set(_prev_board_items(prev_data, board, depth), _repo_key, depth=depth)
|
||||
if prev_data
|
||||
else set()
|
||||
)
|
||||
moves: list[dict[str, Any]] = []
|
||||
for rank, item in enumerate(items[:depth], 1):
|
||||
repo = _repo_key(item)
|
||||
if not repo or not prev_data or repo in prev_ids:
|
||||
continue
|
||||
moves.append(
|
||||
{
|
||||
"kind": "github",
|
||||
"board": board,
|
||||
"repo": repo,
|
||||
"url": item.get("url", ""),
|
||||
"language": item.get("language", ""),
|
||||
"stars_today_fmt": item.get("stars_today_fmt", ""),
|
||||
"total_stars_fmt": item.get("total_stars_fmt", ""),
|
||||
"created_at": item.get("created_at", ""),
|
||||
"description": item.get("description", ""),
|
||||
"rank": rank,
|
||||
"is_new": True,
|
||||
"note": _format_new_note(board, rank, topic_name=topic_name),
|
||||
}
|
||||
)
|
||||
return moves
|
||||
|
||||
|
||||
def _board_summary(
|
||||
*,
|
||||
label: str,
|
||||
baseline_date: str | None,
|
||||
depth: int,
|
||||
moves: list[dict[str, Any]],
|
||||
capped: list[dict[str, Any]],
|
||||
) -> str:
|
||||
if not baseline_date:
|
||||
return f"无历史基准，无法判断 {label} Top{depth} 新增"
|
||||
if not moves:
|
||||
return f"较 {baseline_date} Top{depth} 无新增 {label} 条目"
|
||||
total = len(moves)
|
||||
shown = len(capped)
|
||||
if shown < total:
|
||||
return f"较 {baseline_date} Top{depth} 新增 {total} 条，企微展示前 {shown} 条"
|
||||
return f"较 {baseline_date} Top{depth} 新增 {total} 条"
|
||||
|
||||
|
||||
def build_movement_baseline(
|
||||
*,
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
github_topic: list[dict[str, Any]],
|
||||
depth: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
n = depth if depth is not None else compare_depth()
|
||||
return {
|
||||
"compare_depth": n,
|
||||
"skills_trending": trending[:n],
|
||||
"skills_hot": hot[:n],
|
||||
"github_trending": github_trending[:n],
|
||||
"github_emerging": github_emerging[:n],
|
||||
"github_topic": github_topic[:n],
|
||||
}
|
||||
|
||||
|
||||
def build_movement_context(
|
||||
*,
|
||||
date_str: str,
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
github_topic: list[dict[str, Any]],
|
||||
topic_name: str = "llm",
|
||||
) -> dict[str, Any]:
|
||||
"""生成 Agent 可用的新增榜上下文：分 Trending/Hot/Topic，企微每榜最多 wecom_new_limit 条。"""
|
||||
depth = compare_depth()
|
||||
cap = wecom_new_limit()
|
||||
baseline = find_previous_data(date_str)
|
||||
baseline_date = baseline[0] if baseline else None
|
||||
prev_data = baseline[1] if baseline else None
|
||||
topic = (topic_name or "llm").strip() or "llm"
|
||||
|
||||
skills_trending_all = _build_skill_board_moves(
|
||||
board="trending", items=trending, prev_data=prev_data, depth=depth
|
||||
)
|
||||
skills_hot_all = _build_skill_board_moves(
|
||||
board="hot", items=hot, prev_data=prev_data, depth=depth
|
||||
)
|
||||
github_trending_all = _build_github_board_moves(
|
||||
board="github_trending", items=github_trending, prev_data=prev_data, depth=depth
|
||||
)
|
||||
github_emerging_all = _build_github_board_moves(
|
||||
board="github_emerging", items=github_emerging, prev_data=prev_data, depth=depth
|
||||
)
|
||||
github_topic_all = _build_github_board_moves(
|
||||
board="github_topic",
|
||||
items=github_topic,
|
||||
prev_data=prev_data,
|
||||
depth=depth,
|
||||
topic_name=topic,
|
||||
)
|
||||
|
||||
skills_trending_moves = skills_trending_all[:cap]
|
||||
skills_hot_moves = skills_hot_all[:cap]
|
||||
github_trending_moves = github_trending_all[:cap]
|
||||
github_emerging_moves = github_emerging_all[:cap]
|
||||
github_topic_moves = github_topic_all[:cap]
|
||||
topic_label = f"Topic `{topic}`"
|
||||
|
||||
return {
|
||||
"baseline_date": baseline_date,
|
||||
"compare_depth": depth,
|
||||
"wecom_new_limit": cap,
|
||||
"selection_mode": "top_n",
|
||||
"topic_name": topic,
|
||||
"skills_trending_moves": skills_trending_moves,
|
||||
"skills_hot_moves": skills_hot_moves,
|
||||
"skills_trending_stable": not skills_trending_all,
|
||||
"skills_hot_stable": not skills_hot_all,
|
||||
"skills_trending_summary": _board_summary(
|
||||
label="Skills Trending",
|
||||
baseline_date=baseline_date,
|
||||
depth=depth,
|
||||
moves=skills_trending_all,
|
||||
capped=skills_trending_moves,
|
||||
),
|
||||
"skills_hot_summary": _board_summary(
|
||||
label="Skills Hot",
|
||||
baseline_date=baseline_date,
|
||||
depth=depth,
|
||||
moves=skills_hot_all,
|
||||
capped=skills_hot_moves,
|
||||
),
|
||||
"github_trending_moves": github_trending_moves,
|
||||
"github_emerging_moves": github_emerging_moves,
|
||||
"github_topic_moves": github_topic_moves,
|
||||
"github_trending_stable": not github_trending_all,
|
||||
"github_emerging_stable": not github_emerging_all,
|
||||
"github_topic_stable": not github_topic_all,
|
||||
"github_trending_summary": _board_summary(
|
||||
label="GitHub Trending",
|
||||
baseline_date=baseline_date,
|
||||
depth=depth,
|
||||
moves=github_trending_all,
|
||||
capped=github_trending_moves,
|
||||
),
|
||||
"github_emerging_summary": _board_summary(
|
||||
label="GitHub 新兴",
|
||||
baseline_date=baseline_date,
|
||||
depth=depth,
|
||||
moves=github_emerging_all,
|
||||
capped=github_emerging_moves,
|
||||
),
|
||||
"github_topic_summary": _board_summary(
|
||||
label=topic_label,
|
||||
baseline_date=baseline_date,
|
||||
depth=depth,
|
||||
moves=github_topic_all,
|
||||
capped=github_topic_moves,
|
||||
),
|
||||
# 兼容旧字段（合并，仅供调试）
|
||||
"skills_moves": skills_trending_moves + skills_hot_moves,
|
||||
"github_moves": github_trending_moves + github_emerging_moves + github_topic_moves,
|
||||
"skills_stable": not skills_trending_all and not skills_hot_all,
|
||||
"github_stable": not github_trending_all and not github_emerging_all and not github_topic_all,
|
||||
}
|
||||
281
daily/format_wecom.py
Normal file
281
daily/format_wecom.py
Normal file
@@ -0,0 +1,281 @@
|
||||
"""企微早报排版。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from daily.config import wecom_skill_desc_limit
|
||||
from daily.localize import LocalizeJob, localize_brief_descriptions, needs_chinese
|
||||
from daily.text_utils import trim_brief
|
||||
|
||||
ICONS = {
|
||||
"header": "📰",
|
||||
"highlights": "💡",
|
||||
"trending": "📈",
|
||||
"hot": "🔥",
|
||||
"github": "🐙",
|
||||
"emerging": "🌱",
|
||||
"topic": "🤖",
|
||||
"ainews": "🌍",
|
||||
"pick": "📦",
|
||||
"theme": "🎯",
|
||||
"file": "📄",
|
||||
}
|
||||
|
||||
|
||||
def _skill_line(rank: int, item: dict[str, Any], *, badge: str = "") -> list[str]:
|
||||
source = item.get("source", "?")
|
||||
installs = item.get("installs_fmt", "?")
|
||||
link = item.get("link", "")
|
||||
desc = item.get("desc_short", "")
|
||||
badge_prefix = f"{badge} " if badge else ""
|
||||
if item.get("cluster"):
|
||||
count = int(item.get("cluster_count") or 1)
|
||||
sample = item.get("cluster_titles") or item.get("title", "")
|
||||
label = f"**{source}** · {count} skills · **{installs}**"
|
||||
if link:
|
||||
head = f"{rank}. {badge_prefix}[{label}]({link})"
|
||||
else:
|
||||
head = f"{rank}. {badge_prefix}{label}"
|
||||
lines = [head]
|
||||
if sample or desc:
|
||||
hint = desc or sample
|
||||
lines.append(f" > {hint}")
|
||||
return lines
|
||||
title = item.get("title", "?")
|
||||
if link:
|
||||
head = f"{rank}. {badge_prefix}[**{title}**]({link}) · `{source}` · **{installs}**"
|
||||
else:
|
||||
head = f"{rank}. {badge_prefix}**{title}** · `{source}` · **{installs}**"
|
||||
lines = [head]
|
||||
if desc:
|
||||
lines.append(f" > {desc}")
|
||||
return lines
|
||||
|
||||
|
||||
def _ai_news_lines(items: list[dict[str, Any]]) -> list[str]:
|
||||
lines: list[str] = []
|
||||
for i, item in enumerate(items, 1):
|
||||
title = item.get("title", "?")
|
||||
link = item.get("link", "")
|
||||
source = item.get("source_name", "?")
|
||||
pub = item.get("published_fmt", "")
|
||||
desc = item.get("desc_short", "")
|
||||
pub_suffix = f" · {pub}" if pub else ""
|
||||
if link:
|
||||
head = f"{i}. [**{title}**]({link}) · `{source}`{pub_suffix}"
|
||||
else:
|
||||
head = f"{i}. **{title}** · `{source}`{pub_suffix}"
|
||||
lines.append(head)
|
||||
if desc:
|
||||
lines.append(f" > {desc}")
|
||||
return lines
|
||||
|
||||
|
||||
def _github_repo_lines(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
|
||||
lines: list[str] = []
|
||||
for i, repo in enumerate(repos, 1):
|
||||
name = repo["repo"]
|
||||
url = repo["url"]
|
||||
lang = repo.get("language", "")
|
||||
stars_today = repo.get("stars_today_fmt", "")
|
||||
total = repo.get("total_stars_fmt", "")
|
||||
created = repo.get("created_at", "")
|
||||
meta_parts: list[str] = []
|
||||
if lang:
|
||||
meta_parts.append(lang)
|
||||
if stars_today:
|
||||
meta_parts.append(f"+{stars_today} today")
|
||||
elif total:
|
||||
meta_parts.append(f"⭐{total}")
|
||||
if show_created and created:
|
||||
meta_parts.append(f"创建于 {created}")
|
||||
meta = f" · {' · '.join(meta_parts)}" if meta_parts else ""
|
||||
lines.append(f"{i}. [{name}]({url}){meta}")
|
||||
desc = repo.get("desc_short") or repo.get("description", "")
|
||||
if desc:
|
||||
lines.append(f" > {desc}")
|
||||
return lines
|
||||
|
||||
|
||||
def _fallback_skill_desc(item: dict[str, Any]) -> str:
|
||||
if item.get("cluster"):
|
||||
count = int(item.get("cluster_count") or 1)
|
||||
source = item.get("source") or "unknown"
|
||||
sample = item.get("cluster_titles") or item.get("title") or ""
|
||||
return f"{count} agent skills from {source}, including {sample}"
|
||||
title = item.get("title") or "skill"
|
||||
source = item.get("source") or "unknown"
|
||||
return f"{title} skill from {source}"
|
||||
|
||||
|
||||
def _skill_group_key(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
|
||||
|
||||
|
||||
def finalize_wecom_skill_groups(
|
||||
items: list[dict[str, Any]],
|
||||
*,
|
||||
desc_limit: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""为企微 Skills 榜生成一句简要中文简介（与完整版 .md 长描述分离）。"""
|
||||
if desc_limit is None:
|
||||
desc_limit = wecom_skill_desc_limit()
|
||||
limit = desc_limit if desc_limit > 0 else 48
|
||||
copies: list[tuple[str, dict[str, Any]]] = []
|
||||
jobs: list[LocalizeJob] = []
|
||||
for item in items:
|
||||
copy = dict(item)
|
||||
desc = (copy.get("description") or "").strip()
|
||||
if not desc:
|
||||
desc = _fallback_skill_desc(copy)
|
||||
key = _skill_group_key(copy)
|
||||
job_key = f"wecom:{key}"
|
||||
if needs_chinese(desc) or len(desc) > limit:
|
||||
jobs.append(LocalizeJob(job_key, desc, limit))
|
||||
else:
|
||||
copy["wecom_desc"] = desc
|
||||
copies.append((job_key, copy))
|
||||
|
||||
zh_map = localize_brief_descriptions(jobs, archive=True)
|
||||
out: list[dict[str, Any]] = []
|
||||
for job_key, copy in copies:
|
||||
if job_key in zh_map:
|
||||
copy["wecom_desc"] = zh_map[job_key]
|
||||
elif "wecom_desc" not in copy:
|
||||
copy["wecom_desc"] = _brief_fallback_desc(
|
||||
(copy.get("description") or "").strip() or _fallback_skill_desc(copy),
|
||||
limit,
|
||||
)
|
||||
out.append(copy)
|
||||
return out
|
||||
|
||||
|
||||
def _brief_fallback_desc(text: str, limit: int) -> str:
|
||||
return trim_brief(text, limit)
|
||||
|
||||
|
||||
def _grouped_skill_to_wecom_item(
|
||||
item: dict[str, Any],
|
||||
*,
|
||||
desc_limit: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
if desc_limit is None:
|
||||
desc_limit = wecom_skill_desc_limit()
|
||||
limit = desc_limit if desc_limit > 0 else 48
|
||||
installs = int(item.get("installs") or 0)
|
||||
installs_fmt = item.get("installs_fmt") or str(installs)
|
||||
desc = (item.get("wecom_desc") or item.get("description") or "").strip()
|
||||
if not desc and item.get("cluster"):
|
||||
desc = item.get("cluster_titles") or ""
|
||||
if not item.get("wecom_desc"):
|
||||
desc = _brief_fallback_desc(desc, limit)
|
||||
return {
|
||||
"title": item.get("title", ""),
|
||||
"source": item.get("source", "?"),
|
||||
"installs_fmt": installs_fmt,
|
||||
"link": item.get("link", ""),
|
||||
"desc_short": desc,
|
||||
"cluster": bool(item.get("cluster")),
|
||||
"cluster_count": item.get("cluster_count"),
|
||||
"cluster_titles": item.get("cluster_titles"),
|
||||
}
|
||||
|
||||
|
||||
def build_skills_board_section(icon_key: str, board_label: str, items: list[dict[str, Any]]) -> str:
|
||||
prepared = finalize_wecom_skill_groups(items)
|
||||
wecom_items = [_grouped_skill_to_wecom_item(x) for x in prepared]
|
||||
lines = [f"{ICONS[icon_key]} **{board_label} Top {len(wecom_items)}**"]
|
||||
for rank, item in enumerate(wecom_items, 1):
|
||||
lines.extend(_skill_line(rank, item))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
_SKILL_SECTIONS = re.compile(
|
||||
r"📈 \*\*Skills Trending.*?(?=🐙 \*\*GitHub Trending)",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
def replace_wecom_skill_sections(
|
||||
md: str,
|
||||
*,
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
) -> str:
|
||||
"""用 Python 合并后的 Skills 榜替换或插入 Agent 早报中的对应区块。"""
|
||||
trending_sec = build_skills_board_section("trending", "Skills Trending", trending)
|
||||
hot_sec = build_skills_board_section("hot", "Skills Hot", hot)
|
||||
replacement = f"{trending_sec}\n\n{hot_sec}\n\n"
|
||||
if _SKILL_SECTIONS.search(md):
|
||||
return _SKILL_SECTIONS.sub(replacement, md)
|
||||
github_marker = "🐙 **GitHub Trending"
|
||||
idx = md.find(github_marker)
|
||||
if idx >= 0:
|
||||
return md[:idx] + replacement + md[idx:]
|
||||
return md.rstrip() + "\n\n" + replacement
|
||||
|
||||
|
||||
def build_wecom_report(
|
||||
*,
|
||||
date_str: str,
|
||||
time_str: str,
|
||||
updated: str,
|
||||
highlights: list[str],
|
||||
theme_line: str,
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
repos: list[dict[str, Any]],
|
||||
emerging: list[dict[str, Any]],
|
||||
topic_name: str,
|
||||
topic_repos: list[dict[str, Any]],
|
||||
ai_news: list[dict[str, Any]] | None = None,
|
||||
pick_command: str,
|
||||
) -> str:
|
||||
lines = [
|
||||
f"{ICONS['header']} **早报 · {date_str}**",
|
||||
f"> ⏱ {time_str} · 数据截至 {updated}",
|
||||
"",
|
||||
f"{ICONS['highlights']} **今日速览**",
|
||||
]
|
||||
for point in highlights[:3]:
|
||||
lines.append(f"> {point}")
|
||||
lines.append("")
|
||||
lines.append(f"{ICONS['theme']} {theme_line}")
|
||||
lines.append("")
|
||||
|
||||
if ai_news:
|
||||
lines.append(f"{ICONS['ainews']} **国际 AI 时讯 Top {len(ai_news)}**")
|
||||
lines.extend(_ai_news_lines(ai_news))
|
||||
lines.append("")
|
||||
|
||||
lines.append(f"{ICONS['trending']} **Skills Trending Top {len(trending)}**")
|
||||
for rank, item in enumerate(trending, 1):
|
||||
lines.extend(_skill_line(rank, item, badge=item.get("badge", "")))
|
||||
lines.append("")
|
||||
|
||||
lines.append(f"{ICONS['hot']} **Skills Hot Top {len(hot)}**")
|
||||
for rank, item in enumerate(hot, 1):
|
||||
lines.extend(_skill_line(rank, item, badge=item.get("badge", "")))
|
||||
lines.append("")
|
||||
|
||||
if repos:
|
||||
lines.append(f"{ICONS['github']} **GitHub Trending Top {len(repos)}**")
|
||||
lines.extend(_github_repo_lines(repos))
|
||||
lines.append("")
|
||||
|
||||
if emerging:
|
||||
lines.append(f"{ICONS['emerging']} **新兴项目 Top {len(emerging)}**")
|
||||
lines.extend(_github_repo_lines(emerging, show_created=True))
|
||||
lines.append("")
|
||||
|
||||
if topic_repos:
|
||||
lines.append(f"{ICONS['topic']} **Topic `{topic_name}` Top {len(topic_repos)}**")
|
||||
lines.extend(_github_repo_lines(topic_repos))
|
||||
lines.append("")
|
||||
|
||||
lines.append(f"{ICONS['pick']} **今日首推**")
|
||||
lines.append(f"`{pick_command}`")
|
||||
|
||||
return "\n".join(lines)
|
||||
635
daily/generate.py
Normal file
635
daily/generate.py
Normal file
@@ -0,0 +1,635 @@
|
||||
"""生成早报 Markdown（完整版 + 企微短版）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
from daily.config import (
|
||||
CACHE_DIR,
|
||||
LOG_DIR,
|
||||
OUTPUT_DIR,
|
||||
SNAPSHOT_FILE,
|
||||
ensure_bot_on_path,
|
||||
env,
|
||||
env_int,
|
||||
full_desc_limit,
|
||||
news_summary_limit,
|
||||
wecom_skill_desc_limit,
|
||||
)
|
||||
from daily.format_wecom import build_wecom_report, finalize_wecom_skill_groups, replace_wecom_skill_sections
|
||||
from daily.agent_workflow import is_agent_mode, run_agent_workflow
|
||||
from daily.delta import compare_depth
|
||||
from daily.cursor_editor import (
|
||||
apply_descriptions,
|
||||
is_enabled as cursor_editor_enabled,
|
||||
run_editorial,
|
||||
theme_line_from_editorial,
|
||||
)
|
||||
from daily.github.auth import github_html_headers
|
||||
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos
|
||||
from daily.github.trending import fetch_github_trending, trending_data_source_note
|
||||
from daily.localize import LocalizeJob, localize_descriptions, needs_chinese
|
||||
from daily.news.fetch import fetch_ai_news, format_news_section, prepare_wecom_news_items
|
||||
from daily.report_data import (
|
||||
build_full_payload,
|
||||
build_llm_input,
|
||||
data_json_path,
|
||||
save_json,
|
||||
)
|
||||
from daily.skills_board import load_boards
|
||||
from daily.skills_group import group_skills_by_source
|
||||
|
||||
ensure_bot_on_path()
|
||||
from skills_service import _format_installs, load_feed # noqa: E402
|
||||
|
||||
THEME_RULES: list[tuple[str, str, list[str]]] = [
|
||||
("🎬", "AI 多媒体 / 视频", ["runcomfy", "remotion", "video", "seedance", "inpaint", "lipsync"]),
|
||||
("🔧", "工程协作 / Skill 元能力", ["grill", "tdd", "architecture", "find-skills", "to-issues"]),
|
||||
("📱", "飞书 / Lark", ["lark", "feishu"]),
|
||||
("📣", "内容营销", ["viral", "tiktok", "instagram", "reels"]),
|
||||
("🎨", "设计 / 前端", ["frontend", "design", "ui-ux", "tailwind"]),
|
||||
]
|
||||
|
||||
|
||||
def _now_cst() -> datetime:
|
||||
return datetime.now(timezone(timedelta(hours=8)))
|
||||
|
||||
|
||||
def _short_desc(text: str, limit: int = 72) -> str:
|
||||
text = re.sub(r"\s+", " ", text or "").strip()
|
||||
if limit <= 0 or len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 3] + "..."
|
||||
|
||||
|
||||
def _archive_desc(text: str) -> str:
|
||||
return _short_desc(text, full_desc_limit())
|
||||
|
||||
|
||||
def _wecom_desc(text: str, limit: int = 36) -> str:
|
||||
return _short_desc(text, limit)
|
||||
|
||||
|
||||
def _localize_descriptions_in_place(
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
github_topic: list[dict[str, Any]],
|
||||
ai_news: dict[str, Any],
|
||||
) -> None:
|
||||
full_limit = full_desc_limit()
|
||||
news_limit = news_summary_limit()
|
||||
jobs: list[LocalizeJob] = []
|
||||
seen_skill: set[str] = set()
|
||||
for item in trending + hot:
|
||||
sid = _skill_id(item)
|
||||
if sid in seen_skill:
|
||||
continue
|
||||
seen_skill.add(sid)
|
||||
desc = (item.get("description") or "").strip()
|
||||
if desc:
|
||||
jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
|
||||
|
||||
seen_repo: set[str] = set()
|
||||
for repo_list in (github_trending, github_emerging, github_topic):
|
||||
for item in repo_list:
|
||||
repo = item.get("repo", "")
|
||||
if not repo or repo in seen_repo:
|
||||
continue
|
||||
seen_repo.add(repo)
|
||||
desc = (item.get("description") or "").strip()
|
||||
if desc:
|
||||
jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
|
||||
|
||||
if ai_news.get("enabled"):
|
||||
seen_news: set[str] = set()
|
||||
for item in ai_news.get("flat") or []:
|
||||
link = item.get("link", "")
|
||||
if not link or link in seen_news:
|
||||
continue
|
||||
seen_news.add(link)
|
||||
summary = (item.get("summary") or "").strip()
|
||||
if summary:
|
||||
jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
|
||||
|
||||
zh_map = localize_descriptions(jobs, archive=True)
|
||||
if not zh_map and not jobs:
|
||||
return
|
||||
|
||||
def _apply_zh(mapping: dict[str, str]) -> None:
|
||||
for item in trending + hot:
|
||||
key = f"skill:{_skill_id(item)}"
|
||||
if key in mapping:
|
||||
item["description"] = mapping[key]
|
||||
for repo_list in (github_trending, github_emerging, github_topic):
|
||||
for item in repo_list:
|
||||
key = f"github:{item.get('repo', '')}"
|
||||
if key in mapping:
|
||||
item["description"] = mapping[key]
|
||||
if ai_news.get("enabled"):
|
||||
for cat in ai_news.get("categories") or []:
|
||||
for item in cat.get("items") or []:
|
||||
key = f"news:{item.get('link', '')}"
|
||||
if key in mapping:
|
||||
item["summary"] = mapping[key]
|
||||
for item in ai_news.get("flat") or []:
|
||||
key = f"news:{item.get('link', '')}"
|
||||
if key in mapping:
|
||||
item["summary"] = mapping[key]
|
||||
|
||||
_apply_zh(zh_map)
|
||||
|
||||
# 仍为英文的条目再译一轮（长描述或批次失败时）
|
||||
retry_jobs: list[LocalizeJob] = []
|
||||
seen_skill.clear()
|
||||
for item in trending + hot:
|
||||
sid = _skill_id(item)
|
||||
if sid in seen_skill:
|
||||
continue
|
||||
seen_skill.add(sid)
|
||||
desc = (item.get("description") or "").strip()
|
||||
if needs_chinese(desc):
|
||||
retry_jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
|
||||
seen_repo.clear()
|
||||
for repo_list in (github_trending, github_emerging, github_topic):
|
||||
for item in repo_list:
|
||||
repo = item.get("repo", "")
|
||||
if not repo or repo in seen_repo:
|
||||
continue
|
||||
seen_repo.add(repo)
|
||||
desc = (item.get("description") or "").strip()
|
||||
if needs_chinese(desc):
|
||||
retry_jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
|
||||
if ai_news.get("enabled"):
|
||||
seen_news.clear()
|
||||
for item in ai_news.get("flat") or []:
|
||||
link = item.get("link", "")
|
||||
if not link or link in seen_news:
|
||||
continue
|
||||
seen_news.add(link)
|
||||
summary = (item.get("summary") or "").strip()
|
||||
if needs_chinese(summary):
|
||||
retry_jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
|
||||
|
||||
if retry_jobs:
|
||||
_apply_zh(localize_descriptions(retry_jobs, archive=True))
|
||||
|
||||
|
||||
def _skill_id(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
|
||||
|
||||
|
||||
def _load_snapshot() -> set[str]:
|
||||
if not SNAPSHOT_FILE.exists():
|
||||
return set()
|
||||
try:
|
||||
data = json.loads(SNAPSHOT_FILE.read_text(encoding="utf-8"))
|
||||
return set(str(x) for x in (data.get("skill_ids") or []))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return set()
|
||||
|
||||
|
||||
def _save_snapshot(feed: dict[str, Any], date_str: str) -> None:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
ids: list[str] = []
|
||||
for board in ("topTrending", "topHot"):
|
||||
for item in feed.get(board, [])[:20]:
|
||||
sid = _skill_id(item)
|
||||
if sid not in ids:
|
||||
ids.append(sid)
|
||||
SNAPSHOT_FILE.write_text(
|
||||
json.dumps({"date": date_str, "skill_ids": ids}, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _prepare_skill_item(item: dict[str, Any], prev_ids: set[str], rank: int) -> dict[str, Any]:
|
||||
badge = ""
|
||||
sid = _skill_id(item)
|
||||
if sid not in prev_ids and prev_ids:
|
||||
badge = "🆕"
|
||||
elif rank == 1:
|
||||
badge = "👑"
|
||||
installs_fmt = item.get("installs_fmt") or _format_installs(item.get("installs", 0))
|
||||
title = item.get("source", "?") if item.get("cluster") else item.get("title", "?")
|
||||
desc = item.get("wecom_desc") or item.get("description") or item.get("cluster_titles") or ""
|
||||
limit = wecom_skill_desc_limit()
|
||||
desc_short = desc if item.get("wecom_desc") or limit <= 0 else _wecom_desc(desc, limit)
|
||||
return {
|
||||
"title": title,
|
||||
"source": item.get("source", "?"),
|
||||
"installs_fmt": installs_fmt,
|
||||
"link": item.get("link", ""),
|
||||
"desc_short": desc_short,
|
||||
"badge": badge,
|
||||
"cluster": bool(item.get("cluster")),
|
||||
"cluster_count": item.get("cluster_count"),
|
||||
"cluster_titles": item.get("cluster_titles"),
|
||||
}
|
||||
|
||||
|
||||
def _detect_theme_line(feed: dict[str, Any]) -> str:
|
||||
scores: dict[str, int] = defaultdict(int)
|
||||
for board in ("topTrending", "topHot"):
|
||||
for rank, item in enumerate(feed.get(board, [])[:10], 1):
|
||||
haystack = " ".join(
|
||||
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
|
||||
).lower()
|
||||
for _icon, label, keywords in THEME_RULES:
|
||||
if any(k in haystack for k in keywords):
|
||||
scores[label] += max(1, 11 - rank)
|
||||
break
|
||||
if not scores:
|
||||
return "**今日主题**：Agent Skills 生态持续活跃"
|
||||
return f"**今日主题**：{max(scores.items(), key=lambda x: x[1])[0]}"
|
||||
|
||||
|
||||
def _build_highlights(
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
ai_news: dict[str, Any] | None = None,
|
||||
) -> list[str]:
|
||||
points: list[str] = []
|
||||
if ai_news and ai_news.get("enabled"):
|
||||
top_news = prepare_wecom_news_items(ai_news)
|
||||
if top_news:
|
||||
n0 = top_news[0]
|
||||
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
|
||||
points.append(
|
||||
f"🌍 AI 时讯 [{n0['title']}]({n0['link']})（`{n0.get('source_name', '?')}`{pub}）"
|
||||
)
|
||||
elif ai_news.get("flat"):
|
||||
n0 = ai_news["flat"][0]
|
||||
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
|
||||
points.append(f"🌍 AI 时讯 [{n0['title']}]({n0['link']})（`{n0.get('source_name', '?')}`{pub}）")
|
||||
if trending:
|
||||
t0 = trending[0]
|
||||
points.append(f"📈 Skills 榜首 **{t0.get('title')}**（{_format_installs(t0.get('installs', 0))}）")
|
||||
if github_trending:
|
||||
g0 = github_trending[0]
|
||||
stars = g0.get("stars_today_fmt", "")
|
||||
total = g0.get("total_stars_fmt", "")
|
||||
star_hint = f"+{stars} today · " if stars else (f"⭐{total} · " if total else "")
|
||||
points.append(f"🐙 GitHub Trending [{g0['repo']}]({g0['url']})（{star_hint}{g0.get('language', '')}）")
|
||||
if github_emerging:
|
||||
e0 = github_emerging[0]
|
||||
points.append(f"🌱 新兴 [{e0['repo']}]({e0['url']})（⭐ {e0.get('total_stars_fmt', '?')}）")
|
||||
elif hot:
|
||||
h0 = hot[0]
|
||||
points.append(f"🔥 Skills Hot 榜首 **{h0.get('title')}**（1H {_format_installs(h0.get('installs', 0))}）")
|
||||
while len(points) < 3 and len(trending) > len(points):
|
||||
item = trending[len(points)]
|
||||
points.append(f"✨ **{item.get('title')}** · `{item.get('source')}`")
|
||||
return points[:3]
|
||||
|
||||
|
||||
def _prepare_github_item(item: dict[str, Any]) -> dict[str, Any]:
|
||||
return {**item, "desc_short": _wecom_desc(item.get("description", ""), 40)}
|
||||
|
||||
|
||||
def _fetch_latest_release_title(repo: str) -> str | None:
|
||||
atom_url = f"https://github.com/{repo}/releases.atom"
|
||||
try:
|
||||
with httpx.Client(
|
||||
timeout=12.0,
|
||||
verify=certifi.where(),
|
||||
follow_redirects=True,
|
||||
headers=github_html_headers(),
|
||||
) as client:
|
||||
resp = client.get(atom_url)
|
||||
if resp.status_code != 200:
|
||||
return None
|
||||
root = ET.fromstring(resp.text)
|
||||
ns = {"a": "http://www.w3.org/2005/Atom"}
|
||||
entry = root.find("a:entry", ns)
|
||||
if entry is None:
|
||||
return None
|
||||
title = entry.find("a:title", ns)
|
||||
return title.text.strip() if title is not None and title.text else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _theme_clusters(feed: dict[str, Any], limit: int = 5) -> list[tuple[str, list[str]]]:
|
||||
buckets: dict[str, list[str]] = defaultdict(list)
|
||||
seen: set[str] = set()
|
||||
for board in ("topTrending", "topHot"):
|
||||
for item in feed.get(board, [])[:20]:
|
||||
item_id = _skill_id(item)
|
||||
if item_id in seen:
|
||||
continue
|
||||
seen.add(item_id)
|
||||
haystack = " ".join(
|
||||
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
|
||||
).lower()
|
||||
for _icon, theme, keywords in THEME_RULES:
|
||||
if any(k in haystack for k in keywords):
|
||||
label = f"**{item.get('title')}** (`{item.get('source')}`)"
|
||||
if label not in buckets[theme]:
|
||||
buckets[theme].append(label)
|
||||
break
|
||||
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
|
||||
|
||||
|
||||
def _format_github_repo_section(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
|
||||
lines: list[str] = []
|
||||
for i, repo in enumerate(repos, 1):
|
||||
lang = repo.get("language") or "—"
|
||||
stars_today = repo.get("stars_today_fmt") or ""
|
||||
total = repo.get("total_stars_fmt") or ""
|
||||
created = repo.get("created_at") or ""
|
||||
meta_parts = [lang]
|
||||
if stars_today:
|
||||
meta_parts.append(f"+{stars_today} today")
|
||||
if total:
|
||||
meta_parts.append(f"总 ⭐ {total}")
|
||||
if show_created and created:
|
||||
meta_parts.append(f"创建于 {created}")
|
||||
lines.append(f"{i}. **[{repo['repo']}]({repo['url']})** · {' · '.join(meta_parts)}")
|
||||
desc = _archive_desc(repo.get("description", ""))
|
||||
if desc:
|
||||
lines.append(f" - {desc}")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def _format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) -> list[str]:
|
||||
lines: list[str] = []
|
||||
for i, item in enumerate(items, 1):
|
||||
skill_id = item.get("id") or f"{item.get('source', '?')}/{item.get('title', '?')}"
|
||||
link = item.get("link", "")
|
||||
installs = _format_installs(item.get("installs", 0))
|
||||
meta = f"1H {installs}" if hot else f"总安装 {installs}"
|
||||
if link:
|
||||
lines.append(f"{i}. **[{skill_id}]({link})** · {meta}")
|
||||
else:
|
||||
lines.append(f"{i}. **{skill_id}** · {meta}")
|
||||
desc = _archive_desc(item.get("description", ""))
|
||||
if desc:
|
||||
lines.append(f" - {desc}")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def generate_report() -> tuple[str, str, Path, Path]:
|
||||
trending_n = env_int("DAILY_TRENDING_LIMIT", 150)
|
||||
hot_n = max(env_int("DAILY_HOT_LIMIT", 150), compare_depth())
|
||||
compare_n = compare_depth()
|
||||
skill_pool = max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
|
||||
wecom_trending = env_int("DAILY_WECOM_TRENDING", 10)
|
||||
wecom_hot = env_int("DAILY_WECOM_HOT", 10)
|
||||
github_limit = env_int("DAILY_GITHUB_TRENDING_LIMIT", 10)
|
||||
wecom_github = env_int("DAILY_WECOM_GITHUB_TRENDING", env_int("DAILY_WECOM_REPOS", 10))
|
||||
github_fetch_n = max(github_limit, compare_n, wecom_github)
|
||||
emerging_limit = env_int("DAILY_GITHUB_EMERGING_LIMIT", 10)
|
||||
wecom_emerging = env_int("DAILY_WECOM_GITHUB_EMERGING", 10)
|
||||
emerging_fetch_n = max(emerging_limit, compare_n, wecom_emerging)
|
||||
topic_limit = env_int("DAILY_GITHUB_TOPIC_LIMIT", 10)
|
||||
wecom_topic = env_int("DAILY_WECOM_GITHUB_TOPIC", 10)
|
||||
topic_fetch_n = max(topic_limit, compare_n, wecom_topic)
|
||||
|
||||
feed = load_feed(force=True)
|
||||
prev_ids = _load_snapshot()
|
||||
now = _now_cst()
|
||||
date_str = now.strftime("%Y-%m-%d")
|
||||
time_str = now.strftime("%H:%M") + " (UTC+8)"
|
||||
updated = (feed.get("updatedAt") or "")[:10]
|
||||
|
||||
trending, hot = load_boards(feed, trending_limit=trending_n, hot_limit=hot_n)
|
||||
github_trending = fetch_github_trending(github_fetch_n)
|
||||
seen_repos = {r["repo"] for r in github_trending}
|
||||
github_emerging = fetch_emerging_repos(emerging_fetch_n, exclude=seen_repos)
|
||||
seen_repos.update(r["repo"] for r in github_emerging)
|
||||
topic_name, github_topic = fetch_topic_hot_repos(topic_fetch_n, exclude=seen_repos)
|
||||
ai_news = fetch_ai_news()
|
||||
|
||||
wecom_limits = {
|
||||
"trending": wecom_trending,
|
||||
"hot": wecom_hot,
|
||||
"trending_pool": skill_pool,
|
||||
"hot_pool": skill_pool,
|
||||
"github": wecom_github,
|
||||
"emerging": wecom_emerging,
|
||||
"topic": wecom_topic,
|
||||
"ai_news": env_int("DAILY_WECOM_AI_NEWS", 10),
|
||||
}
|
||||
llm_input = build_llm_input(
|
||||
date_str=date_str,
|
||||
updated=updated,
|
||||
trending=trending,
|
||||
hot=hot,
|
||||
github_trending=github_trending,
|
||||
github_emerging=github_emerging,
|
||||
github_topic=github_topic,
|
||||
topic_name=topic_name,
|
||||
ai_news=ai_news,
|
||||
wecom_limits=wecom_limits,
|
||||
)
|
||||
save_json(
|
||||
data_json_path(date_str),
|
||||
build_full_payload(
|
||||
llm_input,
|
||||
meta={
|
||||
"generated_at": now.isoformat(),
|
||||
"report_mode": "agent" if is_agent_mode() else "classic",
|
||||
"cursor_editor": cursor_editor_enabled() and not is_agent_mode(),
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
agent_wecom: str | None = None
|
||||
if is_agent_mode():
|
||||
agent_wecom = run_agent_workflow(
|
||||
llm_input,
|
||||
date_str=date_str,
|
||||
time_str=time_str,
|
||||
updated=updated,
|
||||
)
|
||||
if not agent_wecom:
|
||||
logger.warning("Agent 工作流失败，回退 classic 模式")
|
||||
|
||||
editorial_theme: str | None = None
|
||||
editorial_highlights: list[str] | None = None
|
||||
if agent_wecom is None:
|
||||
editorial = run_editorial(llm_input, date_str=date_str)
|
||||
if editorial:
|
||||
apply_descriptions(
|
||||
trending=trending,
|
||||
hot=hot,
|
||||
github_trending=github_trending,
|
||||
github_emerging=github_emerging,
|
||||
github_topic=github_topic,
|
||||
ai_news=ai_news,
|
||||
descriptions=editorial.get("descriptions") or {},
|
||||
)
|
||||
editorial_theme = theme_line_from_editorial(editorial) or None
|
||||
hl = editorial.get("highlights") or []
|
||||
editorial_highlights = hl if hl else None
|
||||
|
||||
# 完整版归档：中文化 + 加长摘要（已是中文的条目会跳过翻译）
|
||||
_localize_descriptions_in_place(
|
||||
trending, hot, github_trending, github_emerging, github_topic, ai_news
|
||||
)
|
||||
|
||||
themes = _theme_clusters(feed)
|
||||
|
||||
lines = [
|
||||
f"# 早报 · {date_str}",
|
||||
"",
|
||||
f"> 生成时间：{now.strftime('%Y-%m-%d %H:%M')} (UTC+8) ",
|
||||
f"> skills 数据更新：{updated} ",
|
||||
"> 数据来源：[skills.sh/trending](https://skills.sh/trending) · [skills.sh/hot](https://skills.sh/hot) · 国际 AI RSS",
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
f"## 一、Skills Trending Top {trending_n}",
|
||||
"",
|
||||
*_format_skill_section(trending),
|
||||
"---",
|
||||
"",
|
||||
f"## 二、Skills Hot Top {hot_n}",
|
||||
"",
|
||||
*_format_skill_section(hot, hot=True),
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
f"## 三、GitHub Trending Top {github_limit}",
|
||||
"",
|
||||
trending_data_source_note(),
|
||||
"",
|
||||
]
|
||||
|
||||
if github_trending:
|
||||
lines.extend(_format_github_repo_section(github_trending))
|
||||
else:
|
||||
lines.append("*GitHub Trending 获取失败，请检查网络或配置 GITHUB_TOKEN。*")
|
||||
lines.append("")
|
||||
|
||||
lines.extend(["---", "", f"## 四、新兴项目 Top {emerging_limit}", "", "> 数据来源：GitHub Search API（需 `GITHUB_TOKEN`）", ""])
|
||||
if github_emerging:
|
||||
lines.extend(_format_github_repo_section(github_emerging, show_created=True))
|
||||
else:
|
||||
lines.append("*新兴项目获取失败或未配置 GITHUB_TOKEN。*")
|
||||
lines.append("")
|
||||
|
||||
lines.extend(["---", "", f"## 五、Topic `{topic_name}` Top {topic_limit}", "", "> 数据来源：GitHub Search API（需 `GITHUB_TOKEN`）", ""])
|
||||
if github_topic:
|
||||
lines.extend(_format_github_repo_section(github_topic))
|
||||
else:
|
||||
lines.append(f"*Topic `{topic_name}` 热点获取失败或未配置 GITHUB_TOKEN。*")
|
||||
lines.append("")
|
||||
|
||||
section_no = 6
|
||||
lines.extend(format_news_section(ai_news, section_no=section_no))
|
||||
section_no += 1
|
||||
|
||||
watch = (env("GITHUB_REPOS") or "").strip()
|
||||
if watch:
|
||||
lines.extend(["---", "", f"## {section_no}、关注仓库 Release", ""])
|
||||
section_no += 1
|
||||
for repo in [r.strip() for r in watch.split(",") if r.strip()]:
|
||||
release = _fetch_latest_release_title(repo)
|
||||
lines.append(f"- **{repo}**：{release or '暂无 release'}")
|
||||
lines.append("")
|
||||
|
||||
lines.extend(["---", "", f"## {section_no}、主题聚类", ""])
|
||||
for theme, examples in themes:
|
||||
lines.append(f"### {theme}")
|
||||
for ex in examples:
|
||||
lines.append(f"- {ex}")
|
||||
lines.append("")
|
||||
|
||||
pick_src = trending[0].get("source", "") if trending else ""
|
||||
pick_name = trending[0].get("title", "") if trending else ""
|
||||
pick_command = (
|
||||
f"npx skills add {pick_src}/{pick_name}"
|
||||
if pick_src and pick_name
|
||||
else "npx skills add vercel-labs/skills/find-skills"
|
||||
)
|
||||
|
||||
lines.extend(["---", "", "## 安装示例", "", "```bash"])
|
||||
for item in trending[:4]:
|
||||
src, name = item.get("source", ""), item.get("title", "")
|
||||
if src and name:
|
||||
lines.append(f"npx skills add {src}/{name}")
|
||||
lines.extend(["```", "", f"*企微短版见 `output/{date_str}.wecom.md`*"])
|
||||
|
||||
markdown = "\n".join(lines)
|
||||
if agent_wecom:
|
||||
gt = group_skills_by_source(trending, limit=wecom_trending, pool_size=skill_pool)
|
||||
gh = group_skills_by_source(hot, limit=wecom_hot, pool_size=skill_pool)
|
||||
wecom_md = replace_wecom_skill_sections(agent_wecom, trending=gt, hot=gh)
|
||||
else:
|
||||
wecom_md = build_wecom_report(
|
||||
date_str=date_str,
|
||||
time_str=time_str,
|
||||
updated=updated,
|
||||
highlights=editorial_highlights
|
||||
or _build_highlights(trending, hot, github_trending, github_emerging, ai_news),
|
||||
theme_line=editorial_theme or _detect_theme_line(feed),
|
||||
ai_news=prepare_wecom_news_items(ai_news),
|
||||
trending=[
|
||||
_prepare_skill_item(item, prev_ids, r)
|
||||
for r, item in enumerate(
|
||||
finalize_wecom_skill_groups(
|
||||
group_skills_by_source(trending, limit=wecom_trending, pool_size=skill_pool)
|
||||
),
|
||||
1,
|
||||
)
|
||||
],
|
||||
hot=[
|
||||
_prepare_skill_item(item, prev_ids, r)
|
||||
for r, item in enumerate(
|
||||
finalize_wecom_skill_groups(
|
||||
group_skills_by_source(hot, limit=wecom_hot, pool_size=skill_pool)
|
||||
),
|
||||
1,
|
||||
)
|
||||
],
|
||||
repos=[_prepare_github_item(item) for item in github_trending[:wecom_github]],
|
||||
emerging=[_prepare_github_item(item) for item in github_emerging[:wecom_emerging]],
|
||||
topic_name=topic_name,
|
||||
topic_repos=[_prepare_github_item(item) for item in github_topic[:wecom_topic]],
|
||||
pick_command=pick_command,
|
||||
)
|
||||
|
||||
_save_snapshot(feed, date_str)
|
||||
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
out_md = OUTPUT_DIR / f"{date_str}.md"
|
||||
out_wecom = OUTPUT_DIR / f"{date_str}.wecom.md"
|
||||
out_md.write_text(markdown, encoding="utf-8")
|
||||
out_wecom.write_text(wecom_md, encoding="utf-8")
|
||||
return markdown, wecom_md, out_md, out_wecom
|
||||
|
||||
|
||||
def main() -> int:
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
log_file = LOG_DIR / f"{_now_cst():%Y-%m-%d}.log"
|
||||
try:
|
||||
_, wecom_md, out_md, out_wecom = generate_report()
|
||||
nbytes = len(wecom_md.encode("utf-8"))
|
||||
msg = f"[{_now_cst():%H:%M:%S}] OK -> {out_md}, {out_wecom} ({nbytes} bytes)\n"
|
||||
log_file.write_text(msg, encoding="utf-8")
|
||||
print(msg.strip())
|
||||
return 0
|
||||
except Exception as exc:
|
||||
msg = f"[{_now_cst():%H:%M:%S}] FAIL: {exc}\n"
|
||||
log_file.write_text(msg, encoding="utf-8")
|
||||
print(msg.strip(), file=sys.stderr)
|
||||
return 1
|
||||
9
daily/github/__init__.py
Normal file
9
daily/github/__init__.py
Normal file
@@ -0,0 +1,9 @@
|
||||
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos
|
||||
from daily.github.trending import fetch_github_trending, trending_data_source_note
|
||||
|
||||
__all__ = [
|
||||
"fetch_emerging_repos",
|
||||
"fetch_topic_hot_repos",
|
||||
"fetch_github_trending",
|
||||
"trending_data_source_note",
|
||||
]
|
||||
81
daily/github/auth.py
Normal file
81
daily/github/auth.py
Normal file
@@ -0,0 +1,81 @@
|
||||
"""GitHub 请求共用：GITHUB_TOKEN、请求头、仓库 API。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_USER_AGENT = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
|
||||
def github_token() -> str | None:
|
||||
raw = (env("GITHUB_TOKEN") or "").strip()
|
||||
return raw or None
|
||||
|
||||
|
||||
def github_api_headers() -> dict[str, str]:
|
||||
headers = {
|
||||
"User-Agent": env("GITHUB_TRENDING_USER_AGENT", DEFAULT_USER_AGENT),
|
||||
"Accept": "application/vnd.github+json",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
}
|
||||
token = github_token()
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
return headers
|
||||
|
||||
|
||||
def github_html_headers() -> dict[str, str]:
|
||||
headers = {
|
||||
"User-Agent": env("GITHUB_TRENDING_USER_AGENT", DEFAULT_USER_AGENT),
|
||||
"Accept": "text/html,application/xhtml+xml",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
token = github_token()
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
return headers
|
||||
|
||||
|
||||
def fetch_repo_api(repo: str) -> dict[str, Any] | None:
|
||||
url = f"https://api.github.com/repos/{repo}"
|
||||
try:
|
||||
with httpx.Client(
|
||||
timeout=12.0,
|
||||
verify=certifi.where(),
|
||||
headers=github_api_headers(),
|
||||
) as client:
|
||||
resp = client.get(url)
|
||||
if resp.status_code != 200:
|
||||
return None
|
||||
data = resp.json()
|
||||
return {
|
||||
"description": data.get("description") or "",
|
||||
"language": data.get("language") or "",
|
||||
"stars": data.get("stargazers_count", 0),
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.debug("GitHub API repo %s 失败: %s", repo, exc)
|
||||
return None
|
||||
|
||||
|
||||
def format_star_count(value: int | float | str) -> str:
|
||||
try:
|
||||
n = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
if n >= 1_000_000:
|
||||
return f"{n / 1_000_000:.1f}M".replace(".0M", "M")
|
||||
if n >= 1_000:
|
||||
return f"{n / 1_000:.1f}K".replace(".0K", "K")
|
||||
return f"{n:,}"
|
||||
121
daily/github/search.py
Normal file
121
daily/github/search.py
Normal file
@@ -0,0 +1,121 @@
|
||||
"""GitHub Search API：新兴项目、Topic 热点。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env, env_int
|
||||
from daily.github.auth import format_star_count, github_api_headers, github_token
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _repo_from_api_item(item: dict[str, Any], *, source: str) -> dict[str, Any]:
|
||||
full_name = item.get("full_name") or ""
|
||||
stars = item.get("stargazers_count", 0)
|
||||
created = (item.get("created_at") or "")[:10]
|
||||
return {
|
||||
"repo": full_name,
|
||||
"url": item.get("html_url") or f"https://github.com/{full_name}",
|
||||
"description": item.get("description") or "",
|
||||
"language": item.get("language") or "",
|
||||
"stars_today": None,
|
||||
"stars_today_fmt": "",
|
||||
"total_stars_fmt": format_star_count(stars),
|
||||
"created_at": created,
|
||||
"source": source,
|
||||
}
|
||||
|
||||
|
||||
def search_github_repos(
|
||||
query: str,
|
||||
limit: int,
|
||||
*,
|
||||
sort: str = "stars",
|
||||
require_token: bool = True,
|
||||
) -> list[dict[str, Any]]:
|
||||
if require_token and not github_token():
|
||||
logger.warning("GitHub Search 需要 GITHUB_TOKEN: %s", query[:80])
|
||||
return []
|
||||
|
||||
try:
|
||||
with httpx.Client(
|
||||
timeout=20.0,
|
||||
verify=certifi.where(),
|
||||
headers=github_api_headers(),
|
||||
) as client:
|
||||
resp = client.get(
|
||||
"https://api.github.com/search/repositories",
|
||||
params={
|
||||
"q": query,
|
||||
"sort": sort,
|
||||
"order": "desc",
|
||||
"per_page": min(max(limit, 1), 30),
|
||||
},
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
logger.warning("GitHub Search 失败 (%s): %s", resp.status_code, query[:80])
|
||||
return []
|
||||
items = resp.json().get("items") or []
|
||||
except Exception as exc:
|
||||
logger.warning("GitHub Search 异常: %s", exc)
|
||||
return []
|
||||
|
||||
repos: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
full_name = item.get("full_name") or ""
|
||||
if not full_name:
|
||||
continue
|
||||
repos.append(_repo_from_api_item(item, source="api-search"))
|
||||
if len(repos) >= limit:
|
||||
break
|
||||
return repos
|
||||
|
||||
|
||||
def _date_days_ago(days: int) -> str:
|
||||
return (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def fetch_emerging_repos(
|
||||
limit: int = 3,
|
||||
*,
|
||||
days: int | None = None,
|
||||
min_stars: int | None = None,
|
||||
exclude: set[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
days = days if days is not None else env_int("GITHUB_EMERGING_DAYS", 14)
|
||||
min_stars = min_stars if min_stars is not None else env_int("GITHUB_EMERGING_MIN_STARS", 200)
|
||||
created_after = _date_days_ago(days)
|
||||
query = f"created:>{created_after} stars:>{min_stars} fork:false"
|
||||
repos = search_github_repos(query, limit + len(exclude or set()))
|
||||
if exclude:
|
||||
repos = [r for r in repos if r["repo"] not in exclude]
|
||||
for item in repos:
|
||||
item["source"] = "api-emerging"
|
||||
return repos[:limit]
|
||||
|
||||
|
||||
def fetch_topic_hot_repos(
|
||||
limit: int = 3,
|
||||
*,
|
||||
topic: str | None = None,
|
||||
pushed_days: int | None = None,
|
||||
min_stars: int | None = None,
|
||||
exclude: set[str] | None = None,
|
||||
) -> tuple[str, list[dict[str, Any]]]:
|
||||
topic = (topic or env("GITHUB_TOPIC") or "llm").strip()
|
||||
pushed_days = pushed_days if pushed_days is not None else env_int("GITHUB_TOPIC_PUSHED_DAYS", 7)
|
||||
min_stars = min_stars if min_stars is not None else env_int("GITHUB_TOPIC_MIN_STARS", 50)
|
||||
pushed_after = _date_days_ago(pushed_days)
|
||||
query = f"topic:{topic} pushed:>{pushed_after} stars:>{min_stars} fork:false"
|
||||
repos = search_github_repos(query, limit + len(exclude or set()))
|
||||
if exclude:
|
||||
repos = [r for r in repos if r["repo"] not in exclude]
|
||||
for item in repos:
|
||||
item["source"] = "api-topic"
|
||||
return topic, repos[:limit]
|
||||
206
daily/github/trending.py
Normal file
206
daily/github/trending.py
Normal file
@@ -0,0 +1,206 @@
|
||||
"""GitHub Trending：页面爬取或 Search API。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from html import unescape
|
||||
from typing import Any, Literal
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env
|
||||
from daily.github.auth import (
|
||||
fetch_repo_api,
|
||||
format_star_count,
|
||||
github_html_headers,
|
||||
github_token,
|
||||
)
|
||||
from daily.github.search import search_github_repos
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TrendingSince = Literal["daily", "weekly", "monthly"]
|
||||
TrendingMode = Literal["scrape", "api"]
|
||||
|
||||
DEFAULT_LIMIT = 5
|
||||
DEFAULT_SINCE: TrendingSince = "daily"
|
||||
DEFAULT_MODE: TrendingMode = "scrape"
|
||||
TRENDING_URL = "https://github.com/trending"
|
||||
|
||||
_ARTICLE_RE = re.compile(r'<article class="Box-row">.*?</article>', re.S)
|
||||
_REPO_HREF_RE = re.compile(r'h2[^>]*>\s*<a[^>]+href="([^"]+)"')
|
||||
_DESC_RE = re.compile(r'<p class="col-9[^"]*"[^>]*>([^<]*)</p>')
|
||||
_STARS_TODAY_RE = re.compile(r"([\d,]+)\s+stars?\s+today", re.I)
|
||||
_LANG_RE = re.compile(r'itemprop="programmingLanguage"[^>]*>([^<]+)<')
|
||||
_TOTAL_STARS_RE = re.compile(
|
||||
r'href="/[^/]+/[^/]+/stargazers"[^>]*>\s*<svg[^>]*octicon-star[^>]*>.*?</svg>\s*([\d.,kKmM]+)',
|
||||
re.S,
|
||||
)
|
||||
|
||||
|
||||
def _strip_html(text: str) -> str:
|
||||
return unescape(re.sub(r"\s+", " ", text or "")).strip()
|
||||
|
||||
|
||||
def trending_mode() -> TrendingMode:
|
||||
raw = (env("GITHUB_TRENDING_MODE") or DEFAULT_MODE).strip().lower()
|
||||
if raw in {"api", "token", "search"}:
|
||||
return "api"
|
||||
return "scrape"
|
||||
|
||||
|
||||
def trending_data_source_note() -> str:
|
||||
if trending_mode() == "api":
|
||||
return "> 数据来源：GitHub Search API（`GITHUB_TRENDING_MODE=api`，需 `GITHUB_TOKEN`）"
|
||||
return "> 数据来源：[github.com/trending](https://github.com/trending?since=daily)（页面爬取，失败时 API 降级）"
|
||||
|
||||
|
||||
def _parse_article(article_html: str) -> dict[str, Any] | None:
|
||||
href_match = _REPO_HREF_RE.search(article_html)
|
||||
if not href_match:
|
||||
return None
|
||||
href = href_match.group(1).strip("/")
|
||||
if href.count("/") != 1:
|
||||
return None
|
||||
owner, name = href.split("/", 1)
|
||||
repo = f"{owner}/{name}"
|
||||
desc_match = _DESC_RE.search(article_html)
|
||||
stars_today_match = _STARS_TODAY_RE.search(article_html)
|
||||
lang_match = _LANG_RE.search(article_html)
|
||||
total_stars_match = _TOTAL_STARS_RE.search(article_html)
|
||||
stars_today_raw = stars_today_match.group(1).replace(",", "") if stars_today_match else ""
|
||||
stars_today = int(stars_today_raw) if stars_today_raw.isdigit() else None
|
||||
return {
|
||||
"repo": repo,
|
||||
"url": f"https://github.com/{repo}",
|
||||
"description": _strip_html(desc_match.group(1)) if desc_match else "",
|
||||
"language": _strip_html(lang_match.group(1)) if lang_match else "",
|
||||
"stars_today": stars_today,
|
||||
"stars_today_fmt": stars_today_match.group(1) if stars_today_match else "",
|
||||
"total_stars_fmt": _strip_html(total_stars_match.group(1)) if total_stars_match else "",
|
||||
"source": "scrape",
|
||||
}
|
||||
|
||||
|
||||
def _build_trending_url(*, since: TrendingSince = DEFAULT_SINCE, language: str = "") -> str:
|
||||
if language:
|
||||
return f"{TRENDING_URL}/{language}?{urlencode({'since': since})}"
|
||||
return f"{TRENDING_URL}?{urlencode({'since': since})}"
|
||||
|
||||
|
||||
def _since_push_date(since: TrendingSince) -> str:
|
||||
now = datetime.now(timezone.utc)
|
||||
if since == "weekly":
|
||||
delta = timedelta(days=7)
|
||||
elif since == "monthly":
|
||||
delta = timedelta(days=30)
|
||||
else:
|
||||
delta = timedelta(days=1)
|
||||
return (now - delta).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def _enrich_repo_from_api(item: dict[str, Any]) -> dict[str, Any]:
|
||||
if not github_token() and env("GITHUB_API_ENRICH", "1") != "1":
|
||||
return item
|
||||
meta = fetch_repo_api(item["repo"])
|
||||
if not meta:
|
||||
return item
|
||||
enriched = dict(item)
|
||||
if not enriched.get("description"):
|
||||
enriched["description"] = meta["description"]
|
||||
if not enriched.get("language"):
|
||||
enriched["language"] = meta["language"]
|
||||
if not enriched.get("total_stars_fmt") and meta["stars"]:
|
||||
enriched["total_stars_fmt"] = format_star_count(meta["stars"])
|
||||
enriched["source"] = enriched.get("source", "scrape") + "+api"
|
||||
return enriched
|
||||
|
||||
|
||||
def _fetch_trending_html(url: str) -> str | None:
|
||||
try:
|
||||
with httpx.Client(
|
||||
timeout=20.0,
|
||||
verify=certifi.where(),
|
||||
follow_redirects=True,
|
||||
headers=github_html_headers(),
|
||||
) as client:
|
||||
resp = client.get(url)
|
||||
if resp.status_code in {403, 429} and not github_token():
|
||||
logger.warning("GitHub Trending %s（匿名可能被限），可配置 GITHUB_TOKEN", resp.status_code)
|
||||
resp.raise_for_status()
|
||||
return resp.text
|
||||
except Exception as exc:
|
||||
logger.warning("GitHub Trending 页面抓取失败: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _parse_trending_html(html: str, limit: int) -> list[dict[str, Any]]:
|
||||
repos: list[dict[str, Any]] = []
|
||||
for article_html in _ARTICLE_RE.findall(html):
|
||||
item = _parse_article(article_html)
|
||||
if item:
|
||||
repos.append(_enrich_repo_from_api(item))
|
||||
if len(repos) >= limit:
|
||||
break
|
||||
return repos
|
||||
|
||||
|
||||
def _fetch_trending_via_api(limit: int, since: TrendingSince, language: str) -> list[dict[str, Any]]:
|
||||
pushed_after = _since_push_date(since)
|
||||
parts = [f"pushed:>{pushed_after}", "stars:>50", "fork:false"]
|
||||
if language:
|
||||
parts.append(f"language:{language}")
|
||||
query = " ".join(parts)
|
||||
repos = search_github_repos(query, limit, require_token=True)
|
||||
for item in repos:
|
||||
item["source"] = "api-search"
|
||||
return repos
|
||||
|
||||
|
||||
def fetch_github_trending(
|
||||
limit: int = DEFAULT_LIMIT,
|
||||
*,
|
||||
since: TrendingSince | None = None,
|
||||
language: str = "",
|
||||
) -> list[dict[str, Any]]:
|
||||
since = since or env("GITHUB_TRENDING_SINCE", DEFAULT_SINCE) # type: ignore[assignment]
|
||||
if since not in ("daily", "weekly", "monthly"):
|
||||
since = DEFAULT_SINCE
|
||||
lang = (language or env("GITHUB_TRENDING_LANGUAGE") or "").strip()
|
||||
mode = trending_mode()
|
||||
|
||||
if mode == "api":
|
||||
repos = _fetch_trending_via_api(limit, since, lang)
|
||||
if not repos and not github_token():
|
||||
logger.warning("GITHUB_TRENDING_MODE=api 需要配置 GITHUB_TOKEN")
|
||||
elif repos:
|
||||
logger.info("GitHub Trending 使用 API 模式，共 %d 条", len(repos))
|
||||
return repos[:limit]
|
||||
|
||||
url = _build_trending_url(since=since, language=lang)
|
||||
html = _fetch_trending_html(url)
|
||||
repos: list[dict[str, Any]] = []
|
||||
if html:
|
||||
repos = _parse_trending_html(html, limit)
|
||||
if not repos:
|
||||
logger.warning("GitHub Trending 页面解析为空: %s", url)
|
||||
|
||||
if len(repos) < limit:
|
||||
before = len(repos)
|
||||
fallback = _fetch_trending_via_api(limit, since, lang)
|
||||
seen = {r["repo"] for r in repos}
|
||||
for item in fallback:
|
||||
if item["repo"] in seen:
|
||||
continue
|
||||
repos.append(item)
|
||||
seen.add(item["repo"])
|
||||
if len(repos) >= limit:
|
||||
break
|
||||
if len(repos) > before:
|
||||
logger.info("已用 GitHub API 补充 %d 条 Trending 数据", len(repos) - before)
|
||||
|
||||
return repos[:limit]
|
||||
119
daily/llm_client.py
Normal file
119
daily/llm_client.py
Normal file
@@ -0,0 +1,119 @@
|
||||
"""LLM 调用共享工具（OpenAI 兼容 API / Cursor SDK）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import ROOT, env, env_int
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_JSON_BLOCK = re.compile(r"```(?:json)?\s*([\s\S]*?)```", re.IGNORECASE)
|
||||
|
||||
|
||||
def extract_json_object(text: str) -> dict[str, Any]:
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(text)
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
match = _JSON_BLOCK.search(text)
|
||||
if match:
|
||||
try:
|
||||
data = json.loads(match.group(1).strip())
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
start, end = text.find("{"), text.rfind("}")
|
||||
if start >= 0 and end > start:
|
||||
try:
|
||||
data = json.loads(text[start : end + 1])
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def _openai_chat(system: str, user: str) -> str:
|
||||
api_key = (env("DAILY_LLM_API_KEY") or env("OPENAI_API_KEY") or "").strip()
|
||||
if not api_key:
|
||||
return ""
|
||||
base = (env("DAILY_LLM_API_BASE") or env("OPENAI_API_BASE") or "https://api.openai.com/v1").rstrip("/")
|
||||
model = env("DAILY_LLM_MODEL") or env("OPENAI_MODEL") or "gpt-4o-mini"
|
||||
timeout = env_int("DAILY_LLM_TIMEOUT", 120)
|
||||
payload = {
|
||||
"model": model,
|
||||
"temperature": 0.2,
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
}
|
||||
with httpx.Client(timeout=timeout, verify=certifi.where()) as client:
|
||||
resp = client.post(
|
||||
f"{base}/chat/completions",
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
json=payload,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return str(data["choices"][0]["message"]["content"] or "").strip()
|
||||
|
||||
|
||||
def _cursor_chat(system: str, user: str) -> str:
|
||||
api_key = (env("CURSOR_API_KEY") or "").strip()
|
||||
if not api_key:
|
||||
return ""
|
||||
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
|
||||
|
||||
from daily.config import ensure_bot_on_path
|
||||
|
||||
ensure_bot_on_path()
|
||||
try:
|
||||
from bridge_manager import warm_cursor_bridge
|
||||
except ImportError:
|
||||
warm_cursor_bridge = lambda: None # noqa: E731
|
||||
|
||||
cwd = env("DAILY_CURSOR_CWD") or str(ROOT)
|
||||
# bridge_manager 读 bot env_config 的 CURSOR_CWD，早报侧须先对齐工作目录
|
||||
os.environ["CURSOR_CWD"] = cwd
|
||||
warm_cursor_bridge()
|
||||
model = env("CURSOR_MODEL") or "composer-2.5"
|
||||
prompt = f"{system}\n\n{user}"
|
||||
try:
|
||||
result = Agent.prompt(
|
||||
prompt,
|
||||
AgentOptions(
|
||||
api_key=api_key,
|
||||
model=model,
|
||||
local=LocalAgentOptions(cwd=cwd),
|
||||
),
|
||||
)
|
||||
except CursorAgentError as exc:
|
||||
raise RuntimeError(f"LLM 调用失败：{exc.message}") from exc
|
||||
if result.status == "error":
|
||||
raise RuntimeError(f"LLM 调用失败：{result.result or '未知错误'}")
|
||||
return (result.result or "").strip()
|
||||
|
||||
|
||||
def llm_chat(system: str, user: str) -> str:
|
||||
"""优先 OpenAI 兼容 API，否则 Cursor SDK。"""
|
||||
if env("DAILY_LLM_API_KEY") or env("OPENAI_API_KEY"):
|
||||
return _openai_chat(system, user)
|
||||
if env("CURSOR_API_KEY"):
|
||||
return _cursor_chat(system, user)
|
||||
return ""
|
||||
|
||||
|
||||
def has_llm_configured() -> bool:
|
||||
return bool(env("DAILY_LLM_API_KEY") or env("OPENAI_API_KEY") or env("CURSOR_API_KEY"))
|
||||
236
daily/localize.py
Normal file
236
daily/localize.py
Normal file
@@ -0,0 +1,236 @@
|
||||
"""将英文描述批量改写为简短中文（大模型 + 本地缓存）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
from daily.config import CACHE_DIR, env, env_int
|
||||
from daily.cursor_editor import is_enabled as cursor_editor_enabled
|
||||
from daily.llm_client import extract_json_object, llm_chat
|
||||
from daily.text_utils import clip_text, trim_brief
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_CACHE_FILE = CACHE_DIR / "zh-desc-cache.json"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LocalizeJob:
|
||||
key: str
|
||||
text: str
|
||||
limit: int
|
||||
|
||||
|
||||
def _enabled(*, archive: bool = False) -> bool:
|
||||
if not archive and cursor_editor_enabled():
|
||||
return False
|
||||
raw = (env("DAILY_ZH_DESC") or "1").strip().lower()
|
||||
return raw not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def _is_mostly_chinese(text: str) -> bool:
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return True
|
||||
cjk = sum(1 for c in text if "\u4e00" <= c <= "\u9fff")
|
||||
latin = sum(1 for c in text if c.isascii() and c.isalpha())
|
||||
return cjk >= max(latin, 1)
|
||||
|
||||
|
||||
def needs_chinese(text: str) -> bool:
|
||||
"""文本非空且尚未以中文为主。"""
|
||||
text = (text or "").strip()
|
||||
if not text:
|
||||
return False
|
||||
return not _is_mostly_chinese(text)
|
||||
|
||||
|
||||
def _cache_key(text: str) -> str:
|
||||
return hashlib.sha1(text.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _legacy_cache_key(text: str, limit: int) -> str:
|
||||
return hashlib.sha1(f"{limit}:{text}".encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _cache_is_truncated(text: str) -> bool:
|
||||
t = (text or "").rstrip()
|
||||
return t.endswith("…") or t.endswith("...")
|
||||
|
||||
|
||||
def _lookup_cached(cache: dict[str, str], text: str, limit: int) -> str | None:
|
||||
hit = cache.get(_cache_key(text))
|
||||
if hit and not (limit <= 0 and _cache_is_truncated(hit)):
|
||||
return hit
|
||||
for legacy_limit in (72, 120, 200):
|
||||
legacy = cache.get(_legacy_cache_key(text, legacy_limit))
|
||||
if legacy and not (limit <= 0 and _cache_is_truncated(legacy)):
|
||||
cache[_cache_key(text)] = legacy
|
||||
return legacy
|
||||
return None
|
||||
|
||||
|
||||
def _load_cache() -> dict[str, str]:
|
||||
if not _CACHE_FILE.exists():
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(_CACHE_FILE.read_text(encoding="utf-8"))
|
||||
return {str(k): str(v) for k, v in (data.get("entries") or {}).items()}
|
||||
except (OSError, json.JSONDecodeError, TypeError):
|
||||
return {}
|
||||
|
||||
|
||||
def _save_cache(entries: dict[str, str]) -> None:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
_CACHE_FILE.write_text(
|
||||
json.dumps({"entries": entries}, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _build_prompt(jobs: list[LocalizeJob], *, brief: bool = False) -> tuple[str, str]:
|
||||
if brief:
|
||||
system = (
|
||||
"你是技术早报编辑。把输入 JSON 中每条描述改写为**一句**中文简要介绍。"
|
||||
"要求：输出必须是中文；只保留核心能力与典型场景；不要逐字翻译；不要加引号或编号；"
|
||||
"每条必须语义完整、可独立阅读；limit 为建议最大字数，请控制在 limit 以内且不要用省略号截断；"
|
||||
"已是中文且足够简短时可适度精简；"
|
||||
"只输出 JSON 对象，key 与输入一致，value 为中文简介字符串。"
|
||||
)
|
||||
else:
|
||||
system = (
|
||||
"你是技术早报编辑。把输入 JSON 中每条英文描述改写为**完整**中文介绍。"
|
||||
"要求：输出必须是中文；保留关键能力与使用场景；不要逐字翻译；不要加引号或编号；"
|
||||
"不要以省略号截断；已是中文则原样或适度精简；"
|
||||
"只输出 JSON 对象，key 与输入一致，value 为中文简介字符串。"
|
||||
)
|
||||
payload = {job.key: {"text": job.text, "limit": job.limit} for job in jobs}
|
||||
user = json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
return system, user
|
||||
|
||||
|
||||
def _brief_cache_key(text: str, limit: int) -> str:
|
||||
return hashlib.sha1(f"brief:{limit}:{text}".encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _lookup_brief_cached(cache: dict[str, str], text: str, limit: int) -> str | None:
|
||||
hit = cache.get(_brief_cache_key(text, limit))
|
||||
if hit and not _cache_is_truncated(hit):
|
||||
return hit
|
||||
return None
|
||||
|
||||
|
||||
def _translate_batch(jobs: list[LocalizeJob], *, brief: bool = False) -> dict[str, str]:
|
||||
if not jobs:
|
||||
return {}
|
||||
system, user = _build_prompt(jobs, brief=brief)
|
||||
raw = llm_chat(system, user)
|
||||
if not raw:
|
||||
return {}
|
||||
parsed = extract_json_object(raw)
|
||||
out: dict[str, str] = {}
|
||||
for job in jobs:
|
||||
value = parsed.get(job.key)
|
||||
if isinstance(value, str) and value.strip():
|
||||
trimmed = value.strip()
|
||||
if brief and job.limit > 0:
|
||||
out[job.key] = trim_brief(trimmed, job.limit)
|
||||
else:
|
||||
out[job.key] = clip_text(trimmed, job.limit) if job.limit > 0 else trimmed
|
||||
return out
|
||||
|
||||
|
||||
def localize_brief_descriptions(jobs: list[LocalizeJob], *, archive: bool = False) -> dict[str, str]:
|
||||
"""企微用：将描述改写为一句简要中文。"""
|
||||
if not _enabled(archive=archive) or not jobs:
|
||||
return {}
|
||||
|
||||
cache = _load_cache()
|
||||
result: dict[str, str] = {}
|
||||
pending: list[LocalizeJob] = []
|
||||
|
||||
for job in jobs:
|
||||
if not job.text.strip():
|
||||
continue
|
||||
limit = job.limit if job.limit > 0 else 48
|
||||
text = job.text.strip()
|
||||
if _is_mostly_chinese(text) and (limit <= 0 or len(text) <= limit):
|
||||
result[job.key] = text
|
||||
continue
|
||||
cached = _lookup_brief_cached(cache, text, limit)
|
||||
if cached:
|
||||
result[job.key] = trim_brief(cached, limit) if limit > 0 else cached
|
||||
else:
|
||||
pending.append(LocalizeJob(job.key, text, limit))
|
||||
|
||||
if not pending:
|
||||
return result
|
||||
|
||||
batch_size = max(3, min(10, env_int("DAILY_ZH_DESC_BATCH", 20)))
|
||||
for i in range(0, len(pending), batch_size):
|
||||
chunk = pending[i : i + batch_size]
|
||||
try:
|
||||
translated = _translate_batch(chunk, brief=True)
|
||||
except Exception as exc:
|
||||
logger.warning("企微简要摘要批次失败，保留原文：%s", exc)
|
||||
continue
|
||||
for job in chunk:
|
||||
zh = translated.get(job.key)
|
||||
if not zh:
|
||||
continue
|
||||
ck = _brief_cache_key(job.text, job.limit if job.limit > 0 else 48)
|
||||
cache[ck] = zh
|
||||
result[job.key] = zh
|
||||
|
||||
if cache:
|
||||
_save_cache(cache)
|
||||
return result
|
||||
|
||||
|
||||
def localize_descriptions(jobs: list[LocalizeJob], *, archive: bool = False) -> dict[str, str]:
|
||||
"""返回 job.key -> 中文简介。archive=True 时用于完整版 .md，不受 Cursor 编辑层开关影响。"""
|
||||
if not _enabled(archive=archive) or not jobs:
|
||||
return {}
|
||||
|
||||
cache = _load_cache()
|
||||
result: dict[str, str] = {}
|
||||
pending: list[LocalizeJob] = []
|
||||
|
||||
for job in jobs:
|
||||
if not job.text.strip():
|
||||
continue
|
||||
if _is_mostly_chinese(job.text):
|
||||
result[job.key] = clip_text(job.text, job.limit)
|
||||
continue
|
||||
ck = _cache_key(job.text)
|
||||
cached = _lookup_cached(cache, job.text, job.limit)
|
||||
if cached:
|
||||
result[job.key] = clip_text(cached, job.limit)
|
||||
else:
|
||||
pending.append(job)
|
||||
|
||||
if not pending:
|
||||
return result
|
||||
|
||||
batch_size = max(5, env_int("DAILY_ZH_DESC_BATCH", 20))
|
||||
for i in range(0, len(pending), batch_size):
|
||||
chunk = pending[i : i + batch_size]
|
||||
try:
|
||||
translated = _translate_batch(chunk, brief=False)
|
||||
except Exception as exc:
|
||||
logger.warning("中文摘要批次失败，保留英文：%s", exc)
|
||||
continue
|
||||
for job in chunk:
|
||||
zh = translated.get(job.key)
|
||||
if not zh:
|
||||
continue
|
||||
ck = _cache_key(job.text)
|
||||
cache[ck] = zh
|
||||
result[job.key] = clip_text(zh, job.limit)
|
||||
|
||||
if cache:
|
||||
_save_cache(cache)
|
||||
return result
|
||||
3
daily/news/__init__.py
Normal file
3
daily/news/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from daily.news.fetch import fetch_ai_news, format_news_section
|
||||
|
||||
__all__ = ["fetch_ai_news", "format_news_section"]
|
||||
124
daily/news/feeds.py
Normal file
124
daily/news/feeds.py
Normal file
@@ -0,0 +1,124 @@
|
||||
"""国际 AI 时讯 RSS 源定义（按类别分组）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeed:
|
||||
name: str
|
||||
url: str
|
||||
slow: bool = False # 限速源（如 Reddit）串行抓取
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsCategory:
|
||||
id: str
|
||||
name: str
|
||||
icon: str
|
||||
feeds: tuple[NewsFeed, ...]
|
||||
|
||||
|
||||
NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
|
||||
NewsCategory(
|
||||
id="official",
|
||||
name="厂商官方",
|
||||
icon="🏢",
|
||||
feeds=(
|
||||
NewsFeed("Anthropic Claude 更新", "https://docs.anthropic.com/en/release-notes/feed"),
|
||||
NewsFeed("OpenAI", "https://openai.com/news/rss.xml"),
|
||||
NewsFeed("Google AI", "https://blog.google/technology/ai/rss/"),
|
||||
NewsFeed("DeepMind", "https://deepmind.google/blog/rss.xml"),
|
||||
NewsFeed("Meta Engineering", "https://engineering.fb.com/feed/"),
|
||||
NewsFeed("Microsoft Research", "https://www.microsoft.com/en-us/research/feed/"),
|
||||
NewsFeed("Microsoft Blog", "https://blogs.microsoft.com/feed/"),
|
||||
NewsFeed("Cohere", "https://cohere.com/blog/rss.xml"),
|
||||
NewsFeed("Cursor Changelog", "https://cursor.com/changelog/rss.xml"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="developer",
|
||||
name="Agent / LLM 开发者",
|
||||
icon="🛠",
|
||||
feeds=(
|
||||
NewsFeed("LangChain", "https://blog.langchain.dev/rss/"),
|
||||
NewsFeed("Hugging Face", "https://huggingface.co/blog/feed.xml"),
|
||||
NewsFeed("Vercel Changelog", "https://vercel.com/changelog/rss.xml"),
|
||||
NewsFeed("GitHub Copilot", "https://github.blog/changelog/label/copilot/feed/"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="media",
|
||||
name="综合科技媒体",
|
||||
icon="📰",
|
||||
feeds=(
|
||||
NewsFeed("The Verge AI", "https://www.theverge.com/rss/ai-artificial-intelligence/index.xml"),
|
||||
NewsFeed("TechCrunch AI", "https://techcrunch.com/category/artificial-intelligence/feed/"),
|
||||
NewsFeed("Ars Technica AI", "https://arstechnica.com/ai/feed/"),
|
||||
NewsFeed("Wired AI", "https://www.wired.com/feed/tag/ai/latest/rss"),
|
||||
NewsFeed("MIT Tech Review", "https://www.technologyreview.com/feed/"),
|
||||
NewsFeed("VentureBeat AI", "https://venturebeat.com/category/ai/feed/"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="newsletter",
|
||||
name="Newsletter 日报",
|
||||
icon="✉️",
|
||||
feeds=(
|
||||
NewsFeed("Ben's Bites", "https://bensbites.substack.com/feed"),
|
||||
NewsFeed("The Rundown AI", "https://therundown.substack.com/feed"),
|
||||
NewsFeed("Latent Space", "https://www.latent.space/feed"),
|
||||
NewsFeed("Simon Willison", "https://simonwillison.net/atom/everything/"),
|
||||
NewsFeed("Import AI", "https://importai.substack.com/feed"),
|
||||
NewsFeed("Last Week in AI", "https://lastweekin.ai/feed"),
|
||||
NewsFeed("The Neuron", "https://www.theneuron.ai/feed"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="research",
|
||||
name="研究 / 论文",
|
||||
icon="📚",
|
||||
feeds=(
|
||||
NewsFeed("arXiv cs.CL", "https://arxiv.org/rss/cs.CL"),
|
||||
NewsFeed("arXiv cs.AI", "https://arxiv.org/rss/cs.AI"),
|
||||
NewsFeed("arXiv cs.LG", "https://arxiv.org/rss/cs.LG"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="trending",
|
||||
name="热点 / 趋势",
|
||||
icon="🔥",
|
||||
feeds=(
|
||||
NewsFeed(
|
||||
"Google News · AI",
|
||||
"https://news.google.com/rss/search?q=artificial+intelligence+OR+LLM+OR+Claude+OR+GPT&hl=en-US&gl=US&ceid=US:en",
|
||||
),
|
||||
NewsFeed(
|
||||
"Google News · Technology",
|
||||
"https://news.google.com/rss/headlines/section/topic/TECHNOLOGY?hl=en-US&gl=US&ceid=US:en",
|
||||
),
|
||||
NewsFeed("Techmeme", "https://www.techmeme.com/feed.xml"),
|
||||
NewsFeed("HN · Front Page", "https://hnrss.org/frontpage"),
|
||||
NewsFeed("HN · 100+ Points", "https://hnrss.org/newest?points=100"),
|
||||
NewsFeed("Dev.to · AI", "https://dev.to/feed/tag/ai"),
|
||||
NewsFeed("Lobsters", "https://lobste.rs/rss"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="community",
|
||||
name="社区讨论",
|
||||
icon="💬",
|
||||
feeds=(
|
||||
NewsFeed(
|
||||
"HN · AI/LLM/Agent",
|
||||
"https://hnrss.org/newest?q=AI+OR+LLM+OR+Claude+OR+agent+OR+GPT+OR+Gemini",
|
||||
),
|
||||
NewsFeed(
|
||||
"Reddit · LLM/Claude/ML",
|
||||
"https://old.reddit.com/r/LocalLLaMA+ClaudeAI+MachineLearning+OpenAI/.rss?limit=25",
|
||||
slow=True,
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
462
daily/news/fetch.py
Normal file
462
daily/news/fetch.py
Normal file
@@ -0,0 +1,462 @@
|
||||
"""抓取并整理国际 AI 时讯 RSS。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
import html
|
||||
import xml.etree.ElementTree as ET
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from email.utils import parsedate_to_datetime
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env, env_int, news_summary_limit
|
||||
from daily.news.feeds import NEWS_CATEGORIES, NewsCategory
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)"
|
||||
BROWSER_USER_AGENT = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/131.0.0.0 Safari/537.36"
|
||||
)
|
||||
STRIP_HTML = re.compile(r"<[^>]+>")
|
||||
WS = re.compile(r"\s+")
|
||||
|
||||
ATOM_NS = {"a": "http://www.w3.org/2005/Atom"}
|
||||
RSS_NS = {"r": "http://purl.org/rss/1.0/modules/content/"}
|
||||
|
||||
|
||||
def _enabled() -> bool:
|
||||
raw = (env("DAILY_AI_NEWS") or "1").strip().lower()
|
||||
return raw not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def _hours_window() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_HOURS", 72))
|
||||
|
||||
|
||||
def _per_feed_limit() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_PER_FEED", 3))
|
||||
|
||||
|
||||
def _per_category_limit() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_PER_CATEGORY", 5))
|
||||
|
||||
|
||||
def _wecom_limit() -> int:
|
||||
return max(1, env_int("DAILY_WECOM_AI_NEWS", 10))
|
||||
|
||||
|
||||
def _now_utc() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _parse_datetime(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
text = value.strip()
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
dt = parsedate_to_datetime(text)
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt.astimezone(timezone.utc)
|
||||
except (TypeError, ValueError, OverflowError):
|
||||
pass
|
||||
for fmt in (
|
||||
"%Y-%m-%dT%H:%M:%SZ",
|
||||
"%Y-%m-%dT%H:%M:%S%z",
|
||||
"%Y-%m-%d",
|
||||
):
|
||||
try:
|
||||
dt = datetime.strptime(text[: len(fmt.replace("%z", "+0000"))], fmt.replace("%z", ""))
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt.astimezone(timezone.utc)
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _clean_text(text: str | None, limit: int = 200) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
plain = STRIP_HTML.sub(" ", html.unescape(text))
|
||||
plain = WS.sub(" ", plain).strip()
|
||||
if limit <= 0 or len(plain) <= limit:
|
||||
return plain
|
||||
return plain[: limit - 3] + "..."
|
||||
|
||||
|
||||
def _normalize_link(link: str) -> str:
|
||||
parsed = urlparse(link.strip())
|
||||
query = parse_qs(parsed.query, keep_blank_values=False)
|
||||
for key in list(query.keys()):
|
||||
if key.lower().startswith("utm_") or key.lower() in {"ref", "source"}:
|
||||
query.pop(key, None)
|
||||
clean_query = urlencode({k: v[0] for k, v in query.items() if v}, doseq=False)
|
||||
return urlunparse((parsed.scheme, parsed.netloc, parsed.path.rstrip("/"), "", clean_query, ""))
|
||||
|
||||
|
||||
def _normalize_title(title: str) -> str:
|
||||
return WS.sub(" ", title.strip().lower())
|
||||
|
||||
|
||||
def _entry_datetime(entry: dict[str, Any]) -> datetime | None:
|
||||
for key in ("published", "updated"):
|
||||
dt = _parse_datetime(entry.get(key))
|
||||
if dt:
|
||||
return dt
|
||||
return None
|
||||
|
||||
|
||||
def _parse_atom(content: str, feed_name: str, category: NewsCategory) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
try:
|
||||
root = ET.fromstring(content)
|
||||
except ET.ParseError:
|
||||
return items
|
||||
|
||||
for entry in root.findall("a:entry", ATOM_NS):
|
||||
title_el = entry.find("a:title", ATOM_NS)
|
||||
link_el = entry.find("a:link", ATOM_NS)
|
||||
summary_el = entry.find("a:summary", ATOM_NS) or entry.find("a:content", ATOM_NS)
|
||||
updated_el = entry.find("a:updated", ATOM_NS) or entry.find("a:published", ATOM_NS)
|
||||
title = title_el.text.strip() if title_el is not None and title_el.text else ""
|
||||
link = ""
|
||||
if link_el is not None:
|
||||
link = link_el.get("href") or (link_el.text or "").strip()
|
||||
if not title or not link:
|
||||
continue
|
||||
items.append(
|
||||
{
|
||||
"title": title,
|
||||
"link": link,
|
||||
"summary": _clean_text(summary_el.text if summary_el is not None else ""),
|
||||
"published": updated_el.text.strip() if updated_el is not None and updated_el.text else "",
|
||||
"source_name": feed_name,
|
||||
"category_id": category.id,
|
||||
"category_name": category.name,
|
||||
"category_icon": category.icon,
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _parse_rss(content: str, feed_name: str, category: NewsCategory) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
try:
|
||||
root = ET.fromstring(content)
|
||||
except ET.ParseError:
|
||||
return items
|
||||
|
||||
channel = root.find("channel")
|
||||
if channel is None:
|
||||
return items
|
||||
|
||||
for item in channel.findall("item"):
|
||||
title_el = item.find("title")
|
||||
link_el = item.find("link")
|
||||
desc_el = item.find("description")
|
||||
pub_el = item.find("pubDate")
|
||||
title = title_el.text.strip() if title_el is not None and title_el.text else ""
|
||||
link = link_el.text.strip() if link_el is not None and link_el.text else ""
|
||||
if not title or not link:
|
||||
continue
|
||||
items.append(
|
||||
{
|
||||
"title": title,
|
||||
"link": link,
|
||||
"summary": _clean_text(desc_el.text if desc_el is not None else ""),
|
||||
"published": pub_el.text.strip() if pub_el is not None and pub_el.text else "",
|
||||
"source_name": feed_name,
|
||||
"category_id": category.id,
|
||||
"category_name": category.name,
|
||||
"category_icon": category.icon,
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _parse_feed(content: str, feed_name: str, category: NewsCategory) -> list[dict[str, Any]]:
|
||||
text = content.lstrip("\ufeff").strip()
|
||||
if not text:
|
||||
return []
|
||||
if text.startswith("<rss") or "<channel>" in text[:500]:
|
||||
return _parse_rss(text, feed_name, category)
|
||||
if text.startswith("<feed") or "<entry>" in text[:500]:
|
||||
return _parse_atom(text, feed_name, category)
|
||||
if "<item>" in text[:2000]:
|
||||
return _parse_rss(text, feed_name, category)
|
||||
return _parse_atom(text, feed_name, category)
|
||||
|
||||
|
||||
def _is_reddit_url(url: str) -> bool:
|
||||
host = urlparse(url).netloc.lower()
|
||||
return host.endswith("reddit.com")
|
||||
|
||||
|
||||
def _reddit_auth_params() -> dict[str, str]:
|
||||
user = (env("REDDIT_RSS_USER") or "").strip()
|
||||
feed = (env("REDDIT_RSS_FEED") or "").strip()
|
||||
if user and feed:
|
||||
return {"user": user, "feed": feed}
|
||||
return {}
|
||||
|
||||
|
||||
def _with_query_params(url: str, extra: dict[str, str]) -> str:
|
||||
if not extra:
|
||||
return url
|
||||
parsed = urlparse(url)
|
||||
query = parse_qs(parsed.query, keep_blank_values=True)
|
||||
for key, value in extra.items():
|
||||
if value and key not in query:
|
||||
query[key] = [value]
|
||||
clean_query = urlencode({k: v[0] for k, v in query.items() if v and v[0]}, doseq=False)
|
||||
return urlunparse((parsed.scheme, parsed.netloc, parsed.path, "", clean_query, ""))
|
||||
|
||||
|
||||
def _reddit_old_url(url: str) -> str:
|
||||
parsed = urlparse(url)
|
||||
host = parsed.netloc.lower()
|
||||
if host.startswith("old."):
|
||||
return url
|
||||
if host in {"www.reddit.com", "reddit.com"}:
|
||||
return urlunparse((parsed.scheme, "old.reddit.com", parsed.path, "", parsed.query, ""))
|
||||
return url
|
||||
|
||||
|
||||
def _request_headers(base: dict[str, str], url: str) -> dict[str, str]:
|
||||
if not _is_reddit_url(url):
|
||||
return base
|
||||
return {
|
||||
**base,
|
||||
"User-Agent": BROWSER_USER_AGENT,
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
|
||||
|
||||
def _reddit_fetch_urls(feed_url: str) -> list[str]:
|
||||
primary = _with_query_params(feed_url, _reddit_auth_params())
|
||||
if not _is_reddit_url(primary):
|
||||
return [primary]
|
||||
fallback = _reddit_old_url(primary)
|
||||
if fallback == primary:
|
||||
return [primary]
|
||||
return [primary, fallback]
|
||||
|
||||
|
||||
def _fetch_one(client: httpx.Client, category: NewsCategory, feed_url: str, feed_name: str) -> list[dict[str, Any]]:
|
||||
last_exc: Exception | None = None
|
||||
for url in _reddit_fetch_urls(feed_url):
|
||||
try:
|
||||
headers = _request_headers(dict(client.headers), url)
|
||||
resp = client.get(url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
return _parse_feed(resp.text, feed_name, category)
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
continue
|
||||
logger.warning("RSS fetch failed [%s] %s: %s", feed_name, feed_url, last_exc)
|
||||
return []
|
||||
|
||||
|
||||
def _dedupe_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
seen_links: set[str] = set()
|
||||
seen_titles: set[str] = set()
|
||||
result: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
link_key = _normalize_link(item["link"])
|
||||
title_key = _normalize_title(item["title"])
|
||||
if link_key in seen_links or title_key in seen_titles:
|
||||
continue
|
||||
seen_links.add(link_key)
|
||||
seen_titles.add(title_key)
|
||||
result.append(item)
|
||||
return result
|
||||
|
||||
|
||||
def _within_window(item: dict[str, Any], cutoff: datetime) -> bool:
|
||||
dt = _entry_datetime(item)
|
||||
if dt is None:
|
||||
return True
|
||||
return dt >= cutoff
|
||||
|
||||
|
||||
def _sort_key(item: dict[str, Any]) -> tuple[int, datetime]:
|
||||
dt = _entry_datetime(item)
|
||||
if dt is None:
|
||||
return (1, datetime.min.replace(tzinfo=timezone.utc))
|
||||
return (0, dt)
|
||||
|
||||
|
||||
def fetch_ai_news() -> dict[str, Any]:
|
||||
"""按类别抓取 AI 时讯，返回 {enabled, hours, categories, flat, stats}。"""
|
||||
if not _enabled():
|
||||
return {"enabled": False, "categories": [], "flat": [], "stats": {}}
|
||||
|
||||
hours = _hours_window()
|
||||
per_feed = _per_feed_limit()
|
||||
per_category = _per_category_limit()
|
||||
cutoff = _now_utc() - timedelta(hours=hours)
|
||||
|
||||
headers = {"User-Agent": USER_AGENT, "Accept": "application/rss+xml, application/atom+xml, application/xml, text/xml, */*"}
|
||||
tasks: list[tuple[NewsCategory, str, str, bool]] = []
|
||||
for category in NEWS_CATEGORIES:
|
||||
for feed in category.feeds:
|
||||
tasks.append((category, feed.url, feed.name, feed.slow))
|
||||
|
||||
raw_by_category: dict[str, list[dict[str, Any]]] = {c.id: [] for c in NEWS_CATEGORIES}
|
||||
stats = {"feeds_total": len(tasks), "feeds_ok": 0, "items_raw": 0}
|
||||
|
||||
with httpx.Client(timeout=15.0, verify=certifi.where(), follow_redirects=True, headers=headers) as client:
|
||||
fast_tasks = [t for t in tasks if not t[3]]
|
||||
slow_tasks = [t for t in tasks if t[3]]
|
||||
|
||||
with ThreadPoolExecutor(max_workers=8) as pool:
|
||||
futures = {
|
||||
pool.submit(_fetch_one, client, cat, url, name): (cat.id, name)
|
||||
for cat, url, name, _slow in fast_tasks
|
||||
}
|
||||
for future in as_completed(futures):
|
||||
cat_id, feed_name = futures[future]
|
||||
try:
|
||||
entries = future.result()
|
||||
except Exception as exc:
|
||||
logger.warning("RSS 任务异常 [%s]: %s", feed_name, exc)
|
||||
continue
|
||||
if entries:
|
||||
stats["feeds_ok"] += 1
|
||||
stats["items_raw"] += len(entries)
|
||||
raw_by_category[cat_id].extend(entries[:per_feed])
|
||||
|
||||
for cat, url, name, _slow in slow_tasks:
|
||||
entries = _fetch_one(client, cat, url, name)
|
||||
if entries:
|
||||
stats["feeds_ok"] += 1
|
||||
stats["items_raw"] += len(entries)
|
||||
raw_by_category[cat.id].extend(entries[:per_feed])
|
||||
if _is_reddit_url(url):
|
||||
time.sleep(2.0)
|
||||
else:
|
||||
time.sleep(1.0)
|
||||
|
||||
categories_out: list[dict[str, Any]] = []
|
||||
flat: list[dict[str, Any]] = []
|
||||
|
||||
for category in NEWS_CATEGORIES:
|
||||
items = raw_by_category[category.id]
|
||||
items = [i for i in items if _within_window(i, cutoff)]
|
||||
items.sort(key=_sort_key, reverse=True)
|
||||
items = _dedupe_items(items)[:per_category]
|
||||
for item in items:
|
||||
dt = _entry_datetime(item)
|
||||
item["published_fmt"] = dt.astimezone(timezone(timedelta(hours=8))).strftime("%m-%d %H:%M") if dt else ""
|
||||
if items:
|
||||
categories_out.append(
|
||||
{
|
||||
"id": category.id,
|
||||
"name": category.name,
|
||||
"icon": category.icon,
|
||||
"items": items,
|
||||
}
|
||||
)
|
||||
flat.extend(items)
|
||||
|
||||
flat.sort(key=_sort_key, reverse=True)
|
||||
flat = _dedupe_items(flat)
|
||||
|
||||
return {
|
||||
"enabled": True,
|
||||
"hours": hours,
|
||||
"categories": categories_out,
|
||||
"flat": flat,
|
||||
"stats": stats,
|
||||
}
|
||||
|
||||
|
||||
def format_news_section(news: dict[str, Any], *, section_no: int, wecom_limit: int | None = None) -> list[str]:
|
||||
if not news.get("enabled"):
|
||||
return ["---", "", f"## {section_no}、国际 AI 时讯", "", "*AI 时讯已关闭（`DAILY_AI_NEWS=0`）。*", ""]
|
||||
|
||||
categories = news.get("categories") or []
|
||||
hours = news.get("hours", 72)
|
||||
lines = [
|
||||
"---",
|
||||
"",
|
||||
f"## {section_no}、国际 AI 时讯",
|
||||
"",
|
||||
f"> 近 **{hours}h** · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用",
|
||||
"",
|
||||
]
|
||||
|
||||
if not categories:
|
||||
lines.append("*暂无可用条目（网络/RSS 源异常或时间窗口内无更新）。*")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
if wecom_limit is not None:
|
||||
flat = (news.get("flat") or [])[:wecom_limit]
|
||||
for i, item in enumerate(flat, 1):
|
||||
pub = f" · {item['published_fmt']}" if item.get("published_fmt") else ""
|
||||
lines.append(
|
||||
f"{i}. [{item['title']}]({item['link']}) · `{item['source_name']}`{pub}"
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
for cat in categories:
|
||||
lines.append(f"### {cat['icon']} {cat['name']}")
|
||||
lines.append("")
|
||||
for i, item in enumerate(cat["items"], 1):
|
||||
pub = f" · {item['published_fmt']}" if item.get("published_fmt") else ""
|
||||
lines.append(f"{i}. **[{item['title']}]({item['link']})** · `{item['source_name']}`{pub}")
|
||||
summary = item.get("summary", "")
|
||||
if summary:
|
||||
lines.append(f" - {_clean_text(summary, news_summary_limit())}")
|
||||
lines.append("")
|
||||
|
||||
return lines
|
||||
|
||||
|
||||
def prepare_wecom_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
if not news.get("enabled"):
|
||||
return []
|
||||
limit = _wecom_limit()
|
||||
flat = _dedupe_items(news.get("flat") or [])
|
||||
flat.sort(key=_sort_key, reverse=True)
|
||||
preferred = ("media", "newsletter", "official", "community", "research", "developer")
|
||||
picked: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for cat in preferred:
|
||||
for item in flat:
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
if item.get("category_id") != cat or link in seen:
|
||||
continue
|
||||
picked.append(item)
|
||||
seen.add(link)
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
items: list[dict[str, Any]] = []
|
||||
for item in picked[:limit]:
|
||||
items.append(
|
||||
{
|
||||
"title": item.get("title", "?"),
|
||||
"link": item.get("link", ""),
|
||||
"source_name": item.get("source_name", "?"),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"desc_short": _clean_text(item.get("summary", ""), 36),
|
||||
}
|
||||
)
|
||||
return items
|
||||
183
daily/report_data.py
Normal file
183
daily/report_data.py
Normal file
@@ -0,0 +1,183 @@
|
||||
"""早报结构化数据：抓取结果 → JSON 中间层。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from daily.config import OUTPUT_DIR, env_int
|
||||
from daily.delta import build_movement_baseline, build_movement_context, compare_depth
|
||||
from daily.news.fetch import prepare_wecom_news_items
|
||||
from daily.skills_group import group_skills_by_source
|
||||
|
||||
|
||||
def skill_id(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
|
||||
|
||||
|
||||
def _slim_skill(item: dict[str, Any]) -> dict[str, Any]:
|
||||
payload = {
|
||||
"id": skill_id(item),
|
||||
"title": item.get("title", ""),
|
||||
"source": item.get("source", ""),
|
||||
"installs": item.get("installs", 0),
|
||||
"link": item.get("link", ""),
|
||||
"description": item.get("description", ""),
|
||||
}
|
||||
if item.get("cluster"):
|
||||
payload.update(
|
||||
{
|
||||
"cluster": True,
|
||||
"cluster_count": item.get("cluster_count", 1),
|
||||
"cluster_skills": item.get("cluster_skills", []),
|
||||
"cluster_titles": item.get("cluster_titles", ""),
|
||||
"installs_fmt": item.get("installs_fmt", ""),
|
||||
"installs_min": item.get("installs_min"),
|
||||
"installs_max": item.get("installs_max"),
|
||||
}
|
||||
)
|
||||
elif item.get("installs_fmt"):
|
||||
payload["installs_fmt"] = item.get("installs_fmt")
|
||||
return payload
|
||||
|
||||
|
||||
def _slim_github(item: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"repo": item.get("repo", ""),
|
||||
"url": item.get("url", ""),
|
||||
"language": item.get("language", ""),
|
||||
"stars_today_fmt": item.get("stars_today_fmt", ""),
|
||||
"total_stars_fmt": item.get("total_stars_fmt", ""),
|
||||
"created_at": item.get("created_at", ""),
|
||||
"description": item.get("description", ""),
|
||||
}
|
||||
|
||||
|
||||
def _slim_news_items(ai_news: dict[str, Any], limit: int) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
for item in prepare_wecom_news_items(ai_news):
|
||||
items.append(
|
||||
{
|
||||
"link": item.get("link", ""),
|
||||
"title": item.get("title", ""),
|
||||
"source_name": item.get("source_name", ""),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"summary": item.get("desc_short") or "",
|
||||
}
|
||||
)
|
||||
if len(items) >= limit:
|
||||
break
|
||||
if items:
|
||||
return items
|
||||
for item in (ai_news.get("flat") or [])[:limit]:
|
||||
items.append(
|
||||
{
|
||||
"link": item.get("link", ""),
|
||||
"title": item.get("title", ""),
|
||||
"source_name": item.get("source_name", ""),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"summary": item.get("summary", ""),
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _wecom_skill_pool() -> int:
|
||||
return max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
|
||||
|
||||
|
||||
def build_llm_input(
|
||||
*,
|
||||
date_str: str,
|
||||
updated: str,
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
github_topic: list[dict[str, Any]],
|
||||
topic_name: str,
|
||||
ai_news: dict[str, Any],
|
||||
wecom_limits: dict[str, int],
|
||||
) -> dict[str, Any]:
|
||||
"""供 Cursor 编辑的精简 JSON（不含完整 markdown）。"""
|
||||
news_limit = wecom_limits.get("ai_news", 10)
|
||||
depth = compare_depth()
|
||||
trend_cmp = trending[:depth]
|
||||
hot_cmp = hot[:depth]
|
||||
github_cmp = github_trending[:depth]
|
||||
emerging_cmp = github_emerging[:depth]
|
||||
topic_cmp = github_topic[:depth]
|
||||
|
||||
trending_slice = group_skills_by_source(
|
||||
trending,
|
||||
limit=wecom_limits.get("trending", 10),
|
||||
pool_size=wecom_limits.get("trending_pool", _wecom_skill_pool()),
|
||||
)
|
||||
hot_slice = group_skills_by_source(
|
||||
hot,
|
||||
limit=wecom_limits.get("hot", 10),
|
||||
pool_size=wecom_limits.get("hot_pool", _wecom_skill_pool()),
|
||||
)
|
||||
github_slice = github_trending[: wecom_limits.get("github", 5)]
|
||||
emerging_slice = github_emerging[: wecom_limits.get("emerging", 3)]
|
||||
topic_slice = github_topic[: wecom_limits.get("topic", 3)]
|
||||
|
||||
movement = build_movement_context(
|
||||
date_str=date_str,
|
||||
trending=[_slim_skill(x) for x in trend_cmp],
|
||||
hot=[_slim_skill(x) for x in hot_cmp],
|
||||
github_trending=[_slim_github(x) for x in github_cmp],
|
||||
github_emerging=[_slim_github(x) for x in emerging_cmp],
|
||||
github_topic=[_slim_github(x) for x in topic_cmp],
|
||||
topic_name=topic_name,
|
||||
)
|
||||
movement_baseline = build_movement_baseline(
|
||||
trending=[_slim_skill(x) for x in trend_cmp],
|
||||
hot=[_slim_skill(x) for x in hot_cmp],
|
||||
github_trending=[_slim_github(x) for x in github_cmp],
|
||||
github_emerging=[_slim_github(x) for x in emerging_cmp],
|
||||
github_topic=[_slim_github(x) for x in topic_cmp],
|
||||
depth=depth,
|
||||
)
|
||||
|
||||
return {
|
||||
"date": date_str,
|
||||
"data_updated": updated,
|
||||
"skills_trending": [_slim_skill(x) for x in trending_slice],
|
||||
"skills_hot": [_slim_skill(x) for x in hot_slice],
|
||||
"github_trending": [_slim_github(x) for x in github_slice],
|
||||
"github_emerging": [_slim_github(x) for x in emerging_slice],
|
||||
"github_topic": {
|
||||
"topic": topic_name,
|
||||
"repos": [_slim_github(x) for x in topic_slice],
|
||||
},
|
||||
"ai_news": _slim_news_items(ai_news, news_limit) if ai_news.get("enabled") else [],
|
||||
"movement": movement,
|
||||
"movement_baseline": movement_baseline,
|
||||
}
|
||||
|
||||
|
||||
def build_full_payload(
|
||||
llm_input: dict[str, Any],
|
||||
*,
|
||||
meta: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
return {"meta": meta, "data": llm_input}
|
||||
|
||||
|
||||
def data_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.data.json"
|
||||
|
||||
|
||||
def editorial_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.editorial.json"
|
||||
|
||||
|
||||
def save_json(path: Path, data: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def load_json(path: Path) -> dict[str, Any]:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
120
daily/skills_board.py
Normal file
120
daily/skills_board.py
Normal file
@@ -0,0 +1,120 @@
|
||||
"""从 skills.sh 官网抓取 Trending / Hot 完整榜单（突破 feed.json 50 条限制）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any, Literal
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
Board = Literal["trending", "hot"]
|
||||
SKILLS_SITE = "https://www.skills.sh"
|
||||
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)"
|
||||
|
||||
_SKILL_RE = re.compile(
|
||||
r'\{"source":"(?P<source>[^"]+)","skillId":"(?P<skill_id>[^"]+)",'
|
||||
r'"name":"(?P<name>[^"]+)","installs":(?P<installs>\d+)'
|
||||
)
|
||||
_RSC_CHUNK_RE = re.compile(r"self\.__next_f\.push\(\[1,\"(.*?)\"\]\)", re.DOTALL)
|
||||
|
||||
|
||||
def board_source() -> str:
|
||||
return (env("SKILLS_BOARD_SOURCE") or "website").strip().lower()
|
||||
|
||||
|
||||
def _fetch_html(path: str) -> str:
|
||||
url = f"{SKILLS_SITE}{path}"
|
||||
headers = {"User-Agent": USER_AGENT, "Accept": "text/html"}
|
||||
with httpx.Client(timeout=30.0, verify=certifi.where(), follow_redirects=True) as client:
|
||||
resp = client.get(url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
return resp.text
|
||||
|
||||
|
||||
def _rsc_blob(html: str) -> str:
|
||||
chunks = _RSC_CHUNK_RE.findall(html)
|
||||
blob = "\n".join(chunks)
|
||||
return blob.encode("utf-8").decode("unicode_escape", errors="ignore")
|
||||
|
||||
|
||||
def _parse_initial_skills(blob: str, *, limit: int) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for match in _SKILL_RE.finditer(blob):
|
||||
source = match.group("source")
|
||||
skill_id = match.group("skill_id")
|
||||
uid = f"{source}/{skill_id}"
|
||||
if uid in seen:
|
||||
continue
|
||||
seen.add(uid)
|
||||
items.append(
|
||||
{
|
||||
"id": uid,
|
||||
"title": skill_id,
|
||||
"source": source,
|
||||
"installs": int(match.group("installs")),
|
||||
"link": f"{SKILLS_SITE}/{source}/{skill_id}",
|
||||
"description": "",
|
||||
}
|
||||
)
|
||||
if len(items) >= limit:
|
||||
break
|
||||
return items
|
||||
|
||||
|
||||
def fetch_board(board: Board, *, limit: int) -> list[dict[str, Any]]:
|
||||
path = "/trending" if board == "trending" else "/hot"
|
||||
try:
|
||||
html = _fetch_html(path)
|
||||
items = _parse_initial_skills(_rsc_blob(html), limit=limit)
|
||||
if items:
|
||||
logger.info("skills.sh %s: %d items (limit=%d)", board, len(items), limit)
|
||||
return items
|
||||
except Exception as exc:
|
||||
logger.warning("skills.sh %s fetch failed, fallback to feed.json: %s", board, exc)
|
||||
return []
|
||||
|
||||
|
||||
def enrich_from_feed(items: list[dict[str, Any]], feed: dict[str, Any]) -> None:
|
||||
desc_by_id: dict[str, str] = {}
|
||||
for key in ("topTrending", "topHot", "topAllTime"):
|
||||
for row in feed.get(key, []):
|
||||
uid = str(row.get("id") or f"{row.get('source')}/{row.get('title')}")
|
||||
desc = (row.get("description") or "").strip()
|
||||
if desc:
|
||||
desc_by_id[uid] = desc
|
||||
for item in items:
|
||||
if not item.get("description"):
|
||||
item["description"] = desc_by_id.get(item["id"], "")
|
||||
|
||||
|
||||
def load_boards(
|
||||
feed: dict[str, Any],
|
||||
*,
|
||||
trending_limit: int,
|
||||
hot_limit: int,
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""优先 skills.sh 官网；失败时回退 feed.json。"""
|
||||
if board_source() == "feed":
|
||||
return (
|
||||
list(feed.get("topTrending", [])[:trending_limit]),
|
||||
list(feed.get("topHot", [])[:hot_limit]),
|
||||
)
|
||||
|
||||
trending = fetch_board("trending", limit=trending_limit)
|
||||
hot = fetch_board("hot", limit=hot_limit)
|
||||
if not trending:
|
||||
trending = list(feed.get("topTrending", [])[:trending_limit])
|
||||
else:
|
||||
enrich_from_feed(trending, feed)
|
||||
if not hot:
|
||||
hot = list(feed.get("topHot", [])[:hot_limit])
|
||||
else:
|
||||
enrich_from_feed(hot, feed)
|
||||
return trending, hot
|
||||
103
daily/skills_group.py
Normal file
103
daily/skills_group.py
Normal file
@@ -0,0 +1,103 @@
|
||||
"""Skills 榜单：同 source 合并为一条（企微 Top N）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
def skill_id(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
|
||||
|
||||
|
||||
def format_installs(n: int | float) -> str:
|
||||
if n >= 1_000_000:
|
||||
return f"{n / 1_000_000:.1f}M"
|
||||
if n >= 1_000:
|
||||
return f"{n / 1_000:.1f}K"
|
||||
return str(int(n))
|
||||
|
||||
|
||||
def _installs_range(items: list[dict[str, Any]]) -> tuple[int, int, str]:
|
||||
values = [int(item.get("installs") or 0) for item in items]
|
||||
lo, hi = min(values), max(values)
|
||||
if lo == hi:
|
||||
return lo, hi, format_installs(hi)
|
||||
return lo, hi, f"{format_installs(lo)}–{format_installs(hi)}"
|
||||
|
||||
|
||||
def _best_description(items: list[dict[str, Any]]) -> str:
|
||||
for item in items:
|
||||
desc = (item.get("description") or "").strip()
|
||||
if desc:
|
||||
return desc
|
||||
return ""
|
||||
|
||||
|
||||
def _cluster_skill(items: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
ranked = sorted(items, key=lambda x: int(x.get("installs") or 0), reverse=True)
|
||||
top = ranked[0]
|
||||
_lo, _hi, installs_fmt = _installs_range(ranked)
|
||||
titles = [str(x.get("title") or "") for x in ranked if x.get("title")]
|
||||
sample = ", ".join(titles[:4])
|
||||
if len(titles) > 4:
|
||||
sample = f"{sample}…"
|
||||
return {
|
||||
"id": skill_id(top),
|
||||
"title": top.get("title", ""),
|
||||
"source": top.get("source", ""),
|
||||
"installs": int(top.get("installs") or 0),
|
||||
"installs_min": _lo,
|
||||
"installs_max": _hi,
|
||||
"installs_fmt": installs_fmt,
|
||||
"link": top.get("link", ""),
|
||||
"description": _best_description(ranked),
|
||||
"cluster": True,
|
||||
"cluster_count": len(ranked),
|
||||
"cluster_skills": titles,
|
||||
"cluster_titles": sample,
|
||||
}
|
||||
|
||||
|
||||
def _single_skill(item: dict[str, Any]) -> dict[str, Any]:
|
||||
installs = int(item.get("installs") or 0)
|
||||
return {
|
||||
"id": skill_id(item),
|
||||
"title": item.get("title", ""),
|
||||
"source": item.get("source", ""),
|
||||
"installs": installs,
|
||||
"installs_fmt": format_installs(installs),
|
||||
"link": item.get("link", ""),
|
||||
"description": item.get("description", ""),
|
||||
"cluster": False,
|
||||
}
|
||||
|
||||
|
||||
def group_skills_by_source(
|
||||
items: list[dict[str, Any]],
|
||||
*,
|
||||
limit: int = 10,
|
||||
pool_size: int = 50,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""按 source 去重合并；保留各 source 在榜内的最佳名次顺序。"""
|
||||
if not items or limit <= 0:
|
||||
return []
|
||||
|
||||
pool = items[: max(pool_size, limit)]
|
||||
by_source: dict[str, list[dict[str, Any]]] = {}
|
||||
first_rank: dict[str, int] = {}
|
||||
for rank, item in enumerate(pool, 1):
|
||||
source = (item.get("source") or "?").strip() or "?"
|
||||
by_source.setdefault(source, []).append(item)
|
||||
first_rank.setdefault(source, rank)
|
||||
|
||||
ordered_sources = sorted(by_source.keys(), key=lambda s: first_rank[s])
|
||||
result: list[dict[str, Any]] = []
|
||||
for source in ordered_sources:
|
||||
group = by_source[source]
|
||||
if len(group) == 1:
|
||||
result.append(_single_skill(group[0]))
|
||||
else:
|
||||
result.append(_cluster_skill(group))
|
||||
if len(result) >= limit:
|
||||
break
|
||||
return result
|
||||
30
daily/text_utils.py
Normal file
30
daily/text_utils.py
Normal file
@@ -0,0 +1,30 @@
|
||||
"""文本裁剪等轻量工具。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_WS = re.compile(r"\s+")
|
||||
|
||||
|
||||
def clip_text(text: str, limit: int) -> str:
|
||||
text = _WS.sub(" ", (text or "").strip())
|
||||
if limit <= 0 or len(text) <= limit:
|
||||
return text
|
||||
return text[: max(1, limit - 1)] + "…"
|
||||
|
||||
|
||||
def trim_brief(text: str, limit: int) -> str:
|
||||
"""企微简要：控制在 limit 内，优先在句读处截断，不加省略号。"""
|
||||
text = _WS.sub(" ", (text or "").strip())
|
||||
if not text or limit <= 0 or len(text) <= limit:
|
||||
return text
|
||||
for sep in ("。", "！", "？", "；"):
|
||||
pos = text.find(sep)
|
||||
if pos != -1 and pos + 1 <= limit + 8:
|
||||
return text[: pos + 1]
|
||||
for sep in ("，", "、"):
|
||||
pos = text.find(sep)
|
||||
if pos != -1 and pos + 1 <= limit:
|
||||
return text[: pos + 1]
|
||||
return text[:limit].rstrip("，、；： ")
|
||||
96
daily/webhook.py
Normal file
96
daily/webhook.py
Normal file
@@ -0,0 +1,96 @@
|
||||
"""推送早报至企业微信群 webhook（超长自动分多条）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import OUTPUT_DIR, ROOT, env_int, wecom_chunk_bytes
|
||||
from daily.wecom_split import split_wecom_messages
|
||||
|
||||
_PUSH_GAP_MS = 300
|
||||
|
||||
|
||||
def _load_webhook_key() -> str:
|
||||
from daily.config import env
|
||||
|
||||
key = (env("WECOM_WEBHOOK_KEY") or "").strip()
|
||||
if not key:
|
||||
raise RuntimeError("请设置 WECOM_WEBHOOK_KEY（项目根 .env）")
|
||||
return key
|
||||
|
||||
|
||||
def _resolve_report_path(arg: str | None) -> Path:
|
||||
if arg:
|
||||
path = Path(arg)
|
||||
if not path.is_absolute():
|
||||
path = ROOT / path
|
||||
return path
|
||||
candidates = sorted(
|
||||
OUTPUT_DIR.glob("*.wecom.md"),
|
||||
key=lambda p: p.stat().st_mtime,
|
||||
reverse=True,
|
||||
)
|
||||
if candidates:
|
||||
return candidates[0]
|
||||
legacy = sorted(ROOT.glob("*.wecom.md"), key=lambda p: p.stat().st_mtime, reverse=True)
|
||||
if legacy:
|
||||
return legacy[0]
|
||||
raise RuntimeError("未找到 .wecom.md 报告，请先运行 python -m daily")
|
||||
|
||||
|
||||
def _post_markdown(client: httpx.Client, url: str, content: str) -> None:
|
||||
payload = {"msgtype": "markdown", "markdown": {"content": content}}
|
||||
resp = client.post(url, json=payload)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
if data.get("errcode", 0) != 0:
|
||||
raise RuntimeError(f"推送失败: errcode={data.get('errcode')} errmsg={data.get('errmsg')}")
|
||||
|
||||
|
||||
def send_report(report_path: Path | None = None) -> None:
|
||||
path = _resolve_report_path(str(report_path) if report_path else None)
|
||||
if not path.exists():
|
||||
raise RuntimeError(f"报告文件不存在: {path}")
|
||||
|
||||
content = path.read_text(encoding="utf-8").strip()
|
||||
if not content:
|
||||
raise RuntimeError(f"报告内容为空: {path}")
|
||||
|
||||
chunk_limit = wecom_chunk_bytes()
|
||||
max_parts = env_int("DAILY_WECOM_MAX_PARTS", 5)
|
||||
parts = split_wecom_messages(content, chunk_limit)
|
||||
if len(parts) > max_parts:
|
||||
raise RuntimeError(
|
||||
f"早报需 {len(parts)} 条消息，超过 DAILY_WECOM_MAX_PARTS={max_parts}，请调小各区块条数"
|
||||
)
|
||||
|
||||
key = _load_webhook_key()
|
||||
url = f"https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key={key}"
|
||||
|
||||
with httpx.Client(timeout=20.0, verify=certifi.where()) as client:
|
||||
for i, part in enumerate(parts, 1):
|
||||
if i > 1:
|
||||
time.sleep(_PUSH_GAP_MS / 1000.0)
|
||||
_post_markdown(client, url, part)
|
||||
|
||||
total_bytes = len(content.encode("utf-8"))
|
||||
if len(parts) == 1:
|
||||
print(f"已推送至企业微信: {path.name} ({total_bytes} bytes)")
|
||||
else:
|
||||
sizes = ", ".join(str(len(p.encode("utf-8"))) for p in parts)
|
||||
print(f"已推送至企业微信: {path.name} ({total_bytes} bytes → {len(parts)} 条: {sizes})")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = argv if argv is not None else sys.argv[1:]
|
||||
try:
|
||||
send_report(Path(args[0]) if args else None)
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
94
daily/wecom_split.py
Normal file
94
daily/wecom_split.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""企微 markdown 按字节上限拆分为多条消息（按区块，不截断正文）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_SECTION_START = re.compile(r"^[📰💡🎯🌍📈🔥🐙🌱🤖📦📄]")
|
||||
|
||||
|
||||
def _utf8_len(text: str) -> int:
|
||||
return len(text.encode("utf-8"))
|
||||
|
||||
|
||||
def _split_lines_by_budget(text: str, limit: int) -> list[str]:
|
||||
lines = text.splitlines()
|
||||
chunks: list[str] = []
|
||||
buf: list[str] = []
|
||||
for line in lines:
|
||||
candidate = "\n".join(buf + [line]) if buf else line
|
||||
if _utf8_len(candidate) <= limit:
|
||||
buf.append(line)
|
||||
continue
|
||||
if buf:
|
||||
chunks.append("\n".join(buf))
|
||||
buf = []
|
||||
if _utf8_len(line) <= limit:
|
||||
buf = [line]
|
||||
else:
|
||||
encoded = line.encode("utf-8")
|
||||
start = 0
|
||||
while start < len(encoded):
|
||||
piece = encoded[start : start + limit].decode("utf-8", errors="ignore")
|
||||
chunks.append(piece)
|
||||
start += len(piece.encode("utf-8"))
|
||||
if buf:
|
||||
chunks.append("\n".join(buf))
|
||||
return chunks
|
||||
|
||||
|
||||
def _split_sections(text: str) -> list[str]:
|
||||
sections: list[str] = []
|
||||
current: list[str] = []
|
||||
for line in text.splitlines():
|
||||
if _SECTION_START.match(line) and current:
|
||||
sections.append("\n".join(current))
|
||||
current = [line]
|
||||
else:
|
||||
current.append(line)
|
||||
if current:
|
||||
sections.append("\n".join(current))
|
||||
return sections
|
||||
|
||||
|
||||
def split_wecom_messages(text: str, limit: int = 4096) -> list[str]:
|
||||
"""超长时拆成多条；每条不超过 limit 字节，按区块边界优先。"""
|
||||
text = text.strip()
|
||||
if not text or _utf8_len(text) <= limit:
|
||||
return [text] if text else []
|
||||
|
||||
footer_reserve = 40
|
||||
pack_limit = max(512, limit - footer_reserve)
|
||||
|
||||
sections: list[str] = []
|
||||
for sec in _split_sections(text):
|
||||
if _utf8_len(sec) <= pack_limit:
|
||||
sections.append(sec)
|
||||
else:
|
||||
sections.extend(_split_lines_by_budget(sec, pack_limit))
|
||||
|
||||
packed: list[str] = []
|
||||
buf: list[str] = []
|
||||
for sec in sections:
|
||||
candidate = "\n\n".join(buf + [sec]) if buf else sec
|
||||
if _utf8_len(candidate) <= pack_limit:
|
||||
buf.append(sec)
|
||||
else:
|
||||
if buf:
|
||||
packed.append("\n\n".join(buf))
|
||||
buf = [sec]
|
||||
if buf:
|
||||
packed.append("\n\n".join(buf))
|
||||
|
||||
total = len(packed)
|
||||
if total <= 1:
|
||||
return packed
|
||||
|
||||
result: list[str] = []
|
||||
for i, chunk in enumerate(packed, 1):
|
||||
suffix = f"\n\n> 📄 {i}/{total}"
|
||||
body = chunk
|
||||
while body and _utf8_len(body + suffix) > limit:
|
||||
body = body.rsplit("\n", 1)[0] if "\n" in body else body[:-1]
|
||||
result.append(body + suffix)
|
||||
return result
|
||||
72
register-daily-task.ps1
Normal file
72
register-daily-task.ps1
Normal file
@@ -0,0 +1,72 @@
|
||||
# Register a Windows scheduled task to run run-daily.ps1
|
||||
# Usage:
|
||||
# .\register-daily-task.ps1
|
||||
# .\register-daily-task.ps1 -Time "08:50"
|
||||
# .\register-daily-task.ps1 -Unregister
|
||||
|
||||
param(
|
||||
[string]$Time = "08:50",
|
||||
[string]$TaskName = "DailyRobots",
|
||||
[switch]$Unregister
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
$RunScript = Join-Path $Root "run-daily.ps1"
|
||||
$LogDir = Join-Path $Root "logs"
|
||||
$LogFile = Join-Path $LogDir "scheduled-run.log"
|
||||
|
||||
if ($Unregister) {
|
||||
Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false -ErrorAction SilentlyContinue
|
||||
Write-Host "Removed scheduled task: $TaskName"
|
||||
exit 0
|
||||
}
|
||||
|
||||
if (-not (Test-Path $RunScript)) {
|
||||
throw "Not found: $RunScript"
|
||||
}
|
||||
|
||||
if (-not (Test-Path $LogDir)) {
|
||||
New-Item -ItemType Directory -Path $LogDir | Out-Null
|
||||
}
|
||||
|
||||
# Append stdout/stderr to logs/scheduled-run.log for troubleshooting
|
||||
$Argument = @(
|
||||
"-NoProfile",
|
||||
"-ExecutionPolicy", "Bypass",
|
||||
"-Command",
|
||||
"& { Set-Location '$Root'; & '$RunScript' *>&1 | Tee-Object -FilePath '$LogFile' -Append; exit `$LASTEXITCODE }"
|
||||
) -join " "
|
||||
|
||||
$Action = New-ScheduledTaskAction `
|
||||
-Execute "powershell.exe" `
|
||||
-Argument $Argument `
|
||||
-WorkingDirectory $Root
|
||||
|
||||
$Trigger = New-ScheduledTaskTrigger -Daily -At $Time
|
||||
|
||||
$Settings = New-ScheduledTaskSettingsSet `
|
||||
-AllowStartIfOnBatteries `
|
||||
-DontStopIfGoingOnBatteries `
|
||||
-StartWhenAvailable `
|
||||
-ExecutionTimeLimit (New-TimeSpan -Hours 2)
|
||||
|
||||
$Principal = New-ScheduledTaskPrincipal -UserId $env:USERNAME -LogonType Interactive -RunLevel Limited
|
||||
|
||||
Register-ScheduledTask `
|
||||
-TaskName $TaskName `
|
||||
-Action $Action `
|
||||
-Trigger $Trigger `
|
||||
-Settings $Settings `
|
||||
-Principal $Principal `
|
||||
-Description "Generate and push daily report (run-daily.ps1)" `
|
||||
-Force | Out-Null
|
||||
|
||||
Write-Host "Scheduled task registered:"
|
||||
Write-Host " Name: $TaskName"
|
||||
Write-Host " Time: daily at $Time"
|
||||
Write-Host " Script: $RunScript"
|
||||
Write-Host " Log: $LogFile"
|
||||
Write-Host ""
|
||||
Write-Host "Test now: Start-ScheduledTask -TaskName '$TaskName'"
|
||||
Write-Host "Remove: .\register-daily-task.ps1 -Unregister"
|
||||
5
requirements.txt
Normal file
5
requirements.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
python-dotenv>=1.0.0
|
||||
httpx>=0.27.0
|
||||
certifi>=2024.0.0
|
||||
# 英文描述转中文（使用 CURSOR_API_KEY 时需安装）
|
||||
cursor-sdk>=0.1.0
|
||||
58
run-daily.ps1
Normal file
58
run-daily.ps1
Normal file
@@ -0,0 +1,58 @@
|
||||
# Daily report: generate + push to WeCom webhook
|
||||
# Usage:
|
||||
# .\run-daily.ps1
|
||||
# .\run-daily.ps1 -SkipPush
|
||||
# .\run-daily.ps1 -SkipGenerate
|
||||
|
||||
param(
|
||||
[switch]$SkipPush,
|
||||
[switch]$SkipGenerate
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
Set-Location $Root
|
||||
|
||||
function Import-DotEnvFile {
|
||||
param([string]$Path)
|
||||
if (-not (Test-Path $Path)) { return }
|
||||
Get-Content $Path -Encoding UTF8 | ForEach-Object {
|
||||
if ($_ -match '^\s*#' -or $_ -notmatch '=') { return }
|
||||
$pair = $_ -split '=', 2
|
||||
if ($pair.Count -eq 2) {
|
||||
$name = $pair[0].Trim()
|
||||
$value = $pair[1].Trim().Trim('"').Trim("'")
|
||||
if ($name -and $value) {
|
||||
Set-Item -Path "Env:$name" -Value $value
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Import-DotEnvFile (Join-Path $Root ".env")
|
||||
Import-DotEnvFile (Join-Path $Root ".env.local")
|
||||
|
||||
$python = "python"
|
||||
$date = Get-Date -Format "yyyy-MM-dd"
|
||||
$reportWecom = Join-Path $Root "output\$date.wecom.md"
|
||||
|
||||
if (-not $SkipGenerate) {
|
||||
Write-Host "Generating daily report: $date"
|
||||
& $python -m daily generate
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "daily generate failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $SkipPush) {
|
||||
if (-not (Test-Path $reportWecom)) {
|
||||
throw "Report not found: $reportWecom"
|
||||
}
|
||||
Write-Host "Pushing to WeCom webhook..."
|
||||
& $python -m daily push $reportWecom
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "daily push failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "Done: $date"
|
||||
17
send-wecom.ps1
Normal file
17
send-wecom.ps1
Normal file
@@ -0,0 +1,17 @@
|
||||
# Push daily report to WeCom group webhook
|
||||
param(
|
||||
[string]$ReportPath
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
Set-Location $Root
|
||||
|
||||
$python = "python"
|
||||
$args = @("-m", "daily", "push")
|
||||
if ($ReportPath) { $args += $ReportPath }
|
||||
|
||||
& $python @args
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "daily push failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
23
skills-lock.json
Normal file
23
skills-lock.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"version": 1,
|
||||
"skills": {
|
||||
"brand-voice": {
|
||||
"source": "affaan-m/everything-claude-code",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/brand-voice/SKILL.md",
|
||||
"computedHash": "f07173b4a886e800df150f9824535c0bfceed289f00a27ad4a3b1df5226dfa4d"
|
||||
},
|
||||
"python-patterns": {
|
||||
"source": "affaan-m/everything-claude-code",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/python-patterns/SKILL.md",
|
||||
"computedHash": "f06cd8c1f350b4efd3bb0afdf56420a245952d5c7b7f42303ad5c5844ef90db8"
|
||||
},
|
||||
"python-testing": {
|
||||
"source": "affaan-m/everything-claude-code",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/python-testing/SKILL.md",
|
||||
"computedHash": "bd9f35bf0518963313be72b6581c465d28e04aea67725cbe9d157589a61ae0bb"
|
||||
}
|
||||
}
|
||||
}
|
||||
180
skills/daily-agent/SKILL.md
Normal file
180
skills/daily-agent/SKILL.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# 早报 Agent 工作流
|
||||
|
||||
你是 **Skills / GitHub / AI 时讯早报** 的主编 Agent。Python 已完成数据抓取；你分 **两步** 产出可读性强的企微早报。
|
||||
|
||||
## 工作流
|
||||
|
||||
```
|
||||
Step 1 读话题 → 识别热门趋势（输出 trends JSON）
|
||||
Step 2 基于趋势 + 原始数据 → 写企微 Markdown 早报
|
||||
```
|
||||
|
||||
下游 Python 负责分条推送，你 **不** 推送、 **不** 改数字/URL/榜单顺序。
|
||||
|
||||
---
|
||||
|
||||
## Step 1：趋势分析
|
||||
|
||||
阅读输入 JSON，识别今日 2–4 个**交叉主题**（Skills 集群、GitHub 方向、AI 时讯的交集）。
|
||||
|
||||
输入含 **各榜 Top N**（企微默认每榜 10 条，见 `skills_trending` / `skills_hot` / `github_trending` / `github_emerging` / `github_topic.repos`）及可选 **`movement`**（较昨日 Top15 新增，仅供导语与 signals 引用）：
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `skills_trending` | Skills **Trending** 当前 Top N |
|
||||
| `skills_hot` | Skills **Hot** 当前 Top N |
|
||||
| `github_trending` | GitHub **Trending** 当前 Top N |
|
||||
| `github_emerging` | GitHub **新兴** 当前 Top N |
|
||||
| `github_topic.repos` | GitHub **Topic** 当前 Top N |
|
||||
| `github_topic.topic` | Topic 名称，用于区块标题（如 `llm`） |
|
||||
| `movement.*_moves` | 较昨日新增（**仅**用于 opening / signals，**不**用于列表区块） |
|
||||
| `movement.*_summary` | 新增摘要（可选写入 signals） |
|
||||
|
||||
**禁止**使用已合并的 `skills_moves` / `github_moves` 自行扩写；**禁止**排名变化、安装涨跌。
|
||||
|
||||
Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目、movement 新增与 AI 时讯**。
|
||||
|
||||
**只输出 JSON：**
|
||||
|
||||
```json
|
||||
{
|
||||
"headline": "8–16字焦点标题",
|
||||
"opening": "2–3句中文导语；首句必须是具体证据（榜首 skill+安装量 / 头条新闻 / GitHub #1），再解释为什么值得看",
|
||||
"themes": [
|
||||
{
|
||||
"title": "主题名",
|
||||
"summary": "1–2句，说明证据来自哪些榜单/新闻",
|
||||
"keywords": ["remotion", "video"]
|
||||
}
|
||||
],
|
||||
"top_picks": {
|
||||
"skill": { "id": "owner/repo/skill", "title": "...", "why": "中文，为什么今天首推" },
|
||||
"github": { "repo": "owner/repo", "why": "中文" },
|
||||
"news": { "link": "完整URL", "title_zh": "中文标题", "why": "中文一句话" }
|
||||
},
|
||||
"signals": [
|
||||
"📈 Skills：find-skills 1H 安装暴涨，元能力需求上升",
|
||||
"🐙 GitHub：openclaw 持续霸榜，本地 Agent 助手热度不减"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
要求：
|
||||
|
||||
- 所有结论必须能在输入 JSON 中找到依据，禁止编造
|
||||
- `opening` 遵循 **article-writing Newsletter** 规则：首句用数字/条目名/新闻标题开头，不用「今天有三条线」「值得关注」等空框架
|
||||
- `signals` 3–5 条，每条单行，可含 emoji 前缀；与 `opening` 不重复同一句信息
|
||||
- `top_picks.why` 用「事实/数字 + 一句判断」，不用空泛形容词
|
||||
- `top_picks` 必须引用输入中真实存在的 id/repo/link
|
||||
|
||||
---
|
||||
|
||||
## Step 2：撰写企微早报
|
||||
|
||||
你会收到 **原始数据 JSON** + **Step 1 的 trends JSON**。
|
||||
|
||||
**只输出企微 Markdown 正文**（不要代码块包裹，不要解释）。
|
||||
|
||||
### 版式（友好、叙事，避免机械长列表）
|
||||
|
||||
```markdown
|
||||
📰 **早报 · {date}**
|
||||
> ⏱ {time} · 数据截至 {data_updated}
|
||||
|
||||
{opening — 用 2–3 行引用块或普通段落，中文}
|
||||
|
||||
🎯 **{headline}**
|
||||
|
||||
💡 **今日信号**
|
||||
> {signal 1}
|
||||
> {signal 2}
|
||||
> {signal 3}
|
||||
|
||||
📦 **今日首推**
|
||||
`npx skills add {source}/{skill}`
|
||||
> {top_picks.skill.why}
|
||||
|
||||
🌍 **国际 AI · 精选 10**
|
||||
1. [{title_zh}]({link}) — {why 或摘要}
|
||||
2. ...（**必须 10 条**，来自 `ai_news`，按重要性排序）
|
||||
|
||||
<!-- **不要写** Skills Trending / Skills Hot 区块，Python 会在推送前按 source 合并后自动插入 -->
|
||||
|
||||
🐙 **GitHub Trending Top {N}**
|
||||
<!-- 只列 data.github_trending -->
|
||||
1. [{repo}]({url}) · {lang} · ⭐{stars} — {中文一句话}
|
||||
|
||||
🌱 **GitHub 新兴 Top {N}**
|
||||
<!-- 只列 data.github_emerging -->
|
||||
1. ...
|
||||
|
||||
🤖 **Topic `{topic}` Top {N}**
|
||||
<!-- 只列 data.github_topic.repos；topic 来自 data.github_topic.topic -->
|
||||
1. [{repo}]({url}) · {lang} · ⭐{stars} — {中文一句话}
|
||||
```
|
||||
|
||||
### 榜单选取规则（top_n）
|
||||
|
||||
1. **五个 GitHub 区块分开写**：GitHub Trending / 新兴 / Topic，**禁止合并**
|
||||
2. **Skills Trending / Hot 由 Python 自动插入**，Agent 不要写这两段
|
||||
3. **禁止**改用 `movement.*_moves` 作为列表来源；movement 仅用于 opening / signals 描述「今日新增」
|
||||
4. **禁止**在条目后写 `（新入 … #n）` 类括号标注
|
||||
5. **即使某榜较昨日无新增，仍须完整列出 Top 榜条目**
|
||||
6. **国际 AI 必须 10 条**
|
||||
7. 禁止排名变化、安装涨跌、连霸描述
|
||||
|
||||
```markdown
|
||||
📈 **Skills Trending Top 10**
|
||||
1. **`halt-catch-fire/skills` · 5 skills · **21.4K–21.4K** — remotion-render, ai-video-generation… 程序化视频/图像/社媒自动化
|
||||
|
||||
🔥 **Skills Hot Top 10**
|
||||
1. **`larksuite/cli` · 8 skills · **141–144** — lark-wiki, lark-doc… 飞书办公 CLI 能力集群
|
||||
```
|
||||
|
||||
### 写作原则
|
||||
|
||||
1. **全中文叙述**：新闻用 `title_zh`；Skill/GitHub 名保留英文，说明用中文
|
||||
2. **Skills/GitHub 分榜 Top**：每榜最多 10 条（Python 已截断），不自行追加或删减
|
||||
3. **链接必留**：所有 `[文字](url)` 来自输入，URL 不改
|
||||
4. **数字必真**：installs、star 与输入一致
|
||||
5. **语气**：简洁、有判断，像技术媒体晨报，避免「据悉」
|
||||
6. **篇幅**：整篇尽量 ≤3500 字节（UTF-8），便于单条或双条推送
|
||||
|
||||
### Newsletter 写作规范（article-writing）
|
||||
|
||||
受众：国内 Agent / 全栈开发者；企微首屏必须「有料」，像写给同事的晨报，不是 AI 摘要。
|
||||
|
||||
**必做：**
|
||||
|
||||
1. `opening` 第一句用具体证据开头（榜首 skill + 安装量 / 头条新闻 / GitHub #1 repo），先例子后解释
|
||||
2. 每条 `why` 用数字或事实支撑判断，不用「game-changer」「cutting-edge」等形容词
|
||||
3. `signals` 与 `opening` 分工：`opening` 定调，`signals` 只补充增量信息
|
||||
4. 段落短、句子紧；删掉不推进信息的过渡句
|
||||
|
||||
**禁止（Banned Patterns）：**
|
||||
|
||||
- 「据悉」「值得关注」「快速演进」「In today's rapidly evolving landscape」
|
||||
- 「今天有三条线叠在一起」这类无证据的空框架开场
|
||||
- 无证据的「为什么这很重要」「 here's why this matters」
|
||||
- 结尾硬塞互动问句（如「你怎么看？」「值得花十分钟扫一眼」）
|
||||
- `opening` 与 `signals` 重复同一件事
|
||||
|
||||
**输出前自检：**
|
||||
|
||||
- [ ] 首屏 3 行内出现至少 1 个具体数字或条目名
|
||||
- [ ] 无编造事实（只来自 data / trends JSON）
|
||||
- [ ] 无 AI 套话与空泛过渡
|
||||
- [ ] 每条 why 有证据，不是纯形容词
|
||||
|
||||
**Voice 范例（模仿语气与节奏，不抄具体内容）：**
|
||||
|
||||
> remotion-render 21307 安装拿下 Trending 第一——Skills 榜今天被「用代码出片」占满。Hot 榜另一边是 find-skills：大家不是在装某个 skill，是在找「有没有能做 X 的 skill」。
|
||||
|
||||
> openclaw 381K star 仍居 GitHub Trending 榜首；新闻侧 Anthropic Fable 限制解除，模型政策与本地 Agent 基建同屏升温。
|
||||
|
||||
### 禁止
|
||||
|
||||
- 不要输出 JSON
|
||||
- 不要表格
|
||||
- 不要用 movement 新增列表替代 Top 榜列表
|
||||
- 不要编造未在输入或 trends 中出现的事实
|
||||
151
skills/daily-editor/SKILL.md
Normal file
151
skills/daily-editor/SKILL.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# 早报 Cursor 编辑规范
|
||||
|
||||
你是 **Skills / GitHub / AI 时讯早报** 的编辑。Python 已完成数据抓取与排序；你只负责把输入 JSON 改写成 **企微早报可用的中文编辑稿**，并输出 **严格 JSON**。
|
||||
|
||||
## 场景
|
||||
|
||||
- 读者：国内 Agent / 全栈开发者
|
||||
- 渠道：企业微信 Markdown（不支持复杂表格）
|
||||
- 下游：`daily/cursor_editor.py` 解析你的 JSON，填入 `format_wecom.py` 模板后推送
|
||||
- 你 **不** 抓取数据、 **不** 改榜单顺序、 **不** 输出 Markdown 正文
|
||||
|
||||
## 输入
|
||||
|
||||
你会收到一个 JSON，结构如下（字段可能为空数组）：
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `date` | 早报日期 |
|
||||
| `data_updated` | skills feed 更新日期 |
|
||||
| `skills_trending` / `skills_hot` | 各含 `id`, `title`, `source`, `installs`, `link`, `description` |
|
||||
| `github_trending` / `github_emerging` | 各含 `repo`, `url`, `language`, `stars_today_fmt`, `total_stars_fmt`, `description` |
|
||||
| `github_topic.repos` | 同上 |
|
||||
| `ai_news` | 各含 `link`, `title`, `source_name`, `published_fmt`, `summary` |
|
||||
|
||||
**铁律（违反即失败）：**
|
||||
|
||||
- 不得修改、编造：`installs`、star 数、仓库名、skill 名、URL、语言
|
||||
- 不得调整榜单顺序或增删条目
|
||||
- 新闻 **`title` 保持英文原文**（只在 highlights 里引用）；仅 `summary` 译成中文
|
||||
|
||||
## 输出
|
||||
|
||||
**只输出一个 JSON 对象**，不要 markdown 代码块，不要任何解释文字。
|
||||
|
||||
```json
|
||||
{
|
||||
"theme_line": "AI 视频与监管",
|
||||
"highlights": [
|
||||
"🌍 AI 时讯 [English Title](url)（`Source`）",
|
||||
"📈 Skills 榜首 **skill-name**（22.3K）",
|
||||
"🐙 GitHub Trending [owner/repo](url)（⭐380.5K · TypeScript）"
|
||||
],
|
||||
"descriptions": {
|
||||
"skill:owner/repo/skill-name": "中文简介",
|
||||
"github:owner/repo": "中文简介",
|
||||
"news:https://完整链接": "中文摘要"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### theme_line
|
||||
|
||||
- **8–16 个汉字**（含标点），概括今日最强信号
|
||||
- 只写主题短语，**不要**加「今日主题：」前缀（模板会自动加）
|
||||
- 优先综合：AI 时讯热点 + Skills 集群 + GitHub 趋势
|
||||
- ✅ `AI 视频与监管` · `Agent Skills 爆发` · `开源 OCR 与本地 LLM`
|
||||
- ❌ `今日值得关注的有...` · `据悉 AI 行业...`
|
||||
|
||||
### highlights（恰好 3 条）
|
||||
|
||||
企微 Markdown，每条 **单行**，格式固定：
|
||||
|
||||
| 优先级 | 前缀 | 内容来源 | 格式 |
|
||||
|--------|------|----------|------|
|
||||
| 1 | 🌍 | `ai_news[0]`（若有） | `🌍 AI 时讯 [title](link)（\`source_name\`）` |
|
||||
| 2 | 📈 | `skills_trending[0]` | `📈 Skills 榜首 **{title}**（{installs_fmt}）` |
|
||||
| 3 | 🐙 / 🌱 / 🔥 | GitHub 或 Hot | 见下 |
|
||||
|
||||
第 3 条选择规则（取第一个有数据的）：
|
||||
|
||||
1. `github_trending[0]` → `🐙 GitHub Trending [repo](url)（⭐{total_stars_fmt} · {language}）`
|
||||
2. 否则 `github_emerging[0]` → `🌱 新兴 [repo](url)（⭐ {total_stars_fmt}）`
|
||||
3. 否则 `skills_hot[0]` → `🔥 Skills Hot 榜首 **{title}**（1H {installs_fmt}）`
|
||||
|
||||
**installs / star 显示：**
|
||||
|
||||
- 直接使用输入中的数字；若需缩写：≥1000 用 `22.3K`，≥1000000 用 `1.2M`（与输入一致即可，不要自行换算错）
|
||||
|
||||
### descriptions
|
||||
|
||||
**覆盖规则：** 输入中每一条 **非空** `description` 或 `summary`，都必须在 `descriptions` 里有对应 key。
|
||||
|
||||
| 类型 | key 格式 | 字数上限（字符） | 要求 |
|
||||
|------|----------|------------------|------|
|
||||
| skill | `skill:{id}` | 36 | 说清「做什么 + 用什么」 |
|
||||
| github | `github:{repo}` | 40 | 说清项目定位 + 技术栈/场景 |
|
||||
| news | `news:{link}` 完整 URL | 36 | 一句话新闻摘要，中文 |
|
||||
|
||||
写法：
|
||||
|
||||
- 已是中文 → 精简，去掉冗余
|
||||
- 英文 → 意译，**不**逐字翻译，**不**保留大段英文
|
||||
- 用动词开头：`集成…` `支持…` `用于…`
|
||||
- 禁止：`据悉` `据了解` `据报道` 等空话
|
||||
- 禁止编造输入中不存在的产品/数字/结论
|
||||
|
||||
## 编辑原则
|
||||
|
||||
1. **事实优先**：所有判断必须能在输入 JSON 中找到依据
|
||||
2. **安装导向**：Skills 描述突出「能帮你做什么」
|
||||
3. **差异化**：GitHub 新兴项目强调「新在哪」；Topic 项目强调与 LLM/Agent 的关系
|
||||
4. **时讯克制**：摘要 1 句说清「谁 + 做了什么 + 影响」
|
||||
|
||||
## 输出前自检（必须全部满足）
|
||||
|
||||
- [ ] 仅有 JSON，无 markdown 围栏、无前后说明
|
||||
- [ ] `highlights` 恰好 3 条，各有正确 emoji 前缀
|
||||
- [ ] `theme_line` 8–16 字，无「今日主题」前缀
|
||||
- [ ] 每条有 description/summary 的输入项，都在 `descriptions` 中有 key
|
||||
- [ ] 所有 key 与输入 `id` / `repo` / `link` **完全一致**（含大小写、协议、路径）
|
||||
- [ ] 未修改任何 URL、数字、仓库名、skill 名
|
||||
- [ ] 新闻 title 未翻译成中文
|
||||
|
||||
## 示例
|
||||
|
||||
**输入片段：**
|
||||
|
||||
```json
|
||||
{
|
||||
"skills_trending": [{
|
||||
"id": "halt-catch-fire/skills/remotion-render",
|
||||
"title": "remotion-render",
|
||||
"installs": 22300,
|
||||
"link": "https://skills.sh/...",
|
||||
"description": "Render videos from React/Remotion TSX code..."
|
||||
}],
|
||||
"ai_news": [{
|
||||
"link": "https://example.com/news",
|
||||
"title": "OpenAI delays GPT-5.6",
|
||||
"source_name": "The Verge AI",
|
||||
"summary": "The administration asked OpenAI to delay..."
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
**输出片段：**
|
||||
|
||||
```json
|
||||
{
|
||||
"theme_line": "AI 视频与监管",
|
||||
"highlights": [
|
||||
"🌍 AI 时讯 [OpenAI delays GPT-5.6](https://example.com/news)（`The Verge AI`）",
|
||||
"📈 Skills 榜首 **remotion-render**（22.3K）",
|
||||
"🐙 GitHub Trending [openclaw/openclaw](https://github.com/openclaw/openclaw)（⭐380.5K · TypeScript）"
|
||||
],
|
||||
"descriptions": {
|
||||
"skill:halt-catch-fire/skills/remotion-render": "用 React/Remotion 代码渲染 MP4 视频，支持动画与导出配置",
|
||||
"news:https://example.com/news": "美方要求 OpenAI 推迟 GPT-5.6 发布，出于安全考量"
|
||||
}
|
||||
}
|
||||
```
|
||||
14
tmp_check_desc.py
Normal file
14
tmp_check_desc.py
Normal file
@@ -0,0 +1,14 @@
|
||||
import re
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
url = "https://skills.sh/vercel-labs/skills/find-skills"
|
||||
r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20, verify=certifi.where())
|
||||
chunks = re.findall(r'self\.__next_f\.push\(\[1,"(.*?)"\]\)', r.text, re.DOTALL)
|
||||
blob = "\n".join(chunks).encode("utf-8").decode("unicode_escape", errors="ignore")
|
||||
for needle in ("description", "Helps users", "SKILL.md", "summary"):
|
||||
print(needle, blob.count(needle))
|
||||
idx = blob.find("Helps users")
|
||||
if idx >= 0:
|
||||
print(blob[idx : idx + 300])
|
||||
BIN
tmp_desc_snip.txt
Normal file
BIN
tmp_desc_snip.txt
Normal file
Binary file not shown.
Reference in New Issue
Block a user