Compare commits
29 Commits
main
...
dev_26_07_
| Author | SHA1 | Date | |
|---|---|---|---|
| 85614f5921 | |||
| e2f609a027 | |||
| 0384e2c7a9 | |||
| f166d1e504 | |||
| 36085f107d | |||
| d66f2c716c | |||
| d9aef4b340 | |||
| d7992e7a7a | |||
| 3321a307a0 | |||
| 33bfed0e79 | |||
| ea8de9fe61 | |||
| cd92644be4 | |||
| 92e0ed9a27 | |||
| 6ea2a4e4c6 | |||
| 6192dd4e2a | |||
| 4a128b0fa6 | |||
| dcd0608b5d | |||
| 54f164cdbf | |||
| d448002e7a | |||
| 0c324f9ace | |||
| f563239e0b | |||
| 02e97e057f | |||
| 3ad2e7c090 | |||
| 588def8eb2 | |||
| b2dd8721d0 | |||
| e3b5623860 | |||
| 2f1fac9308 | |||
| ba8631c867 | |||
| aff75141cb |
61
.agents/skills/cavecrew/README.md
Normal file
61
.agents/skills/cavecrew/README.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# cavecrew
|
||||
|
||||
Decision guide. When to delegate to caveman subagents instead of doing the work inline.
|
||||
|
||||
## What it does
|
||||
|
||||
Tells the main thread when to spawn a caveman-style subagent versus the vanilla equivalent. The win: subagent tool-results inject back into main context verbatim, and caveman output is roughly 1/3 the size of vanilla prose. Across 20 delegations in one session, that is the difference between context exhaustion and finishing the task.
|
||||
|
||||
Three subagents:
|
||||
|
||||
| Subagent | Job | Use when |
|
||||
|----------|-----|----------|
|
||||
| `cavecrew-investigator` | Locate code (read-only) | "Where is X defined / what calls Y / list uses of Z" |
|
||||
| `cavecrew-builder` | Surgical edit, 1-2 files | Scope is obvious, ≤2 files. Refuses 3+ file scope. |
|
||||
| `cavecrew-reviewer` | Diff/file review | One-line findings with severity emoji |
|
||||
|
||||
Use vanilla `Explore` or `Code Reviewer` when you want prose, architecture commentary, or rationale. Use main thread directly for one-line answers and 3+ file refactors.
|
||||
|
||||
This skill is a decision guide, not a slash command. It activates when the conversation mentions delegation.
|
||||
|
||||
## How to invoke
|
||||
|
||||
Triggers on phrases like "delegate to subagent", "use cavecrew", "spawn investigator", "save context", "compressed agent output".
|
||||
|
||||
## Example chaining
|
||||
|
||||
Locate → fix → verify (most common):
|
||||
|
||||
1. `cavecrew-investigator` returns site list (`path:line — symbol — note`)
|
||||
2. Main thread picks 1-2 sites, hands paths to `cavecrew-builder`
|
||||
3. `cavecrew-reviewer` audits the resulting diff
|
||||
|
||||
Parallel scout: spawn 2-3 `cavecrew-investigator` calls in one message with different angles (defs, callers, tests). Aggregate in main.
|
||||
|
||||
## Model overrides
|
||||
|
||||
By default, `cavecrew-reviewer` and `cavecrew-investigator` pin `model: haiku` in their frontmatter; `cavecrew-builder` has no `model:` line (uses the API session default). Set env vars in your shell before launching Claude Code to override per-agent:
|
||||
|
||||
| Env var | Agent |
|
||||
|---|---|
|
||||
| `CAVECREW_REVIEWER_MODEL` | `cavecrew-reviewer` |
|
||||
| `CAVECREW_BUILDER_MODEL` | `cavecrew-builder` |
|
||||
| `CAVECREW_INVESTIGATOR_MODEL` | `cavecrew-investigator` |
|
||||
|
||||
Example — run reviewer on sonnet, keep others on default:
|
||||
|
||||
```sh
|
||||
export CAVECREW_REVIEWER_MODEL=sonnet
|
||||
```
|
||||
|
||||
Use the same model name strings you'd use in any Claude Code agent frontmatter (e.g. `haiku`, `sonnet`, `opus`).
|
||||
|
||||
Overrides patch only the `model:` line in the installed agent's frontmatter; the prompt body is untouched and keeps receiving upstream updates. Plugin installs only — standalone hook installs have no local agent files to patch. Unset or blank = no change. The patch persists in the installed file until the plugin is updated or reinstalled.
|
||||
|
||||
## See also
|
||||
|
||||
- [`SKILL.md`](./SKILL.md) — full decision matrix and output contracts
|
||||
- [`agents/cavecrew-investigator.md`](../../agents/cavecrew-investigator.md)
|
||||
- [`agents/cavecrew-builder.md`](../../agents/cavecrew-builder.md)
|
||||
- [`agents/cavecrew-reviewer.md`](../../agents/cavecrew-reviewer.md)
|
||||
- [Caveman README](../../README.md) — repo overview
|
||||
82
.agents/skills/cavecrew/SKILL.md
Normal file
82
.agents/skills/cavecrew/SKILL.md
Normal file
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: cavecrew
|
||||
description: >
|
||||
Decision guide for delegating to caveman-style subagents. Tells the main
|
||||
thread WHEN to spawn `cavecrew-investigator` (locate code), `cavecrew-builder`
|
||||
(1-2 file edit), or `cavecrew-reviewer` (diff review) instead of doing the
|
||||
work inline or using vanilla `Explore`. Subagent output is caveman-compressed
|
||||
so the tool-result injected back into main context is ~60% smaller — main
|
||||
context lasts longer across long sessions.
|
||||
Trigger: "delegate to subagent", "use cavecrew", "spawn investigator/builder/reviewer",
|
||||
"save context", "compressed agent output".
|
||||
---
|
||||
|
||||
Cavecrew = three subagent presets that emit caveman output. Same job as Anthropic defaults (`Explore`, edit-style agents, reviewer); difference is the tool-result they return is compressed, so main context shrinks per delegation.
|
||||
|
||||
## When to use cavecrew vs alternatives
|
||||
|
||||
| Task | Use |
|
||||
|---|---|
|
||||
| "Where is X defined / what calls Y / list uses of Z" | `cavecrew-investigator` |
|
||||
| Same but you also want suggestions/architecture commentary | `Explore` (vanilla) |
|
||||
| Surgical edit, ≤2 files, scope obvious | `cavecrew-builder` |
|
||||
| New feature / 3+ files / cross-cutting refactor | Main thread or `feature-dev:code-architect` |
|
||||
| Review diff, branch, or file for bugs | `cavecrew-reviewer` |
|
||||
| Deep code review with rationale + alternatives | `Code Reviewer` (vanilla) |
|
||||
| One-line answer you already know | Main thread, no subagent |
|
||||
|
||||
Rule of thumb: **if you'd want the subagent's output in 1/3 the tokens, pick cavecrew. If you'd want prose, pick vanilla.**
|
||||
|
||||
## Why this exists (the real win)
|
||||
|
||||
Subagent tool results get injected into main context verbatim. A vanilla `Explore` that returns 2k tokens of prose costs 2k tokens of main-context budget every time. The same finding from `cavecrew-investigator` returns ~700 tokens. Across 20 delegations in one session that's the difference between context exhaustion and finishing the task.
|
||||
|
||||
## Output contracts
|
||||
|
||||
What main thread can rely on per agent:
|
||||
|
||||
**`cavecrew-investigator`**
|
||||
```
|
||||
<Header>:
|
||||
- path:line — `symbol` — short note
|
||||
totals: <counts>.
|
||||
```
|
||||
Or `No match.` Always file-path-first, line-number-attached, backticked symbols. Safe to grep with `path:\d+`.
|
||||
|
||||
**`cavecrew-builder`**
|
||||
```
|
||||
<path:line-range> — <change ≤10 words>.
|
||||
verified: <re-read OK | mismatch @ path:line>.
|
||||
```
|
||||
Or one of: `too-big.` / `needs-confirm.` / `ambiguous.` / `regressed.` (terminal first token).
|
||||
|
||||
**`cavecrew-reviewer`**
|
||||
```
|
||||
path:line: <emoji> <severity>: <problem>. <fix>.
|
||||
totals: N🔴 N🟡 N🔵 N❓
|
||||
```
|
||||
Or `No issues.` Findings sorted file → line ascending.
|
||||
|
||||
## Chaining patterns
|
||||
|
||||
**Locate → fix → verify** (most common):
|
||||
1. `cavecrew-investigator` returns site list.
|
||||
2. Main thread picks 1-2 sites, hands paths to `cavecrew-builder`.
|
||||
3. `cavecrew-reviewer` audits the diff.
|
||||
|
||||
**Parallel scout** (when investigation is broad):
|
||||
Spawn 2-3 `cavecrew-investigator` calls in one message (different angles: defs vs callers vs tests). Aggregate in main thread.
|
||||
|
||||
**Single-shot edit** (when site is already known):
|
||||
Skip investigator. Hand exact path:line to `cavecrew-builder` directly.
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- Don't use `cavecrew-builder` when you don't already know the file. Spawn investigator first or main thread will eat tokens passing context.
|
||||
- Don't chain `cavecrew-investigator → cavecrew-builder` for a 5-file refactor. Builder will return `too-big.` and you'll have wasted a turn.
|
||||
- Don't ask `cavecrew-reviewer` for "general feedback" — it returns findings only, no architecture opinions. Use `Code Reviewer` for that.
|
||||
- Don't expect prose. Cavecrew output is structured, sometimes terse to the point of cryptic. If a human will read it directly, paraphrase.
|
||||
|
||||
## Auto-clarity (inherited)
|
||||
|
||||
Subagents drop caveman → normal English for security warnings, irreversible-action confirmations, and any output where fragment ambiguity could be misread. Resume caveman after.
|
||||
44
.agents/skills/caveman-commit/README.md
Normal file
44
.agents/skills/caveman-commit/README.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# caveman-commit
|
||||
|
||||
Terse Conventional Commits. Why over what.
|
||||
|
||||
## What it does
|
||||
|
||||
Generates commit messages in Conventional Commits format. Subject ≤50 chars, hard cap 72. Imperative mood. Body only when the *why* is non-obvious or there are breaking changes. No AI attribution, no "this commit does X", no emoji unless the project uses them. Body always required for breaking changes, security fixes, data migrations, and reverts — future debuggers need the context.
|
||||
|
||||
Outputs only the message. Does not stage, commit, or amend.
|
||||
|
||||
## How to invoke
|
||||
|
||||
```
|
||||
/caveman-commit
|
||||
```
|
||||
|
||||
Also triggers on phrases like "write a commit", "commit message", "generate commit".
|
||||
|
||||
## Example output
|
||||
|
||||
Diff: new endpoint for user profile.
|
||||
|
||||
```
|
||||
feat(api): add GET /users/:id/profile
|
||||
|
||||
Mobile client needs profile data without the full user payload
|
||||
to reduce LTE bandwidth on cold-launch screens.
|
||||
|
||||
Closes #128
|
||||
```
|
||||
|
||||
Diff: breaking API rename.
|
||||
|
||||
```
|
||||
feat(api)!: rename /v1/orders to /v1/checkout
|
||||
|
||||
BREAKING CHANGE: clients on /v1/orders must migrate to /v1/checkout
|
||||
before 2026-06-01. Old route returns 410 after that date.
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
|
||||
- [Caveman README](../../README.md) — repo overview
|
||||
65
.agents/skills/caveman-commit/SKILL.md
Normal file
65
.agents/skills/caveman-commit/SKILL.md
Normal file
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: caveman-commit
|
||||
description: >
|
||||
Ultra-compressed commit message generator. Cuts noise from commit messages while preserving
|
||||
intent and reasoning. Conventional Commits format. Subject ≤50 chars, body only when "why"
|
||||
isn't obvious. Use when user says "write a commit", "commit message", "generate commit",
|
||||
"/commit", or invokes /caveman-commit. Auto-triggers when staging changes.
|
||||
---
|
||||
|
||||
Write commit messages terse and exact. Conventional Commits format. No fluff. Why over what.
|
||||
|
||||
## Rules
|
||||
|
||||
**Subject line:**
|
||||
- `<type>(<scope>): <imperative summary>` — `<scope>` optional
|
||||
- Types: `feat`, `fix`, `refactor`, `perf`, `docs`, `test`, `chore`, `build`, `ci`, `style`, `revert`
|
||||
- Imperative mood: "add", "fix", "remove" — not "added", "adds", "adding"
|
||||
- ≤50 chars when possible, hard cap 72
|
||||
- No trailing period
|
||||
- Match project convention for capitalization after the colon
|
||||
|
||||
**Body (only if needed):**
|
||||
- Skip entirely when subject is self-explanatory
|
||||
- Add body only for: non-obvious *why*, breaking changes, migration notes, linked issues
|
||||
- Wrap at 72 chars
|
||||
- Bullets `-` not `*`
|
||||
- Reference issues/PRs at end: `Closes #42`, `Refs #17`
|
||||
|
||||
**What NEVER goes in:**
|
||||
- "This commit does X", "I", "we", "now", "currently" — the diff says what
|
||||
- "As requested by..." — use Co-authored-by trailer
|
||||
- "Generated with Claude Code" or any AI attribution — unless the user's own rule requires an `Assisted-by`/AI-attribution trailer, then add it as a trailer
|
||||
- Emoji (unless project convention requires)
|
||||
- Restating the file name when scope already says it
|
||||
|
||||
## Examples
|
||||
|
||||
Diff: new endpoint for user profile with body explaining the why
|
||||
- ❌ "feat: add a new endpoint to get user profile information from the database"
|
||||
- ✅
|
||||
```
|
||||
feat(api): add GET /users/:id/profile
|
||||
|
||||
Mobile client needs profile data without the full user payload
|
||||
to reduce LTE bandwidth on cold-launch screens.
|
||||
|
||||
Closes #128
|
||||
```
|
||||
|
||||
Diff: breaking API change
|
||||
- ✅
|
||||
```
|
||||
feat(api)!: rename /v1/orders to /v1/checkout
|
||||
|
||||
BREAKING CHANGE: clients on /v1/orders must migrate to /v1/checkout
|
||||
before 2026-06-01. Old route returns 410 after that date.
|
||||
```
|
||||
|
||||
## Auto-Clarity
|
||||
|
||||
Always include body for: breaking changes, security fixes, data migrations, anything reverting a prior commit. Never compress these into subject-only — future debuggers need the context.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Only generates the commit message. Does not run `git commit`, does not stage files, does not amend. Output the message as a code block ready to paste. "stop caveman-commit" or "normal mode": revert to verbose commit style.
|
||||
163
.agents/skills/caveman-compress/README.md
Normal file
163
.agents/skills/caveman-compress/README.md
Normal file
@@ -0,0 +1,163 @@
|
||||
<p align="center">
|
||||
<img src="https://em-content.zobj.net/source/apple/391/rock_1faa8.png" width="80" />
|
||||
</p>
|
||||
|
||||
<h1 align="center">caveman-compress</h1>
|
||||
|
||||
<p align="center">
|
||||
<strong>shrink memory file. save token every session.</strong>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
A Claude Code skill that compresses your project memory files (`CLAUDE.md`, todos, preferences) into caveman format — so every session loads fewer tokens automatically.
|
||||
|
||||
Claude read `CLAUDE.md` on every session start. If file big, cost big. Caveman make file small. Cost go down forever.
|
||||
|
||||
## What It Do
|
||||
|
||||
```
|
||||
/caveman-compress CLAUDE.md
|
||||
```
|
||||
|
||||
```
|
||||
CLAUDE.md ← compressed (Claude reads this — fewer tokens every session)
|
||||
CLAUDE.original.md ← human-readable backup (you edit this)
|
||||
```
|
||||
|
||||
Original never lost. You can read and edit `.original.md`. Run skill again to re-compress after edits.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Real results on real project files:
|
||||
|
||||
| File | Original | Compressed | Saved |
|
||||
|------|----------:|----------:|------:|
|
||||
| `claude-md-preferences.md` | 706 | 285 | **59.6%** |
|
||||
| `project-notes.md` | 1145 | 535 | **53.3%** |
|
||||
| `claude-md-project.md` | 1122 | 636 | **43.3%** |
|
||||
| `todo-list.md` | 627 | 388 | **38.1%** |
|
||||
| `mixed-with-code.md` | 888 | 560 | **36.9%** |
|
||||
| **Average** | **898** | **481** | **46%** |
|
||||
|
||||
All validations passed ✅ — headings, code blocks, URLs, file paths preserved exactly.
|
||||
|
||||
## Before / After
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
|
||||
### 📄 Original (706 tokens)
|
||||
|
||||
> "I strongly prefer TypeScript with strict mode enabled for all new code. Please don't use `any` type unless there's genuinely no way around it, and if you do, leave a comment explaining the reasoning. I find that taking the time to properly type things catches a lot of bugs before they ever make it to runtime."
|
||||
|
||||
</td>
|
||||
<td width="50%">
|
||||
|
||||
### <img src="../../docs/assets/dancing-rock.svg" width="20" height="20" alt="rock"/> Caveman (285 tokens)
|
||||
|
||||
> "Prefer TypeScript strict mode always. No `any` unless unavoidable — comment why if used. Proper types catch bugs early."
|
||||
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
**Same instructions. 60% fewer tokens. Every. Single. Session.**
|
||||
|
||||
## Security
|
||||
|
||||
`caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O patterns detected by static analysis. This is a false positive — see [SECURITY.md](./SECURITY.md) for a full explanation of what the skill does and does not do.
|
||||
|
||||
## Install
|
||||
|
||||
Compress is built in with the `caveman` plugin. Install `caveman` once, then use `/caveman-compress`.
|
||||
|
||||
If you need local files, the compress skill lives at:
|
||||
|
||||
```bash
|
||||
caveman-compress/
|
||||
```
|
||||
|
||||
**Requires:** Python 3.10+
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/caveman-compress <filepath>
|
||||
```
|
||||
|
||||
Examples:
|
||||
```
|
||||
/caveman-compress CLAUDE.md
|
||||
/caveman-compress docs/preferences.md
|
||||
/caveman-compress todos.md
|
||||
```
|
||||
|
||||
### What files work
|
||||
|
||||
| Type | Compress? |
|
||||
|------|-----------|
|
||||
| `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` | ✅ Yes |
|
||||
| Extensionless natural language | ✅ Yes |
|
||||
| `.py`, `.js`, `.ts`, `.json`, `.yaml` | ❌ Skip (code/config) |
|
||||
| `*.original.md` | ❌ Skip (backup files) |
|
||||
|
||||
## How It Work
|
||||
|
||||
```
|
||||
/caveman-compress CLAUDE.md
|
||||
↓
|
||||
detect file type (no tokens)
|
||||
↓
|
||||
Claude compresses (tokens — one call)
|
||||
↓
|
||||
validate output (no tokens)
|
||||
checks: headings, code blocks, URLs, file paths, bullets
|
||||
↓
|
||||
if errors: Claude fixes cherry-picked issues only (tokens — targeted fix)
|
||||
does NOT recompress — only patches broken parts
|
||||
↓
|
||||
retry up to 2 times
|
||||
↓
|
||||
write compressed → CLAUDE.md
|
||||
write original → CLAUDE.original.md
|
||||
```
|
||||
|
||||
Only two things use tokens: initial compression + targeted fix if validation fails. Everything else is local Python.
|
||||
|
||||
## What Is Preserved
|
||||
|
||||
Caveman compress natural language. It never touch:
|
||||
|
||||
- Code blocks (` ``` ` fenced or indented)
|
||||
- Inline code (`` `backtick content` ``)
|
||||
- URLs and links
|
||||
- File paths (`/src/components/...`)
|
||||
- Commands (`npm install`, `git commit`)
|
||||
- Technical terms, library names, API names
|
||||
- Headings (exact text preserved)
|
||||
- Tables (structure preserved, cell text compressed)
|
||||
- Dates, version numbers, numeric values
|
||||
|
||||
## Why This Matter
|
||||
|
||||
`CLAUDE.md` loads on **every session start**. A 1000-token project memory file costs tokens every single time you open a project. Over 100 sessions that's 100,000 tokens of overhead — just for context you already wrote.
|
||||
|
||||
Caveman cut that by ~46% on average. Same instructions. Same accuracy. Less waste.
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────┐
|
||||
│ TOKEN SAVINGS PER FILE █████ 46% │
|
||||
│ SESSIONS THAT BENEFIT ██████████ 100% │
|
||||
│ INFORMATION PRESERVED ██████████ 100% │
|
||||
│ SETUP TIME █ 1x │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Part of Caveman
|
||||
|
||||
This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit — making Claude use fewer tokens without losing accuracy.
|
||||
|
||||
- **caveman** — make Claude *speak* like caveman (cuts response tokens ~65%)
|
||||
- **caveman-compress** — make Claude *read* less (cuts context tokens ~46%)
|
||||
31
.agents/skills/caveman-compress/SECURITY.md
Normal file
31
.agents/skills/caveman-compress/SECURITY.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Security
|
||||
|
||||
## Snyk High Risk Rating
|
||||
|
||||
`caveman-compress` receives a Snyk High Risk rating due to static analysis heuristics. This document explains what the skill does and does not do.
|
||||
|
||||
### What triggers the rating
|
||||
|
||||
1. **subprocess usage**: The skill calls the `claude` CLI via `subprocess.run()` as a fallback when `ANTHROPIC_API_KEY` is not set. The subprocess call uses a fixed argument list — no shell interpolation occurs. User file content is passed via stdin, not as a shell argument.
|
||||
|
||||
2. **File read/write**: The skill reads the file the user explicitly points it at, compresses it, and writes the result back to the same path. A `.original.md` backup is saved alongside it. No files outside the user-specified path are read or written.
|
||||
|
||||
### What the skill does NOT do
|
||||
|
||||
- Does not execute user file content as code
|
||||
- Does not make network requests except to Anthropic's API (via SDK or CLI)
|
||||
- Does not access files outside the path the user provides
|
||||
- Does not use shell=True or string interpolation in subprocess calls
|
||||
- Does not collect or transmit any data beyond the file being compressed
|
||||
|
||||
### Auth behavior
|
||||
|
||||
If `ANTHROPIC_API_KEY` is set, the skill uses the Anthropic Python SDK directly (no subprocess). If not set, it falls back to the `claude` CLI, which uses the user's existing Claude desktop authentication.
|
||||
|
||||
### File size limit
|
||||
|
||||
Files larger than 500KB are rejected before any API call is made.
|
||||
|
||||
### Reporting a vulnerability
|
||||
|
||||
If you believe you've found a genuine security issue, please open a GitHub issue with the label `security`.
|
||||
111
.agents/skills/caveman-compress/SKILL.md
Normal file
111
.agents/skills/caveman-compress/SKILL.md
Normal file
@@ -0,0 +1,111 @@
|
||||
---
|
||||
name: caveman-compress
|
||||
description: >
|
||||
Compress natural language memory files (CLAUDE.md, todos, preferences) into caveman format
|
||||
to save input tokens. Preserves all technical substance, code, URLs, and structure.
|
||||
Compressed version overwrites the original file. Human-readable backup saved as FILE.original.md.
|
||||
Trigger: /caveman-compress FILEPATH or "compress memory file"
|
||||
---
|
||||
|
||||
# Caveman Compress
|
||||
|
||||
## Purpose
|
||||
|
||||
Compress natural language files (CLAUDE.md, todos, preferences) into caveman-speak to reduce input tokens. Compressed version overwrites original. Human-readable backup saved as `<filename>.original.md`.
|
||||
|
||||
## Trigger
|
||||
|
||||
`/caveman-compress <filepath>` or when user asks to compress a memory file.
|
||||
|
||||
## Process
|
||||
|
||||
1. The compression scripts live in `scripts/` (adjacent to this SKILL.md). If the path is not immediately available, search for `scripts/__main__.py` next to this SKILL.md.
|
||||
|
||||
2. From the directory containing this SKILL.md, run:
|
||||
|
||||
python3 -m scripts <absolute_filepath>
|
||||
|
||||
3. The CLI will:
|
||||
- detect file type (no tokens)
|
||||
- call Claude to compress
|
||||
- validate output (no tokens)
|
||||
- if errors: cherry-pick fix with Claude (targeted fixes only, no recompression)
|
||||
- retry up to 2 times
|
||||
- if still failing after 2 retries: report error to user, leave original file untouched
|
||||
|
||||
4. Return result to user
|
||||
|
||||
## Compression Rules
|
||||
|
||||
### Remove
|
||||
- Articles: a, an, the
|
||||
- Filler: just, really, basically, actually, simply, essentially, generally
|
||||
- Pleasantries: "sure", "certainly", "of course", "happy to", "I'd recommend"
|
||||
- Hedging: "it might be worth", "you could consider", "it would be good to"
|
||||
- Redundant phrasing: "in order to" → "to", "make sure to" → "ensure", "the reason is because" → "because"
|
||||
- Connective fluff: "however", "furthermore", "additionally", "in addition"
|
||||
|
||||
### Preserve EXACTLY (never modify)
|
||||
- Code blocks (fenced ``` and indented)
|
||||
- Inline code (`backtick content`)
|
||||
- URLs and links (full URLs, markdown links)
|
||||
- File paths (`/src/components/...`, `./config.yaml`)
|
||||
- Commands (`npm install`, `git commit`, `docker build`)
|
||||
- Technical terms (library names, API names, protocols, algorithms)
|
||||
- Proper nouns (project names, people, companies)
|
||||
- Dates, version numbers, numeric values
|
||||
- Environment variables (`$HOME`, `NODE_ENV`)
|
||||
|
||||
### Preserve Structure
|
||||
- All markdown headings (keep exact heading text, compress body below)
|
||||
- Bullet point hierarchy (keep nesting level)
|
||||
- Numbered lists (keep numbering)
|
||||
- Tables (compress cell text, keep structure)
|
||||
- Frontmatter/YAML headers in markdown files
|
||||
|
||||
### Compress
|
||||
- Use short synonyms: "big" not "extensive", "fix" not "implement a solution for", "use" not "utilize"
|
||||
- Fragments OK: "Run tests before commit" not "You should always run tests before committing"
|
||||
- Drop "you should", "make sure to", "remember to" — just state the action
|
||||
- Merge redundant bullets that say the same thing differently
|
||||
- Keep one example where multiple examples show the same pattern
|
||||
|
||||
CRITICAL RULE:
|
||||
Anything inside ``` ... ``` must be copied EXACTLY.
|
||||
Do not:
|
||||
- remove comments
|
||||
- remove spacing
|
||||
- reorder lines
|
||||
- shorten commands
|
||||
- simplify anything
|
||||
|
||||
Inline code (`...`) must be preserved EXACTLY.
|
||||
Do not modify anything inside backticks.
|
||||
|
||||
If file contains code blocks:
|
||||
- Treat code blocks as read-only regions
|
||||
- Only compress text outside them
|
||||
- Do not merge sections around code
|
||||
|
||||
## Pattern
|
||||
|
||||
Original:
|
||||
> You should always make sure to run the test suite before pushing any changes to the main branch. This is important because it helps catch bugs early and prevents broken builds from being deployed to production.
|
||||
|
||||
Compressed:
|
||||
> Run tests before push to main. Catch bugs early, prevent broken prod deploys.
|
||||
|
||||
Original:
|
||||
> The application uses a microservices architecture with the following components. The API gateway handles all incoming requests and routes them to the appropriate service. The authentication service is responsible for managing user sessions and JWT tokens.
|
||||
|
||||
Compressed:
|
||||
> Microservices architecture. API gateway route all requests to services. Auth service manage user sessions + JWT tokens.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- ONLY compress natural language files (.md, .txt, .typ, .typst, .tex, extensionless)
|
||||
- NEVER modify: .py, .js, .ts, .json, .yaml, .yml, .toml, .env, .lock, .css, .html, .xml, .sql, .sh
|
||||
- If file has mixed content (prose + code), compress ONLY the prose sections
|
||||
- If unsure whether something is code or prose, leave it unchanged
|
||||
- Original file is backed up as FILE.original.md before overwriting
|
||||
- Never compress FILE.original.md (skip it)
|
||||
9
.agents/skills/caveman-compress/scripts/__init__.py
Normal file
9
.agents/skills/caveman-compress/scripts/__init__.py
Normal file
@@ -0,0 +1,9 @@
|
||||
"""Caveman compress scripts.
|
||||
|
||||
This package provides tools to compress natural language markdown files
|
||||
into caveman format to save input tokens.
|
||||
"""
|
||||
|
||||
__all__ = ["cli", "compress", "detect", "validate"]
|
||||
|
||||
__version__ = "1.0.0"
|
||||
3
.agents/skills/caveman-compress/scripts/__main__.py
Normal file
3
.agents/skills/caveman-compress/scripts/__main__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from .cli import main
|
||||
|
||||
main()
|
||||
80
.agents/skills/caveman-compress/scripts/benchmark.py
Normal file
80
.agents/skills/caveman-compress/scripts/benchmark.py
Normal file
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
# Support both direct execution and module import
|
||||
try:
|
||||
from .validate import validate
|
||||
except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from validate import validate
|
||||
|
||||
try:
|
||||
import tiktoken
|
||||
_enc = tiktoken.get_encoding("o200k_base")
|
||||
except ImportError:
|
||||
_enc = None
|
||||
|
||||
|
||||
def count_tokens(text):
|
||||
if _enc is None:
|
||||
return len(text.split()) # fallback: word count
|
||||
return len(_enc.encode(text))
|
||||
|
||||
|
||||
def benchmark_pair(orig_path: Path, comp_path: Path):
|
||||
orig_text = orig_path.read_text()
|
||||
comp_text = comp_path.read_text()
|
||||
|
||||
orig_tokens = count_tokens(orig_text)
|
||||
comp_tokens = count_tokens(comp_text)
|
||||
saved = 100 * (orig_tokens - comp_tokens) / orig_tokens if orig_tokens > 0 else 0.0
|
||||
result = validate(orig_path, comp_path)
|
||||
|
||||
return (comp_path.name, orig_tokens, comp_tokens, saved, result.is_valid)
|
||||
|
||||
|
||||
def print_table(rows):
|
||||
print("\n| File | Original | Compressed | Saved % | Valid |")
|
||||
print("|------|----------|------------|---------|-------|")
|
||||
for r in rows:
|
||||
print(f"| {r[0]} | {r[1]} | {r[2]} | {r[3]:.1f}% | {'✅' if r[4] else '❌'} |")
|
||||
|
||||
|
||||
def main():
|
||||
# Direct file pair: python3 benchmark.py original.md compressed.md
|
||||
if len(sys.argv) == 3:
|
||||
orig = Path(sys.argv[1]).resolve()
|
||||
comp = Path(sys.argv[2]).resolve()
|
||||
if not orig.exists():
|
||||
print(f"❌ Not found: {orig}")
|
||||
sys.exit(1)
|
||||
if not comp.exists():
|
||||
print(f"❌ Not found: {comp}")
|
||||
sys.exit(1)
|
||||
print_table([benchmark_pair(orig, comp)])
|
||||
return
|
||||
|
||||
# Glob mode: repo_root/tests/caveman-compress/
|
||||
# __file__ lives at <repo_root>/skills/caveman-compress/scripts/benchmark.py
|
||||
# Walk up four dirs: scripts → caveman-compress → skills → repo_root.
|
||||
tests_dir = Path(__file__).resolve().parents[3] / "tests" / "caveman-compress"
|
||||
if not tests_dir.exists():
|
||||
print(f"❌ Tests dir not found: {tests_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
rows = []
|
||||
for orig in sorted(tests_dir.glob("*.original.md")):
|
||||
comp = orig.with_name(orig.stem.removesuffix(".original") + ".md")
|
||||
if comp.exists():
|
||||
rows.append(benchmark_pair(orig, comp))
|
||||
|
||||
if not rows:
|
||||
print("No compressed file pairs found.")
|
||||
return
|
||||
|
||||
print_table(rows)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
85
.agents/skills/caveman-compress/scripts/cli.py
Normal file
85
.agents/skills/caveman-compress/scripts/cli.py
Normal file
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Caveman Compress CLI
|
||||
|
||||
Usage:
|
||||
caveman <filepath>
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
# Force UTF-8 on stdout/stderr before any code can print. Windows consoles
|
||||
# default to cp1252 and crash on the ❌ glyphs in error/validation branches,
|
||||
# masking the real error and leaving the user with a half-compressed file.
|
||||
for _stream in (sys.stdout, sys.stderr):
|
||||
reconfigure = getattr(_stream, "reconfigure", None)
|
||||
if callable(reconfigure):
|
||||
try:
|
||||
reconfigure(encoding="utf-8", errors="replace")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from .compress import backup_dir_for, compress_file
|
||||
from .detect import detect_file_type, should_compress
|
||||
|
||||
|
||||
def print_usage():
|
||||
print("Usage: caveman <filepath>")
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 2:
|
||||
print_usage()
|
||||
sys.exit(1)
|
||||
|
||||
filepath = Path(sys.argv[1])
|
||||
|
||||
# Check file exists
|
||||
if not filepath.exists():
|
||||
print(f"❌ File not found: {filepath}")
|
||||
sys.exit(1)
|
||||
|
||||
if not filepath.is_file():
|
||||
print(f"❌ Not a file: {filepath}")
|
||||
sys.exit(1)
|
||||
|
||||
filepath = filepath.resolve()
|
||||
|
||||
# Detect file type
|
||||
file_type = detect_file_type(filepath)
|
||||
|
||||
print(f"Detected: {file_type}")
|
||||
|
||||
# Check if compressible
|
||||
if not should_compress(filepath):
|
||||
print("Skipping: file is not natural language (code/config)")
|
||||
sys.exit(0)
|
||||
|
||||
print("Starting caveman compression...\n")
|
||||
|
||||
try:
|
||||
success = compress_file(filepath)
|
||||
|
||||
if success:
|
||||
print("\nCompression completed successfully")
|
||||
backup_path = backup_dir_for(filepath) / (filepath.stem + ".original.md")
|
||||
print(f"Compressed: {filepath}")
|
||||
print(f"Original: {backup_path}")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print("\n❌ Compression failed after retries")
|
||||
sys.exit(2)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted by user")
|
||||
sys.exit(130)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
342
.agents/skills/caveman-compress/scripts/compress.py
Normal file
342
.agents/skills/caveman-compress/scripts/compress.py
Normal file
@@ -0,0 +1,342 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Caveman Memory Compression Orchestrator
|
||||
|
||||
Usage:
|
||||
python scripts/compress.py <filepath>
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
OUTER_FENCE_REGEX = re.compile(
|
||||
r"\A\s*(`{3,}|~{3,})[^\n]*\n(.*)\n\1\s*\Z", re.DOTALL
|
||||
)
|
||||
|
||||
# YAML frontmatter: starts at file start with --- on its own line, ends with --- on its own line.
|
||||
# Captures the entire block (including delimiters and trailing newline) and the body after.
|
||||
FRONTMATTER_REGEX = re.compile(
|
||||
r"\A(---\r?\n.*?\r?\n---\r?\n)(.*)", re.DOTALL
|
||||
)
|
||||
|
||||
|
||||
def split_frontmatter(text: str):
|
||||
"""Split YAML frontmatter from body. Returns (frontmatter, body).
|
||||
|
||||
Memory files (and many other markdown docs) start with a YAML frontmatter
|
||||
block delimited by `---` lines. The compression LLM has a habit of stripping
|
||||
or rewriting these despite preserve-structure rules in the prompt — so we
|
||||
surgically remove the frontmatter before compression and prepend it back
|
||||
verbatim to the output. Files without frontmatter pass through unchanged.
|
||||
"""
|
||||
m = FRONTMATTER_REGEX.match(text)
|
||||
if m:
|
||||
return m.group(1), m.group(2)
|
||||
return "", text
|
||||
|
||||
# Filenames and paths that almost certainly hold secrets or PII. Compressing
|
||||
# them ships raw bytes to the Anthropic API — a third-party data boundary that
|
||||
# developers on sensitive codebases cannot cross. detect.py already skips .env
|
||||
# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would
|
||||
# slip through the natural-language filter. This is a hard refuse before read.
|
||||
SENSITIVE_BASENAME_REGEX = re.compile(
|
||||
r"(?ix)^("
|
||||
r"\.env(\..+)?"
|
||||
r"|\.netrc"
|
||||
r"|credentials(\..+)?"
|
||||
r"|secrets?(\..+)?"
|
||||
r"|passwords?(\..+)?"
|
||||
r"|id_(rsa|dsa|ecdsa|ed25519)(\.pub)?"
|
||||
r"|authorized_keys"
|
||||
r"|known_hosts"
|
||||
r"|.*\.(pem|key|p12|pfx|crt|cer|jks|keystore|asc|gpg)"
|
||||
r")$"
|
||||
)
|
||||
|
||||
SENSITIVE_PATH_COMPONENTS = frozenset({".ssh", ".aws", ".gnupg", ".kube", ".docker"})
|
||||
|
||||
SENSITIVE_NAME_TOKENS = (
|
||||
"secret", "credential", "password", "passwd",
|
||||
"apikey", "accesskey", "token", "privatekey",
|
||||
)
|
||||
|
||||
|
||||
def backup_dir_for(filepath: Path) -> Path:
|
||||
"""Resolve the out-of-tree backup directory for a given source file.
|
||||
|
||||
Backups must live OUTSIDE the source directory so skill auto-loaders
|
||||
(Claude Code rules/, opencode instructions/, etc.) stop re-ingesting the
|
||||
`.original.md` copies as live files. Base dir is platform-aware:
|
||||
- Windows: %LOCALAPPDATA%\\caveman-compress\\backups
|
||||
- else: $XDG_DATA_HOME/caveman-compress/backups if set,
|
||||
else ~/.local/share/caveman-compress/backups
|
||||
|
||||
The source file's parent-dir name is mirrored under the base to reduce
|
||||
cross-project collisions (e.g. two `task.md` files in different repos).
|
||||
"""
|
||||
if os.name == "nt" or sys.platform == "win32":
|
||||
local_appdata = os.environ.get("LOCALAPPDATA")
|
||||
base = Path(local_appdata) if local_appdata else Path.home() / "AppData" / "Local"
|
||||
base = base / "caveman-compress" / "backups"
|
||||
else:
|
||||
xdg = os.environ.get("XDG_DATA_HOME")
|
||||
base = Path(xdg) if xdg else Path.home() / ".local" / "share"
|
||||
base = base / "caveman-compress" / "backups"
|
||||
return base / filepath.parent.name
|
||||
|
||||
|
||||
def is_sensitive_path(filepath: Path) -> bool:
|
||||
"""Heuristic denylist for files that must never be shipped to a third-party API."""
|
||||
name = filepath.name
|
||||
if SENSITIVE_BASENAME_REGEX.match(name):
|
||||
return True
|
||||
lowered_parts = {p.lower() for p in filepath.parts}
|
||||
if lowered_parts & SENSITIVE_PATH_COMPONENTS:
|
||||
return True
|
||||
# Normalize separators so "api-key" and "api_key" both match "apikey".
|
||||
lower = re.sub(r"[_\-\s.]", "", name.lower())
|
||||
return any(tok in lower for tok in SENSITIVE_NAME_TOKENS)
|
||||
|
||||
|
||||
def strip_llm_wrapper(text: str) -> str:
|
||||
"""Strip outer ```markdown ... ``` fence when it wraps the entire output."""
|
||||
m = OUTER_FENCE_REGEX.match(text)
|
||||
if m:
|
||||
return m.group(2)
|
||||
return text
|
||||
|
||||
from .detect import should_compress
|
||||
from .validate import validate
|
||||
|
||||
MAX_RETRIES = 2
|
||||
|
||||
|
||||
# ---------- Claude Calls ----------
|
||||
|
||||
|
||||
def call_claude(prompt: str) -> str:
|
||||
"""Send a prompt to Claude.
|
||||
|
||||
Prefers the Anthropic SDK when ANTHROPIC_API_KEY is set; otherwise falls
|
||||
back to the ``claude --print`` CLI (which handles desktop auth).
|
||||
|
||||
On Windows the CLI subprocess decoding defaults to the system codepage
|
||||
(cp1251 / cp1252) and crashes on UTF-8 output — see issue #152. Pinning
|
||||
``encoding="utf-8"`` with ``errors="replace"`` matches the CLI's actual
|
||||
native I/O and prevents the UnicodeDecodeError before validation can
|
||||
report. Windows users with non-ASCII content can also set
|
||||
``ANTHROPIC_API_KEY`` to route through the SDK and skip the subprocess.
|
||||
"""
|
||||
api_key = os.environ.get("ANTHROPIC_API_KEY")
|
||||
if api_key:
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic(api_key=api_key)
|
||||
msg = client.messages.create(
|
||||
model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),
|
||||
max_tokens=8192,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
return strip_llm_wrapper(msg.content[0].text.strip())
|
||||
except ImportError:
|
||||
pass # anthropic not installed, fall back to CLI
|
||||
# Fallback: use claude CLI (handles desktop auth).
|
||||
# Resolve binary via shutil.which so Windows .cmd/.bat shims (e.g.
|
||||
# %APPDATA%\npm\claude.CMD) work without shell=True. On POSIX,
|
||||
# shutil.which returns the same absolute path as the implicit lookup,
|
||||
# so this is a no-op there. Falls back to bare "claude" if not found
|
||||
# on PATH so subprocess raises a clear FileNotFoundError.
|
||||
claude_bin = shutil.which("claude") or "claude"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[claude_bin, "--print"],
|
||||
input=prompt,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
return strip_llm_wrapper(result.stdout.strip())
|
||||
except subprocess.CalledProcessError as e:
|
||||
raise RuntimeError(f"Claude call failed:\n{e.stderr}")
|
||||
|
||||
|
||||
def build_compress_prompt(original: str) -> str:
|
||||
return f"""
|
||||
Compress this markdown into caveman format.
|
||||
|
||||
STRICT RULES:
|
||||
- Do NOT modify anything inside ``` code blocks
|
||||
- Do NOT modify anything inside inline backticks
|
||||
- Preserve ALL URLs exactly
|
||||
- Preserve ALL headings exactly
|
||||
- Preserve file paths and commands
|
||||
- Return ONLY the compressed markdown body — do NOT wrap the entire output in a ```markdown fence or any other fence. Inner code blocks from the original stay as-is; do not add a new outer fence around the whole file.
|
||||
|
||||
Only compress natural language.
|
||||
|
||||
TEXT:
|
||||
{original}
|
||||
"""
|
||||
|
||||
|
||||
def build_fix_prompt(original: str, compressed: str, errors: List[str]) -> str:
|
||||
errors_str = "\n".join(f"- {e}" for e in errors)
|
||||
return f"""You are fixing a caveman-compressed markdown file. Specific validation errors were found.
|
||||
|
||||
CRITICAL RULES:
|
||||
- DO NOT recompress or rephrase the file
|
||||
- ONLY fix the listed errors — leave everything else exactly as-is
|
||||
- The ORIGINAL is provided as reference only (to restore missing content)
|
||||
- Preserve caveman style in all untouched sections
|
||||
|
||||
ERRORS TO FIX:
|
||||
{errors_str}
|
||||
|
||||
HOW TO FIX:
|
||||
- Missing URL: find it in ORIGINAL, restore it exactly where it belongs in COMPRESSED
|
||||
- Code block mismatch: find the exact code block in ORIGINAL, restore it in COMPRESSED
|
||||
- Heading mismatch: restore the exact heading text from ORIGINAL into COMPRESSED
|
||||
- Do not touch any section not mentioned in the errors
|
||||
|
||||
ORIGINAL (reference only):
|
||||
{original}
|
||||
|
||||
COMPRESSED (fix this):
|
||||
{compressed}
|
||||
|
||||
Return ONLY the fixed compressed file. No explanation.
|
||||
"""
|
||||
|
||||
|
||||
# ---------- Core Logic ----------
|
||||
|
||||
|
||||
def compress_file(filepath: Path) -> bool:
|
||||
# Resolve and validate path
|
||||
filepath = filepath.resolve()
|
||||
MAX_FILE_SIZE = 500_000 # 500KB
|
||||
if not filepath.exists():
|
||||
raise FileNotFoundError(f"File not found: {filepath}")
|
||||
if filepath.stat().st_size > MAX_FILE_SIZE:
|
||||
raise ValueError(f"File too large to compress safely (max 500KB): {filepath}")
|
||||
|
||||
# Refuse files that look like they contain secrets or PII. Compressing ships
|
||||
# the raw bytes to the Anthropic API — a third-party boundary — so we fail
|
||||
# loudly rather than silently exfiltrate credentials or keys. Override is
|
||||
# intentional: the user must rename the file if the heuristic is wrong.
|
||||
if is_sensitive_path(filepath):
|
||||
raise ValueError(
|
||||
f"Refusing to compress {filepath}: filename looks sensitive "
|
||||
"(credentials, keys, secrets, or known private paths). "
|
||||
"Compression sends file contents to the Anthropic API. "
|
||||
"Rename the file if this is a false positive."
|
||||
)
|
||||
|
||||
print(f"Processing: {filepath}")
|
||||
|
||||
if not should_compress(filepath):
|
||||
print("Skipping (not natural language)")
|
||||
return False
|
||||
|
||||
original_text = filepath.read_text(errors="ignore")
|
||||
# Store backup outside the source directory so skill auto-loaders don't
|
||||
# re-ingest the `.original.md` copy as a live file. Mirror the source's
|
||||
# parent-dir name + stem under a platform-aware base to reduce collisions.
|
||||
backup_dir = backup_dir_for(filepath)
|
||||
backup_dir.mkdir(parents=True, exist_ok=True)
|
||||
backup_path = backup_dir / (filepath.stem + ".original.md")
|
||||
|
||||
if not original_text.strip():
|
||||
print("❌ Refusing to compress: file is empty or whitespace-only.")
|
||||
return False
|
||||
|
||||
# Check if backup already exists to prevent accidental overwriting
|
||||
if backup_path.exists():
|
||||
print(f"⚠️ Backup file already exists: {backup_path}")
|
||||
print("The original backup may contain important content.")
|
||||
print("Aborting to prevent data loss. Please remove or rename the backup file if you want to proceed.")
|
||||
return False
|
||||
|
||||
# Split YAML frontmatter off before compression. Claude tends to strip or
|
||||
# rewrite frontmatter despite preserve-structure rules; we keep it verbatim
|
||||
# by removing it from the input and re-prepending it to the output.
|
||||
frontmatter, body = split_frontmatter(original_text)
|
||||
if frontmatter:
|
||||
print(f"Detected YAML frontmatter ({len(frontmatter)} chars) — preserving verbatim")
|
||||
|
||||
if not body.strip():
|
||||
print("❌ Refusing to compress: body is empty after frontmatter removal.")
|
||||
return False
|
||||
|
||||
# Step 1: Compress (body only, frontmatter excluded)
|
||||
print("Compressing with Claude...")
|
||||
compressed_body = call_claude(build_compress_prompt(body))
|
||||
|
||||
if compressed_body is None or not compressed_body.strip():
|
||||
print("❌ Compression aborted: Claude returned an empty response.")
|
||||
print(" Original file is untouched (no backup created).")
|
||||
return False
|
||||
|
||||
# Compare the BODY (not the whole file) — frontmatter is preserved verbatim
|
||||
# and would never change, so identity must be judged on the compressible part.
|
||||
if compressed_body.strip() == body.strip():
|
||||
print("❌ Compression aborted: output is identical to input.")
|
||||
print(" Likely causes: Claude refused, returned the prompt verbatim, or the file is")
|
||||
print(" already in caveman form. Original file is untouched (no backup created).")
|
||||
return False
|
||||
|
||||
# Reassemble: frontmatter (verbatim) + compressed body
|
||||
compressed = frontmatter + compressed_body
|
||||
|
||||
# Save original as backup, then verify the backup readback before
|
||||
# touching the input file. If the filesystem dropped bytes (encoding,
|
||||
# antivirus, disk full), unlink the bad backup and abort instead of
|
||||
# leaving the user with a corrupt backup + compressed primary.
|
||||
backup_path.write_text(original_text)
|
||||
backup_readback = backup_path.read_text(errors="ignore")
|
||||
if backup_readback != original_text:
|
||||
print(f"❌ Backup write verification failed: {backup_path}")
|
||||
print(" In-memory original differs from on-disk backup. Aborting before touching the input file.")
|
||||
try:
|
||||
backup_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
return False
|
||||
filepath.write_text(compressed)
|
||||
|
||||
# Step 2: Validate + Retry
|
||||
for attempt in range(MAX_RETRIES):
|
||||
print(f"\nValidation attempt {attempt + 1}")
|
||||
|
||||
result = validate(backup_path, filepath)
|
||||
|
||||
if result.is_valid:
|
||||
print("Validation passed")
|
||||
break
|
||||
|
||||
print("❌ Validation failed:")
|
||||
for err in result.errors:
|
||||
print(f" - {err}")
|
||||
|
||||
if attempt == MAX_RETRIES - 1:
|
||||
# Restore original on failure
|
||||
filepath.write_text(original_text)
|
||||
backup_path.unlink(missing_ok=True)
|
||||
print("❌ Failed after retries — original restored")
|
||||
return False
|
||||
|
||||
print("Fixing with Claude...")
|
||||
compressed = call_claude(
|
||||
build_fix_prompt(original_text, compressed, result.errors)
|
||||
)
|
||||
filepath.write_text(compressed)
|
||||
|
||||
return True
|
||||
139
.agents/skills/caveman-compress/scripts/detect.py
Normal file
139
.agents/skills/caveman-compress/scripts/detect.py
Normal file
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Detect whether a file is natural language (compressible) or code/config (skip)."""
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
# Extensions that are natural language and compressible
|
||||
COMPRESSIBLE_EXTENSIONS = {".md", ".txt", ".markdown", ".rst", ".typ", ".typst", ".tex"}
|
||||
|
||||
# Extensions that are code/config and should be skipped
|
||||
SKIP_EXTENSIONS = {
|
||||
".py", ".js", ".ts", ".tsx", ".jsx", ".json", ".yaml", ".yml",
|
||||
".toml", ".env", ".lock", ".css", ".scss", ".html", ".xml",
|
||||
".sql", ".sh", ".bash", ".zsh", ".go", ".rs", ".java", ".c",
|
||||
".cpp", ".h", ".hpp", ".rb", ".php", ".swift", ".kt", ".lua",
|
||||
".dockerfile", ".makefile", ".csv", ".ini", ".cfg",
|
||||
}
|
||||
|
||||
# Well-known build/config files that carry no (or a misleading) extension —
|
||||
# `Dockerfile` has no suffix so `.dockerfile` above never matches it, and
|
||||
# `CMakeLists.txt` would ride the compressible `.txt` rule. Checked by
|
||||
# basename before any extension rule.
|
||||
KNOWN_CODE_FILENAMES = {
|
||||
"dockerfile", "makefile", "gnumakefile", "jenkinsfile", "vagrantfile",
|
||||
"rakefile", "gemfile", "justfile", "procfile", "brewfile",
|
||||
"cmakelists.txt",
|
||||
}
|
||||
|
||||
# Patterns that indicate a line is code
|
||||
CODE_PATTERNS = [
|
||||
re.compile(r"^\s*(import |from .+ import |require\(|const |let |var )"),
|
||||
re.compile(r"^\s*(def |class |function |async function |export )"),
|
||||
re.compile(r"^\s*(if\s*\(|for\s*\(|while\s*\(|switch\s*\(|try\s*\{)"),
|
||||
re.compile(r"^\s*[\}\]\);]+\s*$"), # closing braces/brackets
|
||||
re.compile(r"^\s*@\w+"), # decorators/annotations
|
||||
re.compile(r'^\s*"[^"]+"\s*:\s*'), # JSON-like key-value
|
||||
re.compile(r"^\s*\w+\s*=\s*[{\[\(\"']"), # assignment with literal
|
||||
]
|
||||
|
||||
|
||||
def _is_code_line(line: str) -> bool:
|
||||
"""Check if a line looks like code."""
|
||||
return any(p.match(line) for p in CODE_PATTERNS)
|
||||
|
||||
|
||||
def _is_json_content(text: str) -> bool:
|
||||
"""Check if content is valid JSON."""
|
||||
try:
|
||||
json.loads(text)
|
||||
return True
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _is_yaml_content(lines: list[str]) -> bool:
|
||||
"""Heuristic: check if content looks like YAML."""
|
||||
yaml_indicators = 0
|
||||
for line in lines[:30]:
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("---"):
|
||||
yaml_indicators += 1
|
||||
elif re.match(r"^\w[\w\s]*:\s", stripped):
|
||||
yaml_indicators += 1
|
||||
elif stripped.startswith("- ") and ":" in stripped:
|
||||
yaml_indicators += 1
|
||||
# If most non-empty lines look like YAML
|
||||
non_empty = sum(1 for l in lines[:30] if l.strip())
|
||||
return non_empty > 0 and yaml_indicators / non_empty > 0.6
|
||||
|
||||
|
||||
def detect_file_type(filepath: Path) -> str:
|
||||
"""Classify a file as 'natural_language', 'code', 'config', or 'unknown'.
|
||||
|
||||
Returns:
|
||||
One of: 'natural_language', 'code', 'config', 'unknown'
|
||||
"""
|
||||
ext = filepath.suffix.lower()
|
||||
|
||||
# Known code filenames win over any extension rule
|
||||
if filepath.name.lower() in KNOWN_CODE_FILENAMES:
|
||||
return "code"
|
||||
|
||||
# Extension-based classification
|
||||
if ext in COMPRESSIBLE_EXTENSIONS:
|
||||
return "natural_language"
|
||||
if ext in SKIP_EXTENSIONS:
|
||||
return "code" if ext not in {".json", ".yaml", ".yml", ".toml", ".ini", ".cfg", ".env"} else "config"
|
||||
|
||||
# Extensionless files (like CLAUDE.md, TODO) — check content
|
||||
if not ext:
|
||||
try:
|
||||
text = filepath.read_text(errors="ignore")
|
||||
except (OSError, PermissionError):
|
||||
return "unknown"
|
||||
|
||||
lines = text.splitlines()[:50]
|
||||
|
||||
# Shebang means executable script, never prose
|
||||
if text.startswith("#!"):
|
||||
return "code"
|
||||
|
||||
if _is_json_content(text[:10000]):
|
||||
return "config"
|
||||
if _is_yaml_content(lines):
|
||||
return "config"
|
||||
|
||||
code_lines = sum(1 for l in lines if l.strip() and _is_code_line(l))
|
||||
non_empty = sum(1 for l in lines if l.strip())
|
||||
if non_empty > 0 and code_lines / non_empty > 0.4:
|
||||
return "code"
|
||||
|
||||
return "natural_language"
|
||||
|
||||
return "unknown"
|
||||
|
||||
|
||||
def should_compress(filepath: Path) -> bool:
|
||||
"""Return True if the file is natural language and should be compressed."""
|
||||
if not filepath.is_file():
|
||||
return False
|
||||
# Skip backup files
|
||||
if filepath.name.endswith(".original.md"):
|
||||
return False
|
||||
return detect_file_type(filepath) == "natural_language"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python detect.py <file1> [file2] ...")
|
||||
sys.exit(1)
|
||||
|
||||
for path_str in sys.argv[1:]:
|
||||
p = Path(path_str).resolve()
|
||||
file_type = detect_file_type(p)
|
||||
compress = should_compress(p)
|
||||
print(f" {p.name:30s} type={file_type:20s} compress={compress}")
|
||||
213
.agents/skills/caveman-compress/scripts/validate.py
Normal file
213
.agents/skills/caveman-compress/scripts/validate.py
Normal file
@@ -0,0 +1,213 @@
|
||||
#!/usr/bin/env python3
|
||||
import re
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
URL_REGEX = re.compile(r"https?://[^\s)]+")
|
||||
FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
|
||||
HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
|
||||
BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
|
||||
|
||||
# crude but effective path detection
|
||||
# Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
|
||||
PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
|
||||
|
||||
|
||||
class ValidationResult:
|
||||
def __init__(self):
|
||||
self.is_valid = True
|
||||
self.errors = []
|
||||
self.warnings = []
|
||||
|
||||
def add_error(self, msg):
|
||||
self.is_valid = False
|
||||
self.errors.append(msg)
|
||||
|
||||
def add_warning(self, msg):
|
||||
self.warnings.append(msg)
|
||||
|
||||
|
||||
def read_file(path: Path) -> str:
|
||||
return path.read_text(errors="ignore")
|
||||
|
||||
|
||||
# ---------- Extractors ----------
|
||||
|
||||
|
||||
def extract_headings(text):
|
||||
return [(level, title.strip()) for level, title in HEADING_REGEX.findall(text)]
|
||||
|
||||
|
||||
def extract_code_blocks(text):
|
||||
"""Line-based fenced code block extractor.
|
||||
|
||||
Handles ``` and ~~~ fences with variable length (CommonMark: closing
|
||||
fence must use same char and be at least as long as opening). Supports
|
||||
nested fences (e.g. an outer 4-backtick block wrapping inner 3-backtick
|
||||
content).
|
||||
"""
|
||||
blocks = []
|
||||
lines = text.split("\n")
|
||||
i = 0
|
||||
n = len(lines)
|
||||
while i < n:
|
||||
m = FENCE_OPEN_REGEX.match(lines[i])
|
||||
if not m:
|
||||
i += 1
|
||||
continue
|
||||
fence_char = m.group(2)[0]
|
||||
fence_len = len(m.group(2))
|
||||
open_line = lines[i]
|
||||
block_lines = [open_line]
|
||||
i += 1
|
||||
closed = False
|
||||
while i < n:
|
||||
close_m = FENCE_OPEN_REGEX.match(lines[i])
|
||||
if (
|
||||
close_m
|
||||
and close_m.group(2)[0] == fence_char
|
||||
and len(close_m.group(2)) >= fence_len
|
||||
and close_m.group(3).strip() == ""
|
||||
):
|
||||
block_lines.append(lines[i])
|
||||
closed = True
|
||||
i += 1
|
||||
break
|
||||
block_lines.append(lines[i])
|
||||
i += 1
|
||||
if closed:
|
||||
blocks.append("\n".join(block_lines))
|
||||
# Unclosed fences are silently skipped — they indicate malformed markdown
|
||||
# and including them would cause false-positive validation failures.
|
||||
return blocks
|
||||
|
||||
|
||||
def extract_urls(text):
|
||||
return set(URL_REGEX.findall(text))
|
||||
|
||||
|
||||
def extract_paths(text):
|
||||
return set(PATH_REGEX.findall(text))
|
||||
|
||||
|
||||
def count_bullets(text):
|
||||
return len(BULLET_REGEX.findall(text))
|
||||
|
||||
|
||||
def extract_inline_codes(text):
|
||||
text_without_fences = re.sub(r"^```[\s\S]*?^```", "", text, flags=re.MULTILINE)
|
||||
text_without_fences = re.sub(r"^~~~[\s\S]*?^~~~", "", text_without_fences, flags=re.MULTILINE)
|
||||
return re.findall(r"`([^`]+)`", text_without_fences)
|
||||
|
||||
|
||||
# ---------- Validators ----------
|
||||
|
||||
|
||||
def validate_headings(orig, comp, result):
|
||||
h1 = extract_headings(orig)
|
||||
h2 = extract_headings(comp)
|
||||
|
||||
if len(h1) != len(h2):
|
||||
result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
|
||||
|
||||
if h1 != h2:
|
||||
result.add_warning("Heading text/order changed")
|
||||
|
||||
|
||||
def validate_code_blocks(orig, comp, result):
|
||||
c1 = extract_code_blocks(orig)
|
||||
c2 = extract_code_blocks(comp)
|
||||
|
||||
if c1 != c2:
|
||||
result.add_error("Code blocks not preserved exactly")
|
||||
|
||||
|
||||
def validate_urls(orig, comp, result):
|
||||
u1 = extract_urls(orig)
|
||||
u2 = extract_urls(comp)
|
||||
|
||||
if u1 != u2:
|
||||
result.add_error(f"URL mismatch: lost={u1 - u2}, added={u2 - u1}")
|
||||
|
||||
|
||||
def validate_paths(orig, comp, result):
|
||||
p1 = extract_paths(orig)
|
||||
p2 = extract_paths(comp)
|
||||
|
||||
if p1 != p2:
|
||||
result.add_warning(f"Path mismatch: lost={p1 - p2}, added={p2 - p1}")
|
||||
|
||||
|
||||
def validate_bullets(orig, comp, result):
|
||||
b1 = count_bullets(orig)
|
||||
b2 = count_bullets(comp)
|
||||
|
||||
if b1 == 0:
|
||||
return
|
||||
|
||||
diff = abs(b1 - b2) / b1
|
||||
|
||||
if diff > 0.15:
|
||||
result.add_warning(f"Bullet count changed too much: {b1} -> {b2}")
|
||||
|
||||
|
||||
def validate_inline_codes(orig, comp, result):
|
||||
c1 = Counter(extract_inline_codes(orig))
|
||||
c2 = Counter(extract_inline_codes(comp))
|
||||
|
||||
if c1 != c2:
|
||||
lost = set(c1.keys()) - set(c2.keys())
|
||||
added = set(c2.keys()) - set(c1.keys())
|
||||
for code, count in c1.items():
|
||||
if code in c2 and c2[code] < count:
|
||||
lost.add(f"{code} (lost {count - c2[code]} of {count} occurrences)")
|
||||
if lost:
|
||||
result.add_error(f"Inline code lost: {lost}")
|
||||
if added:
|
||||
result.add_warning(f"Inline code added: {added}")
|
||||
|
||||
|
||||
# ---------- Main ----------
|
||||
|
||||
|
||||
def validate(original_path: Path, compressed_path: Path) -> ValidationResult:
|
||||
result = ValidationResult()
|
||||
|
||||
orig = read_file(original_path)
|
||||
comp = read_file(compressed_path)
|
||||
|
||||
validate_headings(orig, comp, result)
|
||||
validate_code_blocks(orig, comp, result)
|
||||
validate_urls(orig, comp, result)
|
||||
validate_paths(orig, comp, result)
|
||||
validate_bullets(orig, comp, result)
|
||||
validate_inline_codes(orig, comp, result)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------- CLI ----------
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) != 3:
|
||||
print("Usage: python validate.py <original> <compressed>")
|
||||
sys.exit(1)
|
||||
|
||||
orig = Path(sys.argv[1]).resolve()
|
||||
comp = Path(sys.argv[2]).resolve()
|
||||
|
||||
res = validate(orig, comp)
|
||||
|
||||
print(f"\nValid: {res.is_valid}")
|
||||
|
||||
if res.errors:
|
||||
print("\nErrors:")
|
||||
for e in res.errors:
|
||||
print(f" - {e}")
|
||||
|
||||
if res.warnings:
|
||||
print("\nWarnings:")
|
||||
for w in res.warnings:
|
||||
print(f" - {w}")
|
||||
38
.agents/skills/caveman-help/README.md
Normal file
38
.agents/skills/caveman-help/README.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# caveman-help
|
||||
|
||||
Quick-reference card. One shot, no mode change.
|
||||
|
||||
## What it does
|
||||
|
||||
Prints a cheat sheet of all caveman modes, sibling skills, deactivation triggers, and how to set the default mode via env var or config file. One-shot display — does not flip the active mode, write flag files, or persist anything. Use when you forget the slash commands.
|
||||
|
||||
## How to invoke
|
||||
|
||||
```
|
||||
/caveman-help
|
||||
```
|
||||
|
||||
Also triggers on "caveman help", "what caveman commands", "how do I use caveman".
|
||||
|
||||
## Example output
|
||||
|
||||
```
|
||||
Modes:
|
||||
/caveman full (default)
|
||||
/caveman lite lighter
|
||||
/caveman ultra extreme
|
||||
/caveman wenyan classical Chinese
|
||||
|
||||
Skills:
|
||||
/caveman-commit terse Conventional Commits
|
||||
/caveman-review one-line PR comments
|
||||
/caveman-stats session token savings
|
||||
|
||||
Deactivate:
|
||||
"stop caveman" or "normal mode"
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [`SKILL.md`](./SKILL.md) — full reference card
|
||||
- [Caveman README](../../README.md) — repo overview
|
||||
63
.agents/skills/caveman-help/SKILL.md
Normal file
63
.agents/skills/caveman-help/SKILL.md
Normal file
@@ -0,0 +1,63 @@
|
||||
---
|
||||
name: caveman-help
|
||||
description: >
|
||||
Quick-reference card for all caveman modes, skills, and commands.
|
||||
One-shot display, not a persistent mode. Trigger: /caveman-help,
|
||||
"caveman help", "what caveman commands", "how do I use caveman".
|
||||
---
|
||||
|
||||
# Caveman Help
|
||||
|
||||
Display this reference card when invoked. One-shot — do NOT change mode, write flag files, or persist anything. Output in caveman style.
|
||||
|
||||
## Modes
|
||||
|
||||
| Mode | Trigger | What change |
|
||||
|------|---------|-------------|
|
||||
| **Lite** | `/caveman lite` | Drop filler. Keep sentence structure. |
|
||||
| **Full** | `/caveman` | Drop articles, filler, pleasantries, hedging. Fragments OK. Default. |
|
||||
| **Ultra** | `/caveman ultra` | Extreme compression. Bare fragments. Tables over prose. |
|
||||
| **Wenyan-Lite** | `/caveman wenyan-lite` | Classical Chinese style, light compression. |
|
||||
| **Wenyan-Full** | `/caveman wenyan` | Full 文言文. Maximum classical terseness. |
|
||||
| **Wenyan-Ultra** | `/caveman wenyan-ultra` | Extreme. Ancient scholar on a budget. |
|
||||
|
||||
Mode stick until changed or session end.
|
||||
|
||||
## Skills
|
||||
|
||||
| Skill | Trigger | What it do |
|
||||
|-------|---------|-----------|
|
||||
| **caveman-commit** | `/caveman-commit` | Terse commit messages. Conventional Commits. ≤50 char subject. |
|
||||
| **caveman-review** | `/caveman-review` | One-line PR comments: `L42: bug: user null. Add guard.` |
|
||||
| **caveman-compress** | `/caveman-compress <file>` | Compress .md files to caveman prose. Saves ~46% input tokens. |
|
||||
| **caveman-help** | `/caveman-help` | This card. |
|
||||
|
||||
## Deactivate
|
||||
|
||||
Say "stop caveman" or "normal mode". Resume anytime with `/caveman`.
|
||||
|
||||
## Language
|
||||
|
||||
Keep user's language by default. User write Portuguese → reply Portuguese caveman. Compress the style, not the language. Technical terms, code, commands, commit types, and exact error strings stay verbatim unless user ask for translation.
|
||||
|
||||
## Configure Default Mode
|
||||
|
||||
Default mode = `full`. Change it:
|
||||
|
||||
**Environment variable** (highest priority):
|
||||
```bash
|
||||
export CAVEMAN_DEFAULT_MODE=ultra
|
||||
```
|
||||
|
||||
**Config file** (`~/.config/caveman/config.json`):
|
||||
```json
|
||||
{ "defaultMode": "lite" }
|
||||
```
|
||||
|
||||
Set `"off"` to disable auto-activation on session start. User can still activate manually with `/caveman`.
|
||||
|
||||
Resolution: env var > config file > `full`.
|
||||
|
||||
## More
|
||||
|
||||
Full docs: https://github.com/JuliusBrussee/caveman
|
||||
33
.agents/skills/caveman-review/README.md
Normal file
33
.agents/skills/caveman-review/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# caveman-review
|
||||
|
||||
One-line PR comments. Location, problem, fix. No throat-clearing.
|
||||
|
||||
## What it does
|
||||
|
||||
Generates code review comments in `L<line>: <severity> <problem>. <fix>.` format. One line per finding. Severity emoji: 🔴 bug, 🟡 risk, 🔵 nit, ❓ question. Drops "I noticed that...", hedging, and restating what the diff already shows. Keeps exact line numbers, backticked symbols, and concrete fixes.
|
||||
|
||||
Auto-clarity: drops terse mode for CVE-class security findings, architectural disagreements, and onboarding contexts where the author needs the *why*. Resumes terse for the rest.
|
||||
|
||||
Output only — does not approve, request changes, or run linters.
|
||||
|
||||
## How to invoke
|
||||
|
||||
```
|
||||
/caveman-review
|
||||
```
|
||||
|
||||
Also triggers on "review this PR", "code review", "review the diff".
|
||||
|
||||
## Example output
|
||||
|
||||
```
|
||||
L42: 🔴 bug: user can be null after .find(). Add guard before .email.
|
||||
L88-140: 🔵 nit: 50-line fn does 4 things. Extract validate/normalize/persist.
|
||||
L23: 🟡 risk: no retry on 429. Wrap in withBackoff(3).
|
||||
L107: ❓ q: why drop the cache here? Reads on next request will miss.
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
|
||||
- [Caveman README](../../README.md) — repo overview
|
||||
55
.agents/skills/caveman-review/SKILL.md
Normal file
55
.agents/skills/caveman-review/SKILL.md
Normal file
@@ -0,0 +1,55 @@
|
||||
---
|
||||
name: caveman-review
|
||||
description: >
|
||||
Ultra-compressed code review comments. Cuts noise from PR feedback while preserving
|
||||
the actionable signal. Each comment is one line: location, problem, fix. Use when user
|
||||
says "review this PR", "code review", "review the diff", "/review", or invokes
|
||||
/caveman-review. Auto-triggers when reviewing pull requests.
|
||||
---
|
||||
|
||||
Write code review comments terse and actionable. One line per finding. Location, problem, fix. No throat-clearing.
|
||||
|
||||
## Rules
|
||||
|
||||
**Format:** `L<line>: <problem>. <fix>.` — or `<file>:L<line>: ...` when reviewing multi-file diffs.
|
||||
|
||||
**Severity prefix (optional, when mixed):**
|
||||
- `🔴 bug:` — broken behavior, will cause incident
|
||||
- `🟡 risk:` — works but fragile (race, missing null check, swallowed error)
|
||||
- `🔵 nit:` — style, naming, micro-optim. Author can ignore
|
||||
- `❓ q:` — genuine question, not a suggestion
|
||||
|
||||
**Drop:**
|
||||
- "I noticed that...", "It seems like...", "You might want to consider..."
|
||||
- "This is just a suggestion but..." — use `nit:` instead
|
||||
- "Great work!", "Looks good overall but..." — say it once at the top, not per comment
|
||||
- Restating what the line does — the reviewer can read the diff
|
||||
- Hedging ("perhaps", "maybe", "I think") — if unsure use `q:`
|
||||
|
||||
**Keep:**
|
||||
- Exact line numbers
|
||||
- Exact symbol/function/variable names in backticks
|
||||
- Concrete fix, not "consider refactoring this"
|
||||
- The *why* if the fix isn't obvious from the problem statement
|
||||
|
||||
## Examples
|
||||
|
||||
❌ "I noticed that on line 42 you're not checking if the user object is null before accessing the email property. This could potentially cause a crash if the user is not found in the database. You might want to add a null check here."
|
||||
|
||||
✅ `L42: 🔴 bug: user can be null after .find(). Add guard before .email.`
|
||||
|
||||
❌ "It looks like this function is doing a lot of things and might benefit from being broken up into smaller functions for readability."
|
||||
|
||||
✅ `L88-140: 🔵 nit: 50-line fn does 4 things. Extract validate/normalize/persist.`
|
||||
|
||||
❌ "Have you considered what happens if the API returns a 429? I think we should probably handle that case."
|
||||
|
||||
✅ `L23: 🟡 risk: no retry on 429. Wrap in withBackoff(3).`
|
||||
|
||||
## Auto-Clarity
|
||||
|
||||
Drop terse mode for: security findings (CVE-class bugs need full explanation + reference), architectural disagreements (need rationale, not just a one-liner), and onboarding contexts where the author is new and needs the "why". In those cases write a normal paragraph, then resume terse for the rest.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Reviews only — does not write the code fix, does not approve/request-changes, does not run linters. Output the comment(s) ready to paste into the PR. "stop caveman-review" or "normal mode": revert to verbose review style.
|
||||
30
.agents/skills/caveman-stats/README.md
Normal file
30
.agents/skills/caveman-stats/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# caveman-stats
|
||||
|
||||
Real session token receipts. No AI estimation.
|
||||
|
||||
## What it does
|
||||
|
||||
Reads the current Claude Code session log directly and reports actual input/output token usage plus estimated savings versus a non-caveman baseline. Numbers come from the JSONL session log on disk — the model itself does not compute or estimate them. Output is injected by the `caveman-mode-tracker` hook, which intercepts `/caveman-stats` and returns the formatted stats as a blocked-decision reason.
|
||||
|
||||
Each run also writes a lifetime-savings suffix file used by the statusline badge (`⛏ 12.4k`).
|
||||
|
||||
## How to invoke
|
||||
|
||||
```
|
||||
/caveman-stats
|
||||
```
|
||||
|
||||
## Example output
|
||||
|
||||
```
|
||||
Session: 47 turns
|
||||
Input: 12,304 tokens
|
||||
Output: 3,891 tokens (caveman)
|
||||
Baseline: 11,247 tokens (estimated without caveman)
|
||||
Saved: 7,356 tokens (~65%)
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [`SKILL.md`](./SKILL.md) — hook contract and mechanics
|
||||
- [Caveman README](../../README.md) — repo overview
|
||||
10
.agents/skills/caveman-stats/SKILL.md
Normal file
10
.agents/skills/caveman-stats/SKILL.md
Normal file
@@ -0,0 +1,10 @@
|
||||
---
|
||||
name: caveman-stats
|
||||
description: >
|
||||
Show real token usage and estimated savings for the current session.
|
||||
Reads directly from the Claude Code session log — no AI estimation.
|
||||
Triggers on /caveman-stats. Output is injected by the mode-tracker hook;
|
||||
the model itself does not compute the numbers.
|
||||
---
|
||||
|
||||
This skill is delivered by `hooks/caveman-stats.js` (read by `hooks/caveman-mode-tracker.js` on `/caveman-stats`). The model does not need to do anything when this skill fires — the hook returns `decision: "block"` with the formatted stats as the reason. The user sees the numbers immediately.
|
||||
48
.agents/skills/caveman/README.md
Normal file
48
.agents/skills/caveman/README.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# caveman
|
||||
|
||||
Talk like smart caveman. Same brain, fewer tokens.
|
||||
|
||||
## What it does
|
||||
|
||||
Compress every model response to caveman-style prose. Drops articles, filler, pleasantries, and hedging. Keeps every technical detail, code block, error string, and symbol exact. Cuts 65% of output tokens (measured) with full accuracy preserved. Mode persists for the whole session until changed or stopped.
|
||||
|
||||
Six intensity levels:
|
||||
|
||||
| Level | What change |
|
||||
|-------|-------------|
|
||||
| `lite` | Drop filler/hedging. Sentences stay full. Professional but tight. |
|
||||
| `full` | Default. Drop articles, fragments OK, short synonyms. |
|
||||
| `ultra` | Bare fragments. Abbreviations (DB, auth, fn). Arrows for causality. |
|
||||
| `wenyan-lite` | Classical Chinese register, light compression. |
|
||||
| `wenyan-full` | Maximum 文言文. 80-90% character reduction. |
|
||||
| `wenyan-ultra` | Extreme classical compression. |
|
||||
|
||||
Auto-clarity rule: caveman drops to normal prose for security warnings, irreversible-action confirmations, multi-step sequences where fragment ambiguity risks misread, and when user repeats a question. Resumes after the clear part.
|
||||
|
||||
## How to invoke
|
||||
|
||||
```
|
||||
/caveman # full mode (default)
|
||||
/caveman lite # lighter compression
|
||||
/caveman ultra # extreme compression
|
||||
/caveman wenyan # classical Chinese
|
||||
stop caveman # back to normal prose
|
||||
```
|
||||
|
||||
## Example output
|
||||
|
||||
Question: "Why does my React component re-render?"
|
||||
|
||||
Normal prose:
|
||||
> Your component re-renders because you create a new object reference each render. Wrapping it in `useMemo` will fix the issue.
|
||||
|
||||
Caveman (full):
|
||||
> New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`.
|
||||
|
||||
Caveman (ultra):
|
||||
> Inline obj prop → new ref → re-render. `useMemo`.
|
||||
|
||||
## See also
|
||||
|
||||
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
|
||||
- [Caveman README](../../README.md) — repo overview, install, benchmarks
|
||||
78
.agents/skills/caveman/SKILL.md
Normal file
78
.agents/skills/caveman/SKILL.md
Normal file
@@ -0,0 +1,78 @@
|
||||
---
|
||||
name: caveman
|
||||
description: >
|
||||
Ultra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman
|
||||
while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra,
|
||||
wenyan-lite, wenyan-full, wenyan-ultra.
|
||||
Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens",
|
||||
"be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested.
|
||||
---
|
||||
|
||||
Respond terse like smart caveman. All technical substance stay. Only fluff die.
|
||||
|
||||
## Persistence
|
||||
|
||||
ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode".
|
||||
|
||||
Default: **full**. Switch: `/caveman lite|full|ultra`.
|
||||
|
||||
## Rules
|
||||
|
||||
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). No tool-call narration, no decorative tables/emoji, no dumping long raw error logs unless asked — quote shortest decisive line. Standard well-known tech acronyms OK (DB/API/HTTP); never invent new abbreviations (cfg/impl/req/res/fn) — tokenizer split them same as full word: zero token saved, reader still decode. Full word cheaper AND clearer. No causal arrows (→) either — own token, save nothing. Technical terms exact. Code blocks unchanged. Errors quoted exact.
|
||||
|
||||
Preserve user's dominant language. User write Portuguese → reply Portuguese caveman. User write Spanish → reply Spanish caveman. Compress the style, not the language. No forced English openings or status phrases. ALWAYS keep technical terms, code, API names, CLI commands, commit-type keywords (feat/fix/...), and exact error strings verbatim — unless user explicitly ask for translation.
|
||||
|
||||
No self-reference. Never name or announce the style. No "caveman mode on", "me caveman think", no third-person caveman tags. Output caveman-only — never normal answer plus "Caveman:" recap. Exception: user explicitly ask what the mode is.
|
||||
|
||||
Pattern: `[thing] [action] [reason]. [next step].`
|
||||
|
||||
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
|
||||
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
|
||||
|
||||
## Intensity
|
||||
|
||||
| Level | What change |
|
||||
|-------|------------|
|
||||
| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |
|
||||
| **full** | Drop articles, fragments OK, short synonyms. Classic caveman. No tool-call narration, no decorative tables/emoji, no long raw error-log dumps unless asked. Standard acronyms OK; no invented abbreviations |
|
||||
| **ultra** | Strip conjunctions when cause-then-effect stay unambiguous. One word when one word enough. State each fact once. NO prose abbreviations (cfg/impl/req/res/fn/auth), NO arrows (X → Y) — measured zero token saving under tokenizer, cost decode clarity. Code symbols, function names, API names, error strings: never touch |
|
||||
| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |
|
||||
| **wenyan-full** | Maximum classical terseness. Fully 文言文. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (之/乃/為/其) |
|
||||
| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |
|
||||
|
||||
Example — "Why React component re-render?"
|
||||
- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`."
|
||||
- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`."
|
||||
- ultra: "Inline obj prop, new ref, re-render. `useMemo`."
|
||||
- wenyan-lite: "組件頻重繪,以每繪新生對象參照故。以 useMemo 包之。"
|
||||
- wenyan-full: "每繪新生對象參照,故重繪;以 useMemo 包之則免。"
|
||||
- wenyan-ultra: "新參照則重繪。useMemo 包之。"
|
||||
|
||||
Example — "Explain database connection pooling."
|
||||
- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead."
|
||||
- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead."
|
||||
- ultra: "Pool reuse open DB connections. No per-request handshake."
|
||||
- wenyan-full: "池蓄已開之連,不逐請而新開,省握手之費。"
|
||||
- wenyan-ultra: "池蓄連,免逐請新開,省握手。"
|
||||
|
||||
## Auto-Clarity
|
||||
|
||||
Drop caveman when:
|
||||
- Security warnings
|
||||
- Irreversible action confirmations
|
||||
- Multi-step sequences where fragment order or omitted conjunctions risk misread
|
||||
- Compression itself creates technical ambiguity (e.g., `"migrate table drop column backup first"` — order unclear without articles/conjunctions)
|
||||
- User asks to clarify or repeats question
|
||||
|
||||
Resume caveman after clear part done.
|
||||
|
||||
Example — destructive op:
|
||||
> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.
|
||||
> ```sql
|
||||
> DROP TABLE users;
|
||||
> ```
|
||||
> Caveman resume. Verify backup exist first.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.
|
||||
74
.env.example
74
.env.example
@@ -1,4 +1,4 @@
|
||||
# 企微群机器人 webhook(早报推送,与 bot API 模式凭证不同)
|
||||
# 企微群机器人 webhook(早报推送)
|
||||
WECOM_WEBHOOK_KEY=your-webhook-key
|
||||
|
||||
# 早报内容
|
||||
@@ -8,6 +8,10 @@ DAILY_GITHUB_TRENDING_LIMIT=10
|
||||
DAILY_GITHUB_EMERGING_LIMIT=10
|
||||
DAILY_GITHUB_TOPIC_LIMIT=10
|
||||
|
||||
# 企微版可选行(默认均开)
|
||||
# DAILY_WECOM_TOP_LINE=1 # 顶部「今日主题」行,无评分命中时回退主题名
|
||||
# DAILY_FEATURED_REASON=1 # 首推下「为什么值得点开」理由行
|
||||
|
||||
# GitHub Trending
|
||||
GITHUB_TRENDING_SINCE=daily
|
||||
# GITHUB_TRENDING_LANGUAGE=python
|
||||
@@ -25,14 +29,23 @@ GITHUB_TRENDING_SINCE=daily
|
||||
# GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxx
|
||||
# GITHUB_API_ENRICH=1
|
||||
|
||||
# 企微短版(各榜 Top N,默认 10)
|
||||
DAILY_WECOM_TRENDING=10
|
||||
DAILY_WECOM_HOT=10
|
||||
DAILY_WECOM_GITHUB_TRENDING=10
|
||||
DAILY_WECOM_GITHUB_EMERGING=10
|
||||
DAILY_WECOM_GITHUB_TOPIC=10
|
||||
# 企微短版(各榜 Top N,默认 5;周内不重复见 DAILY_BOARD_DEDUP_DAYS)
|
||||
DAILY_WECOM_TRENDING=5
|
||||
DAILY_WECOM_HOT=5
|
||||
DAILY_WECOM_GITHUB_TRENDING=5
|
||||
DAILY_WECOM_GITHUB_EMERGING=5
|
||||
DAILY_WECOM_GITHUB_TOPIC=5
|
||||
DAILY_WECOM_AI_NEWS=10
|
||||
DAILY_WECOM_CN_AI_NEWS=8
|
||||
# research 模式额外技术类时讯条数(叠加在 AI 时讯精选之上)
|
||||
DAILY_WECOM_AI_NEWS_TECH=5
|
||||
# research 主列表国内最少条数;0=按约 30% 推算(10→3)
|
||||
# DAILY_WECOM_AI_NEWS_CN_MIN=3
|
||||
# research 去重后候选池(独立事件数;默认展示×2;0=自动)
|
||||
# DAILY_AI_NEWS_RESEARCH_POOL=20
|
||||
# DAILY_AI_NEWS_RESEARCH_TECH_POOL=10
|
||||
DAILY_WECOM_CN_AI_NEWS=10
|
||||
# 企微新闻摘要字数(句读/词边界截断,不加省略号)
|
||||
# DAILY_WECOM_NEWS_DESC_LIMIT=72
|
||||
# 企微 Skills 合并前扫描池大小(同 source 合并后仍凑满 Top N)
|
||||
# DAILY_WECOM_SKILL_POOL=200
|
||||
|
||||
@@ -42,10 +55,40 @@ DAILY_WECOM_CN_AI_NEWS=8
|
||||
DAILY_WECOM_CHUNK_BYTES=4096
|
||||
# DAILY_WECOM_MAX_PARTS=5
|
||||
|
||||
# 企微列表模式:delta=仅展示新入榜 | full=全量 Top 榜(回退)
|
||||
DAILY_WECOM_MODE=delta
|
||||
# Delta 模式下新入榜优先,不足时用当日 Top 榜补满各区块条数(0=仅展示变化)
|
||||
# 补榜时会排除近 N 天 baseline 已出现过的条目,避免周内重复(默认 7 天)
|
||||
DAILY_WECOM_DELTA_PAD=1
|
||||
# 补榜时从更大候选池选取(默认展示条数×5,至少 50)
|
||||
# DAILY_WECOM_PAD_POOL=50
|
||||
# 无历史 data.json 时:full=首日全量一次 | empty=列表为空
|
||||
DAILY_DELTA_BASELINE_FALLBACK=full
|
||||
# 推送闸门不满足时跳过 webhook(仍写 output)
|
||||
DAILY_SKIP_PUSH_WHEN_SILENT=1
|
||||
# DAILY_FORCE_PUSH=1
|
||||
# 已推送新闻 link 去重天数
|
||||
DAILY_NEWS_DEDUP_DAYS=7
|
||||
|
||||
# 常驻调度(python -m daily schedule)
|
||||
DAILY_SCHEDULE_TZ=Asia/Shanghai
|
||||
DAILY_SCHEDULE_GENERATE_AT=08:50
|
||||
DAILY_SCHEDULE_PUSH_AT=09:00
|
||||
# 仅工作日生成/推送(法定节假日、周末跳过;调休补班日照常)
|
||||
# 节假日数据取自 xiaoai.me,缓存于 .cache/holidays-<year>.json,每年首次自动获取一次
|
||||
DAILY_WORKDAY_ONLY=1
|
||||
|
||||
# 编辑指定今日首推(可选):关键词,或 关键词|URL
|
||||
# Python Step 0 检索 → featured.json;Agent / classic 企微「今日首推」优先使用
|
||||
# DAILY_FEATURED_PICK=gstack
|
||||
# DAILY_FEATURED_PICK=gstack|https://github.com/you/gstack
|
||||
|
||||
# 国际 AI 时讯(RSS,见 daily/news/feeds.py)
|
||||
DAILY_AI_NEWS=1
|
||||
# 国内 AI 时讯(RSS,见 daily/news/feeds_cn.py)
|
||||
# 国内 AI 时讯(RSS,见 daily/news/feeds_cn.py;research 模式下忽略)
|
||||
DAILY_CN_AI_NEWS=1
|
||||
# AI 时讯来源:rss=RSS 抓取 | research=Cursor SDK + deep-research(WebSearch)
|
||||
# DAILY_AI_NEWS_MODE=research
|
||||
# 英文描述 → 简短中文(DAILY_CURSOR_EDITOR=0 时生效)
|
||||
# DAILY_ZH_DESC=1
|
||||
# DAILY_ZH_DESC_BATCH=20
|
||||
@@ -69,10 +112,21 @@ DAILY_CN_AI_NEWS=1
|
||||
# DAILY_DELTA_LOOKBACK_DAYS=7
|
||||
# DAILY_FULL_DESC_LIMIT=0
|
||||
# DAILY_FULL_NEWS_SUMMARY_LIMIT=0
|
||||
DAILY_AI_NEWS_HOURS=72
|
||||
DAILY_AI_NEWS_HOURS=24
|
||||
DAILY_AI_NEWS_PER_FEED=3
|
||||
DAILY_AI_NEWS_PER_CATEGORY=5
|
||||
|
||||
# 多样性 / 去重(见 docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md)
|
||||
DAILY_BOARD_DEDUP_DAYS=7
|
||||
# DAILY_BOARD_POOL_SIZE=50
|
||||
# DAILY_FEATURED_DEDUP_DAYS=30
|
||||
# DAILY_THEME_BAN_DAYS=7
|
||||
# DAILY_NARRATIVE_AXIS_DAYS=3
|
||||
DAILY_NEWS_BACKFILL=0
|
||||
# 国际时讯:在 24h 滚动窗口基础上,不早于今日 0 点(DAILY_AI_NEWS_TZ)
|
||||
DAILY_AI_NEWS_FLOOR_TODAY=1
|
||||
# DAILY_AI_NEWS_TZ=Asia/Shanghai
|
||||
|
||||
# Reddit RSS(403/429 时在 Reddit 偏好设置 → RSS feeds 复制 user / feed 参数)
|
||||
# REDDIT_RSS_USER=your_username
|
||||
# REDDIT_RSS_FEED=your_feed_token
|
||||
|
||||
8
.gitignore
vendored
8
.gitignore
vendored
@@ -2,7 +2,7 @@
|
||||
.env.local
|
||||
logs/
|
||||
.cache/
|
||||
bot/.env
|
||||
bot/.venv/
|
||||
bot/.cache/
|
||||
output
|
||||
output
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.cursor/
|
||||
202
README.md
202
README.md
@@ -1,6 +1,6 @@
|
||||
# skills-hot-daily
|
||||
|
||||
Skills / GitHub 早报推送 + 企微对话机器人(同一仓库、两套企微接入)。
|
||||
Skills / GitHub 早报推送(企微 Webhook)。
|
||||
|
||||
## 项目结构
|
||||
|
||||
@@ -8,7 +8,7 @@ Skills / GitHub 早报推送 + 企微对话机器人(同一仓库、两套企
|
||||
skills-hot-daily/
|
||||
├── README.md
|
||||
├── .env.example # 早报 webhook、GitHub 等
|
||||
├── requirements.txt # 早报 Python 依赖
|
||||
├── requirements.txt # Python 依赖
|
||||
├── run-daily.ps1 # 生成 + 推送一条龙
|
||||
├── send-wecom.ps1 # 仅推送
|
||||
├── daily/ # 早报 Python 包
|
||||
@@ -32,24 +32,15 @@ skills-hot-daily/
|
||||
├── skills/daily-editor/ # 早报 Cursor 编辑 Skill
|
||||
│ └── SKILL.md
|
||||
├── logs/
|
||||
├── .cache/
|
||||
└── bot/ # 企微 API 模式对话机器人(独立 venv)
|
||||
├── main.py
|
||||
├── skills_service.py
|
||||
└── scenarios/
|
||||
└── .cache/
|
||||
```
|
||||
|
||||
| 模块 | 配置文件 | 启动方式 |
|
||||
|------|----------|----------|
|
||||
| **早报推送** | 根目录 `.env`(`WECOM_WEBHOOK_KEY` 等) | `.\run-daily.ps1` |
|
||||
| **对话 Bot** | `bot/.env`(`WECOM_BOT_ID` / `SECRET` 等) | `cd bot` → `python main.py` |
|
||||
|
||||
---
|
||||
|
||||
## 一、早报推送
|
||||
## 早报推送
|
||||
|
||||
```powershell
|
||||
cd d:\LY\test\tech\skills-hot-daily
|
||||
cd d:\LY\diy\daily-robots
|
||||
pip install -r requirements.txt
|
||||
copy .env.example .env
|
||||
.\run-daily.ps1
|
||||
@@ -84,15 +75,14 @@ WECOM_WEBHOOK_KEY=your-key
|
||||
| 研究 / 论文 | arXiv cs.CL/AI/LG、HF Papers |
|
||||
| 社区讨论 | HN、Reddit r/LocalLLaMA / ClaudeAI / ML 等 |
|
||||
|
||||
环境变量:`DAILY_AI_NEWS=1` · `DAILY_CN_AI_NEWS=1` · `DAILY_AI_NEWS_HOURS=72` · `DAILY_WECOM_AI_NEWS=10` · `DAILY_WECOM_CN_AI_NEWS=8`
|
||||
环境变量:`DAILY_AI_NEWS=1` · `DAILY_CN_AI_NEWS=1` · `DAILY_AI_NEWS_HOURS=24` · `DAILY_WECOM_AI_NEWS=10` · `DAILY_WECOM_CN_AI_NEWS=10`
|
||||
|
||||
**国内 AI 时讯**(RSS,见 `daily/news/feeds_cn.py`):
|
||||
|
||||
| 类别 | 覆盖 |
|
||||
|------|------|
|
||||
| AI 专业媒体 | 量子位、InfoQ 中文 |
|
||||
| AI 专业媒体 | 量子位 |
|
||||
| 综合科技 | 36氪、雷锋网、Google News 中文 |
|
||||
| 开发者社区 | 掘金(标题 AI 关键词过滤) |
|
||||
|
||||
|
||||
### 生成架构(Tier B · Cursor 编辑层)
|
||||
@@ -135,7 +125,9 @@ Python 抓取 → Step1 趋势分析 → Step2 叙事写稿 → Python 分条推
|
||||
|
||||
```env
|
||||
DAILY_REPORT_MODE=agent
|
||||
DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily # 早报 LLM 工作目录(与 bot 的 CURSOR_CWD 独立)
|
||||
CURSOR_API_KEY=cursor_...
|
||||
CURSOR_MODEL=composer-2.5
|
||||
DAILY_CURSOR_CWD=d:\LY\diy\daily-robots
|
||||
```
|
||||
|
||||
| 文件 | 说明 |
|
||||
@@ -146,169 +138,11 @@ DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily # 早报 LLM 工作目录(与 bo
|
||||
- 完整版 `YYYY-MM-DD.md` 仍为数据表格归档;企微版由 Agent 直接写 Markdown
|
||||
- Agent 失败自动回退 `classic`,不影响 `run-daily.ps1`
|
||||
|
||||
定时推送:Windows 任务计划程序或 `/loop 1d` 执行 `run-daily.ps1`。
|
||||
定时推送:
|
||||
|
||||
---
|
||||
|
||||
## 二、可对话 Skills 助手(企业微信智能机器人)
|
||||
|
||||
在企微里 @ 机器人即可:
|
||||
- **快查**:`trending 10`、`hot 10`、`搜索 react`(本地 skills 数据,秒回)
|
||||
- **截图预览**:`preview` / `截图`(基于 `.env` 的 `CURSOR_CWD` 启动前端并发图)
|
||||
- **通用任务**:任意自然语言需求,由 **Cursor Agent** 执行并回传结果
|
||||
|
||||
### 1. 创建 API 模式机器人
|
||||
|
||||
1. [企业微信管理后台](https://work.weixin.qq.com/) → **安全与管理** → **管理工具** → **智能机器人** → **创建机器人**
|
||||
2. 选择 **API 模式创建** → **使用长连接**
|
||||
3. 记录 **Bot ID** 和 **Secret**(Secret 只显示一次,请立即保存)
|
||||
4. 设置可见范围,将机器人 **添加到目标群** 或允许成员单聊
|
||||
|
||||
普通成员路径:工作台 → 智能机器人 → 手动创建 → API 模式 → 长连接
|
||||
|
||||
### 2. 启动本地服务
|
||||
|
||||
```powershell
|
||||
cd d:\LY\test\tech\skills-hot-daily\bot
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium
|
||||
copy .env.example .env
|
||||
# 编辑 .env:WECOM_BOT_ID / WECOM_BOT_SECRET / CURSOR_API_KEY
|
||||
python main.py
|
||||
```
|
||||
|
||||
服务需 **常驻运行**(本机、服务器或 Docker)。长连接模式下机器人进程须在线才能收消息。
|
||||
|
||||
### 3. 路由模式(ROUTING_MODE)
|
||||
|
||||
| 模式 | 行为 |
|
||||
|------|------|
|
||||
| `hybrid`(默认) | `trending`/`hot`/`搜索`/`详情` 走本地快查;其余 @ 消息交给 Cursor |
|
||||
| `cursor` | 所有消息都交给 Cursor 执行 |
|
||||
| `skills` | 仅本地 skills 快查(旧行为) |
|
||||
|
||||
**Cursor 任务示例**(群里发送):
|
||||
|
||||
```
|
||||
@test 总结 trending top10,并推荐 3 个适合前端团队的 skill
|
||||
@test 对比 mattpocock/skills 和 obra/superpowers 各有哪些热门 skill
|
||||
@test 帮我写一段 npx skills add 的安装说明
|
||||
```
|
||||
|
||||
Cursor 在本机 `CURSOR_CWD` 目录下运行,默认 `d:\LY\test\tech`。复杂任务可能需要 1–10 分钟,流式消息会显示「Cursor 正在执行任务…」。
|
||||
|
||||
### 4. 前端截图预览(API 模式发图)
|
||||
|
||||
项目路径读取 `.env` 中的 **`CURSOR_CWD`**。机器人会:
|
||||
|
||||
1. 在 `CURSOR_CWD` 检测 `package.json`,若有 `dev` 脚本则执行 `PREVIEW_DEV_COMMAND`(默认 `npm run dev`)
|
||||
2. 等待 `PREVIEW_PORT`(默认 `5173`)就绪,或用 `PREVIEW_URL` 直接访问
|
||||
3. Playwright 打开页面并截图
|
||||
4. 通过 API 模式 **上传图片 + 回复 image 消息** 到群
|
||||
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `preview` / `截图` / `预览` | 访问 `http://127.0.0.1:5173/` 并截图 |
|
||||
| `preview /login` | 指定路径 |
|
||||
| `preview / 3000` | 指定端口 |
|
||||
| `preview http://127.0.0.1:8080/` | 指定完整 URL |
|
||||
|
||||
**多步网页操作**(登录、点菜单、再截图)见下一节,不再写死在代码里。
|
||||
|
||||
`.env` 可选配置:
|
||||
|
||||
```env
|
||||
CURSOR_CWD=d:\LY\test\tech
|
||||
PREVIEW_PORT=5173
|
||||
PREVIEW_URL=http://127.0.0.1:5173/
|
||||
PREVIEW_DEV_COMMAND=npm run dev
|
||||
PREVIEW_STARTUP_TIMEOUT=120
|
||||
```
|
||||
|
||||
若 `CURSOR_CWD` 下暂无前端项目,可先手动启动 dev server,或设置 `PREVIEW_URL` 指向已运行地址。
|
||||
|
||||
### 4b. 网页操作(Playwright 步骤引擎)
|
||||
|
||||
支持三种方式定义操作流程,**无需改 Python 代码**:
|
||||
|
||||
**1. 自然语言(企微里直接说)**
|
||||
|
||||
```
|
||||
@test 访问登录页,输入账号密码,点击登录后进入主页,点击智能体管理菜单然后截图
|
||||
```
|
||||
|
||||
账号密码从 `.env` 读取(`{{PREVIEW_LOGIN_USER}}` / `{{PREVIEW_LOGIN_PASSWORD}}`),勿在群里发密码。
|
||||
|
||||
**2. 场景文件 YAML**
|
||||
|
||||
`bot/scenarios/xiaobao-agent-manage.yaml` 示例:
|
||||
|
||||
```yaml
|
||||
name: xiaobao-agent-manage
|
||||
steps:
|
||||
- goto: /login
|
||||
- fill:
|
||||
field: 账号
|
||||
value: "{{PREVIEW_LOGIN_USER}}"
|
||||
- fill:
|
||||
field: 密码
|
||||
value: "{{PREVIEW_LOGIN_PASSWORD}}"
|
||||
- click: 登录
|
||||
- wait:
|
||||
url: "**/app/**"
|
||||
- click: 智能体管理
|
||||
- wait: 1500
|
||||
- screenshot
|
||||
```
|
||||
|
||||
触发:`@test browser xiaobao-agent-manage`
|
||||
|
||||
场景搜索路径:`bot/scenarios/`、`CURSOR_CWD/.browser-scenarios/`、环境变量 `BROWSER_SCENARIOS_DIR`。
|
||||
|
||||
**3. 消息内 DSL**
|
||||
|
||||
```
|
||||
browser:
|
||||
goto /login
|
||||
fill 账号 {{PREVIEW_LOGIN_USER}}
|
||||
fill 密码 {{PREVIEW_LOGIN_PASSWORD}}
|
||||
click 登录
|
||||
click 智能体管理
|
||||
screenshot
|
||||
```
|
||||
|
||||
**支持的步骤**:`goto` · `fill` · `click` · `wait` · `screenshot` · `press`
|
||||
|
||||
`.env` 登录与场景配置:
|
||||
|
||||
```env
|
||||
PREVIEW_LOGIN_USER=test_account
|
||||
PREVIEW_LOGIN_PASSWORD=your_password
|
||||
# BROWSER_DEFAULT_SCENARIO=xiaobao-agent-manage
|
||||
```
|
||||
|
||||
### 5. 支持的快查命令
|
||||
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `trending 10` / `趋势 10` | 近期增长榜 Top N(默认 10,最大 30) |
|
||||
| `hot 10` / `实时 10` | 实时热度榜 |
|
||||
| `all 10` / `总榜 10` | 历史总安装榜 |
|
||||
| `搜索 react` / `search tdd` | 关键词搜索 |
|
||||
| `详情 find-skills` | 单个 skill 详情 + 安装命令 |
|
||||
| `preview` / `截图` | 启动 CURSOR_CWD 前端并截图发群 |
|
||||
| `帮助` | 命令列表 |
|
||||
|
||||
自然语言(非显式快查命令)会交给 **Cursor** 处理,例如 `@test 查 trending 并写推荐` 。
|
||||
|
||||
### 6. 本地测试(无需企微凭证)
|
||||
|
||||
```powershell
|
||||
cd d:\LY\test\tech\skills-hot-daily\bot
|
||||
python -c "from skills_service import handle_command; print(handle_command('trending 5'))"
|
||||
```
|
||||
- **常驻调度(推荐)**:`python -m daily schedule` 或 `.\run-scheduler.ps1`(默认 08:50 生成、09:00 推送,见 `DAILY_SCHEDULE_*`)
|
||||
- Windows 任务计划:`.\register-daily-task.ps1`
|
||||
- Cursor:`/loop 1d`(时间会漂移,仅临时用)
|
||||
|
||||
---
|
||||
|
||||
@@ -321,14 +155,6 @@ python -c "from skills_service import handle_command; print(handle_command('tren
|
||||
| **邮件 + 企业微信邮箱** | 已有 SMTP | 中 |
|
||||
| **PushPlus / Server酱** | 个人微信中转 | 低(第三方) |
|
||||
|
||||
### 应用消息 API(简要)
|
||||
|
||||
适合「推送给某个人」而非群聊。需在 [企业微信管理后台](https://work.weixin.qq.com/) 创建自建应用,调用:
|
||||
|
||||
`POST https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=TOKEN`
|
||||
|
||||
消息体支持 `text` / `markdown` / `news` 等。需先 `gettoken` 再发消息,并维护 access_token 缓存。
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
# 企业微信智能机器人(API 模式 · 长连接)
|
||||
# 管理后台 → 安全与管理 → 管理工具 → 智能机器人 → 创建 → API 模式 → 使用长连接
|
||||
WECOM_BOT_ID=your-bot-id
|
||||
WECOM_BOT_SECRET=your-bot-secret
|
||||
|
||||
# Cursor SDK(@ 机器人后的通用任务由 Cursor 执行)
|
||||
CURSOR_API_KEY=cursor_...
|
||||
CURSOR_CWD=d:\LY\test\tech
|
||||
CURSOR_MODEL=composer-2.5
|
||||
CURSOR_TIMEOUT=600
|
||||
|
||||
# 前端截图预览(基于 CURSOR_CWD)
|
||||
PREVIEW_PORT=5173
|
||||
PREVIEW_URL=http://127.0.0.1:5173/
|
||||
# PREVIEW_DEV_COMMAND=npm run dev
|
||||
# PREVIEW_STARTUP_TIMEOUT=120
|
||||
|
||||
# 登录后截图(账号密码只放 .env,切勿发到企微群)
|
||||
# PREVIEW_LOGIN_USER=your_account_or_phone
|
||||
# PREVIEW_LOGIN_PASSWORD=your_password
|
||||
# PREVIEW_AFTER_LOGIN_URL=/app/dashboard
|
||||
# PREVIEW_AUTO_LOGIN=true
|
||||
|
||||
# 网页操作场景目录(可选,默认 bot/scenarios 与 CURSOR_CWD/.browser-scenarios)
|
||||
# BROWSER_SCENARIOS_DIR=d:\path\to\scenarios
|
||||
# BROWSER_DEFAULT_SCENARIO=xiaobao-agent-manage
|
||||
|
||||
# hybrid=快查走本地 / 其余走 Cursor | cursor=全部 Cursor | skills=仅本地
|
||||
ROUTING_MODE=hybrid
|
||||
4
bot/.gitignore
vendored
4
bot/.gitignore
vendored
@@ -1,4 +0,0 @@
|
||||
.cache/
|
||||
.env
|
||||
.venv/
|
||||
.cache/screenshots/
|
||||
@@ -1,12 +0,0 @@
|
||||
"""Bot 内部数据结构。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class RouteResult:
|
||||
source: str
|
||||
text: str
|
||||
image_path: str | None = None
|
||||
@@ -1,49 +0,0 @@
|
||||
"""浏览器场景变量替换与 base URL 解析。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import env_config
|
||||
|
||||
_VAR_PATTERN = re.compile(r"\{\{([A-Z0-9_]+)\}\}")
|
||||
|
||||
|
||||
def interpolate(value: str) -> str:
|
||||
def repl(match: re.Match[str]) -> str:
|
||||
key = match.group(1)
|
||||
resolved = env_config.env(key)
|
||||
if resolved is None:
|
||||
raise RuntimeError(f"场景变量未配置:{key}")
|
||||
return resolved
|
||||
|
||||
return _VAR_PATTERN.sub(repl, value)
|
||||
|
||||
|
||||
def default_base_url() -> str:
|
||||
explicit = (env_config.env("PREVIEW_BASE_URL") or "").strip()
|
||||
if explicit:
|
||||
return interpolate(explicit.rstrip("/"))
|
||||
|
||||
preview = (env_config.env("PREVIEW_URL") or "").strip()
|
||||
if preview:
|
||||
parsed = urlparse(preview)
|
||||
scheme = parsed.scheme or "http"
|
||||
host = parsed.hostname or "127.0.0.1"
|
||||
port = parsed.port
|
||||
if port and port not in (80, 443):
|
||||
return f"{scheme}://{host}:{port}"
|
||||
return f"{scheme}://{host}"
|
||||
|
||||
port = env_config.env("PREVIEW_PORT", "5173") or "5173"
|
||||
return f"http://127.0.0.1:{port}"
|
||||
|
||||
|
||||
def resolve_url(base_url: str, target: str) -> str:
|
||||
target = interpolate(target.strip())
|
||||
if target.startswith("http://") or target.startswith("https://"):
|
||||
return target
|
||||
if not target.startswith("/"):
|
||||
target = "/" + target
|
||||
return base_url.rstrip("/") + target
|
||||
@@ -1,269 +0,0 @@
|
||||
"""通用 Playwright 步骤执行器(不写死业务页面)。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from browser_env import interpolate, resolve_url
|
||||
from browser_models import BrowserResult, BrowserScenario
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCREENSHOT_DIR = Path(__file__).resolve().parent / ".cache" / "screenshots"
|
||||
|
||||
FIELD_HINTS: dict[str, list[str]] = {
|
||||
"账号": [
|
||||
"#login-username",
|
||||
"input#login-username",
|
||||
"input[autocomplete='username']",
|
||||
"username",
|
||||
"account",
|
||||
"phone",
|
||||
"账号",
|
||||
"手机号",
|
||||
"企业账号",
|
||||
],
|
||||
"密码": [
|
||||
"#login-password input",
|
||||
"#login-password",
|
||||
"input#login-password",
|
||||
"input[type='password']",
|
||||
"password",
|
||||
"密码",
|
||||
],
|
||||
"用户名": ["#login-username", "input#login-username", "username", "account", "账号"],
|
||||
}
|
||||
|
||||
def _step_label(step: dict[str, Any], index: int) -> str:
|
||||
action = step.get("action", "?")
|
||||
target = step.get("target") or step.get("field") or step.get("url") or ""
|
||||
return f"{index + 1}. {action} {target}".strip()
|
||||
|
||||
|
||||
def _resolve_fill_locator(page, field: str, step: dict[str, Any]):
|
||||
if step.get("selector"):
|
||||
return page.locator(interpolate(str(step["selector"])))
|
||||
|
||||
field_key = interpolate(str(field))
|
||||
if step.get("label"):
|
||||
return page.get_by_label(interpolate(str(step["label"])), exact=False)
|
||||
if step.get("placeholder"):
|
||||
return page.get_by_placeholder(interpolate(str(step["placeholder"])), exact=False)
|
||||
|
||||
hints = FIELD_HINTS.get(field_key, [field_key])
|
||||
for hint in hints:
|
||||
if hint.startswith("#") or hint.startswith(".") or hint.startswith("["):
|
||||
locator = page.locator(hint)
|
||||
if locator.count() > 0:
|
||||
return locator.first
|
||||
for getter in (
|
||||
lambda h=hint: page.get_by_label(h, exact=False),
|
||||
lambda h=hint: page.get_by_placeholder(h, exact=False),
|
||||
):
|
||||
locator = getter()
|
||||
if locator.count() > 0:
|
||||
return locator.first
|
||||
|
||||
return page.locator("input, textarea").filter(has_text=field_key).first
|
||||
|
||||
|
||||
def _fill_field(page, field: str, step: dict[str, Any]) -> None:
|
||||
value = interpolate(str(step.get("value", "")))
|
||||
locator = _resolve_fill_locator(page, field, step)
|
||||
locator.click(timeout=10_000)
|
||||
locator.fill("", timeout=5_000)
|
||||
locator.fill(value, timeout=10_000)
|
||||
|
||||
|
||||
def _page_error_text(page) -> str | None:
|
||||
for selector in (
|
||||
".ant-message-error",
|
||||
".ant-form-item-explain-error",
|
||||
".ant-alert-error",
|
||||
):
|
||||
try:
|
||||
locator = page.locator(selector).first
|
||||
if locator.is_visible(timeout=300):
|
||||
text = locator.inner_text(timeout=1_000).strip()
|
||||
if text:
|
||||
return text
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _pathname_matches(pattern: str, pathname: str) -> bool:
|
||||
pattern = pattern.strip()
|
||||
if pattern in {"**/app/**", "**/app/*", "/app/**"}:
|
||||
return pathname.startswith("/app")
|
||||
if pattern.endswith("/**"):
|
||||
prefix = pattern[:-3].rstrip("/")
|
||||
if prefix.startswith("**/"):
|
||||
prefix = prefix[3:]
|
||||
if not prefix.startswith("/"):
|
||||
prefix = "/" + prefix
|
||||
return pathname.startswith(prefix)
|
||||
if "**" in pattern or "*" in pattern:
|
||||
regex = "^" + re.escape(pattern).replace(r"\*\*", ".*").replace(r"\*", "[^/]*") + "$"
|
||||
return re.search(regex, pathname) is not None
|
||||
return pathname == pattern or pathname.startswith(pattern)
|
||||
|
||||
|
||||
def _wait_for_url_pattern(page, pattern: str, timeout: int = 60_000) -> None:
|
||||
"""SPA 路由用 pathname 轮询;glob 模式不依赖 navigation 事件。"""
|
||||
deadline = time.monotonic() + timeout / 1000
|
||||
last_error: str | None = None
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
pathname = page.evaluate("() => window.location.pathname")
|
||||
if _pathname_matches(pattern, pathname):
|
||||
try:
|
||||
page.wait_for_load_state("networkidle", timeout=8_000)
|
||||
except Exception:
|
||||
page.wait_for_timeout(800)
|
||||
return
|
||||
|
||||
err = _page_error_text(page)
|
||||
if err and err != last_error:
|
||||
last_error = err
|
||||
logger.warning("页面提示:%s", err)
|
||||
if "/login" in pathname:
|
||||
raise RuntimeError(f"登录失败:{err}")
|
||||
|
||||
page.wait_for_timeout(400)
|
||||
|
||||
err = _page_error_text(page)
|
||||
hint_parts = [f"当前 URL:`{page.url}`"]
|
||||
if err:
|
||||
hint_parts.append(f"页面错误:{err}")
|
||||
elif last_error:
|
||||
hint_parts.append(f"页面错误:{last_error}")
|
||||
hint_parts.append("请确认 PREVIEW_LOGIN_USER/PASSWORD 正确,且登录 API(内网网关)可达。")
|
||||
raise RuntimeError(f"等待 URL 匹配 `{pattern}` 超时({timeout}ms)。{' '.join(hint_parts)}")
|
||||
|
||||
|
||||
def _click_target(page, target: str) -> None:
|
||||
target = interpolate(target.strip())
|
||||
if target.lower() in {"登录", "login"}:
|
||||
for selector in ("button.login-submit", "button[type='submit']"):
|
||||
locator = page.locator(selector)
|
||||
if locator.count() > 0:
|
||||
locator.first.click(timeout=10_000)
|
||||
return
|
||||
|
||||
candidates = [
|
||||
page.get_by_role("menuitem", name=target, exact=True),
|
||||
page.get_by_role("button", name=target, exact=True),
|
||||
page.get_by_role("link", name=target, exact=True),
|
||||
page.get_by_text(target, exact=True),
|
||||
page.get_by_text(target, exact=False),
|
||||
]
|
||||
for locator in candidates:
|
||||
if locator.count() > 0:
|
||||
locator.first.click(timeout=10_000)
|
||||
return
|
||||
raise RuntimeError(f"未找到可点击元素:{target}")
|
||||
|
||||
|
||||
def _execute_step(page, base_url: str, step: dict[str, Any]) -> None:
|
||||
action = str(step.get("action", "")).lower()
|
||||
if action == "goto":
|
||||
target = step.get("target") or step.get("url") or "/"
|
||||
url = resolve_url(base_url, str(target))
|
||||
page.goto(url, wait_until="networkidle", timeout=60_000)
|
||||
return
|
||||
|
||||
if action == "fill":
|
||||
field = str(step.get("field") or step.get("target") or "账号")
|
||||
_fill_field(page, field, step)
|
||||
return
|
||||
|
||||
if action == "click":
|
||||
target = step.get("target") or step.get("text")
|
||||
if not target:
|
||||
raise RuntimeError("click 步骤缺少 target")
|
||||
_click_target(page, str(target))
|
||||
page.wait_for_timeout(800)
|
||||
return
|
||||
|
||||
if action == "wait":
|
||||
timeout = int(step.get("timeout") or 60_000)
|
||||
if step.get("url"):
|
||||
_wait_for_url_pattern(page, str(step["url"]), timeout=timeout)
|
||||
return
|
||||
if step.get("selector"):
|
||||
page.locator(interpolate(str(step["selector"]))).wait_for(timeout=30_000)
|
||||
return
|
||||
if step.get("text"):
|
||||
page.get_by_text(interpolate(str(step["text"])), exact=False).wait_for(timeout=30_000)
|
||||
return
|
||||
ms = int(step.get("ms") or 1500)
|
||||
page.wait_for_timeout(ms)
|
||||
return
|
||||
|
||||
if action == "press":
|
||||
key = str(step.get("key") or step.get("target") or "Enter")
|
||||
page.keyboard.press(key)
|
||||
return
|
||||
|
||||
if action == "screenshot":
|
||||
return
|
||||
|
||||
raise RuntimeError(f"未知步骤 action={action}")
|
||||
|
||||
|
||||
def run_browser_scenario_sync(scenario: BrowserScenario) -> BrowserResult:
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
slug = (scenario.name or "browser").replace(" ", "-")
|
||||
output = SCREENSHOT_DIR / f"{slug}-{stamp}.png"
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
step_log: list[str] = []
|
||||
final_url = scenario.base_url
|
||||
|
||||
with sync_playwright() as playwright:
|
||||
browser = playwright.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 720})
|
||||
|
||||
steps = list(scenario.steps)
|
||||
if steps and steps[-1].get("action") != "screenshot" and not any(
|
||||
s.get("action") == "screenshot" for s in steps
|
||||
):
|
||||
steps.append({"action": "screenshot"})
|
||||
|
||||
for index, step in enumerate(steps):
|
||||
label = _step_label(step, index)
|
||||
logger.info("执行步骤 %s", label)
|
||||
action = str(step.get("action", "")).lower()
|
||||
if action == "screenshot":
|
||||
page.wait_for_timeout(int(step.get("ms") or 1500))
|
||||
page.screenshot(path=str(output), full_page=False, type="png")
|
||||
final_url = page.url
|
||||
step_log.append(label + " ✓")
|
||||
continue
|
||||
try:
|
||||
_execute_step(page, scenario.base_url, step)
|
||||
final_url = page.url
|
||||
step_log.append(label + " ✓")
|
||||
except Exception as exc:
|
||||
err = _page_error_text(page)
|
||||
detail = f"({err})" if err else ""
|
||||
raise RuntimeError(f"步骤失败:{label} @ {page.url}{detail}") from exc
|
||||
|
||||
browser.close()
|
||||
|
||||
return BrowserResult(
|
||||
scenario_name=scenario.name,
|
||||
base_url=scenario.base_url,
|
||||
final_url=final_url,
|
||||
screenshot_path=output,
|
||||
step_count=len(steps),
|
||||
step_log=step_log,
|
||||
)
|
||||
@@ -1,26 +0,0 @@
|
||||
"""浏览器自动化步骤模型。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class BrowserScenario:
|
||||
name: str | None
|
||||
base_url: str
|
||||
steps: list[dict[str, Any]]
|
||||
source: str = "natural"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BrowserResult:
|
||||
scenario_name: str | None
|
||||
base_url: str
|
||||
final_url: str
|
||||
screenshot_path: Path
|
||||
step_count: int
|
||||
started_dev_server: bool = False
|
||||
step_log: list[str] = field(default_factory=list)
|
||||
@@ -1,307 +0,0 @@
|
||||
"""解析自然语言 / YAML / 场景名 → 浏览器步骤。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
import env_config
|
||||
from browser_env import default_base_url, interpolate
|
||||
from browser_models import BrowserScenario
|
||||
|
||||
SCENARIO_DIRS = [
|
||||
Path(__file__).resolve().parent / "scenarios",
|
||||
Path(__file__).resolve().parent.parent / "scenarios",
|
||||
]
|
||||
|
||||
|
||||
def _project_cwd() -> Path:
|
||||
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return Path(raw).resolve()
|
||||
|
||||
|
||||
def _strip_mention(text: str) -> str:
|
||||
return re.sub(r"@\S+\s*", "", text).strip()
|
||||
|
||||
|
||||
def _scenario_search_dirs() -> list[Path]:
|
||||
dirs = list(SCENARIO_DIRS)
|
||||
dirs.append(_project_cwd() / ".browser-scenarios")
|
||||
custom = (env_config.env("BROWSER_SCENARIOS_DIR") or "").strip()
|
||||
if custom:
|
||||
dirs.append(Path(custom).resolve())
|
||||
return dirs
|
||||
|
||||
|
||||
def is_browser_intent(text: str) -> bool:
|
||||
raw = _strip_mention(text)
|
||||
if not raw:
|
||||
return False
|
||||
if re.match(r"^(browser|网页|网页操作|操作)\b", raw, re.IGNORECASE):
|
||||
return True
|
||||
if re.search(r"```(?:yaml|yml)", raw, re.IGNORECASE):
|
||||
return True
|
||||
if re.search(r"(?m)^browser\s*:", raw, re.IGNORECASE):
|
||||
return True
|
||||
|
||||
if re.match(r"^(preview|截图|预览|截屏)\s", raw, re.IGNORECASE):
|
||||
if not re.search(r"[,,。;;]|然后|输入|点击|填写|访问|打开|登录", raw):
|
||||
return False
|
||||
|
||||
if len(_split_segments(text)) >= 2:
|
||||
return True
|
||||
|
||||
verbs = 0
|
||||
for pattern in (r"访问", r"打开", r"输入", r"填写", r"点击", r"点选", r"选择", r"登录"):
|
||||
if re.search(pattern, raw):
|
||||
verbs += 1
|
||||
return verbs >= 2
|
||||
|
||||
|
||||
def _load_yaml_scenario(path: Path) -> BrowserScenario:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise RuntimeError(f"场景文件格式错误:{path}")
|
||||
base_url = interpolate(str(data.get("base_url") or default_base_url()))
|
||||
steps = data.get("steps")
|
||||
if not isinstance(steps, list) or not steps:
|
||||
raise RuntimeError(f"场景缺少 steps:{path}")
|
||||
return BrowserScenario(
|
||||
name=data.get("name") or path.stem,
|
||||
base_url=base_url,
|
||||
steps=_normalize_steps(steps),
|
||||
source=f"file:{path.name}",
|
||||
)
|
||||
|
||||
|
||||
def _find_scenario_file(name: str) -> Path | None:
|
||||
slug = name.strip().replace(" ", "-")
|
||||
for directory in _scenario_search_dirs():
|
||||
for candidate in (directory / f"{slug}.yaml", directory / f"{slug}.yml"):
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_steps(raw_steps: list[Any]) -> list[dict[str, Any]]:
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for item in raw_steps:
|
||||
if isinstance(item, str):
|
||||
normalized.append({"action": item})
|
||||
continue
|
||||
if not isinstance(item, dict) or not item:
|
||||
raise RuntimeError(f"无效步骤:{item!r}")
|
||||
if "action" in item:
|
||||
normalized.append(dict(item))
|
||||
continue
|
||||
if len(item) == 1:
|
||||
action, payload = next(iter(item.items()))
|
||||
step = {"action": action}
|
||||
if payload is not None:
|
||||
if isinstance(payload, dict):
|
||||
step.update(payload)
|
||||
elif action == "wait" and isinstance(payload, int):
|
||||
step["ms"] = payload
|
||||
elif action == "wait" and isinstance(payload, str) and payload.isdigit():
|
||||
step["ms"] = int(payload)
|
||||
else:
|
||||
step["target"] = payload
|
||||
normalized.append(step)
|
||||
continue
|
||||
raise RuntimeError(f"无效步骤:{item!r}")
|
||||
return normalized
|
||||
|
||||
|
||||
def _parse_inline_dsl(text: str) -> BrowserScenario | None:
|
||||
raw = _strip_mention(text)
|
||||
match = re.search(r"(?ms)^browser\s*:\s*\n(.+)$", raw, re.IGNORECASE)
|
||||
if not match:
|
||||
return None
|
||||
|
||||
steps: list[dict[str, Any]] = []
|
||||
for line in match.group(1).splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
line = re.sub(r"^[-*]\s*", "", line)
|
||||
if not line:
|
||||
continue
|
||||
steps.append(_parse_dsl_line(line))
|
||||
|
||||
if not steps:
|
||||
return None
|
||||
return BrowserScenario(
|
||||
name="inline",
|
||||
base_url=default_base_url(),
|
||||
steps=steps,
|
||||
source="inline-dsl",
|
||||
)
|
||||
|
||||
|
||||
def _parse_dsl_line(line: str) -> dict[str, Any]:
|
||||
parts = line.split(None, 2)
|
||||
action = parts[0].lower()
|
||||
if action == "goto":
|
||||
return {"action": "goto", "target": parts[1] if len(parts) > 1 else "/"}
|
||||
if action == "click":
|
||||
return {"action": "click", "target": " ".join(parts[1:])}
|
||||
if action == "fill":
|
||||
if len(parts) < 3:
|
||||
raise RuntimeError(f"fill 语法:fill 字段 值({line})")
|
||||
return {"action": "fill", "field": parts[1], "value": parts[2]}
|
||||
if action == "wait":
|
||||
payload = parts[1] if len(parts) > 1 else "1500"
|
||||
if payload.isdigit():
|
||||
return {"action": "wait", "ms": int(payload)}
|
||||
return {"action": "wait", "url": payload}
|
||||
if action in {"screenshot", "shot"}:
|
||||
return {"action": "screenshot"}
|
||||
raise RuntimeError(f"未知 DSL 步骤:{line}")
|
||||
|
||||
|
||||
def _split_segments(text: str) -> list[str]:
|
||||
raw = _strip_mention(text)
|
||||
raw = re.sub(r"^(browser|网页|网页操作|操作)\s*[::]?\s*", "", raw, flags=re.IGNORECASE)
|
||||
raw = re.sub(r"然后截图|再截图|最后截图", "截图", raw)
|
||||
chunks = re.split(r"[,,。;;]\s*|\s+然后\s+|\s+接着\s+|\s+并\s*", raw)
|
||||
expanded: list[str] = []
|
||||
for chunk in chunks:
|
||||
chunk = chunk.strip()
|
||||
if not chunk:
|
||||
continue
|
||||
subchunks = re.split(r"\s+然后\s+", chunk)
|
||||
if "后" in chunk and len(subchunks) == 1:
|
||||
subchunks = re.split(r"(?<=[登录页表单])后(?=[进入打开等待点击访问])", chunk)
|
||||
for part in subchunks:
|
||||
part = part.strip()
|
||||
if part:
|
||||
expanded.append(part)
|
||||
return expanded
|
||||
|
||||
|
||||
def _parse_segment(segment: str) -> list[dict[str, Any]]:
|
||||
seg = segment.strip()
|
||||
if not seg or seg.lower() in {"browser", "网页操作"}:
|
||||
return []
|
||||
|
||||
if re.fullmatch(r"截图|截屏", seg, re.IGNORECASE):
|
||||
return [{"action": "screenshot"}]
|
||||
|
||||
match = re.search(r"访问登录页|打开登录页|进入登录页|运行登录页", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "goto", "target": "/login"}]
|
||||
|
||||
match = re.search(r"输入账号密码|填写账号密码|输入账号和密码", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [
|
||||
{"action": "fill", "field": "账号", "value": "{{PREVIEW_LOGIN_USER}}"},
|
||||
{"action": "fill", "field": "密码", "value": "{{PREVIEW_LOGIN_PASSWORD}}"},
|
||||
]
|
||||
|
||||
match = re.search(r"输入账号|填写账号|输入用户名|填写用户名", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "fill", "field": "账号", "value": "{{PREVIEW_LOGIN_USER}}"}]
|
||||
|
||||
match = re.search(r"输入密码|填写密码", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "fill", "field": "密码", "value": "{{PREVIEW_LOGIN_PASSWORD}}"}]
|
||||
|
||||
match = re.search(r"进入主页|进入首页|打开主页|打开首页|等待主页", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "wait", "url": "**/app/**"}]
|
||||
|
||||
match = re.search(r"等待\s*(\d+)\s*秒", seg, re.IGNORECASE)
|
||||
if match:
|
||||
return [{"action": "wait", "ms": int(match.group(1)) * 1000}]
|
||||
|
||||
match = re.search(
|
||||
r"(?:点击|点选|选择)\s*(.+?)(?:菜单|按钮|链接)?$",
|
||||
seg,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if match:
|
||||
target = match.group(1).strip()
|
||||
target = re.sub(r"(然后|再|并)?\s*(截图|截屏).*$", "", target, flags=re.IGNORECASE).strip()
|
||||
target = re.sub(r"(然后|再|之后)$", "", target).strip()
|
||||
target = re.sub(r"(菜单|按钮|链接)$", "", target).strip()
|
||||
if target:
|
||||
return [{"action": "click", "target": target}]
|
||||
|
||||
match = re.search(
|
||||
r"(?:访问|打开|进入)\s*(https?://\S+|/\S+|登录页|主页|首页)",
|
||||
seg,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if match:
|
||||
target = match.group(1)
|
||||
mapping = {"登录页": "/login", "主页": "/app/dashboard", "首页": "/app/dashboard"}
|
||||
return [{"action": "goto", "target": mapping.get(target, target)}]
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def parse_natural_language(text: str) -> BrowserScenario | None:
|
||||
segments = _split_segments(text)
|
||||
steps: list[dict[str, Any]] = []
|
||||
for segment in segments:
|
||||
steps.extend(_parse_segment(segment))
|
||||
|
||||
if not steps:
|
||||
return None
|
||||
if not any(step.get("action") == "screenshot" for step in steps):
|
||||
if re.search(r"截图|截屏", text, re.IGNORECASE):
|
||||
steps.append({"action": "screenshot"})
|
||||
if not steps:
|
||||
return None
|
||||
|
||||
return BrowserScenario(
|
||||
name="natural",
|
||||
base_url=default_base_url(),
|
||||
steps=steps,
|
||||
source="natural-language",
|
||||
)
|
||||
|
||||
|
||||
def parse_browser_request(text: str) -> BrowserScenario | None:
|
||||
if not is_browser_intent(text):
|
||||
return None
|
||||
|
||||
raw = _strip_mention(text)
|
||||
yaml_block = re.search(r"```(?:yaml|yml)\s*\n(.+?)```", raw, re.IGNORECASE | re.DOTALL)
|
||||
if yaml_block:
|
||||
data = yaml.safe_load(yaml_block.group(1))
|
||||
if isinstance(data, dict):
|
||||
base_url = interpolate(str(data.get("base_url") or default_base_url()))
|
||||
steps = data.get("steps") or []
|
||||
return BrowserScenario(
|
||||
name=data.get("name") or "yaml-inline",
|
||||
base_url=base_url,
|
||||
steps=_normalize_steps(steps),
|
||||
source="yaml-inline",
|
||||
)
|
||||
|
||||
inline = _parse_inline_dsl(text)
|
||||
if inline:
|
||||
return inline
|
||||
|
||||
match = re.match(r"^(browser|网页|网页操作|操作)\s+([\w\-./]+)\s*$", raw, re.IGNORECASE)
|
||||
if match:
|
||||
path = _find_scenario_file(match.group(2))
|
||||
if not path:
|
||||
raise RuntimeError(f"未找到场景文件:{match.group(2)}.yaml")
|
||||
return _load_yaml_scenario(path)
|
||||
|
||||
scenario = parse_natural_language(text)
|
||||
if scenario:
|
||||
return scenario
|
||||
|
||||
default_name = (env_config.env("BROWSER_DEFAULT_SCENARIO") or "").strip()
|
||||
if default_name:
|
||||
path = _find_scenario_file(default_name)
|
||||
if path:
|
||||
return _load_yaml_scenario(path)
|
||||
|
||||
return None
|
||||
@@ -1,87 +0,0 @@
|
||||
"""浏览器自动化服务:解析场景 + 启动 dev server + 执行步骤。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from browser_executor import run_browser_scenario_sync
|
||||
from browser_models import BrowserResult, BrowserScenario
|
||||
from browser_parser import parse_browser_request
|
||||
from preview_service import (
|
||||
_package_dev_script,
|
||||
_preview_port,
|
||||
_project_cwd,
|
||||
_startup_timeout,
|
||||
_wait_for_port,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _ensure_dev_server(base_url: str) -> bool:
|
||||
parsed = urlparse(base_url)
|
||||
host = parsed.hostname or "127.0.0.1"
|
||||
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||
|
||||
if _wait_for_port(host, port, timeout=3):
|
||||
return False
|
||||
|
||||
cwd = _project_cwd()
|
||||
dev_command = _package_dev_script(cwd)
|
||||
if not dev_command:
|
||||
raise RuntimeError(
|
||||
f"无法访问 {base_url},且未找到可启动的 dev 脚本。"
|
||||
"请先手动启动前端,或设置 PREVIEW_URL。"
|
||||
)
|
||||
|
||||
import subprocess
|
||||
|
||||
logger.info("启动 dev server: %s (cwd=%s)", dev_command, cwd)
|
||||
proc = subprocess.Popen(
|
||||
dev_command,
|
||||
cwd=str(cwd),
|
||||
shell=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
if not _wait_for_port(host, port, timeout=_startup_timeout()):
|
||||
err = ""
|
||||
if proc.stderr:
|
||||
err = proc.stderr.read().decode("utf-8", errors="replace")[-1000:]
|
||||
proc.kill()
|
||||
raise RuntimeError(
|
||||
f"dev server 在 {_startup_timeout()}s 内未就绪 ({base_url})。"
|
||||
f"{(' 日志: ' + err) if err else ''}"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
async def run_browser_automation(text: str) -> BrowserResult:
|
||||
scenario = parse_browser_request(text)
|
||||
if scenario is None:
|
||||
raise RuntimeError("无法解析网页操作步骤")
|
||||
|
||||
started = await asyncio.to_thread(_ensure_dev_server, scenario.base_url)
|
||||
result = await asyncio.to_thread(run_browser_scenario_sync, scenario)
|
||||
result.started_dev_server = started
|
||||
return result
|
||||
|
||||
|
||||
def format_browser_caption(result: BrowserResult) -> str:
|
||||
lines = [
|
||||
"**网页操作完成**",
|
||||
f"> 场景:`{result.scenario_name or '自定义'}`",
|
||||
f"> 起始:`{result.base_url}`",
|
||||
f"> 最终:`{result.final_url}`",
|
||||
f"> 步骤数:{result.step_count}",
|
||||
]
|
||||
if result.started_dev_server:
|
||||
lines.append("> dev server:已自动启动")
|
||||
if result.step_log:
|
||||
lines.append("")
|
||||
lines.append("执行记录:")
|
||||
for item in result.step_log[-8:]:
|
||||
lines.append(f"- {item}")
|
||||
return "\n".join(lines)
|
||||
@@ -1,106 +0,0 @@
|
||||
"""通过 Cursor SDK 执行用户任务。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from typing import Awaitable, Callable
|
||||
|
||||
import env_config
|
||||
from bridge_manager import warm_cursor_bridge
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_cursor_lock = asyncio.Lock()
|
||||
|
||||
WECHAT_SYSTEM_PREFIX = """你是企业微信群里的 Skills 助手,正在回复群成员的消息。
|
||||
|
||||
要求:
|
||||
- 用简洁的中文回答(除非用户用其他语言提问)
|
||||
- 使用企业微信支持的 Markdown 子集(加粗、链接、列表;避免复杂表格)
|
||||
- 直接给出结论,不要冗长铺垫
|
||||
- 若任务涉及 skills.sh,可说明安装命令 `npx skills add owner/repo/skill-name`
|
||||
- **不要**在回复里写 `[图片]` 占位符;企微无法通过 Markdown 显示图片
|
||||
- 若用户要页面截图,请明确告知其发送:`截图` 或 `preview`(由 bot 自动发图)
|
||||
|
||||
用户任务:
|
||||
"""
|
||||
|
||||
|
||||
def _cursor_settings() -> dict[str, str | int]:
|
||||
timeout_raw = env_config.env("CURSOR_TIMEOUT", "600") or "600"
|
||||
return {
|
||||
"api_key": env_config.env("CURSOR_API_KEY"),
|
||||
"cwd": env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech",
|
||||
"model": env_config.env("CURSOR_MODEL", "composer-2.5") or "composer-2.5",
|
||||
"timeout": int(timeout_raw),
|
||||
}
|
||||
|
||||
|
||||
def strip_mention(text: str) -> str:
|
||||
return re.sub(r"@\S+\s*", "", text).strip()
|
||||
|
||||
|
||||
def _build_prompt(task: str) -> str:
|
||||
return WECHAT_SYSTEM_PREFIX + task.strip()
|
||||
|
||||
|
||||
def execute_cursor_task_sync(task: str) -> str:
|
||||
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
|
||||
|
||||
settings = _cursor_settings()
|
||||
api_key = settings["api_key"]
|
||||
if not api_key:
|
||||
raise RuntimeError(
|
||||
"未配置 CURSOR_API_KEY。请在 bot/.env 中设置,"
|
||||
"密钥见 https://cursor.com/dashboard/integrations"
|
||||
)
|
||||
|
||||
warm_cursor_bridge()
|
||||
|
||||
cwd = str(settings["cwd"])
|
||||
prompt = _build_prompt(task)
|
||||
logger.info("Cursor 执行任务 cwd=%s model=%s", cwd, settings["model"])
|
||||
|
||||
try:
|
||||
result = Agent.prompt(
|
||||
prompt,
|
||||
AgentOptions(
|
||||
api_key=api_key,
|
||||
model=settings["model"],
|
||||
local=LocalAgentOptions(cwd=cwd),
|
||||
),
|
||||
)
|
||||
except CursorAgentError as exc:
|
||||
raise RuntimeError(
|
||||
f"Cursor 启动失败:{exc.message}"
|
||||
+ ("(可重试)" if exc.is_retryable else "")
|
||||
) from exc
|
||||
|
||||
if result.status == "error":
|
||||
detail = result.result or "运行失败,无详细错误"
|
||||
raise RuntimeError(f"Cursor 执行失败:{detail}")
|
||||
|
||||
text = (result.result or "").strip()
|
||||
if not text:
|
||||
return "Cursor 已完成任务,但没有返回文本内容。"
|
||||
return text
|
||||
|
||||
|
||||
async def run_cursor_task(
|
||||
task: str,
|
||||
on_progress: Callable[[str], Awaitable[None]] | None = None,
|
||||
) -> str:
|
||||
timeout = int(_cursor_settings()["timeout"])
|
||||
if on_progress:
|
||||
await on_progress("Cursor 正在执行任务,请稍候…")
|
||||
|
||||
async with _cursor_lock:
|
||||
try:
|
||||
return await asyncio.wait_for(
|
||||
asyncio.to_thread(execute_cursor_task_sync, task),
|
||||
timeout=timeout,
|
||||
)
|
||||
except asyncio.TimeoutError as exc:
|
||||
raise RuntimeError(f"Cursor 执行超时(>{timeout}s)") from exc
|
||||
@@ -1,16 +0,0 @@
|
||||
"""加载 bot/.env,供各模块在 import 时统一读取环境变量。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
_BOT_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(_BOT_DIR / ".env")
|
||||
load_dotenv(_BOT_DIR / ".env.local", override=True)
|
||||
|
||||
|
||||
def env(key: str, default: str | None = None) -> str | None:
|
||||
return os.getenv(key, default)
|
||||
@@ -1,58 +0,0 @@
|
||||
"""从文本/Cursor 回复中解析本地截图路径。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import env_config
|
||||
|
||||
IMAGE_SUFFIXES = (".png", ".jpg", ".jpeg", ".webp")
|
||||
|
||||
|
||||
def _project_cwd() -> Path:
|
||||
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return Path(raw).resolve()
|
||||
|
||||
|
||||
def _resolve_candidate(raw: str, cwd: Path) -> Path | None:
|
||||
cleaned = raw.strip().strip("`\"'[]()")
|
||||
if not cleaned or cleaned.startswith("http"):
|
||||
return None
|
||||
path = Path(cleaned)
|
||||
if not path.is_absolute():
|
||||
path = cwd / path
|
||||
try:
|
||||
resolved = path.resolve()
|
||||
except OSError:
|
||||
return None
|
||||
if resolved.is_file() and resolved.suffix.lower() in IMAGE_SUFFIXES:
|
||||
return resolved
|
||||
return None
|
||||
|
||||
|
||||
def find_image_paths(text: str) -> list[Path]:
|
||||
cwd = _project_cwd()
|
||||
seen: set[Path] = set()
|
||||
found: list[Path] = []
|
||||
|
||||
patterns = [
|
||||
r"(?:保存(?:至|到)|saved\s+to|screenshot\s*[::])\s*([^\s\n\]]+\.(?:png|jpe?g|webp))",
|
||||
r"([A-Za-z]:\\[^\s\n\]]+\.(?:png|jpe?g|webp))",
|
||||
r"([^\s\n\]]+\.(?:png|jpe?g|webp))",
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
for match in re.finditer(pattern, text, re.IGNORECASE):
|
||||
path = _resolve_candidate(match.group(1), cwd)
|
||||
if path and path not in seen:
|
||||
seen.add(path)
|
||||
found.append(path)
|
||||
|
||||
return found
|
||||
|
||||
|
||||
def strip_fake_image_markdown(text: str) -> str:
|
||||
text = re.sub(r"^\s*\[图片\]\s*$", "", text, flags=re.MULTILINE)
|
||||
text = re.sub(r"\n{3,}", "\n\n", text)
|
||||
return text.strip()
|
||||
156
bot/main.py
156
bot/main.py
@@ -1,156 +0,0 @@
|
||||
"""企业微信智能机器人 · Skills 助手(skills 快查 + 截图预览 + Cursor 执行任务)。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
|
||||
import env_config
|
||||
from bridge_manager import shutdown_cursor_bridge, warm_cursor_bridge
|
||||
from router import route_message, routing_mode
|
||||
from skills_service import handle_command, warm_feed_cache
|
||||
from wecom_media import reply_image, upload_image
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
)
|
||||
logger = logging.getLogger("skills-bot")
|
||||
|
||||
BOT_ID = env_config.env("WECOM_BOT_ID") or env_config.env("WECHAT_BOT_ID")
|
||||
BOT_SECRET = env_config.env("WECOM_BOT_SECRET") or env_config.env("WECHAT_BOT_SECRET")
|
||||
|
||||
|
||||
def _require_credentials() -> None:
|
||||
if not BOT_ID or not BOT_SECRET:
|
||||
print(
|
||||
"请设置环境变量 WECOM_BOT_ID 和 WECOM_BOT_SECRET\n"
|
||||
"(企业微信 → 智能机器人 → API 模式 → 长连接)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def create_client():
|
||||
from aibot import WSClient, WSClientOptions, generate_req_id
|
||||
|
||||
ws_client = WSClient(
|
||||
WSClientOptions(
|
||||
bot_id=BOT_ID,
|
||||
secret=BOT_SECRET,
|
||||
)
|
||||
)
|
||||
|
||||
@ws_client.on("authenticated")
|
||||
def on_authenticated():
|
||||
logger.info("企业微信长连接认证成功,路由模式=%s", routing_mode())
|
||||
cursor_key = env_config.env("CURSOR_API_KEY")
|
||||
if cursor_key:
|
||||
logger.info("CURSOR_API_KEY 已加载(%s…)", cursor_key[:8])
|
||||
try:
|
||||
warm_cursor_bridge()
|
||||
logger.info("Cursor bridge 预启动完成")
|
||||
except Exception as exc:
|
||||
logger.warning("Cursor bridge 预启动失败(Cursor 任务时会重试): %s", exc)
|
||||
else:
|
||||
logger.warning("CURSOR_API_KEY 未配置,Cursor 任务将失败")
|
||||
try:
|
||||
warm_feed_cache()
|
||||
logger.info("skills 数据预加载完成")
|
||||
except Exception as exc:
|
||||
logger.warning("skills 数据预加载失败: %s", exc)
|
||||
|
||||
@ws_client.on("event.enter_chat")
|
||||
async def on_enter_chat(frame):
|
||||
help_text = handle_command("help")
|
||||
extra = (
|
||||
"\n\n---\n"
|
||||
"**单页截图**:`preview` / `截图` / `预览 [路径或URL]`\n"
|
||||
"**网页操作**:自然语言多步操作,或 `browser 场景名`\n"
|
||||
"例:`访问登录页,输入账号密码,点击登录,点击智能体管理,截图`\n"
|
||||
"场景文件:`bot/scenarios/*.yaml`(可用 `browser xiaobao-agent-manage`)"
|
||||
)
|
||||
await ws_client.reply_welcome(
|
||||
frame,
|
||||
{
|
||||
"msgtype": "markdown",
|
||||
"markdown": {"content": help_text + extra},
|
||||
},
|
||||
)
|
||||
|
||||
@ws_client.on("message.text")
|
||||
async def on_text(frame):
|
||||
body = frame.get("body", {})
|
||||
content = body.get("text", {}).get("content", "")
|
||||
logger.info("收到消息: %s", content)
|
||||
|
||||
stream_id = generate_req_id("stream")
|
||||
last_progress = ""
|
||||
|
||||
async def on_progress(message: str) -> None:
|
||||
nonlocal last_progress
|
||||
if message != last_progress:
|
||||
last_progress = message
|
||||
await ws_client.reply_stream(frame, stream_id, message, False)
|
||||
|
||||
await ws_client.reply_stream(frame, stream_id, "收到,正在处理…", False)
|
||||
|
||||
try:
|
||||
result = await route_message(content, on_progress=on_progress)
|
||||
reply = result.text
|
||||
logger.info(
|
||||
"回复来源: %s, 文本长度=%d, 图片=%s",
|
||||
result.source,
|
||||
len(reply),
|
||||
result.image_path or "-",
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.exception("处理失败")
|
||||
reply = f"处理失败:{exc}"
|
||||
result = None
|
||||
|
||||
if len(reply) > 3800:
|
||||
reply = reply[:3800] + "\n\n> …内容已截断"
|
||||
|
||||
await ws_client.reply_stream(frame, stream_id, reply, True)
|
||||
|
||||
if result and result.image_path:
|
||||
try:
|
||||
media_id = await upload_image(ws_client, result.image_path)
|
||||
await reply_image(ws_client, frame, media_id)
|
||||
logger.info("图片已发送到企微: %s", result.image_path)
|
||||
except Exception as exc:
|
||||
logger.exception("发送图片失败")
|
||||
await ws_client.reply(
|
||||
frame,
|
||||
{
|
||||
"msgtype": "markdown",
|
||||
"markdown": {
|
||||
"content": f"截图文件:`{result.image_path}`\n发图失败:{exc}\n\n请确认 bot 已重启,或发送 `截图` 重试。",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
@ws_client.on("error")
|
||||
def on_error(error):
|
||||
logger.error("连接错误: %s", error)
|
||||
|
||||
@ws_client.on("disconnected")
|
||||
def on_disconnected(reason):
|
||||
logger.warning("连接断开: %s", reason)
|
||||
|
||||
return ws_client
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import atexit
|
||||
|
||||
atexit.register(shutdown_cursor_bridge)
|
||||
_require_credentials()
|
||||
client = create_client()
|
||||
logger.info("启动 Skills 助手,Bot ID=%s…", BOT_ID[:8] if BOT_ID else "?")
|
||||
client.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,255 +0,0 @@
|
||||
"""在 CURSOR_CWD 启动/访问前端并截图(单页,不含多步操作)。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import env_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCREENSHOT_DIR = Path(__file__).resolve().parent / ".cache" / "screenshots"
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewResult:
|
||||
url: str
|
||||
screenshot_path: Path
|
||||
started_dev_server: bool
|
||||
final_url: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewRequest:
|
||||
url: str | None
|
||||
port: int | None
|
||||
|
||||
|
||||
def _project_cwd() -> Path:
|
||||
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return Path(raw).resolve()
|
||||
|
||||
|
||||
def _preview_port() -> int:
|
||||
raw = env_config.env("PREVIEW_PORT", "5173") or "5173"
|
||||
return int(raw)
|
||||
|
||||
|
||||
def _startup_timeout() -> int:
|
||||
raw = env_config.env("PREVIEW_STARTUP_TIMEOUT", "120") or "120"
|
||||
return int(raw)
|
||||
|
||||
|
||||
def _dev_command() -> str:
|
||||
return env_config.env("PREVIEW_DEV_COMMAND", "npm run dev") or "npm run dev"
|
||||
|
||||
|
||||
def parse_preview_command(text: str) -> tuple[str | None, int | None] | None:
|
||||
raw = re.sub(r"@\S+\s*", "", text).strip()
|
||||
if not raw:
|
||||
return None
|
||||
|
||||
m = re.match(
|
||||
r"^(preview|截图|预览|截屏)(?:\s+(https?://\S+|/\S*))?(?:\s+(\d{2,5}))?$",
|
||||
raw,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if not m:
|
||||
return None
|
||||
|
||||
url_part = m.group(2)
|
||||
port_part = m.group(3)
|
||||
port = int(port_part) if port_part else None
|
||||
|
||||
if url_part and url_part.startswith("/"):
|
||||
port = port or _preview_port()
|
||||
return f"http://127.0.0.1:{port}{url_part}", port
|
||||
|
||||
return url_part, port
|
||||
|
||||
|
||||
def resolve_preview_request(text: str) -> PreviewRequest | None:
|
||||
explicit = parse_preview_command(text)
|
||||
if explicit is not None:
|
||||
url_override, port_override = explicit
|
||||
return PreviewRequest(url=url_override, port=port_override)
|
||||
|
||||
if not is_preview_intent(text):
|
||||
return None
|
||||
|
||||
url_override = extract_url_from_text(text)
|
||||
if not url_override:
|
||||
env_url = env_config.env("PREVIEW_URL")
|
||||
url_override = env_url.strip() if env_url else f"http://127.0.0.1:{_preview_port()}/"
|
||||
|
||||
return PreviewRequest(url=url_override, port=None)
|
||||
|
||||
|
||||
_PREVIEW_INTENT = re.compile(
|
||||
r"^(preview|截图|预览|截屏)\b|"
|
||||
r"(页面预览|运行.*(前端|项目|页面)|"
|
||||
r"打开.*(前端|页面|项目)|"
|
||||
r"访问.*(并)?.*(截图|截屏)|"
|
||||
r"启动.*(前端|项目|dev|服务).*(截图|截屏)?)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def is_preview_intent(text: str) -> bool:
|
||||
raw = re.sub(r"@\S+\s*", "", text).strip()
|
||||
if parse_preview_command(text) is not None:
|
||||
return True
|
||||
return bool(_PREVIEW_INTENT.search(raw))
|
||||
|
||||
|
||||
def extract_url_from_text(text: str) -> str | None:
|
||||
raw = re.sub(r"@\S+\s*", "", text)
|
||||
match = re.search(
|
||||
r"(https?://[^\s\]`\"']+|localhost:\d+[/\w\-./]*)",
|
||||
raw,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if not match:
|
||||
return None
|
||||
url = match.group(1).rstrip(".,,。")
|
||||
if url.lower().startswith("localhost"):
|
||||
url = "http://" + url
|
||||
return url
|
||||
|
||||
|
||||
def _capture_screenshot_sync(url: str, output: Path) -> str:
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 720})
|
||||
page.goto(url, wait_until="networkidle", timeout=60_000)
|
||||
page.wait_for_timeout(1500)
|
||||
page.screenshot(path=str(output), full_page=False, type="png")
|
||||
final_url = page.url
|
||||
browser.close()
|
||||
return final_url
|
||||
|
||||
|
||||
def _wait_for_port(host: str, port: int, timeout: int) -> bool:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with socket.create_connection((host, port), timeout=2):
|
||||
return True
|
||||
except OSError:
|
||||
time.sleep(1)
|
||||
return False
|
||||
|
||||
|
||||
def _resolve_target_url(url_override: str | None, port_override: int | None) -> tuple[str, str | None]:
|
||||
if url_override:
|
||||
parsed = urlparse(url_override)
|
||||
if parsed.scheme and parsed.netloc:
|
||||
return url_override, None
|
||||
raise RuntimeError(f"无效 URL:{url_override}")
|
||||
|
||||
env_url = env_config.env("PREVIEW_URL")
|
||||
if env_url:
|
||||
return env_url.strip(), None
|
||||
|
||||
port = port_override or _preview_port()
|
||||
cwd = _project_cwd()
|
||||
dev_script = _package_dev_script(cwd)
|
||||
base = f"http://127.0.0.1:{port}/"
|
||||
return base, dev_script
|
||||
|
||||
|
||||
def _package_dev_script(cwd: Path) -> str | None:
|
||||
pkg = cwd / "package.json"
|
||||
if not pkg.exists():
|
||||
return None
|
||||
try:
|
||||
data = json.loads(pkg.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
scripts = data.get("scripts") or {}
|
||||
for key in ("dev", "preview", "start"):
|
||||
if scripts.get(key):
|
||||
cmd = _dev_command()
|
||||
if key != "dev" and cmd == "npm run dev":
|
||||
return f"npm run {key}"
|
||||
return cmd
|
||||
return None
|
||||
|
||||
|
||||
def _capture_preview_sync(url: str, dev_command: str | None) -> PreviewResult:
|
||||
cwd = _project_cwd()
|
||||
parsed = urlparse(url)
|
||||
host = parsed.hostname or "127.0.0.1"
|
||||
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||
|
||||
dev_proc: subprocess.Popen | None = None
|
||||
started = False
|
||||
|
||||
if dev_command:
|
||||
if _wait_for_port(host, port, timeout=3):
|
||||
logger.info("检测到端口 %s 已监听,跳过启动 dev server", port)
|
||||
else:
|
||||
logger.info("启动 dev server: %s (cwd=%s)", dev_command, cwd)
|
||||
dev_proc = subprocess.Popen(
|
||||
dev_command,
|
||||
cwd=str(cwd),
|
||||
shell=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
started = True
|
||||
if not _wait_for_port(host, port, timeout=_startup_timeout()):
|
||||
err = ""
|
||||
if dev_proc.stderr:
|
||||
err = dev_proc.stderr.read().decode("utf-8", errors="replace")[-1000:]
|
||||
raise RuntimeError(
|
||||
f"dev server 在 {_startup_timeout()}s 内未就绪 ({url})。"
|
||||
f"{(' 日志: ' + err) if err else ''}"
|
||||
)
|
||||
else:
|
||||
if not _wait_for_port(host, port, timeout=5):
|
||||
raise RuntimeError(
|
||||
f"无法访问 {url}。请在 CURSOR_CWD 放置前端项目,"
|
||||
"或先手动启动 dev server,或设置 PREVIEW_URL。"
|
||||
)
|
||||
|
||||
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
output = SCREENSHOT_DIR / f"preview-{stamp}.png"
|
||||
|
||||
try:
|
||||
final_url = _capture_screenshot_sync(url, output)
|
||||
finally:
|
||||
if dev_proc and dev_proc.poll() is None:
|
||||
dev_proc.terminate()
|
||||
try:
|
||||
dev_proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
dev_proc.kill()
|
||||
|
||||
return PreviewResult(
|
||||
url=url,
|
||||
screenshot_path=output,
|
||||
started_dev_server=started,
|
||||
final_url=final_url,
|
||||
)
|
||||
|
||||
|
||||
async def capture_preview(
|
||||
url_override: str | None = None,
|
||||
port_override: int | None = None,
|
||||
) -> PreviewResult:
|
||||
url, dev_command = _resolve_target_url(url_override, port_override)
|
||||
return await asyncio.to_thread(_capture_preview_sync, url, dev_command)
|
||||
@@ -1,7 +0,0 @@
|
||||
wecom-aibot-python-sdk>=1.0.2
|
||||
python-dotenv>=1.0.0
|
||||
httpx>=0.27.0
|
||||
certifi>=2024.0.0
|
||||
cursor-sdk>=0.1.0
|
||||
playwright>=1.49.0
|
||||
PyYAML>=6.0.0
|
||||
109
bot/router.py
109
bot/router.py
@@ -1,109 +0,0 @@
|
||||
"""消息路由:skills 快查 / 网页操作 / 截图预览 / Cursor 通用任务。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import env_config
|
||||
from browser_parser import is_browser_intent, parse_browser_request
|
||||
from browser_service import format_browser_caption, run_browser_automation
|
||||
from cursor_runner import run_cursor_task, strip_mention
|
||||
from image_extract import find_image_paths, strip_fake_image_markdown
|
||||
from preview_service import capture_preview, is_preview_intent, resolve_preview_request
|
||||
from skills_service import handle_command, parse_command
|
||||
from bot_types import RouteResult
|
||||
|
||||
|
||||
def routing_mode() -> str:
|
||||
return (env_config.env("ROUTING_MODE", "hybrid") or "hybrid").lower()
|
||||
|
||||
|
||||
def _normalize(text: str) -> str:
|
||||
return re.sub(r"@\S+\s*", "", text).strip().lower()
|
||||
|
||||
|
||||
def is_skills_fast_command(text: str) -> bool:
|
||||
raw = _normalize(text)
|
||||
if not raw:
|
||||
return True
|
||||
if raw in {"help", "帮助", "?", "h"}:
|
||||
return True
|
||||
|
||||
cmd = parse_command(text)
|
||||
if cmd.kind in {"help", "list", "detail"}:
|
||||
return True
|
||||
if cmd.kind == "search" and re.match(r"^(search|搜索|find|查)\s+", raw):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def _run_browser(text: str, on_progress=None) -> RouteResult:
|
||||
if parse_browser_request(text) is None:
|
||||
raise RuntimeError("无法解析网页操作步骤")
|
||||
|
||||
if on_progress:
|
||||
await on_progress("正在按步骤执行网页操作…")
|
||||
|
||||
result = await run_browser_automation(text)
|
||||
return RouteResult(
|
||||
source="browser",
|
||||
text=format_browser_caption(result),
|
||||
image_path=str(result.screenshot_path),
|
||||
)
|
||||
|
||||
|
||||
async def _run_preview(text: str, on_progress=None) -> RouteResult:
|
||||
preview_req = resolve_preview_request(text)
|
||||
if preview_req is None:
|
||||
raise RuntimeError("无法解析截图请求")
|
||||
|
||||
if on_progress:
|
||||
await on_progress(f"正在访问并截图:{preview_req.url or '默认地址'}…")
|
||||
|
||||
result = await capture_preview(preview_req.url, preview_req.port)
|
||||
caption = (
|
||||
f"**页面预览**\n"
|
||||
f"> URL:`{result.final_url or result.url}`\n"
|
||||
f"> 项目:`{env_config.env('CURSOR_CWD', '')}`\n"
|
||||
f"> dev server:{'已自动启动' if result.started_dev_server else '使用已有服务'}"
|
||||
)
|
||||
return RouteResult(
|
||||
source="preview",
|
||||
text=caption,
|
||||
image_path=str(result.screenshot_path),
|
||||
)
|
||||
|
||||
|
||||
async def route_message(text: str, on_progress=None) -> RouteResult:
|
||||
task = strip_mention(text)
|
||||
if not task:
|
||||
return RouteResult("skills", handle_command("help"))
|
||||
|
||||
if is_browser_intent(text):
|
||||
return await _run_browser(text, on_progress=on_progress)
|
||||
|
||||
if resolve_preview_request(text) is not None:
|
||||
return await _run_preview(text, on_progress=on_progress)
|
||||
|
||||
mode = routing_mode()
|
||||
if mode == "skills":
|
||||
return RouteResult("skills", handle_command(text))
|
||||
|
||||
if mode == "cursor" or not is_skills_fast_command(text):
|
||||
reply = await run_cursor_task(task, on_progress=on_progress)
|
||||
reply = strip_fake_image_markdown(reply)
|
||||
|
||||
image_path: str | None = None
|
||||
paths = find_image_paths(reply)
|
||||
if paths:
|
||||
image_path = str(paths[0])
|
||||
elif is_preview_intent(text) or is_browser_intent(text):
|
||||
if on_progress:
|
||||
await on_progress("未找到截图文件,改用 Playwright 自动执行…")
|
||||
if is_browser_intent(text):
|
||||
return await _run_browser(text, on_progress=on_progress)
|
||||
return await _run_preview(text, on_progress=on_progress)
|
||||
|
||||
return RouteResult("cursor", reply, image_path=image_path)
|
||||
|
||||
return RouteResult("skills", handle_command(text))
|
||||
@@ -1,17 +0,0 @@
|
||||
name: xiaobao-agent-manage
|
||||
description: 登录后打开智能体管理并截图
|
||||
steps:
|
||||
- goto: /login
|
||||
- fill:
|
||||
field: 账号
|
||||
value: "{{PREVIEW_LOGIN_USER}}"
|
||||
- fill:
|
||||
field: 密码
|
||||
value: "{{PREVIEW_LOGIN_PASSWORD}}"
|
||||
- click: 登录
|
||||
- wait:
|
||||
url: "**/app/**"
|
||||
timeout: 60000
|
||||
- click: 智能体管理
|
||||
- wait: 1500
|
||||
- screenshot
|
||||
@@ -1,317 +0,0 @@
|
||||
"""skills.sh 数据查询与命令解析。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FEED_URLS = [
|
||||
# jsDelivr 在国内通常比 raw.githubusercontent.com 更稳定
|
||||
"https://cdn.jsdelivr.net/gh/NeverSight/skills.sh_feed@main/data/feed.json",
|
||||
"https://raw.githubusercontent.com/NeverSight/skills.sh_feed/main/data/feed.json",
|
||||
]
|
||||
CACHE_TTL_SECONDS = 600
|
||||
CACHE_DIR = Path(__file__).resolve().parent / ".cache"
|
||||
CACHE_FILE = CACHE_DIR / "feed.json"
|
||||
|
||||
_cache: dict[str, Any] = {"data": None, "fetched_at": 0.0}
|
||||
|
||||
Board = Literal["trending", "hot", "all"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Command:
|
||||
kind: Literal["help", "list", "search", "detail"]
|
||||
board: Board = "trending"
|
||||
limit: int = 10
|
||||
query: str = ""
|
||||
|
||||
|
||||
def _fetch_json(url: str) -> dict[str, Any]:
|
||||
headers = {
|
||||
"User-Agent": "skills-hot-bot/1.0",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
with httpx.Client(
|
||||
timeout=httpx.Timeout(20.0, connect=10.0),
|
||||
verify=certifi.where(),
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
resp = client.get(url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
def _load_disk_cache() -> dict[str, Any] | None:
|
||||
if not CACHE_FILE.exists():
|
||||
return None
|
||||
try:
|
||||
return json.loads(CACHE_FILE.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
logger.warning("读取本地缓存失败: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _save_disk_cache(data: dict[str, Any]) -> None:
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
|
||||
def load_feed(force: bool = False) -> dict[str, Any]:
|
||||
now = time.time()
|
||||
if not force and _cache["data"] and now - _cache["fetched_at"] < CACHE_TTL_SECONDS:
|
||||
return _cache["data"]
|
||||
|
||||
errors: list[str] = []
|
||||
for url in FEED_URLS:
|
||||
for attempt in range(3):
|
||||
try:
|
||||
data = _fetch_json(url)
|
||||
_cache["data"] = data
|
||||
_cache["fetched_at"] = now
|
||||
_save_disk_cache(data)
|
||||
logger.info("skills 数据已更新: %s", url)
|
||||
return data
|
||||
except Exception as exc:
|
||||
msg = f"{url} (#{attempt + 1}): {exc}"
|
||||
errors.append(msg)
|
||||
logger.debug("拉取失败 %s", msg)
|
||||
time.sleep(0.5 * (attempt + 1))
|
||||
|
||||
stale = _load_disk_cache()
|
||||
if stale:
|
||||
logger.warning("网络不可用,回退到本地缓存")
|
||||
_cache["data"] = stale
|
||||
_cache["fetched_at"] = now
|
||||
return stale
|
||||
|
||||
raise RuntimeError(f"无法获取 skills 数据。最近错误: {errors[-1]}")
|
||||
|
||||
|
||||
def warm_feed_cache() -> None:
|
||||
"""启动时预加载,避免首条消息才触发网络请求。"""
|
||||
load_feed(force=True)
|
||||
|
||||
|
||||
def _normalize_text(text: str) -> str:
|
||||
text = re.sub(r"@\S+\s*", "", text)
|
||||
return text.strip().lower()
|
||||
|
||||
|
||||
def _parse_limit(raw: str | None, default: int = 10) -> int:
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
n = int(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
return max(1, min(n, 30))
|
||||
|
||||
|
||||
def _match_list(raw: str, board: Board, aliases: str) -> Command | None:
|
||||
m = re.match(rf"^({aliases})(?:\s+top)?\s*(\d+)?$", raw)
|
||||
if m:
|
||||
return Command(kind="list", board=board, limit=_parse_limit(m.group(2)))
|
||||
m = re.match(rf"^(查|查询)\s+({aliases})(?:\s+top)?\s*(\d+)?$", raw)
|
||||
if m:
|
||||
return Command(kind="list", board=board, limit=_parse_limit(m.group(3)))
|
||||
return None
|
||||
|
||||
|
||||
def parse_command(text: str) -> Command:
|
||||
raw = _normalize_text(text)
|
||||
if not raw or raw in {"help", "帮助", "?", "h"}:
|
||||
return Command(kind="help")
|
||||
|
||||
for board, aliases in (
|
||||
("trending", "trending|趋势|top"),
|
||||
("hot", "hot|实时|热门"),
|
||||
("all", "all|总榜|alltime|all-time"),
|
||||
):
|
||||
cmd = _match_list(raw, board, aliases)
|
||||
if cmd:
|
||||
return cmd
|
||||
|
||||
m = re.match(r"^(search|搜索|find|查)\s+(.+)$", raw)
|
||||
if m:
|
||||
return Command(kind="search", query=m.group(2).strip(), limit=5)
|
||||
|
||||
m = re.match(r"^(detail|详情|skill|info)\s+(.+)$", raw)
|
||||
if m:
|
||||
return Command(kind="detail", query=m.group(2).strip())
|
||||
|
||||
if raw.startswith("trending") or raw.startswith("趋势"):
|
||||
parts = raw.split(maxsplit=1)
|
||||
return Command(kind="list", board="trending", limit=_parse_limit(parts[1] if len(parts) > 1 else None))
|
||||
|
||||
return Command(kind="search", query=raw, limit=5)
|
||||
|
||||
|
||||
def _format_installs(n: int | float) -> str:
|
||||
if n >= 1_000_000:
|
||||
return f"{n / 1_000_000:.1f}M"
|
||||
if n >= 1_000:
|
||||
return f"{n / 1_000:.1f}K"
|
||||
return str(int(n))
|
||||
|
||||
|
||||
def _board_items(feed: dict[str, Any], board: Board) -> list[dict[str, Any]]:
|
||||
key = {"trending": "topTrending", "hot": "topHot", "all": "topAllTime"}[board]
|
||||
return feed.get(key, [])
|
||||
|
||||
|
||||
def _board_title(board: Board) -> str:
|
||||
return {
|
||||
"trending": "Trending(近期增长)",
|
||||
"hot": "Hot(实时热度)",
|
||||
"all": "All Time(总安装榜)",
|
||||
}[board]
|
||||
|
||||
|
||||
def format_list(board: Board, limit: int) -> str:
|
||||
feed = load_feed()
|
||||
items = _board_items(feed, board)[:limit]
|
||||
updated = feed.get("updatedAt", "未知")[:10]
|
||||
|
||||
lines = [
|
||||
f"**skills.sh {_board_title(board)} Top {limit}**",
|
||||
f"> 数据更新:{updated}",
|
||||
"",
|
||||
]
|
||||
|
||||
for i, item in enumerate(items, 1):
|
||||
title = item.get("title", "?")
|
||||
source = item.get("source", "?")
|
||||
installs = _format_installs(item.get("installs", 0))
|
||||
desc = item.get("description", "")
|
||||
if len(desc) > 80:
|
||||
desc = desc[:77] + "..."
|
||||
link = item.get("link", "")
|
||||
lines.append(f"{i}. **{title}** · {installs}")
|
||||
lines.append(f" `{source}`")
|
||||
if desc:
|
||||
lines.append(f" {desc}")
|
||||
if link:
|
||||
lines.append(f" [查看]({link})")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines).strip()
|
||||
|
||||
|
||||
def format_search(query: str, limit: int) -> str:
|
||||
feed = load_feed()
|
||||
q = query.lower()
|
||||
seen: set[str] = set()
|
||||
matches: list[dict[str, Any]] = []
|
||||
|
||||
for board in ("topTrending", "topHot", "topAllTime"):
|
||||
for item in feed.get(board, []):
|
||||
item_id = item.get("id") or item.get("title", "")
|
||||
if item_id in seen:
|
||||
continue
|
||||
haystack = " ".join(
|
||||
[
|
||||
item.get("title", ""),
|
||||
item.get("source", ""),
|
||||
item.get("description", ""),
|
||||
]
|
||||
).lower()
|
||||
if q in haystack:
|
||||
seen.add(item_id)
|
||||
matches.append(item)
|
||||
if len(matches) >= limit:
|
||||
break
|
||||
if len(matches) >= limit:
|
||||
break
|
||||
|
||||
if not matches:
|
||||
return f"未找到与 **{query}** 相关的 skill。\n\n试试:`trending 10` / `hot 10` / `搜索 react`"
|
||||
|
||||
lines = [f"**搜索「{query}」** 共 {len(matches)} 条", ""]
|
||||
for i, item in enumerate(matches, 1):
|
||||
title = item.get("title", "?")
|
||||
source = item.get("source", "?")
|
||||
installs = _format_installs(item.get("installs", 0))
|
||||
link = item.get("link", "")
|
||||
lines.append(f"{i}. **{title}** · {installs} · `{source}`")
|
||||
if link:
|
||||
lines.append(f" [查看]({link})")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_detail(name: str) -> str:
|
||||
feed = load_feed()
|
||||
q = name.lower().strip()
|
||||
best: dict[str, Any] | None = None
|
||||
|
||||
for board in ("topTrending", "topHot", "topAllTime"):
|
||||
for item in feed.get(board, []):
|
||||
title = (item.get("title") or "").lower()
|
||||
item_id = (item.get("id") or "").lower()
|
||||
if title == q or q in title or q in item_id:
|
||||
if best is None or item.get("installs", 0) > best.get("installs", 0):
|
||||
best = item
|
||||
|
||||
if not best:
|
||||
return f"未找到 skill:**{name}**\n\n试试:`搜索 {name}`"
|
||||
|
||||
desc = best.get("description", "无描述")
|
||||
return "\n".join(
|
||||
[
|
||||
f"**{best.get('title', '?')}**",
|
||||
f"`{best.get('source', '?')}`",
|
||||
f"安装量:**{_format_installs(best.get('installs', 0))}**",
|
||||
"",
|
||||
desc,
|
||||
"",
|
||||
f"[skills.sh 详情]({best.get('link', 'https://skills.sh')})",
|
||||
"",
|
||||
f"安装:`npx skills add {best.get('source', '')}/{best.get('title', '')}`",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def format_help() -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
"**Skills 助手 · 命令帮助**",
|
||||
"",
|
||||
"`trending 10` / `趋势 10` — 近期增长榜",
|
||||
"`hot 10` / `实时 10` — 实时热度榜",
|
||||
"`all 10` / `总榜 10` — 历史总安装榜",
|
||||
"`搜索 react` / `search tdd` — 关键词搜索",
|
||||
"`详情 find-skills` — 查看单个 skill",
|
||||
"`preview` / `截图` / `预览` — 单页截图",
|
||||
"`browser 场景名` — 执行 YAML 场景(见 bot/scenarios/)",
|
||||
"自然语言 — 如:访问登录页,输入账号密码,点击登录,点击智能体管理,截图",
|
||||
"`preview /about 5173` — 指定路径和端口",
|
||||
"",
|
||||
"示例:",
|
||||
"• trending top10",
|
||||
"• 查 grill",
|
||||
"• 详情 remotion-render",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def handle_command(text: str) -> str:
|
||||
cmd = parse_command(text)
|
||||
if cmd.kind == "help":
|
||||
return format_help()
|
||||
if cmd.kind == "list":
|
||||
return format_list(cmd.board, cmd.limit)
|
||||
if cmd.kind == "search":
|
||||
return format_search(cmd.query, cmd.limit)
|
||||
if cmd.kind == "detail":
|
||||
return format_detail(cmd.query)
|
||||
return format_help()
|
||||
@@ -1,96 +0,0 @@
|
||||
"""企业微信 API 模式:上传图片并回复。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from aibot import generate_req_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CHUNK_SIZE = 512 * 1024
|
||||
MAX_IMAGE_BYTES = 9 * 1024 * 1024
|
||||
|
||||
|
||||
def _ensure_image_size(path: Path) -> bytes:
|
||||
data = path.read_bytes()
|
||||
if len(data) > MAX_IMAGE_BYTES:
|
||||
raise RuntimeError(
|
||||
f"截图过大({len(data) // 1024}KB),请缩小页面或使用 viewport 截图(上限 9MB)"
|
||||
)
|
||||
return data
|
||||
|
||||
|
||||
def _response_body(frame: dict[str, Any]) -> dict[str, Any]:
|
||||
if frame.get("errcode", 0) != 0:
|
||||
raise RuntimeError(
|
||||
f"企微接口错误 errcode={frame.get('errcode')} errmsg={frame.get('errmsg')}"
|
||||
)
|
||||
body = frame.get("body")
|
||||
return body if isinstance(body, dict) else {}
|
||||
|
||||
|
||||
async def upload_image(ws_client: Any, image_path: str | Path) -> str:
|
||||
path = Path(image_path)
|
||||
if not path.exists():
|
||||
raise RuntimeError(f"截图不存在: {path}")
|
||||
|
||||
data = _ensure_image_size(path)
|
||||
md5 = hashlib.md5(data).hexdigest()
|
||||
chunks = [data[i : i + CHUNK_SIZE] for i in range(0, len(data), CHUNK_SIZE)]
|
||||
total_chunks = len(chunks)
|
||||
|
||||
manager = ws_client._ws_manager
|
||||
|
||||
init_frame = await manager.send_reply(
|
||||
generate_req_id("upload_init"),
|
||||
{
|
||||
"type": "image",
|
||||
"filename": path.name,
|
||||
"total_size": len(data),
|
||||
"total_chunks": total_chunks,
|
||||
"md5": md5,
|
||||
},
|
||||
"aibot_upload_media_init",
|
||||
)
|
||||
upload_id = _response_body(init_frame).get("upload_id")
|
||||
if not upload_id:
|
||||
raise RuntimeError("上传初始化失败:未返回 upload_id")
|
||||
|
||||
for index, chunk in enumerate(chunks):
|
||||
chunk_frame = await manager.send_reply(
|
||||
generate_req_id("upload_chunk"),
|
||||
{
|
||||
"upload_id": upload_id,
|
||||
"chunk_index": index,
|
||||
"base64_data": base64.b64encode(chunk).decode("ascii"),
|
||||
},
|
||||
"aibot_upload_media_chunk",
|
||||
)
|
||||
_response_body(chunk_frame)
|
||||
|
||||
finish_frame = await manager.send_reply(
|
||||
generate_req_id("upload_finish"),
|
||||
{"upload_id": upload_id},
|
||||
"aibot_upload_media_finish",
|
||||
)
|
||||
media_id = _response_body(finish_frame).get("media_id")
|
||||
if not media_id:
|
||||
raise RuntimeError("上传完成但未返回 media_id")
|
||||
|
||||
logger.info("图片已上传 media_id=%s…", str(media_id)[:12])
|
||||
return str(media_id)
|
||||
|
||||
|
||||
async def reply_image(ws_client: Any, frame: dict[str, Any], media_id: str) -> None:
|
||||
await ws_client.reply(
|
||||
frame,
|
||||
{
|
||||
"msgtype": "image",
|
||||
"image": {"media_id": media_id},
|
||||
},
|
||||
)
|
||||
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import sys
|
||||
|
||||
from daily.generate import main as generate_main
|
||||
from daily.scheduler import main as schedule_main
|
||||
from daily.webhook import main as push_main
|
||||
|
||||
|
||||
@@ -14,7 +15,14 @@ def main() -> int:
|
||||
return generate_main()
|
||||
if cmd in {"push", "send", "webhook"}:
|
||||
return push_main(sys.argv[2:])
|
||||
print(f"未知命令: {cmd}\n用法: python -m daily [generate|push] [report_path]", file=sys.stderr)
|
||||
if cmd in {"schedule", "scheduler", "daemon"}:
|
||||
return schedule_main()
|
||||
print(
|
||||
f"未知命令: {cmd}\n"
|
||||
"用法: python -m daily [generate|push|schedule] [report_path]\n"
|
||||
" python -m daily schedule [--once] [--dry-run]",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -59,11 +59,41 @@ def _extract_markdown(text: str) -> str:
|
||||
|
||||
|
||||
def analyze_trends(llm_input: dict[str, Any], *, date_str: str) -> dict[str, Any] | None:
|
||||
from daily.config import theme_ban_days
|
||||
from daily.narrative_axis import (
|
||||
enforce_narrative_axis,
|
||||
load_recent_axes,
|
||||
load_recent_theme_summaries,
|
||||
pick_narrative_axis,
|
||||
)
|
||||
|
||||
skill = _load_skill()
|
||||
featured_note = ""
|
||||
if llm_input.get("featured_pick"):
|
||||
featured_note = (
|
||||
"\n输入已含 **featured_pick**(编辑指定今日首推);"
|
||||
"top_picks.skill 必须以 featured_pick 为准;"
|
||||
"why/opening 不得向读者提及「编辑指定」。\n"
|
||||
)
|
||||
used_axes = set(load_recent_axes(date_str))
|
||||
axis = pick_narrative_axis(used_axes)
|
||||
llm_input["required_narrative_axis"] = axis
|
||||
llm_input["narrative_axis"] = axis
|
||||
theme_ban = load_recent_theme_summaries(date_str, theme_ban_days())
|
||||
ban_note = ""
|
||||
if theme_ban:
|
||||
ban_note = (
|
||||
"\n近几日已用过的主题/导语(请软避开同类开场,勿原样复用):\n- "
|
||||
+ "\n- ".join(theme_ban)
|
||||
+ "\n"
|
||||
)
|
||||
system = (
|
||||
f"{skill}\n\n"
|
||||
f"{featured_note}"
|
||||
f"{ban_note}"
|
||||
"当前执行 **Step 1:趋势分析**。\n"
|
||||
"只输出 trends JSON(headline, opening, themes, top_picks, signals),不要 Markdown。"
|
||||
f"**required_narrative_axis** = `{axis}`;输出 JSON 必须含 `narrative_axis` 且等于该值。\n"
|
||||
"只输出 trends JSON(headline, opening, themes, top_picks, signals, narrative_axis),不要 Markdown。"
|
||||
)
|
||||
user = json.dumps(llm_input, ensure_ascii=False, indent=2)
|
||||
try:
|
||||
@@ -77,8 +107,9 @@ def analyze_trends(llm_input: dict[str, Any], *, date_str: str) -> dict[str, Any
|
||||
if not parsed.get("headline") and not parsed.get("opening"):
|
||||
logger.warning("Agent Step1 JSON 无效")
|
||||
return None
|
||||
parsed = enforce_narrative_axis(parsed, axis)
|
||||
save_json(trends_json_path(date_str), parsed)
|
||||
logger.info("Agent Step1 完成:%s", parsed.get("headline", "?"))
|
||||
logger.info("Agent Step1 完成:%s [%s]", parsed.get("headline", "?"), axis)
|
||||
return parsed
|
||||
|
||||
|
||||
@@ -91,8 +122,17 @@ def write_wecom_report(
|
||||
updated: str,
|
||||
) -> str | None:
|
||||
skill = _load_skill()
|
||||
featured_note = ""
|
||||
if llm_input.get("featured_pick"):
|
||||
featured_note = (
|
||||
"\n输入 data 已含 **featured_pick**;"
|
||||
"今日首推区块须使用 featured_pick.why_today;"
|
||||
"链接行用 Markdown [标题](URL),勿用反引号裸 URL;"
|
||||
"读者可见文案不得出现「编辑指定」等元信息。\n"
|
||||
)
|
||||
system = (
|
||||
f"{skill}\n\n"
|
||||
f"{featured_note}"
|
||||
"当前执行 **Step 2:撰写企微早报**。\n"
|
||||
f"日期={date_str},时间={time_str},数据截至={updated}。\n"
|
||||
"只输出企微 Markdown 正文,不要代码块,不要 JSON。"
|
||||
|
||||
163
daily/board_history.py
Normal file
163
daily/board_history.py
Normal file
@@ -0,0 +1,163 @@
|
||||
"""企微展示历史:读写 data.wecom_shown_keys,与 movement_baseline 严格分离。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from daily.config import OUTPUT_DIR, board_dedup_days
|
||||
from daily.delta import RECENT_BOARD_KEYS, skill_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BOARD_KEYS = RECENT_BOARD_KEYS
|
||||
|
||||
_GITHUB_REPO_RE = re.compile(r"github\.com/([\w.-]+/[\w.-]+)", re.I)
|
||||
_SKILL_SH_RE = re.compile(r"skills\.sh/([\w.-]+/[\w.-]+(?:/[\w.-]+)?)", re.I)
|
||||
|
||||
# 与企微正文榜单标题对齐;顺序用于切分相邻 section
|
||||
_WECOM_SECTION_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
|
||||
("skills_trending", re.compile(r"Skills\s+Trending", re.I)),
|
||||
("skills_hot", re.compile(r"Skills\s+Hot", re.I)),
|
||||
("github_trending", re.compile(r"GitHub\s+Trending", re.I)),
|
||||
("github_emerging", re.compile(r"GitHub\s+新兴", re.I)),
|
||||
("github_topic", re.compile(r"Topic\s+", re.I)),
|
||||
)
|
||||
|
||||
|
||||
def extract_shown_keys(board: str, items: list[dict[str, Any]]) -> list[str]:
|
||||
"""从最终展示 items 抽取稳定 identity key。
|
||||
|
||||
Skills 榜同时写入 skill id 与 source,便于周去重按仓屏蔽。
|
||||
"""
|
||||
keys: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for item in items:
|
||||
if board.startswith("skills_"):
|
||||
candidates = [skill_id(item), str(item.get("source") or "").strip()]
|
||||
else:
|
||||
candidates = [str(item.get("repo") or "")]
|
||||
for key in candidates:
|
||||
if not key or key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
keys.append(key)
|
||||
return keys
|
||||
|
||||
|
||||
def _keys_from_board_items(board: str, data: dict[str, Any]) -> set[str]:
|
||||
if board == "github_topic":
|
||||
topic = data.get("github_topic") or {}
|
||||
items = topic.get("repos") if isinstance(topic, dict) else []
|
||||
else:
|
||||
items = data.get(board) or []
|
||||
if not isinstance(items, list):
|
||||
return set()
|
||||
return set(extract_shown_keys(board, items))
|
||||
|
||||
|
||||
def parse_wecom_shown_keys(md: str) -> dict[str, set[str]]:
|
||||
"""从企微 Markdown 按榜单 section 解析已展示 keys(冷启动兼容)。"""
|
||||
out: dict[str, set[str]] = {board: set() for board in BOARD_KEYS}
|
||||
if not (md or "").strip():
|
||||
return out
|
||||
|
||||
hits: list[tuple[int, str]] = []
|
||||
for board, pattern in _WECOM_SECTION_PATTERNS:
|
||||
for match in pattern.finditer(md):
|
||||
hits.append((match.start(), board))
|
||||
if not hits:
|
||||
return out
|
||||
hits.sort(key=lambda x: x[0])
|
||||
|
||||
for idx, (start, board) in enumerate(hits):
|
||||
end = hits[idx + 1][0] if idx + 1 < len(hits) else len(md)
|
||||
chunk = md[start:end]
|
||||
if board.startswith("skills_"):
|
||||
out[board].update(_SKILL_SH_RE.findall(chunk))
|
||||
else:
|
||||
out[board].update(_GITHUB_REPO_RE.findall(chunk))
|
||||
return out
|
||||
|
||||
|
||||
def _load_shown_keys_for_day(path: Path, date_str: str) -> dict[str, set[str]] | None:
|
||||
"""读一日历史:优先 wecom_shown_keys;缺省则回退 wecom.md,再回退 data 榜字段。"""
|
||||
empty = {board: set() for board in BOARD_KEYS}
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("读取 wecom_shown_keys %s 失败:%s", path, exc)
|
||||
return None
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return empty
|
||||
|
||||
out: dict[str, set[str]] = {board: set() for board in BOARD_KEYS}
|
||||
shown = data.get("wecom_shown_keys")
|
||||
if isinstance(shown, dict):
|
||||
for board in BOARD_KEYS:
|
||||
keys = shown.get(board) or []
|
||||
if isinstance(keys, list):
|
||||
out[board].update(str(k) for k in keys if k)
|
||||
if any(out.values()):
|
||||
return out
|
||||
|
||||
wecom_path = OUTPUT_DIR / f"{date_str}.wecom.md"
|
||||
if wecom_path.exists():
|
||||
try:
|
||||
md = wecom_path.read_text(encoding="utf-8")
|
||||
except OSError as exc:
|
||||
logger.warning("读取 wecom.md 回退 %s 失败:%s", wecom_path, exc)
|
||||
else:
|
||||
parsed = parse_wecom_shown_keys(md)
|
||||
if any(parsed.values()):
|
||||
return parsed
|
||||
|
||||
for board in BOARD_KEYS:
|
||||
out[board].update(_keys_from_board_items(board, data))
|
||||
return out
|
||||
|
||||
|
||||
def load_recent_shown_keys(
|
||||
date_str: str,
|
||||
*,
|
||||
lookback_days: int | None = None,
|
||||
) -> dict[str, set[str]]:
|
||||
"""近 N 日已展示 keys 并集(不含当日)。缺省或读失败视为空集。"""
|
||||
empty = {board: set() for board in BOARD_KEYS}
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
return empty
|
||||
|
||||
days = lookback_days if lookback_days is not None else board_dedup_days()
|
||||
out: dict[str, set[str]] = {board: set() for board in BOARD_KEYS}
|
||||
|
||||
for day_offset in range(1, days + 1):
|
||||
prev_date = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
|
||||
path = OUTPUT_DIR / f"{prev_date}.data.json"
|
||||
if not path.exists():
|
||||
continue
|
||||
day_keys = _load_shown_keys_for_day(path, prev_date)
|
||||
if day_keys is None:
|
||||
continue
|
||||
for board in BOARD_KEYS:
|
||||
out[board].update(day_keys.get(board) or set())
|
||||
return out
|
||||
|
||||
|
||||
def merge_wecom_shown_into_data(
|
||||
data: dict[str, Any],
|
||||
shown: dict[str, list[str]],
|
||||
) -> dict[str, Any]:
|
||||
"""写入 wecom_shown_keys,不修改 movement_baseline。"""
|
||||
merged = dict(data)
|
||||
merged["wecom_shown_keys"] = {
|
||||
board: list(keys) for board, keys in shown.items()
|
||||
}
|
||||
return merged
|
||||
48
daily/board_select.py
Normal file
48
daily/board_select.py
Normal file
@@ -0,0 +1,48 @@
|
||||
"""五榜唯一列表主人:周去重 + 深池补满。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Literal
|
||||
|
||||
from daily.delta import skill_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def board_select(
|
||||
*,
|
||||
board: str,
|
||||
items: list[dict[str, Any]],
|
||||
recent_keys: set[str],
|
||||
limit: int,
|
||||
pool_size: int,
|
||||
kind: Literal["skill", "github"],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""从深池过滤近 N 日已展示 key,按原顺序取满 limit;不足则短榜。"""
|
||||
if kind == "skill":
|
||||
from daily.skills_group import group_skills_by_source
|
||||
|
||||
pool = group_skills_by_source(items, limit=pool_size, pool_size=pool_size)
|
||||
else:
|
||||
pool = items[: max(pool_size, limit)]
|
||||
|
||||
out: list[dict[str, Any]] = []
|
||||
for item in pool:
|
||||
if kind == "skill":
|
||||
key = skill_id(item)
|
||||
source = str(item.get("source") or "").strip()
|
||||
if (key and key in recent_keys) or (source and source in recent_keys):
|
||||
continue
|
||||
if not key and not source:
|
||||
continue
|
||||
else:
|
||||
key = str(item.get("repo") or "")
|
||||
if not key or key in recent_keys:
|
||||
continue
|
||||
out.append(item)
|
||||
if len(out) >= limit:
|
||||
break
|
||||
if len(out) < limit:
|
||||
logger.info("board_short:%s:%s", board, len(out))
|
||||
return out
|
||||
@@ -12,7 +12,7 @@ import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
|
||||
import env_config
|
||||
from daily.config import ROOT, env
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -22,7 +22,7 @@ _bridge_process: subprocess.Popen[bytes] | None = None
|
||||
|
||||
|
||||
def _cursor_cwd() -> str:
|
||||
return env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
|
||||
return env("DAILY_CURSOR_CWD") or env("CURSOR_CWD") or str(ROOT)
|
||||
|
||||
|
||||
def _parse_discovery_line(line: str) -> Mapping[str, Any] | None:
|
||||
@@ -124,7 +124,8 @@ def warm_cursor_bridge(force: bool = False) -> None:
|
||||
)
|
||||
try:
|
||||
discovery = _read_discovery_polling(process)
|
||||
except Exception:
|
||||
except (RuntimeError, OSError, ValueError) as exc:
|
||||
logger.warning("Cursor bridge discovery 失败,终止子进程:%s", exc)
|
||||
process.kill()
|
||||
process.wait(timeout=5)
|
||||
raise
|
||||
109
daily/config.py
109
daily/config.py
@@ -3,13 +3,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
BOT_DIR = ROOT / "bot"
|
||||
OUTPUT_DIR = ROOT / "output"
|
||||
LOG_DIR = ROOT / "logs"
|
||||
CACHE_DIR = ROOT / ".cache"
|
||||
@@ -32,6 +30,24 @@ def wecom_skill_desc_limit() -> int:
|
||||
return env_int("DAILY_WECOM_SKILL_DESC_LIMIT", 56)
|
||||
|
||||
|
||||
def wecom_news_desc_limit() -> int:
|
||||
"""企微新闻摘要建议字数;在句读/词边界截断,不加省略号。"""
|
||||
return max(24, env_int("DAILY_WECOM_NEWS_DESC_LIMIT", 72))
|
||||
|
||||
|
||||
def wecom_ai_news_tech_limit() -> int:
|
||||
"""research 模式下技术类时讯条数(叠加在 DAILY_WECOM_AI_NEWS 之上)。"""
|
||||
return max(0, env_int("DAILY_WECOM_AI_NEWS_TECH", 5))
|
||||
|
||||
|
||||
def wecom_pad_pool_size(display_limit: int) -> int:
|
||||
"""Delta 补榜候选池大小(展示条数之上多取,避免去重后凑不满)。"""
|
||||
explicit = env_int("DAILY_WECOM_PAD_POOL", -1)
|
||||
if explicit > 0:
|
||||
return explicit
|
||||
return max(display_limit * 5, 50)
|
||||
|
||||
|
||||
def full_desc_limit() -> int:
|
||||
"""完整版早报摘要长度;0 表示不截断。"""
|
||||
return env_int("DAILY_FULL_DESC_LIMIT", 0)
|
||||
@@ -45,16 +61,11 @@ def wecom_max_bytes() -> int:
|
||||
"""兼容旧配置名。"""
|
||||
return wecom_chunk_bytes()
|
||||
|
||||
|
||||
load_dotenv(ROOT / ".env")
|
||||
load_dotenv(ROOT / ".env.local", override=True)
|
||||
|
||||
|
||||
def ensure_bot_on_path() -> None:
|
||||
bot = str(BOT_DIR)
|
||||
if bot not in sys.path:
|
||||
sys.path.insert(0, bot)
|
||||
|
||||
|
||||
def _clean_env_value(raw: str | None) -> str | None:
|
||||
if raw is None:
|
||||
return None
|
||||
@@ -79,3 +90,85 @@ def env_int(key: str, default: int) -> int:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def env_bool(key: str, default: bool) -> bool:
|
||||
raw = env(key)
|
||||
if raw is None:
|
||||
return default
|
||||
return raw.strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def wecom_mode() -> str:
|
||||
raw = (env("DAILY_WECOM_MODE") or "delta").strip().lower()
|
||||
return raw if raw in {"full", "delta"} else "delta"
|
||||
|
||||
|
||||
def news_dedup_days() -> int:
|
||||
return max(1, env_int("DAILY_NEWS_DEDUP_DAYS", 7))
|
||||
|
||||
|
||||
def skip_push_when_silent() -> bool:
|
||||
return env_bool("DAILY_SKIP_PUSH_WHEN_SILENT", True)
|
||||
|
||||
|
||||
def delta_baseline_fallback() -> str:
|
||||
raw = (env("DAILY_DELTA_BASELINE_FALLBACK") or "full").strip().lower()
|
||||
return raw if raw in {"full", "empty"} else "full"
|
||||
|
||||
|
||||
def wecom_delta_pad() -> bool:
|
||||
"""Delta 模式下新入榜优先,不足时用当日 Top 榜补满;补榜排除近 N 天 baseline 已出现条目。"""
|
||||
return env_bool("DAILY_WECOM_DELTA_PAD", True)
|
||||
|
||||
|
||||
def delta_pad_lookback_days() -> int:
|
||||
"""补榜时排除近 N 天 baseline 已出现过的条目(默认与异动对比窗口一致)。"""
|
||||
fallback = env_int("DAILY_DELTA_LOOKBACK_DAYS", 7)
|
||||
return max(1, env_int("DAILY_DELTA_PAD_LOOKBACK_DAYS", fallback))
|
||||
|
||||
|
||||
def force_push() -> bool:
|
||||
return env_bool("DAILY_FORCE_PUSH", False)
|
||||
|
||||
|
||||
def schedule_timezone_name() -> str:
|
||||
return (env("DAILY_SCHEDULE_TZ") or "Asia/Shanghai").strip() or "Asia/Shanghai"
|
||||
|
||||
|
||||
def schedule_generate_at() -> str:
|
||||
return (env("DAILY_SCHEDULE_GENERATE_AT") or "08:50").strip() or "08:50"
|
||||
|
||||
|
||||
def schedule_push_at() -> str:
|
||||
return (env("DAILY_SCHEDULE_PUSH_AT") or "09:00").strip() or "09:00"
|
||||
|
||||
|
||||
def board_dedup_days() -> int:
|
||||
return max(1, env_int("DAILY_BOARD_DEDUP_DAYS", 7))
|
||||
|
||||
|
||||
def board_pool_size() -> int:
|
||||
fallback = env_int("DAILY_WECOM_SKILL_POOL", 400)
|
||||
return max(1, env_int("DAILY_BOARD_POOL_SIZE", max(200, fallback)))
|
||||
|
||||
|
||||
def featured_dedup_days() -> int:
|
||||
return max(1, env_int("DAILY_FEATURED_DEDUP_DAYS", 30))
|
||||
|
||||
|
||||
def theme_ban_days() -> int:
|
||||
return max(1, env_int("DAILY_THEME_BAN_DAYS", 7))
|
||||
|
||||
|
||||
def narrative_axis_days() -> int:
|
||||
return max(1, env_int("DAILY_NARRATIVE_AXIS_DAYS", 3))
|
||||
|
||||
|
||||
def news_backfill_enabled() -> bool:
|
||||
return env_bool("DAILY_NEWS_BACKFILL", False)
|
||||
|
||||
|
||||
def workday_only() -> bool:
|
||||
"""仅工作日生成/推送;法定节假日与周末跳过(调休补班日照常)。"""
|
||||
return env_bool("DAILY_WORKDAY_ONLY", True)
|
||||
|
||||
@@ -8,13 +8,22 @@ from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
from daily.config import OUTPUT_DIR, env_int
|
||||
from daily.config import OUTPUT_DIR, delta_baseline_fallback, env_int, wecom_mode
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
KeyFn = Callable[[dict[str, Any]], str]
|
||||
|
||||
|
||||
RECENT_BOARD_KEYS = (
|
||||
"skills_trending",
|
||||
"skills_hot",
|
||||
"github_trending",
|
||||
"github_emerging",
|
||||
"github_topic",
|
||||
)
|
||||
|
||||
|
||||
def compare_depth() -> int:
|
||||
return env_int("DAILY_DELTA_COMPARE_DEPTH", 15)
|
||||
|
||||
@@ -39,6 +48,23 @@ def _key_set(items: list[dict[str, Any]], key_fn: KeyFn, *, depth: int) -> set[s
|
||||
return {key_fn(item) for item in items[:depth] if key_fn(item)}
|
||||
|
||||
|
||||
def load_recent_board_keys(
|
||||
date_str: str,
|
||||
*,
|
||||
lookback_days: int | None = None,
|
||||
) -> dict[str, set[str]]:
|
||||
"""近 N 天各榜已展示过的 skill id / repo(不含当日,供补榜去重)。
|
||||
|
||||
委托 board_history.load_recent_shown_keys,只读 wecom_shown_keys,
|
||||
不读 movement_baseline。
|
||||
"""
|
||||
from daily.board_history import load_recent_shown_keys
|
||||
from daily.config import board_dedup_days
|
||||
|
||||
days = lookback_days if lookback_days is not None else board_dedup_days()
|
||||
return load_recent_shown_keys(date_str, lookback_days=days)
|
||||
|
||||
|
||||
def find_previous_data(date_str: str) -> tuple[str, dict[str, Any]] | None:
|
||||
"""查找最近一份早于 date_str 的 data.json。"""
|
||||
try:
|
||||
@@ -311,3 +337,40 @@ def build_movement_context(
|
||||
"skills_stable": not skills_trending_all and not skills_hot_all,
|
||||
"github_stable": not github_trending_all and not github_emerging_all and not github_topic_all,
|
||||
}
|
||||
|
||||
|
||||
def effective_wecom_mode(*, date_str: str, configured_mode: str | None = None) -> str:
|
||||
mode = configured_mode or wecom_mode()
|
||||
if mode != "delta":
|
||||
return "full"
|
||||
if find_previous_data(date_str) is None and delta_baseline_fallback() == "full":
|
||||
return "full"
|
||||
return "delta"
|
||||
|
||||
|
||||
def partition_skill_moves_for_wecom(
|
||||
trending_moves: list[dict[str, Any]],
|
||||
hot_moves: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
hot_by_id = {skill_id(m): m for m in hot_moves if skill_id(m)}
|
||||
trending_out: list[dict[str, Any]] = []
|
||||
consumed_hot: set[str] = set()
|
||||
for move in trending_moves:
|
||||
sid = skill_id(move)
|
||||
copy = dict(move)
|
||||
badges = [f"Trending #{move.get('rank', '?')}"]
|
||||
hot_match = hot_by_id.get(sid)
|
||||
if hot_match:
|
||||
badges.append(f"Hot #{hot_match.get('rank', '?')}")
|
||||
consumed_hot.add(sid)
|
||||
copy["badge"] = " · ".join(badges)
|
||||
trending_out.append(copy)
|
||||
hot_out: list[dict[str, Any]] = []
|
||||
for move in hot_moves:
|
||||
sid = skill_id(move)
|
||||
if sid in consumed_hot:
|
||||
continue
|
||||
copy = dict(move)
|
||||
copy["badge"] = f"Hot #{move.get('rank', '?')}"
|
||||
hot_out.append(copy)
|
||||
return trending_out, hot_out
|
||||
|
||||
545
daily/featured_pick.py
Normal file
545
daily/featured_pick.py
Normal file
@@ -0,0 +1,545 @@
|
||||
"""今日首推:解析 DAILY_FEATURED_PICK → 定人(月去重)→ LLM 检索 → featured JSON。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import random
|
||||
import re
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from daily.config import OUTPUT_DIR, ROOT, env, featured_dedup_days
|
||||
from daily.llm_client import extract_json_object, has_llm_configured, llm_chat
|
||||
from daily.report_data import featured_json_path, save_json
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SKILL_DIR = ROOT / "skills" / "daily-featured-pick"
|
||||
_GITHUB_REPO_RE = re.compile(r"github\.com/([^/\s#?]+/[^/\s#?]+)", re.I)
|
||||
|
||||
|
||||
def featured_identity_key(featured: dict[str, Any] | None) -> str:
|
||||
"""稳定身份:skill→id,github→repo,兜底从 url 解析。"""
|
||||
if not featured:
|
||||
return ""
|
||||
typ = str(featured.get("type") or "").lower()
|
||||
if typ == "skill" or featured.get("id"):
|
||||
sid = str(featured.get("id") or "").strip()
|
||||
if sid:
|
||||
return sid
|
||||
repo = str(featured.get("repo") or "").strip()
|
||||
if repo:
|
||||
return repo
|
||||
for field in ("url", "command", "link"):
|
||||
url = str(featured.get(field) or "")
|
||||
m = _GITHUB_REPO_RE.search(url)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return ""
|
||||
|
||||
|
||||
def load_recent_featured_keys(date_str: str, days: int | None = None) -> set[str]:
|
||||
"""近 N 日 data.featured_pick_key 并集(不含当日)。"""
|
||||
lookback = days if days is not None else featured_dedup_days()
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
return set()
|
||||
out: set[str] = set()
|
||||
for day_offset in range(1, lookback + 1):
|
||||
prev = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
|
||||
path = OUTPUT_DIR / f"{prev}.data.json"
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("读取 featured_pick_key %s 失败:%s", path, exc)
|
||||
continue
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
key = str(data.get("featured_pick_key") or "").strip()
|
||||
if not key:
|
||||
featured = data.get("featured_pick")
|
||||
if isinstance(featured, dict):
|
||||
key = featured_identity_key(featured)
|
||||
if key:
|
||||
out.add(key)
|
||||
return out
|
||||
|
||||
|
||||
def load_yesterday_featured_key(date_str: str) -> str | None:
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
return None
|
||||
prev = (dt - timedelta(days=1)).strftime("%Y-%m-%d")
|
||||
path = OUTPUT_DIR / f"{prev}.data.json"
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
key = str(data.get("featured_pick_key") or "").strip()
|
||||
if key:
|
||||
return key
|
||||
featured = data.get("featured_pick")
|
||||
if isinstance(featured, dict):
|
||||
return featured_identity_key(featured) or None
|
||||
return None
|
||||
|
||||
|
||||
def _featured_rng(date_str: str) -> random.Random:
|
||||
seed = int(hashlib.sha256(f"{date_str}:featured".encode()).hexdigest()[:16], 16)
|
||||
return random.Random(seed)
|
||||
|
||||
|
||||
def _stub_from_pool_item(item: dict[str, Any]) -> dict[str, Any]:
|
||||
repo = str(item.get("repo") or "").strip()
|
||||
if repo:
|
||||
url = str(item.get("url") or f"https://github.com/{repo}").strip()
|
||||
return {
|
||||
"type": "github",
|
||||
"title": repo.split("/")[-1],
|
||||
"repo": repo,
|
||||
"url": url,
|
||||
"command": url,
|
||||
"summary": str(item.get("description") or "")[:160],
|
||||
"why_today": "",
|
||||
"evidence": [],
|
||||
"tags": [],
|
||||
}
|
||||
sid = str(item.get("id") or "").strip()
|
||||
source = str(item.get("source") or "").strip()
|
||||
title = str(item.get("title") or "").strip()
|
||||
return {
|
||||
"type": "skill",
|
||||
"id": sid,
|
||||
"title": title or sid,
|
||||
"command": _skill_command(item),
|
||||
"url": str(item.get("link") or ""),
|
||||
"summary": str(item.get("description") or "")[:160],
|
||||
"why_today": "",
|
||||
"evidence": [],
|
||||
"tags": [],
|
||||
"source": source,
|
||||
}
|
||||
|
||||
|
||||
def featured_resolve(
|
||||
*,
|
||||
date_str: str,
|
||||
candidate: dict[str, Any] | None,
|
||||
pool_a: list[dict[str, Any]],
|
||||
pool_b: list[dict[str, Any]],
|
||||
recent_featured: set[str] | None = None,
|
||||
yesterday_key: str | None = None,
|
||||
rng: random.Random | None = None,
|
||||
) -> tuple[dict[str, Any] | None, str | None]:
|
||||
"""若与昨日同一身份则改推;返回 (seed_stub, identity_key),不含完整 why。"""
|
||||
if not candidate:
|
||||
return None, None
|
||||
key = featured_identity_key(candidate)
|
||||
if not yesterday_key or key != yesterday_key:
|
||||
return candidate, key or None
|
||||
|
||||
blocked = set(recent_featured or set()) | {yesterday_key}
|
||||
picker = rng or _featured_rng(date_str)
|
||||
|
||||
def _choices(pool: list[dict[str, Any]]) -> list[tuple[str, dict[str, Any]]]:
|
||||
out: list[tuple[str, dict[str, Any]]] = []
|
||||
seen: set[str] = set()
|
||||
for item in pool:
|
||||
ik = featured_identity_key(item)
|
||||
if not ik or ik in blocked or ik in seen:
|
||||
continue
|
||||
seen.add(ik)
|
||||
out.append((ik, item))
|
||||
return out
|
||||
|
||||
options = _choices(pool_a)
|
||||
if not options:
|
||||
options = _choices(pool_b)
|
||||
if not options:
|
||||
logger.info("featured_fallback_exhausted")
|
||||
return candidate, key
|
||||
|
||||
chosen_key, chosen_item = picker.choice(options)
|
||||
return _stub_from_pool_item(chosen_item), chosen_key
|
||||
|
||||
|
||||
def _config_from_candidate(candidate: dict[str, Any]) -> dict[str, str]:
|
||||
typ = str(candidate.get("type") or "").lower()
|
||||
if typ == "skill" or candidate.get("id"):
|
||||
query = str(candidate.get("id") or candidate.get("title") or "").strip()
|
||||
return {"query": query, "url_hint": str(candidate.get("url") or "")}
|
||||
repo = str(candidate.get("repo") or "").strip()
|
||||
if repo:
|
||||
return {
|
||||
"query": repo,
|
||||
"url_hint": str(candidate.get("url") or f"https://github.com/{repo}"),
|
||||
}
|
||||
query = str(candidate.get("title") or candidate.get("url") or "").strip()
|
||||
return {"query": query or "featured", "url_hint": str(candidate.get("url") or "")}
|
||||
|
||||
|
||||
def _seed_candidate_from_config(
|
||||
config: dict[str, str],
|
||||
llm_input: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
matches = match_in_data(llm_input, config["query"])
|
||||
if matches["skills"]:
|
||||
return _stub_from_pool_item(matches["skills"][0])
|
||||
if matches["github"]:
|
||||
return _stub_from_pool_item(matches["github"][0])
|
||||
url = config.get("url_hint") or ""
|
||||
seed: dict[str, Any] = {
|
||||
"type": "other",
|
||||
"title": config["query"],
|
||||
"url": url,
|
||||
"command": url or config["query"],
|
||||
}
|
||||
m = _GITHUB_REPO_RE.search(url)
|
||||
if m:
|
||||
seed["type"] = "github"
|
||||
seed["repo"] = m.group(1)
|
||||
return seed
|
||||
|
||||
|
||||
def parse_featured_pick() -> dict[str, str] | None:
|
||||
"""解析 DAILY_FEATURED_PICK:query 或 query|url。"""
|
||||
raw = (env("DAILY_FEATURED_PICK") or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
if "|" in raw:
|
||||
query, url_hint = raw.split("|", 1)
|
||||
query = query.strip()
|
||||
url_hint = url_hint.strip()
|
||||
if not query:
|
||||
return None
|
||||
payload: dict[str, str] = {"query": query}
|
||||
if url_hint:
|
||||
payload["url_hint"] = url_hint
|
||||
return payload
|
||||
return {"query": raw}
|
||||
|
||||
|
||||
def _matches_query(text: str, query: str) -> bool:
|
||||
return query.lower() in (text or "").lower()
|
||||
|
||||
|
||||
def _skill_matches(item: dict[str, Any], query: str) -> bool:
|
||||
for key in ("id", "title", "source"):
|
||||
if _matches_query(str(item.get(key) or ""), query):
|
||||
return True
|
||||
for sub in item.get("cluster_skills") or []:
|
||||
if isinstance(sub, str) and _matches_query(sub, query):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def match_in_data(llm_input: dict[str, Any], query: str) -> dict[str, list[dict[str, Any]]]:
|
||||
"""在榜单数据中模糊匹配 query。"""
|
||||
skills: list[dict[str, Any]] = []
|
||||
seen_skill: set[str] = set()
|
||||
for board in ("skills_trending", "skills_hot"):
|
||||
for item in llm_input.get(board) or []:
|
||||
sid = str(item.get("id") or "")
|
||||
if sid in seen_skill:
|
||||
continue
|
||||
if _skill_matches(item, query):
|
||||
seen_skill.add(sid)
|
||||
skills.append({**item, "board": board})
|
||||
if len(skills) >= 5:
|
||||
break
|
||||
if len(skills) >= 5:
|
||||
break
|
||||
|
||||
github: list[dict[str, Any]] = []
|
||||
seen_repo: set[str] = set()
|
||||
for board in ("github_trending", "github_emerging"):
|
||||
for item in llm_input.get(board) or []:
|
||||
repo = str(item.get("repo") or "")
|
||||
if not repo or repo in seen_repo:
|
||||
continue
|
||||
if _matches_query(repo, query):
|
||||
seen_repo.add(repo)
|
||||
github.append({**item, "board": board})
|
||||
if len(github) >= 5:
|
||||
break
|
||||
|
||||
topic = llm_input.get("github_topic") or {}
|
||||
for item in topic.get("repos") or []:
|
||||
repo = str(item.get("repo") or "")
|
||||
if not repo or repo in seen_repo:
|
||||
continue
|
||||
if _matches_query(repo, query):
|
||||
seen_repo.add(repo)
|
||||
github.append({**item, "board": "github_topic"})
|
||||
if len(github) >= 5:
|
||||
break
|
||||
|
||||
return {"skills": skills, "github": github}
|
||||
|
||||
|
||||
def _load_skill() -> str:
|
||||
path = _SKILL_DIR / "SKILL.md"
|
||||
if path.exists():
|
||||
return path.read_text(encoding="utf-8").strip()
|
||||
return "你是早报编辑。根据输入检索今日首推信息,只输出 JSON。"
|
||||
|
||||
|
||||
def _skill_command(item: dict[str, Any]) -> str:
|
||||
source = str(item.get("source") or "").strip()
|
||||
title = str(item.get("title") or "").strip()
|
||||
if source and title:
|
||||
return f"npx skills add {source}/{title}"
|
||||
sid = str(item.get("id") or "").strip()
|
||||
if sid.count("/") >= 2:
|
||||
parts = sid.split("/", 2)
|
||||
return f"npx skills add {parts[0]}/{parts[1]}/{parts[2]}"
|
||||
if sid.count("/") == 1:
|
||||
return f"npx skills add {sid}"
|
||||
return ""
|
||||
|
||||
|
||||
def _evidence_from_skill(item: dict[str, Any]) -> list[str]:
|
||||
board = item.get("board", "")
|
||||
board_label = {
|
||||
"skills_trending": "Skills Trending",
|
||||
"skills_hot": "Skills Hot",
|
||||
}.get(str(board), str(board))
|
||||
installs = item.get("installs_fmt") or item.get("installs")
|
||||
title = item.get("title") or item.get("id") or "?"
|
||||
if installs:
|
||||
return [f"{board_label} 匹配 · {title} · {installs}"]
|
||||
return [f"{board_label} 匹配 · {title}"]
|
||||
|
||||
|
||||
def _evidence_from_github(item: dict[str, Any]) -> list[str]:
|
||||
board = item.get("board", "")
|
||||
board_label = {
|
||||
"github_trending": "GitHub Trending",
|
||||
"github_emerging": "GitHub 新兴",
|
||||
"github_topic": "GitHub Topic",
|
||||
}.get(str(board), str(board))
|
||||
repo = item.get("repo") or "?"
|
||||
stars = item.get("total_stars_fmt") or ""
|
||||
if stars:
|
||||
return [f"{board_label} 匹配 · {repo} · ⭐{stars}"]
|
||||
return [f"{board_label} 匹配 · {repo}"]
|
||||
|
||||
|
||||
def _fallback_featured(
|
||||
config: dict[str, str],
|
||||
llm_input: dict[str, Any],
|
||||
*,
|
||||
partial: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""LLM 不可用或失败时,用榜单匹配 + 配置回退。"""
|
||||
partial = partial or {}
|
||||
matches = match_in_data(llm_input, config["query"])
|
||||
skill = matches["skills"][0] if matches["skills"] else None
|
||||
gh = matches["github"][0] if matches["github"] else None
|
||||
|
||||
if skill:
|
||||
featured: dict[str, Any] = {
|
||||
"title": str(skill.get("title") or config["query"]),
|
||||
"type": "skill",
|
||||
"command": _skill_command(skill),
|
||||
"url": str(skill.get("link") or config.get("url_hint") or ""),
|
||||
"summary": str(partial.get("summary") or skill.get("description") or "")[:160],
|
||||
"why_today": str(
|
||||
partial.get("why_today")
|
||||
or f"今日 Skills 榜匹配到 **{skill.get('title') or config['query']}**,适合作为首推。"
|
||||
),
|
||||
"evidence": list(partial.get("evidence") or _evidence_from_skill(skill)),
|
||||
"tags": list(partial.get("tags") or []),
|
||||
}
|
||||
if skill.get("id"):
|
||||
featured["id"] = skill["id"]
|
||||
return featured
|
||||
|
||||
if gh:
|
||||
return {
|
||||
"title": str(gh.get("repo") or config["query"]).split("/")[-1],
|
||||
"type": "github",
|
||||
"command": str(gh.get("url") or config.get("url_hint") or ""),
|
||||
"url": str(gh.get("url") or config.get("url_hint") or ""),
|
||||
"summary": str(partial.get("summary") or gh.get("description") or "")[:160],
|
||||
"why_today": str(
|
||||
partial.get("why_today")
|
||||
or f"今日 GitHub 榜匹配到 **{gh.get('repo')}**,适合作为首推。"
|
||||
),
|
||||
"evidence": list(partial.get("evidence") or _evidence_from_github(gh)),
|
||||
"tags": list(partial.get("tags") or []),
|
||||
"repo": gh.get("repo"),
|
||||
}
|
||||
|
||||
url = config.get("url_hint") or ""
|
||||
return {
|
||||
"title": config["query"],
|
||||
"type": "other",
|
||||
"command": url or config["query"],
|
||||
"url": url,
|
||||
"summary": str(partial.get("summary") or "")[:160],
|
||||
"why_today": str(
|
||||
partial.get("why_today")
|
||||
or f"**{config['query']}** 未出现在今日 Top 榜,仍值得单独关注。"
|
||||
),
|
||||
"evidence": list(partial.get("evidence") or ([f"主推 · {config['query']}"])),
|
||||
"tags": list(partial.get("tags") or []),
|
||||
}
|
||||
|
||||
|
||||
def _normalize_featured(
|
||||
raw: dict[str, Any],
|
||||
config: dict[str, str],
|
||||
llm_input: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""补齐 command / url / evidence,并与榜单数字对齐。"""
|
||||
matches = match_in_data(llm_input, config["query"])
|
||||
skill = matches["skills"][0] if matches["skills"] else None
|
||||
gh = matches["github"][0] if matches["github"] else None
|
||||
|
||||
featured = dict(raw)
|
||||
featured.setdefault("title", config["query"])
|
||||
featured.setdefault("type", "other")
|
||||
|
||||
if skill and featured.get("type") in {"skill", "other", ""}:
|
||||
featured.setdefault("id", skill.get("id"))
|
||||
featured.setdefault("command", _skill_command(skill))
|
||||
featured.setdefault("url", skill.get("link") or config.get("url_hint") or "")
|
||||
if not featured.get("evidence"):
|
||||
featured["evidence"] = _evidence_from_skill(skill)
|
||||
featured["type"] = "skill"
|
||||
elif gh and featured.get("type") in {"github", "other", ""}:
|
||||
featured.setdefault("repo", gh.get("repo"))
|
||||
featured.setdefault("url", gh.get("url") or config.get("url_hint") or "")
|
||||
featured.setdefault("command", featured.get("url") or gh.get("url") or "")
|
||||
if not featured.get("evidence"):
|
||||
featured["evidence"] = _evidence_from_github(gh)
|
||||
featured["type"] = "github"
|
||||
|
||||
featured.setdefault("command", config.get("url_hint") or config["query"])
|
||||
featured.setdefault("url", config.get("url_hint") or "")
|
||||
featured.setdefault("summary", "")
|
||||
featured.setdefault("why_today", featured.get("summary") or "")
|
||||
featured.setdefault("evidence", [])
|
||||
featured.setdefault("tags", [])
|
||||
return featured
|
||||
|
||||
|
||||
def research_featured_pick(
|
||||
llm_input: dict[str, Any],
|
||||
*,
|
||||
date_str: str,
|
||||
config: dict[str, str] | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Step 0:检索今日首推;成功返回 featured dict,未配置返回 None。"""
|
||||
config = config or parse_featured_pick()
|
||||
if not config:
|
||||
return None
|
||||
|
||||
matches = match_in_data(llm_input, config["query"])
|
||||
payload = {
|
||||
"query": config["query"],
|
||||
"url_hint": config.get("url_hint"),
|
||||
"cwd": env("DAILY_CURSOR_CWD") or str(ROOT),
|
||||
"data_matches": matches,
|
||||
}
|
||||
|
||||
if not has_llm_configured():
|
||||
featured = _fallback_featured(config, llm_input)
|
||||
save_json(featured_json_path(date_str), featured)
|
||||
logger.info("Featured pick(无 LLM,规则回退):%s", featured.get("title"))
|
||||
return featured
|
||||
|
||||
skill = _load_skill()
|
||||
system = (
|
||||
f"{skill}\n\n"
|
||||
"当前执行 **Step 0:今日首推检索**。\n"
|
||||
"只输出 featured JSON(title, type, command, url, summary, why_today, evidence, tags),"
|
||||
"不要 Markdown,不要解释。"
|
||||
)
|
||||
user = json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
try:
|
||||
raw = llm_chat(system, user)
|
||||
except Exception as exc:
|
||||
logger.warning("Featured pick LLM 失败,回退规则模式:%s", exc)
|
||||
featured = _fallback_featured(config, llm_input)
|
||||
save_json(featured_json_path(date_str), featured)
|
||||
return featured
|
||||
|
||||
if not raw:
|
||||
featured = _fallback_featured(config, llm_input)
|
||||
save_json(featured_json_path(date_str), featured)
|
||||
return featured
|
||||
|
||||
parsed = extract_json_object(raw)
|
||||
if not parsed.get("why_today") and not parsed.get("summary"):
|
||||
logger.warning("Featured pick JSON 无效,回退规则模式")
|
||||
featured = _fallback_featured(config, llm_input, partial=parsed)
|
||||
save_json(featured_json_path(date_str), featured)
|
||||
return featured
|
||||
|
||||
featured = _normalize_featured(parsed, config, llm_input)
|
||||
save_json(featured_json_path(date_str), featured)
|
||||
logger.info("Featured pick 完成:%s", featured.get("title"))
|
||||
return featured
|
||||
|
||||
|
||||
def apply_featured_pick(
|
||||
llm_input: dict[str, Any],
|
||||
*,
|
||||
date_str: str,
|
||||
pool_a: list[dict[str, Any]] | None = None,
|
||||
pool_b: list[dict[str, Any]] | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""先定人(相对昨日改推 + 月去重),再 research,写入 featured_pick / featured_pick_key。"""
|
||||
config = parse_featured_pick()
|
||||
if not config:
|
||||
return None
|
||||
|
||||
seed = _seed_candidate_from_config(config, llm_input)
|
||||
recent = load_recent_featured_keys(date_str)
|
||||
yesterday = load_yesterday_featured_key(date_str)
|
||||
resolved, identity_key = featured_resolve(
|
||||
date_str=date_str,
|
||||
candidate=seed,
|
||||
pool_a=pool_a or [],
|
||||
pool_b=pool_b or [],
|
||||
recent_featured=recent,
|
||||
yesterday_key=yesterday,
|
||||
rng=_featured_rng(date_str),
|
||||
)
|
||||
research_config = config
|
||||
if resolved and identity_key and featured_identity_key(seed) != identity_key:
|
||||
research_config = _config_from_candidate(resolved)
|
||||
|
||||
featured = research_featured_pick(
|
||||
llm_input, date_str=date_str, config=research_config
|
||||
)
|
||||
if featured:
|
||||
llm_input["featured_pick"] = featured
|
||||
key = identity_key or featured_identity_key(featured)
|
||||
if key:
|
||||
llm_input["featured_pick_key"] = key
|
||||
return featured
|
||||
|
||||
|
||||
def pick_command_from_featured(featured: dict[str, Any] | None) -> str | None:
|
||||
cmd = str((featured or {}).get("command") or "").strip()
|
||||
return cmd or None
|
||||
|
||||
|
||||
def pick_why_from_featured(featured: dict[str, Any] | None) -> str | None:
|
||||
why = str((featured or {}).get("why_today") or "").strip()
|
||||
return why or None
|
||||
File diff suppressed because it is too large
Load Diff
@@ -22,15 +22,39 @@ from daily.config import (
|
||||
LOG_DIR,
|
||||
OUTPUT_DIR,
|
||||
SNAPSHOT_FILE,
|
||||
ensure_bot_on_path,
|
||||
board_pool_size,
|
||||
env,
|
||||
env_bool,
|
||||
env_int,
|
||||
full_desc_limit,
|
||||
news_summary_limit,
|
||||
wecom_delta_pad,
|
||||
wecom_news_desc_limit,
|
||||
wecom_pad_pool_size,
|
||||
wecom_skill_desc_limit,
|
||||
)
|
||||
from daily.format_wecom import build_wecom_report, finalize_wecom_skill_groups, replace_wecom_skill_sections
|
||||
from daily.board_history import (
|
||||
BOARD_KEYS,
|
||||
extract_shown_keys,
|
||||
load_recent_shown_keys,
|
||||
merge_wecom_shown_into_data,
|
||||
)
|
||||
from daily.board_select import board_select
|
||||
from daily.format_wecom import (
|
||||
build_wecom_report,
|
||||
finalize_wecom_skill_groups,
|
||||
replace_wecom_board_sections,
|
||||
replace_wecom_news_sections,
|
||||
replace_wecom_skill_sections,
|
||||
resolve_wecom_board_items,
|
||||
)
|
||||
from daily.agent_workflow import is_agent_mode, run_agent_workflow
|
||||
from daily.featured_pick import (
|
||||
apply_featured_pick,
|
||||
featured_identity_key,
|
||||
pick_command_from_featured,
|
||||
pick_why_from_featured,
|
||||
)
|
||||
from daily.delta import compare_depth
|
||||
from daily.cursor_editor import (
|
||||
apply_descriptions,
|
||||
@@ -42,25 +66,31 @@ from daily.github.auth import github_html_headers
|
||||
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos
|
||||
from daily.github.trending import fetch_github_trending, trending_data_source_note
|
||||
from daily.localize import LocalizeJob, localize_descriptions, needs_chinese
|
||||
from daily.narrative_axis import theme_clusters, theme_names
|
||||
from daily.news.fetch import (
|
||||
fetch_ai_news,
|
||||
fetch_cn_ai_news,
|
||||
format_cn_news_section,
|
||||
format_news_section,
|
||||
finalize_wecom_news_items,
|
||||
prepare_wecom_cn_news_items,
|
||||
prepare_wecom_news_items,
|
||||
sync_wecom_news_rows,
|
||||
)
|
||||
from daily.news.pushed_links import record_pushed_links
|
||||
from daily.news.research import (
|
||||
fetch_ai_news_research,
|
||||
format_research_news_section,
|
||||
is_research_mode,
|
||||
)
|
||||
from daily.push_gate import evaluate_push_gate
|
||||
from daily.report_data import (
|
||||
build_full_payload,
|
||||
build_llm_input,
|
||||
data_json_path,
|
||||
save_json,
|
||||
)
|
||||
from daily.skills_board import load_boards
|
||||
from daily.skills_group import group_skills_by_source
|
||||
|
||||
ensure_bot_on_path()
|
||||
from skills_service import _format_installs, load_feed # noqa: E402
|
||||
from daily.skills_board import format_installs, load_boards, load_feed
|
||||
|
||||
THEME_RULES: list[tuple[str, str, list[str]]] = [
|
||||
("🎬", "AI 多媒体 / 视频", ["runcomfy", "remotion", "video", "seedance", "inpaint", "lipsync"]),
|
||||
@@ -132,7 +162,30 @@ def _localize_descriptions_in_place(
|
||||
seen_news.add(link)
|
||||
summary = (item.get("summary") or "").strip()
|
||||
if summary:
|
||||
jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
|
||||
jobs.append(
|
||||
LocalizeJob(
|
||||
f"news:{link}",
|
||||
summary,
|
||||
news_limit if news_limit > 0 else wecom_news_desc_limit(),
|
||||
)
|
||||
)
|
||||
|
||||
if cn_ai_news and cn_ai_news.get("enabled"):
|
||||
seen_cn: set[str] = set()
|
||||
for item in cn_ai_news.get("flat") or []:
|
||||
link = item.get("link", "")
|
||||
if not link or link in seen_cn:
|
||||
continue
|
||||
seen_cn.add(link)
|
||||
summary = (item.get("summary") or "").strip()
|
||||
if summary and needs_chinese(summary):
|
||||
jobs.append(
|
||||
LocalizeJob(
|
||||
f"news:{link}",
|
||||
summary,
|
||||
news_limit if news_limit > 0 else wecom_news_desc_limit(),
|
||||
)
|
||||
)
|
||||
|
||||
zh_map = localize_descriptions(jobs, archive=True)
|
||||
if not zh_map and not jobs:
|
||||
@@ -158,6 +211,11 @@ def _localize_descriptions_in_place(
|
||||
key = f"news:{item.get('link', '')}"
|
||||
if key in mapping:
|
||||
item["summary"] = mapping[key]
|
||||
if cn_ai_news and cn_ai_news.get("enabled"):
|
||||
for item in cn_ai_news.get("flat") or []:
|
||||
key = f"news:{item.get('link', '')}"
|
||||
if key in mapping:
|
||||
item["summary"] = mapping[key]
|
||||
|
||||
_apply_zh(zh_map)
|
||||
|
||||
@@ -191,7 +249,29 @@ def _localize_descriptions_in_place(
|
||||
seen_news.add(link)
|
||||
summary = (item.get("summary") or "").strip()
|
||||
if needs_chinese(summary):
|
||||
retry_jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
|
||||
retry_jobs.append(
|
||||
LocalizeJob(
|
||||
f"news:{link}",
|
||||
summary,
|
||||
news_limit if news_limit > 0 else wecom_news_desc_limit(),
|
||||
)
|
||||
)
|
||||
if cn_ai_news and cn_ai_news.get("enabled"):
|
||||
seen_cn: set[str] = set()
|
||||
for item in cn_ai_news.get("flat") or []:
|
||||
link = item.get("link", "")
|
||||
if not link or link in seen_cn:
|
||||
continue
|
||||
seen_cn.add(link)
|
||||
summary = (item.get("summary") or "").strip()
|
||||
if needs_chinese(summary):
|
||||
retry_jobs.append(
|
||||
LocalizeJob(
|
||||
f"news:{link}",
|
||||
summary,
|
||||
news_limit if news_limit > 0 else wecom_news_desc_limit(),
|
||||
)
|
||||
)
|
||||
|
||||
if retry_jobs:
|
||||
_apply_zh(localize_descriptions(retry_jobs, archive=True))
|
||||
@@ -232,7 +312,7 @@ def _prepare_skill_item(item: dict[str, Any], prev_ids: set[str], rank: int) ->
|
||||
badge = "🆕"
|
||||
elif rank == 1:
|
||||
badge = "👑"
|
||||
installs_fmt = item.get("installs_fmt") or _format_installs(item.get("installs", 0))
|
||||
installs_fmt = item.get("installs_fmt") or format_installs(item.get("installs", 0))
|
||||
title = item.get("source", "?") if item.get("cluster") else item.get("title", "?")
|
||||
desc = item.get("wecom_desc") or item.get("description") or item.get("cluster_titles") or ""
|
||||
limit = wecom_skill_desc_limit()
|
||||
@@ -266,6 +346,33 @@ def _detect_theme_line(feed: dict[str, Any]) -> str:
|
||||
return f"**今日主题**:{max(scores.items(), key=lambda x: x[1])[0]}"
|
||||
|
||||
|
||||
def _top_line(feed: dict[str, Any]) -> str:
|
||||
"""今日看点行: 优先评分最高的主题, 回退 _theme_clusters 的主题名(非 markdown 示例)。
|
||||
|
||||
DAILY_WECOM_TOP_LINE=0 关闭时退回 _detect_theme_line 原逻辑。
|
||||
与原 _detect_theme_line 的差别仅在「无评分命中」时的兜底文案:
|
||||
用 theme_names 的真实主题取代硬编码「Agent Skills 生态持续活跃」。
|
||||
"""
|
||||
if not env_bool("DAILY_WECOM_TOP_LINE", True):
|
||||
return _detect_theme_line(feed)
|
||||
scores: dict[str, int] = defaultdict(int)
|
||||
for board in ("topTrending", "topHot"):
|
||||
for rank, item in enumerate(feed.get(board, [])[:10], 1):
|
||||
haystack = " ".join(
|
||||
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
|
||||
).lower()
|
||||
for _icon, label, keywords in THEME_RULES:
|
||||
if any(k in haystack for k in keywords):
|
||||
scores[label] += max(1, 11 - rank)
|
||||
break
|
||||
if scores:
|
||||
return f"**今日主题**:{max(scores.items(), key=lambda x: x[1])[0]}"
|
||||
names = theme_names(feed, theme_rules=THEME_RULES, skill_id_fn=_skill_id)
|
||||
if names:
|
||||
return f"**今日主题**:{' · '.join(names)}"
|
||||
return "**今日主题**:Agent Skills 生态持续活跃"
|
||||
|
||||
|
||||
def _build_highlights(
|
||||
trending: list[dict[str, Any]],
|
||||
hot: list[dict[str, Any]],
|
||||
@@ -303,7 +410,7 @@ def _build_highlights(
|
||||
)
|
||||
if trending:
|
||||
t0 = trending[0]
|
||||
points.append(f"📈 Skills 榜首 **{t0.get('title')}**({_format_installs(t0.get('installs', 0))})")
|
||||
points.append(f"📈 Skills 榜首 **{t0.get('title')}**({format_installs(t0.get('installs', 0))})")
|
||||
if github_trending:
|
||||
g0 = github_trending[0]
|
||||
stars = g0.get("stars_today_fmt", "")
|
||||
@@ -315,7 +422,7 @@ def _build_highlights(
|
||||
points.append(f"🌱 新兴 [{e0['repo']}]({e0['url']})(⭐ {e0.get('total_stars_fmt', '?')})")
|
||||
elif hot:
|
||||
h0 = hot[0]
|
||||
points.append(f"🔥 Skills Hot 榜首 **{h0.get('title')}**(1H {_format_installs(h0.get('installs', 0))})")
|
||||
points.append(f"🔥 Skills Hot 榜首 **{h0.get('title')}**(1H {format_installs(h0.get('installs', 0))})")
|
||||
while len(points) < 3 and len(trending) > len(points):
|
||||
item = trending[len(points)]
|
||||
points.append(f"✨ **{item.get('title')}** · `{item.get('source')}`")
|
||||
@@ -326,6 +433,27 @@ def _prepare_github_item(item: dict[str, Any]) -> dict[str, Any]:
|
||||
return {**item, "desc_short": _wecom_desc(item.get("description", ""), 40)}
|
||||
|
||||
|
||||
def _sync_movement_github_descriptions(
|
||||
movement: dict[str, Any],
|
||||
*,
|
||||
github_trending: list[dict[str, Any]],
|
||||
github_emerging: list[dict[str, Any]],
|
||||
github_topic: list[dict[str, Any]],
|
||||
) -> None:
|
||||
"""将已中文化的 GitHub 描述同步到 movement 新入榜条目(供 Delta 企微列表使用)。"""
|
||||
by_repo: dict[str, str] = {}
|
||||
for item in github_trending + github_emerging + github_topic:
|
||||
repo = str(item.get("repo") or "")
|
||||
desc = (item.get("description") or "").strip()
|
||||
if repo and desc:
|
||||
by_repo[repo] = desc
|
||||
for key in ("github_trending_moves", "github_emerging_moves", "github_topic_moves"):
|
||||
for move in movement.get(key) or []:
|
||||
repo = str(move.get("repo") or "")
|
||||
if repo in by_repo:
|
||||
move["description"] = by_repo[repo]
|
||||
|
||||
|
||||
def _fetch_latest_release_title(repo: str) -> str | None:
|
||||
atom_url = f"https://github.com/{repo}/releases.atom"
|
||||
try:
|
||||
@@ -345,31 +473,11 @@ def _fetch_latest_release_title(repo: str) -> str | None:
|
||||
return None
|
||||
title = entry.find("a:title", ns)
|
||||
return title.text.strip() if title is not None and title.text else None
|
||||
except Exception:
|
||||
except (httpx.HTTPError, ET.ParseError, OSError) as exc:
|
||||
logger.warning("读取 %s release 失败:%s", atom_url, exc)
|
||||
return None
|
||||
|
||||
|
||||
def _theme_clusters(feed: dict[str, Any], limit: int = 5) -> list[tuple[str, list[str]]]:
|
||||
buckets: dict[str, list[str]] = defaultdict(list)
|
||||
seen: set[str] = set()
|
||||
for board in ("topTrending", "topHot"):
|
||||
for item in feed.get(board, [])[:20]:
|
||||
item_id = _skill_id(item)
|
||||
if item_id in seen:
|
||||
continue
|
||||
seen.add(item_id)
|
||||
haystack = " ".join(
|
||||
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
|
||||
).lower()
|
||||
for _icon, theme, keywords in THEME_RULES:
|
||||
if any(k in haystack for k in keywords):
|
||||
label = f"**{item.get('title')}** (`{item.get('source')}`)"
|
||||
if label not in buckets[theme]:
|
||||
buckets[theme].append(label)
|
||||
break
|
||||
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
|
||||
|
||||
|
||||
def _format_github_repo_section(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
|
||||
lines: list[str] = []
|
||||
for i, repo in enumerate(repos, 1):
|
||||
@@ -397,7 +505,7 @@ def _format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) ->
|
||||
for i, item in enumerate(items, 1):
|
||||
skill_id = item.get("id") or f"{item.get('source', '?')}/{item.get('title', '?')}"
|
||||
link = item.get("link", "")
|
||||
installs = _format_installs(item.get("installs", 0))
|
||||
installs = format_installs(item.get("installs", 0))
|
||||
meta = f"1H {installs}" if hot else f"总安装 {installs}"
|
||||
if link:
|
||||
lines.append(f"{i}. **[{skill_id}]({link})** · {meta}")
|
||||
@@ -410,22 +518,31 @@ def _format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) ->
|
||||
return lines
|
||||
|
||||
|
||||
def generate_report() -> tuple[str, str, Path, Path]:
|
||||
trending_n = env_int("DAILY_TRENDING_LIMIT", 150)
|
||||
hot_n = max(env_int("DAILY_HOT_LIMIT", 150), compare_depth())
|
||||
def _collect(date_str: str) -> dict[str, Any]:
|
||||
"""抓取 skills/github/news 数据并归一化,计算 wecom 限额与周去重 recent keys。
|
||||
|
||||
无副作用(不写快照/不记已推)。返回供 _select/_render 消费的 bundle。
|
||||
"""
|
||||
# Hot/Trending 前排同 source 极密,需更深抓取才能凑够展示用的唯一 source
|
||||
trending_n = env_int("DAILY_TRENDING_LIMIT", 400)
|
||||
hot_n = max(env_int("DAILY_HOT_LIMIT", 400), compare_depth())
|
||||
compare_n = compare_depth()
|
||||
skill_pool = max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
|
||||
wecom_trending = env_int("DAILY_WECOM_TRENDING", 10)
|
||||
wecom_hot = env_int("DAILY_WECOM_HOT", 10)
|
||||
skill_pool = max(10, env_int("DAILY_WECOM_SKILL_POOL", 400))
|
||||
wecom_trending = env_int("DAILY_WECOM_TRENDING", 5)
|
||||
wecom_hot = env_int("DAILY_WECOM_HOT", 5)
|
||||
pad_pool = wecom_pad_pool_size(max(wecom_trending, wecom_hot, 5))
|
||||
skill_pool = max(skill_pool, pad_pool)
|
||||
github_limit = env_int("DAILY_GITHUB_TRENDING_LIMIT", 10)
|
||||
wecom_github = env_int("DAILY_WECOM_GITHUB_TRENDING", env_int("DAILY_WECOM_REPOS", 10))
|
||||
github_fetch_n = max(github_limit, compare_n, wecom_github)
|
||||
wecom_github = env_int("DAILY_WECOM_GITHUB_TRENDING", env_int("DAILY_WECOM_REPOS", 5))
|
||||
# 周去重后顶刊 sticky,HTML/~30 条不够补满;深池默认 100(Search 已分页)
|
||||
github_pool = max(pad_pool, env_int("DAILY_GITHUB_POOL", 100))
|
||||
github_fetch_n = max(github_limit, compare_n, wecom_github, github_pool)
|
||||
emerging_limit = env_int("DAILY_GITHUB_EMERGING_LIMIT", 10)
|
||||
wecom_emerging = env_int("DAILY_WECOM_GITHUB_EMERGING", 10)
|
||||
emerging_fetch_n = max(emerging_limit, compare_n, wecom_emerging)
|
||||
wecom_emerging = env_int("DAILY_WECOM_GITHUB_EMERGING", 5)
|
||||
emerging_fetch_n = max(emerging_limit, compare_n, wecom_emerging, github_pool)
|
||||
topic_limit = env_int("DAILY_GITHUB_TOPIC_LIMIT", 10)
|
||||
wecom_topic = env_int("DAILY_WECOM_GITHUB_TOPIC", 10)
|
||||
topic_fetch_n = max(topic_limit, compare_n, wecom_topic)
|
||||
wecom_topic = env_int("DAILY_WECOM_GITHUB_TOPIC", 5)
|
||||
topic_fetch_n = max(topic_limit, compare_n, wecom_topic, github_pool)
|
||||
|
||||
feed = load_feed(force=True)
|
||||
prev_ids = _load_snapshot()
|
||||
@@ -440,8 +557,27 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
github_emerging = fetch_emerging_repos(emerging_fetch_n, exclude=seen_repos)
|
||||
seen_repos.update(r["repo"] for r in github_emerging)
|
||||
topic_name, github_topic = fetch_topic_hot_repos(topic_fetch_n, exclude=seen_repos)
|
||||
ai_news = fetch_ai_news()
|
||||
cn_ai_news = fetch_cn_ai_news()
|
||||
|
||||
news_merged = is_research_mode()
|
||||
ai_news_research: dict[str, Any] | None = None
|
||||
wecom_news: list[dict[str, Any]] = []
|
||||
wecom_tech_news: list[dict[str, Any]] = []
|
||||
if news_merged:
|
||||
ai_news_research = fetch_ai_news_research(date_str=date_str)
|
||||
wecom_news = list(ai_news_research.get("items") or [])
|
||||
wecom_tech_news = list(ai_news_research.get("tech_items") or [])
|
||||
ai_news = {
|
||||
"enabled": ai_news_research.get("enabled", False),
|
||||
"mode": "research",
|
||||
"hours": ai_news_research.get("hours", 24),
|
||||
"flat": ai_news_research.get("flat") or [],
|
||||
"categories": [],
|
||||
"stats": ai_news_research.get("stats") or {},
|
||||
}
|
||||
cn_ai_news = {"enabled": False, "categories": [], "flat": [], "stats": {}}
|
||||
else:
|
||||
ai_news = fetch_ai_news()
|
||||
cn_ai_news = fetch_cn_ai_news()
|
||||
|
||||
wecom_limits = {
|
||||
"trending": wecom_trending,
|
||||
@@ -452,7 +588,137 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
"emerging": wecom_emerging,
|
||||
"topic": wecom_topic,
|
||||
"ai_news": env_int("DAILY_WECOM_AI_NEWS", 10),
|
||||
"cn_ai_news": env_int("DAILY_WECOM_CN_AI_NEWS", 8),
|
||||
"cn_ai_news": env_int("DAILY_WECOM_CN_AI_NEWS", 10),
|
||||
}
|
||||
pool = max(board_pool_size(), skill_pool, pad_pool)
|
||||
recent_shown = load_recent_shown_keys(date_str)
|
||||
from daily.skills_group import expand_skill_recent_keys
|
||||
|
||||
skill_recent = expand_skill_recent_keys(
|
||||
recent_shown["skills_trending"] | recent_shown["skills_hot"]
|
||||
)
|
||||
# GitHub 三榜共用周去重:任一类出现过的 repo 各榜都不再展示
|
||||
github_recent = (
|
||||
recent_shown["github_trending"]
|
||||
| recent_shown["github_emerging"]
|
||||
| recent_shown["github_topic"]
|
||||
)
|
||||
return {
|
||||
"trending_n": trending_n,
|
||||
"hot_n": hot_n,
|
||||
"github_limit": github_limit,
|
||||
"emerging_limit": emerging_limit,
|
||||
"topic_limit": topic_limit,
|
||||
"wecom_trending": wecom_trending,
|
||||
"wecom_hot": wecom_hot,
|
||||
"wecom_github": wecom_github,
|
||||
"wecom_emerging": wecom_emerging,
|
||||
"wecom_topic": wecom_topic,
|
||||
"wecom_limits": wecom_limits,
|
||||
"pool": pool,
|
||||
"pad_pool": pad_pool,
|
||||
"feed": feed,
|
||||
"prev_ids": prev_ids,
|
||||
"now": now,
|
||||
"date_str": date_str,
|
||||
"time_str": time_str,
|
||||
"updated": updated,
|
||||
"trending": trending,
|
||||
"hot": hot,
|
||||
"github_trending": github_trending,
|
||||
"github_emerging": github_emerging,
|
||||
"github_topic": github_topic,
|
||||
"topic_name": topic_name,
|
||||
"news_merged": news_merged,
|
||||
"ai_news": ai_news,
|
||||
"cn_ai_news": cn_ai_news,
|
||||
"ai_news_research": ai_news_research,
|
||||
"wecom_news": wecom_news,
|
||||
"wecom_tech_news": wecom_tech_news,
|
||||
"recent_shown": recent_shown,
|
||||
"skill_recent": skill_recent,
|
||||
"github_recent": github_recent,
|
||||
}
|
||||
|
||||
|
||||
def _select(c: dict[str, Any]) -> dict[str, Any]:
|
||||
"""全部「选择 + 数据突变」:9 处 board_select + featured_pick + LLM 步骤 + pad。
|
||||
|
||||
时序约束内聚本段:featured pick 依赖 deep-pool;_localize 在 featured 后、
|
||||
pad 前执行;写回 shown keys/llm_input 也在此。_render 不再做任何选择。
|
||||
副作用: apply_featured_pick 写回 shown keys(保持原时序)。
|
||||
"""
|
||||
date_str = c["date_str"]
|
||||
feed = c["feed"]
|
||||
prev_ids = c["prev_ids"]
|
||||
updated = c["updated"]
|
||||
time_str = c["time_str"]
|
||||
trending = c["trending"]
|
||||
hot = c["hot"]
|
||||
github_trending = c["github_trending"]
|
||||
github_emerging = c["github_emerging"]
|
||||
github_topic = c["github_topic"]
|
||||
topic_name = c["topic_name"]
|
||||
news_merged = c["news_merged"]
|
||||
ai_news = c["ai_news"]
|
||||
cn_ai_news = c["cn_ai_news"]
|
||||
wecom_news = c["wecom_news"]
|
||||
wecom_tech_news = c["wecom_tech_news"]
|
||||
wecom_limits = c["wecom_limits"]
|
||||
pool = c["pool"]
|
||||
pad_pool = c["pad_pool"]
|
||||
recent_shown = c["recent_shown"]
|
||||
skill_recent = c["skill_recent"]
|
||||
github_recent = c["github_recent"]
|
||||
|
||||
selected_trending = board_select(
|
||||
board="skills_trending",
|
||||
items=trending,
|
||||
recent_keys=skill_recent,
|
||||
limit=c["wecom_trending"],
|
||||
pool_size=pool,
|
||||
kind="skill",
|
||||
)
|
||||
selected_hot = board_select(
|
||||
board="skills_hot",
|
||||
items=hot,
|
||||
recent_keys=skill_recent,
|
||||
limit=c["wecom_hot"],
|
||||
pool_size=pool,
|
||||
kind="skill",
|
||||
)
|
||||
selected_github = board_select(
|
||||
board="github_trending",
|
||||
items=github_trending,
|
||||
recent_keys=github_recent,
|
||||
limit=c["wecom_github"],
|
||||
pool_size=pool,
|
||||
kind="github",
|
||||
)
|
||||
github_recent |= {str(r.get("repo") or "") for r in selected_github if r.get("repo")}
|
||||
selected_emerging = board_select(
|
||||
board="github_emerging",
|
||||
items=github_emerging,
|
||||
recent_keys=github_recent,
|
||||
limit=c["wecom_emerging"],
|
||||
pool_size=pool,
|
||||
kind="github",
|
||||
)
|
||||
github_recent |= {str(r.get("repo") or "") for r in selected_emerging if r.get("repo")}
|
||||
selected_topic = board_select(
|
||||
board="github_topic",
|
||||
items=github_topic,
|
||||
recent_keys=github_recent,
|
||||
limit=c["wecom_topic"],
|
||||
pool_size=pool,
|
||||
kind="github",
|
||||
)
|
||||
boards_for_wecom = {
|
||||
"skills_trending": selected_trending,
|
||||
"skills_hot": selected_hot,
|
||||
"github_trending": selected_github,
|
||||
"github_emerging": selected_emerging,
|
||||
"github_topic": selected_topic,
|
||||
}
|
||||
llm_input = build_llm_input(
|
||||
date_str=date_str,
|
||||
@@ -466,18 +732,67 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
ai_news=ai_news,
|
||||
cn_ai_news=cn_ai_news,
|
||||
wecom_limits=wecom_limits,
|
||||
research_items=wecom_news if news_merged else None,
|
||||
research_tech_items=wecom_tech_news if news_merged else None,
|
||||
boards_for_wecom=boards_for_wecom,
|
||||
)
|
||||
save_json(
|
||||
data_json_path(date_str),
|
||||
build_full_payload(
|
||||
llm_input,
|
||||
meta={
|
||||
"generated_at": now.isoformat(),
|
||||
"report_mode": "agent" if is_agent_mode() else "classic",
|
||||
"cursor_editor": cursor_editor_enabled() and not is_agent_mode(),
|
||||
},
|
||||
),
|
||||
pool_a: list[dict[str, Any]] = []
|
||||
pool_a_keys: set[str] = set()
|
||||
for board_name, items in boards_for_wecom.items():
|
||||
for item in items:
|
||||
keyed = dict(item)
|
||||
keyed["board"] = board_name
|
||||
pool_a.append(keyed)
|
||||
ik = featured_identity_key(keyed)
|
||||
if ik:
|
||||
pool_a_keys.add(ik)
|
||||
pool_b: list[dict[str, Any]] = []
|
||||
for board_name, raw_items, kind in (
|
||||
("skills_trending", trending, "skill"),
|
||||
("skills_hot", hot, "skill"),
|
||||
("github_trending", github_trending, "github"),
|
||||
("github_emerging", github_emerging, "github"),
|
||||
("github_topic", github_topic, "github"),
|
||||
):
|
||||
deep = board_select(
|
||||
board=board_name,
|
||||
items=raw_items,
|
||||
recent_keys=set(),
|
||||
limit=pool,
|
||||
pool_size=pool,
|
||||
kind=kind, # type: ignore[arg-type]
|
||||
)
|
||||
for item in deep:
|
||||
keyed = dict(item)
|
||||
keyed["board"] = board_name
|
||||
ik = featured_identity_key(keyed)
|
||||
if ik and ik not in pool_a_keys:
|
||||
pool_b.append(keyed)
|
||||
featured = apply_featured_pick(
|
||||
llm_input,
|
||||
date_str=date_str,
|
||||
pool_a=pool_a,
|
||||
pool_b=pool_b,
|
||||
)
|
||||
movement = llm_input["movement"]
|
||||
eff_mode = llm_input["effective_wecom_mode"]
|
||||
if news_merged:
|
||||
wecom_ai: list[dict[str, Any]] = []
|
||||
wecom_cn: list[dict[str, Any]] = []
|
||||
else:
|
||||
wecom_ai = prepare_wecom_news_items(ai_news, date_str=date_str)
|
||||
wecom_cn = prepare_wecom_cn_news_items(cn_ai_news, date_str=date_str)
|
||||
push_gate = evaluate_push_gate(
|
||||
movement=movement,
|
||||
ai_news_items=wecom_news if news_merged else wecom_ai,
|
||||
cn_ai_news_items=wecom_cn,
|
||||
featured_pick=featured,
|
||||
)
|
||||
llm_input["push_gate"] = {
|
||||
"should_push": push_gate.should_push,
|
||||
"silent": push_gate.silent,
|
||||
"reasons": push_gate.reasons,
|
||||
}
|
||||
|
||||
agent_wecom: str | None = None
|
||||
if is_agent_mode():
|
||||
@@ -513,15 +828,199 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
_localize_descriptions_in_place(
|
||||
trending, hot, github_trending, github_emerging, github_topic, ai_news, cn_ai_news
|
||||
)
|
||||
if not news_merged:
|
||||
sync_wecom_news_rows(wecom_ai, ai_news.get("flat") or [])
|
||||
sync_wecom_news_rows(wecom_cn, cn_ai_news.get("flat") or [])
|
||||
finalize_wecom_news_items(wecom_ai, force_chinese=True)
|
||||
finalize_wecom_news_items(wecom_cn, force_chinese=False)
|
||||
_sync_movement_github_descriptions(
|
||||
movement,
|
||||
github_trending=github_trending,
|
||||
github_emerging=github_emerging,
|
||||
github_topic=github_topic,
|
||||
)
|
||||
|
||||
themes = _theme_clusters(feed)
|
||||
themes = theme_clusters(feed, theme_rules=THEME_RULES, skill_id_fn=_skill_id)
|
||||
|
||||
pick_src = trending[0].get("source", "") if trending else ""
|
||||
pick_name = trending[0].get("title", "") if trending else ""
|
||||
pick_command = pick_command_from_featured(featured) or (
|
||||
f"npx skills add {pick_src}/{pick_name}"
|
||||
if pick_src and pick_name
|
||||
else "npx skills add vercel-labs/skills/find-skills"
|
||||
)
|
||||
pick_why = pick_why_from_featured(featured) or ""
|
||||
pick_title = str((featured or {}).get("title") or pick_name or "").strip()
|
||||
pick_url = str((featured or {}).get("url") or "").strip()
|
||||
|
||||
gt = selected_trending
|
||||
gh = selected_hot
|
||||
gt_pad = board_select(
|
||||
board="skills_trending",
|
||||
items=trending,
|
||||
recent_keys=skill_recent,
|
||||
limit=pad_pool,
|
||||
pool_size=pool,
|
||||
kind="skill",
|
||||
)
|
||||
gh_pad = board_select(
|
||||
board="skills_hot",
|
||||
items=hot,
|
||||
recent_keys=skill_recent,
|
||||
limit=pad_pool,
|
||||
pool_size=pool,
|
||||
kind="skill",
|
||||
)
|
||||
wecom_github_items = [_prepare_github_item(item) for item in selected_github]
|
||||
wecom_emerging_items = [_prepare_github_item(item) for item in selected_emerging]
|
||||
wecom_topic_items = [_prepare_github_item(item) for item in selected_topic]
|
||||
github_pad_recent = (
|
||||
recent_shown["github_trending"]
|
||||
| recent_shown["github_emerging"]
|
||||
| recent_shown["github_topic"]
|
||||
)
|
||||
wecom_github_pad = [
|
||||
_prepare_github_item(item)
|
||||
for item in board_select(
|
||||
board="github_trending",
|
||||
items=github_trending,
|
||||
recent_keys=github_pad_recent,
|
||||
limit=pad_pool,
|
||||
pool_size=pool,
|
||||
kind="github",
|
||||
)
|
||||
]
|
||||
wecom_emerging_pad = [
|
||||
_prepare_github_item(item)
|
||||
for item in board_select(
|
||||
board="github_emerging",
|
||||
items=github_emerging,
|
||||
recent_keys=github_pad_recent,
|
||||
limit=pad_pool,
|
||||
pool_size=pool,
|
||||
kind="github",
|
||||
)
|
||||
]
|
||||
wecom_topic_pad = [
|
||||
_prepare_github_item(item)
|
||||
for item in board_select(
|
||||
board="github_topic",
|
||||
items=github_topic,
|
||||
recent_keys=github_pad_recent,
|
||||
limit=pad_pool,
|
||||
pool_size=pool,
|
||||
kind="github",
|
||||
)
|
||||
]
|
||||
delta_pad = eff_mode == "delta" and wecom_delta_pad()
|
||||
board_kwargs = {
|
||||
"mode": eff_mode,
|
||||
"movement": movement,
|
||||
"trending": gt,
|
||||
"hot": gh,
|
||||
"topic_name": topic_name,
|
||||
"github_trending": wecom_github_items,
|
||||
"github_emerging": wecom_emerging_items,
|
||||
"github_topic": wecom_topic_items,
|
||||
"wecom_trending": c["wecom_trending"],
|
||||
"wecom_hot": c["wecom_hot"],
|
||||
"wecom_github": c["wecom_github"],
|
||||
"wecom_emerging": c["wecom_emerging"],
|
||||
"wecom_topic": c["wecom_topic"],
|
||||
"pad": delta_pad,
|
||||
"date_str": date_str,
|
||||
"trending_pad": gt_pad,
|
||||
"hot_pad": gh_pad,
|
||||
"github_trending_pad": wecom_github_pad,
|
||||
"github_emerging_pad": wecom_emerging_pad,
|
||||
"github_topic_pad": wecom_topic_pad,
|
||||
}
|
||||
return {
|
||||
"selected_trending": selected_trending,
|
||||
"selected_hot": selected_hot,
|
||||
"boards_for_wecom": boards_for_wecom,
|
||||
"llm_input": llm_input,
|
||||
"featured": featured,
|
||||
"movement": movement,
|
||||
"eff_mode": eff_mode,
|
||||
"push_gate": push_gate,
|
||||
"wecom_ai": wecom_ai,
|
||||
"wecom_cn": wecom_cn,
|
||||
"agent_wecom": agent_wecom,
|
||||
"editorial_theme": editorial_theme,
|
||||
"editorial_highlights": editorial_highlights,
|
||||
"themes": themes,
|
||||
"pick_command": pick_command,
|
||||
"pick_why": pick_why,
|
||||
"pick_title": pick_title,
|
||||
"pick_url": pick_url,
|
||||
"board_kwargs": board_kwargs,
|
||||
"prev_ids": prev_ids,
|
||||
"gt": gt,
|
||||
"gh": gh,
|
||||
"wecom_github_items": wecom_github_items,
|
||||
"wecom_emerging_items": wecom_emerging_items,
|
||||
"wecom_topic_items": wecom_topic_items,
|
||||
}
|
||||
|
||||
|
||||
def _render(c: dict[str, Any], s: dict[str, Any]) -> tuple[str, str, Path, Path]:
|
||||
"""纯拼装: 拼完整版 markdown + 企微 wecom_md + 写盘。
|
||||
|
||||
零选择逻辑。副作用: record_pushed_links + _save_snapshot + save_json + 写文件,
|
||||
全部保持原时序(在拼装完成后执行)。
|
||||
"""
|
||||
feed = c["feed"]
|
||||
now = c["now"]
|
||||
date_str = c["date_str"]
|
||||
time_str = c["time_str"]
|
||||
updated = c["updated"]
|
||||
trending = c["trending"]
|
||||
hot = c["hot"]
|
||||
github_trending = c["github_trending"]
|
||||
github_emerging = c["github_emerging"]
|
||||
github_topic = c["github_topic"]
|
||||
topic_name = c["topic_name"]
|
||||
news_merged = c["news_merged"]
|
||||
ai_news = c["ai_news"]
|
||||
cn_ai_news = c["cn_ai_news"]
|
||||
ai_news_research = c["ai_news_research"]
|
||||
wecom_news = c["wecom_news"]
|
||||
wecom_tech_news = c["wecom_tech_news"]
|
||||
trending_n = c["trending_n"]
|
||||
hot_n = c["hot_n"]
|
||||
github_limit = c["github_limit"]
|
||||
emerging_limit = c["emerging_limit"]
|
||||
topic_limit = c["topic_limit"]
|
||||
prev_ids = s["prev_ids"]
|
||||
llm_input = s["llm_input"]
|
||||
featured = s["featured"]
|
||||
eff_mode = s["eff_mode"]
|
||||
push_gate = s["push_gate"]
|
||||
wecom_ai = s["wecom_ai"]
|
||||
wecom_cn = s["wecom_cn"]
|
||||
agent_wecom = s["agent_wecom"]
|
||||
editorial_theme = s["editorial_theme"]
|
||||
editorial_highlights = s["editorial_highlights"]
|
||||
themes = s["themes"]
|
||||
pick_command = s["pick_command"]
|
||||
pick_why = s["pick_why"]
|
||||
pick_title = s["pick_title"]
|
||||
pick_url = s["pick_url"]
|
||||
board_kwargs = s["board_kwargs"]
|
||||
gt = s["gt"]
|
||||
gh = s["gh"]
|
||||
wecom_github_items = s["wecom_github_items"]
|
||||
wecom_emerging_items = s["wecom_emerging_items"]
|
||||
wecom_topic_items = s["wecom_topic_items"]
|
||||
|
||||
lines = [
|
||||
f"# 早报 · {date_str}",
|
||||
"",
|
||||
f"> 生成时间:{now.strftime('%Y-%m-%d %H:%M')} (UTC+8) ",
|
||||
f"> skills 数据更新:{updated} ",
|
||||
"> 数据来源:[skills.sh/trending](https://skills.sh/trending) · [skills.sh/hot](https://skills.sh/hot) · 国际/国内 AI RSS",
|
||||
"> 数据来源:[skills.sh/trending](https://skills.sh/trending) · [skills.sh/hot](https://skills.sh/hot)"
|
||||
+ (" · AI 时讯 Deep Research" if news_merged else " · 国际/国内 AI RSS"),
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
@@ -563,10 +1062,14 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
lines.append("")
|
||||
|
||||
section_no = 6
|
||||
lines.extend(format_news_section(ai_news, section_no=section_no))
|
||||
section_no += 1
|
||||
lines.extend(format_cn_news_section(cn_ai_news, section_no=section_no))
|
||||
section_no += 1
|
||||
if news_merged and ai_news_research is not None:
|
||||
lines.extend(format_research_news_section(ai_news_research, section_no=section_no))
|
||||
section_no += 1
|
||||
else:
|
||||
lines.extend(format_news_section(ai_news, section_no=section_no))
|
||||
section_no += 1
|
||||
lines.extend(format_cn_news_section(cn_ai_news, section_no=section_no))
|
||||
section_no += 1
|
||||
|
||||
watch = (env("GITHUB_REPOS") or "").strip()
|
||||
if watch:
|
||||
@@ -584,14 +1087,6 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
lines.append(f"- {ex}")
|
||||
lines.append("")
|
||||
|
||||
pick_src = trending[0].get("source", "") if trending else ""
|
||||
pick_name = trending[0].get("title", "") if trending else ""
|
||||
pick_command = (
|
||||
f"npx skills add {pick_src}/{pick_name}"
|
||||
if pick_src and pick_name
|
||||
else "npx skills add vercel-labs/skills/find-skills"
|
||||
)
|
||||
|
||||
lines.extend(["---", "", "## 安装示例", "", "```bash"])
|
||||
for item in trending[:4]:
|
||||
src, name = item.get("source", ""), item.get("title", "")
|
||||
@@ -600,10 +1095,9 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
lines.extend(["```", "", f"*企微短版见 `output/{date_str}.wecom.md`*"])
|
||||
|
||||
markdown = "\n".join(lines)
|
||||
|
||||
if agent_wecom:
|
||||
gt = group_skills_by_source(trending, limit=wecom_trending, pool_size=skill_pool)
|
||||
gh = group_skills_by_source(hot, limit=wecom_hot, pool_size=skill_pool)
|
||||
wecom_md = replace_wecom_skill_sections(agent_wecom, trending=gt, hot=gh)
|
||||
wecom_md = replace_wecom_skill_sections(agent_wecom, **board_kwargs)
|
||||
else:
|
||||
wecom_md = build_wecom_report(
|
||||
date_str=date_str,
|
||||
@@ -611,33 +1105,72 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
updated=updated,
|
||||
highlights=editorial_highlights
|
||||
or _build_highlights(trending, hot, github_trending, github_emerging, ai_news, cn_ai_news),
|
||||
theme_line=editorial_theme or _detect_theme_line(feed),
|
||||
ai_news=prepare_wecom_news_items(ai_news),
|
||||
cn_ai_news=prepare_wecom_cn_news_items(cn_ai_news),
|
||||
theme_line=editorial_theme or _top_line(feed),
|
||||
ai_news=wecom_ai if not news_merged else None,
|
||||
cn_ai_news=wecom_cn if not news_merged else None,
|
||||
merged_ai_news=wecom_news if news_merged else None,
|
||||
merged_tech_ai_news=wecom_tech_news if news_merged else None,
|
||||
trending=[
|
||||
_prepare_skill_item(item, prev_ids, r)
|
||||
for r, item in enumerate(
|
||||
finalize_wecom_skill_groups(
|
||||
group_skills_by_source(trending, limit=wecom_trending, pool_size=skill_pool)
|
||||
),
|
||||
1,
|
||||
)
|
||||
for r, item in enumerate(finalize_wecom_skill_groups(gt), 1)
|
||||
],
|
||||
hot=[
|
||||
_prepare_skill_item(item, prev_ids, r)
|
||||
for r, item in enumerate(
|
||||
finalize_wecom_skill_groups(
|
||||
group_skills_by_source(hot, limit=wecom_hot, pool_size=skill_pool)
|
||||
),
|
||||
1,
|
||||
)
|
||||
for r, item in enumerate(finalize_wecom_skill_groups(gh), 1)
|
||||
],
|
||||
repos=[_prepare_github_item(item) for item in github_trending[:wecom_github]],
|
||||
emerging=[_prepare_github_item(item) for item in github_emerging[:wecom_emerging]],
|
||||
repos=wecom_github_items,
|
||||
emerging=wecom_emerging_items,
|
||||
topic_name=topic_name,
|
||||
topic_repos=[_prepare_github_item(item) for item in github_topic[:wecom_topic]],
|
||||
topic_repos=wecom_topic_items,
|
||||
pick_command=pick_command,
|
||||
pick_why=pick_why,
|
||||
pick_title=pick_title,
|
||||
pick_url=pick_url,
|
||||
include_boards=(eff_mode == "full"),
|
||||
)
|
||||
wecom_md = replace_wecom_board_sections(wecom_md, **board_kwargs)
|
||||
|
||||
wecom_md = replace_wecom_news_sections(
|
||||
wecom_md,
|
||||
ai_news=wecom_news if news_merged else wecom_ai,
|
||||
cn_ai_news=None if news_merged else wecom_cn,
|
||||
tech_ai_news=wecom_tech_news if news_merged else None,
|
||||
merged=news_merged,
|
||||
)
|
||||
|
||||
if push_gate.should_push:
|
||||
if news_merged:
|
||||
links = [x["link"] for x in wecom_news + wecom_tech_news if x.get("link")]
|
||||
else:
|
||||
links = [x["link"] for x in wecom_ai + wecom_cn if x.get("link")]
|
||||
record_pushed_links(date_str, links)
|
||||
|
||||
final_boards = resolve_wecom_board_items(**board_kwargs)
|
||||
shown_keys = {
|
||||
board: extract_shown_keys(board, final_boards.get(board) or [])
|
||||
for board in BOARD_KEYS
|
||||
}
|
||||
llm_input = merge_wecom_shown_into_data(llm_input, shown_keys)
|
||||
|
||||
save_json(
|
||||
data_json_path(date_str),
|
||||
build_full_payload(
|
||||
llm_input,
|
||||
meta={
|
||||
"generated_at": now.isoformat(),
|
||||
"report_mode": "agent" if is_agent_mode() else "classic",
|
||||
"ai_news_mode": "research" if news_merged else "rss",
|
||||
"cursor_editor": cursor_editor_enabled() and not is_agent_mode(),
|
||||
"featured_pick": featured.get("title") if featured else None,
|
||||
"effective_wecom_mode": eff_mode,
|
||||
"push_gate": {
|
||||
"should_push": push_gate.should_push,
|
||||
"silent": push_gate.silent,
|
||||
"reasons": push_gate.reasons,
|
||||
},
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
_save_snapshot(feed, date_str)
|
||||
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
@@ -648,6 +1181,13 @@ def generate_report() -> tuple[str, str, Path, Path]:
|
||||
return markdown, wecom_md, out_md, out_wecom
|
||||
|
||||
|
||||
def generate_report() -> tuple[str, str, Path, Path]:
|
||||
"""编排三段: 抓取(_collect) → 选择(_select) → 拼装(_render)。"""
|
||||
collected = _collect(_now_cst().strftime("%Y-%m-%d"))
|
||||
selected = _select(collected)
|
||||
return _render(collected, selected)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
log_file = LOG_DIR / f"{_now_cst():%Y-%m-%d}.log"
|
||||
|
||||
@@ -43,38 +43,59 @@ def search_github_repos(
|
||||
logger.warning("GitHub Search 需要 GITHUB_TOKEN: %s", query[:80])
|
||||
return []
|
||||
|
||||
target = max(1, min(int(limit), 1000))
|
||||
per_page = min(100, target)
|
||||
repos: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
page = 1
|
||||
|
||||
try:
|
||||
with httpx.Client(
|
||||
timeout=20.0,
|
||||
verify=certifi.where(),
|
||||
headers=github_api_headers(),
|
||||
) as client:
|
||||
resp = client.get(
|
||||
"https://api.github.com/search/repositories",
|
||||
params={
|
||||
"q": query,
|
||||
"sort": sort,
|
||||
"order": "desc",
|
||||
"per_page": min(max(limit, 1), 30),
|
||||
},
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
logger.warning("GitHub Search 失败 (%s): %s", resp.status_code, query[:80])
|
||||
return []
|
||||
items = resp.json().get("items") or []
|
||||
while len(repos) < target:
|
||||
resp = client.get(
|
||||
"https://api.github.com/search/repositories",
|
||||
params={
|
||||
"q": query,
|
||||
"sort": sort,
|
||||
"order": "desc",
|
||||
"per_page": per_page,
|
||||
"page": page,
|
||||
},
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
logger.warning(
|
||||
"GitHub Search 失败 (%s page=%s): %s",
|
||||
resp.status_code,
|
||||
page,
|
||||
query[:80],
|
||||
)
|
||||
break
|
||||
items = resp.json().get("items") or []
|
||||
if not items:
|
||||
break
|
||||
for item in items:
|
||||
full_name = item.get("full_name") or ""
|
||||
if not full_name or full_name in seen:
|
||||
continue
|
||||
seen.add(full_name)
|
||||
repos.append(_repo_from_api_item(item, source="api-search"))
|
||||
if len(repos) >= target:
|
||||
break
|
||||
if len(items) < per_page:
|
||||
break
|
||||
page += 1
|
||||
# Search API 最多约 1000 条 / 10 页
|
||||
if page > 10:
|
||||
break
|
||||
except Exception as exc:
|
||||
logger.warning("GitHub Search 异常: %s", exc)
|
||||
return []
|
||||
return repos[:target]
|
||||
|
||||
repos: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
full_name = item.get("full_name") or ""
|
||||
if not full_name:
|
||||
continue
|
||||
repos.append(_repo_from_api_item(item, source="api-search"))
|
||||
if len(repos) >= limit:
|
||||
break
|
||||
return repos
|
||||
return repos[:target]
|
||||
|
||||
|
||||
def _date_days_ago(days: int) -> str:
|
||||
|
||||
106
daily/holiday.py
Normal file
106
daily/holiday.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""法定节假日获取、缓存与工作日判定。
|
||||
|
||||
数据来自 xiaoai.me 公共接口,一次性抓取全年并缓存到 .cache/,
|
||||
后续判定只读本地缓存;缓存缺失或过期时才重新请求。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import ssl
|
||||
import urllib.request
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
from daily.config import CACHE_DIR
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_API_URL = "https://publicapi.xiaoai.me/holiday/year?date={year}"
|
||||
_TIMEOUT = 15
|
||||
|
||||
|
||||
def _cache_file(year: int) -> Path:
|
||||
return CACHE_DIR / f"holidays-{year}.json"
|
||||
|
||||
|
||||
def _ssl_context() -> ssl.SSLContext:
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
return ctx
|
||||
|
||||
|
||||
def _fetch_year(year: int) -> dict[str, dict]:
|
||||
"""请求全年节假日,返回 {date_str: {"rest": bool, "name": str}}。
|
||||
|
||||
rest=1 表示休息(法定节假日/调休放假),rest=0 表示调休补班(需上班)。
|
||||
"""
|
||||
url = _API_URL.format(year=year)
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
||||
with urllib.request.urlopen(req, timeout=_TIMEOUT, context=_ssl_context()) as resp:
|
||||
payload = json.loads(resp.read().decode("utf-8"))
|
||||
items = payload.get("data") or []
|
||||
result: dict[str, dict] = {}
|
||||
for item in items:
|
||||
day = item.get("date")
|
||||
if not day:
|
||||
continue
|
||||
result[day] = {
|
||||
"rest": bool(item.get("rest", 0)),
|
||||
"name": item.get("holiday", ""),
|
||||
}
|
||||
if not result:
|
||||
raise RuntimeError(f"节假日接口返回为空 year={year}")
|
||||
return result
|
||||
|
||||
|
||||
def load_holidays(year: int, *, refresh: bool = False) -> dict[str, dict]:
|
||||
"""加载全年节假日缓存;缺失或 refresh 时联网抓取并写入缓存。
|
||||
|
||||
联网失败时若已有缓存则回退用缓存,保证离线可用。
|
||||
"""
|
||||
cache = _cache_file(year)
|
||||
if cache.exists() and not refresh:
|
||||
try:
|
||||
raw = json.loads(cache.read_text(encoding="utf-8"))
|
||||
if isinstance(raw, dict) and raw:
|
||||
return raw
|
||||
except (OSError, ValueError):
|
||||
logger.warning("节假日缓存损坏,重新获取 %s", cache)
|
||||
|
||||
try:
|
||||
data = _fetch_year(year)
|
||||
except Exception as exc:
|
||||
if cache.exists():
|
||||
logger.warning("节假日获取失败,回退到本地缓存:%s", exc)
|
||||
return json.loads(cache.read_text(encoding="utf-8"))
|
||||
raise
|
||||
|
||||
cache.parent.mkdir(parents=True, exist_ok=True)
|
||||
cache.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
logger.info("已缓存 %d 年节假日 %d 条 -> %s", year, len(data), cache)
|
||||
return data
|
||||
|
||||
|
||||
def is_workday(day: date, holidays: dict[str, dict] | None = None) -> bool:
|
||||
"""判断是否为工作日。
|
||||
|
||||
规则:调休补班日(rest=0 且在表中)→ 上班;
|
||||
法定节假日/调休放假(rest=1)→ 休息;
|
||||
周六日 → 休息;其余 → 上班。
|
||||
"""
|
||||
holidays = holidays if holidays is not None else load_holidays(day.year)
|
||||
key = day.isoformat()
|
||||
entry = holidays.get(key)
|
||||
if entry is not None:
|
||||
return not entry["rest"]
|
||||
return day.weekday() < 5 # 周一~周五
|
||||
|
||||
|
||||
def workday_name(day: date, holidays: dict[str, dict] | None = None) -> str | None:
|
||||
"""返回该天的节假日名(休息或调休补班),无则 None。用于日志。"""
|
||||
holidays = holidays if holidays is not None else load_holidays(day.year)
|
||||
entry = holidays.get(day.isoformat())
|
||||
return entry["name"] if entry else None
|
||||
@@ -74,22 +74,26 @@ def _cursor_chat(system: str, user: str) -> str:
|
||||
api_key = (env("CURSOR_API_KEY") or "").strip()
|
||||
if not api_key:
|
||||
return ""
|
||||
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
|
||||
from cursor_sdk import Agent, AgentOptions, Client, CursorAgentError, LocalAgentOptions
|
||||
|
||||
from daily.config import ensure_bot_on_path
|
||||
|
||||
ensure_bot_on_path()
|
||||
try:
|
||||
from bridge_manager import warm_cursor_bridge
|
||||
except ImportError:
|
||||
warm_cursor_bridge = lambda: None # noqa: E731
|
||||
from daily.bridge_manager import warm_cursor_bridge
|
||||
|
||||
cwd = env("DAILY_CURSOR_CWD") or str(ROOT)
|
||||
# bridge_manager 读 bot env_config 的 CURSOR_CWD,早报侧须先对齐工作目录
|
||||
os.environ["CURSOR_CWD"] = cwd
|
||||
warm_cursor_bridge()
|
||||
model = env("CURSOR_MODEL") or "composer-2.5"
|
||||
prompt = f"{system}\n\n{user}"
|
||||
|
||||
# SDK 默认 unary_timeout 只有 60s,CURSOR_MODEL=auto 时后端首 token 常超时;
|
||||
# 自建带大超时的 Client 绕开 _default_client(),unary/stream 都放大。
|
||||
unary_timeout = env_int("DAILY_CURSOR_UNARY_TIMEOUT", 300)
|
||||
stream_timeout = env_int("DAILY_CURSOR_STREAM_TIMEOUT", 900)
|
||||
client = Client(
|
||||
base_url=os.environ["CURSOR_SDK_BRIDGE_URL"],
|
||||
auth_token=os.environ["CURSOR_SDK_BRIDGE_TOKEN"],
|
||||
unary_timeout=unary_timeout,
|
||||
stream_timeout=stream_timeout,
|
||||
)
|
||||
try:
|
||||
result = Agent.prompt(
|
||||
prompt,
|
||||
@@ -98,9 +102,12 @@ def _cursor_chat(system: str, user: str) -> str:
|
||||
model=model,
|
||||
local=LocalAgentOptions(cwd=cwd),
|
||||
),
|
||||
client=client,
|
||||
)
|
||||
except CursorAgentError as exc:
|
||||
raise RuntimeError(f"LLM 调用失败:{exc.message}") from exc
|
||||
finally:
|
||||
client.close()
|
||||
if result.status == "error":
|
||||
raise RuntimeError(f"LLM 调用失败:{result.result or '未知错误'}")
|
||||
return (result.result or "").strip()
|
||||
@@ -115,5 +122,16 @@ def llm_chat(system: str, user: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def has_cursor_configured() -> bool:
|
||||
return bool((env("CURSOR_API_KEY") or "").strip())
|
||||
|
||||
|
||||
def cursor_agent_prompt(system: str, user: str) -> str:
|
||||
"""仅 Cursor SDK Agent(可用 WebSearch 等工具),不走 OpenAI 兼容 API。"""
|
||||
if not has_cursor_configured():
|
||||
return ""
|
||||
return _cursor_chat(system, user)
|
||||
|
||||
|
||||
def has_llm_configured() -> bool:
|
||||
return bool(env("DAILY_LLM_API_KEY") or env("OPENAI_API_KEY") or env("CURSOR_API_KEY"))
|
||||
|
||||
149
daily/narrative_axis.py
Normal file
149
daily/narrative_axis.py
Normal file
@@ -0,0 +1,149 @@
|
||||
"""叙事轴硬互斥:代码选定轴,注入 Agent Step1 并强制覆写。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import random
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from daily.config import OUTPUT_DIR, narrative_axis_days
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
NARRATIVE_AXES: tuple[str, ...] = (
|
||||
"政策监管",
|
||||
"模型发布",
|
||||
"工具链/Agent",
|
||||
"芯片算力",
|
||||
"开源生态",
|
||||
"应用落地",
|
||||
"安全/诉讼",
|
||||
)
|
||||
|
||||
|
||||
def pick_narrative_axis(
|
||||
used: set[str],
|
||||
*,
|
||||
rng: random.Random | None = None,
|
||||
) -> str:
|
||||
"""从固定轴枚举中排除已用轴后随机选取;全用尽则回退全表。"""
|
||||
available = [a for a in NARRATIVE_AXES if a not in used]
|
||||
pool = available or list(NARRATIVE_AXES)
|
||||
picker = rng or random.Random()
|
||||
return picker.choice(pool)
|
||||
|
||||
|
||||
def load_recent_axes(date_str: str, days: int | None = None) -> list[str]:
|
||||
"""近 N 日 data.narrative_axis(不含当日,按时间从近到远)。"""
|
||||
lookback = days if days is not None else narrative_axis_days()
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
return []
|
||||
axes: list[str] = []
|
||||
for day_offset in range(1, lookback + 1):
|
||||
prev = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
|
||||
path = OUTPUT_DIR / f"{prev}.data.json"
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("读取 narrative_axis %s 失败:%s", path, exc)
|
||||
continue
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
axis = str(data.get("narrative_axis") or "").strip()
|
||||
if axis:
|
||||
axes.append(axis)
|
||||
return axes
|
||||
|
||||
|
||||
def load_recent_theme_summaries(date_str: str, days: int) -> list[str]:
|
||||
"""近 N 日 theme/opening 摘要,供 Step1 软禁参考。"""
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
return []
|
||||
summaries: list[str] = []
|
||||
for day_offset in range(1, days + 1):
|
||||
prev = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
|
||||
path = OUTPUT_DIR / f"{prev}.data.json"
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
theme = str(data.get("theme") or data.get("editorial_theme") or "").strip()
|
||||
opening = ""
|
||||
trends = data.get("trends") if isinstance(data.get("trends"), dict) else {}
|
||||
if isinstance(trends, dict):
|
||||
opening = str(trends.get("opening") or "").strip()
|
||||
if not theme:
|
||||
themes = trends.get("themes") or []
|
||||
if themes and isinstance(themes[0], dict):
|
||||
theme = str(themes[0].get("title") or "").strip()
|
||||
bit = " · ".join(x for x in (prev, theme, opening[:40]) if x)
|
||||
if bit:
|
||||
summaries.append(bit)
|
||||
return summaries
|
||||
|
||||
|
||||
def enforce_narrative_axis(trends: dict[str, Any], axis: str) -> dict[str, Any]:
|
||||
"""强制 trends['narrative_axis'] = axis。"""
|
||||
out = dict(trends)
|
||||
out["narrative_axis"] = axis
|
||||
return out
|
||||
|
||||
|
||||
def theme_clusters(
|
||||
feed: dict[str, Any],
|
||||
*,
|
||||
limit: int = 5,
|
||||
theme_rules: list[tuple[str, str, list[str]]],
|
||||
skill_id_fn: Any,
|
||||
) -> list[tuple[str, list[str]]]:
|
||||
"""按 THEME_RULES 把 feed 的 topTrending/topHot 聚成 (主题, 示例列表)。
|
||||
|
||||
从 generate.py 迁入(原私有 _theme_clusters)。theme_rules 与 skill_id_fn
|
||||
由调用方注入,避免对 generate.py 的反向依赖(防循环 import)。
|
||||
"""
|
||||
buckets: dict[str, list[str]] = defaultdict(list)
|
||||
seen: set[str] = set()
|
||||
for board in ("topTrending", "topHot"):
|
||||
for item in feed.get(board, [])[:20]:
|
||||
item_id = skill_id_fn(item)
|
||||
if item_id in seen:
|
||||
continue
|
||||
seen.add(item_id)
|
||||
haystack = " ".join(
|
||||
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
|
||||
).lower()
|
||||
for _icon, theme, keywords in theme_rules:
|
||||
if any(k in haystack for k in keywords):
|
||||
label = f"**{item.get('title')}** (`{item.get('source')}`)"
|
||||
if label not in buckets[theme]:
|
||||
buckets[theme].append(label)
|
||||
break
|
||||
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
|
||||
|
||||
|
||||
def theme_names(
|
||||
feed: dict[str, Any],
|
||||
*,
|
||||
theme_rules: list[tuple[str, str, list[str]]],
|
||||
skill_id_fn: Any,
|
||||
limit: int = 3,
|
||||
) -> list[str]:
|
||||
"""仅取主题名(不含 markdown 示例),供「今日看点/theme_line」回退文案。"""
|
||||
return [theme for theme, _ in theme_clusters(
|
||||
feed, theme_rules=theme_rules, skill_id_fn=skill_id_fn
|
||||
)][:limit]
|
||||
@@ -1,13 +1,26 @@
|
||||
from daily.news.fetch import (
|
||||
|
||||
fetch_ai_news,
|
||||
|
||||
fetch_cn_ai_news,
|
||||
|
||||
format_cn_news_section,
|
||||
|
||||
format_news_section,
|
||||
|
||||
)
|
||||
|
||||
|
||||
|
||||
__all__ = [
|
||||
|
||||
"fetch_ai_news",
|
||||
|
||||
"fetch_cn_ai_news",
|
||||
|
||||
"format_news_section",
|
||||
|
||||
"format_cn_news_section",
|
||||
|
||||
]
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
|
||||
name="厂商官方",
|
||||
icon="🏢",
|
||||
feeds=(
|
||||
NewsFeed("Anthropic Claude 更新", "https://docs.anthropic.com/en/release-notes/feed"),
|
||||
NewsFeed("Anthropic Claude 更新", "https://platform.claude.com/docs/en/release-notes/overview"),
|
||||
NewsFeed("OpenAI", "https://openai.com/news/rss.xml"),
|
||||
NewsFeed("Google AI", "https://blog.google/technology/ai/rss/"),
|
||||
NewsFeed("DeepMind", "https://deepmind.google/blog/rss.xml"),
|
||||
|
||||
@@ -44,7 +44,6 @@ CN_NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
|
||||
icon="📰",
|
||||
feeds=(
|
||||
NewsFeed("量子位", "https://www.qbitai.com/feed"),
|
||||
NewsFeed("InfoQ 中文", "https://www.infoq.cn/feed/AI"),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
@@ -60,12 +59,4 @@ CN_NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
|
||||
),
|
||||
),
|
||||
),
|
||||
NewsCategory(
|
||||
id="dev",
|
||||
name="开发者社区",
|
||||
icon="💻",
|
||||
feeds=(
|
||||
NewsFeed("掘金", "https://juejin.cn/rss", ai_filter=True),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
@@ -8,17 +8,19 @@ import time
|
||||
import html
|
||||
import xml.etree.ElementTree as ET
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from datetime import datetime, time as dt_time, timezone, timedelta
|
||||
from email.utils import parsedate_to_datetime
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env, env_int, news_summary_limit
|
||||
from daily.config import env, env_int, news_summary_limit, wecom_news_desc_limit
|
||||
from daily.news.feeds import NEWS_CATEGORIES, NewsCategory, NewsFeed
|
||||
from daily.news.feeds_cn import CN_AI_TITLE_KEYWORDS, CN_NEWS_CATEGORIES
|
||||
from daily.text_utils import trim_brief
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -46,15 +48,40 @@ def _cn_enabled() -> bool:
|
||||
|
||||
|
||||
def _hours_window() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_HOURS", 72))
|
||||
return max(1, env_int("DAILY_AI_NEWS_HOURS", 24))
|
||||
|
||||
|
||||
def _news_tz_name() -> str:
|
||||
return (env("DAILY_AI_NEWS_TZ") or env("DAILY_SCHEDULE_TZ") or "Asia/Shanghai").strip()
|
||||
|
||||
|
||||
def _floor_today_enabled() -> bool:
|
||||
raw = env("DAILY_AI_NEWS_FLOOR_TODAY")
|
||||
if raw is None:
|
||||
return True
|
||||
return raw.strip().lower() not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def _cutoff_datetime(*, floor_today: bool) -> datetime:
|
||||
"""滚动 N 小时窗口;国际新闻可叠加「不早于今日 0 点(本地时区)」。"""
|
||||
now = _now_utc()
|
||||
rolling = now - timedelta(hours=_hours_window())
|
||||
if not floor_today:
|
||||
return rolling
|
||||
tz = ZoneInfo(_news_tz_name())
|
||||
local = now.astimezone(tz)
|
||||
start_today = local.replace(hour=0, minute=0, second=0, microsecond=0).astimezone(timezone.utc)
|
||||
return max(rolling, start_today)
|
||||
|
||||
|
||||
def _per_feed_limit() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_PER_FEED", 3))
|
||||
want = max(_wecom_limit(), _wecom_cn_limit())
|
||||
return max(want // 2, env_int("DAILY_AI_NEWS_PER_FEED", 5))
|
||||
|
||||
|
||||
def _per_category_limit() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_PER_CATEGORY", 5))
|
||||
want = max(_wecom_limit(), _wecom_cn_limit())
|
||||
return max(want, env_int("DAILY_AI_NEWS_PER_CATEGORY", 10))
|
||||
|
||||
|
||||
def _wecom_limit() -> int:
|
||||
@@ -62,7 +89,7 @@ def _wecom_limit() -> int:
|
||||
|
||||
|
||||
def _wecom_cn_limit() -> int:
|
||||
return max(1, env_int("DAILY_WECOM_CN_AI_NEWS", 8))
|
||||
return max(1, env_int("DAILY_WECOM_CN_AI_NEWS", 10))
|
||||
|
||||
|
||||
def _matches_cn_ai_title(title: str) -> bool:
|
||||
@@ -102,7 +129,6 @@ def _parse_datetime(value: str | None) -> datetime | None:
|
||||
for fmt in (
|
||||
"%Y-%m-%dT%H:%M:%SZ",
|
||||
"%Y-%m-%dT%H:%M:%S%z",
|
||||
"%Y-%m-%d",
|
||||
):
|
||||
try:
|
||||
dt = datetime.strptime(text[: len(fmt.replace("%z", "+0000"))], fmt.replace("%z", ""))
|
||||
@@ -111,17 +137,103 @@ def _parse_datetime(value: str | None) -> datetime | None:
|
||||
return dt.astimezone(timezone.utc)
|
||||
except ValueError:
|
||||
continue
|
||||
if re.match(r"^\d{4}-\d{2}-\d{2}$", text):
|
||||
try:
|
||||
tz = ZoneInfo(_news_tz_name())
|
||||
day = datetime.strptime(text, "%Y-%m-%d").date()
|
||||
# 仅日期时按本地中午估算,避免 UTC 0 点误判为「前一天」
|
||||
dt = datetime.combine(day, dt_time(12, 0), tzinfo=tz)
|
||||
return dt.astimezone(timezone.utc)
|
||||
except ValueError:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _clean_text(text: str | None, limit: int = 200) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
plain = STRIP_HTML.sub(" ", html.unescape(text))
|
||||
plain = WS.sub(" ", plain).strip()
|
||||
plain = _strip_summary_plain(text)
|
||||
if limit <= 0 or len(plain) <= limit:
|
||||
return plain
|
||||
return plain[: limit - 3] + "..."
|
||||
return trim_brief(plain, limit)
|
||||
|
||||
|
||||
def _strip_summary_plain(text: str | None) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
plain = STRIP_HTML.sub(" ", html.unescape(text))
|
||||
return WS.sub(" ", plain).strip()
|
||||
|
||||
|
||||
_JUNK_SUMMARY_RE = re.compile(
|
||||
r"^(点击查看原文|article url:|comments url:|discussion on hn|read more)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _is_junk_news_summary(text: str) -> bool:
|
||||
if not text:
|
||||
return True
|
||||
if _JUNK_SUMMARY_RE.match(text.strip()):
|
||||
return True
|
||||
if text.strip().endswith(">") and "点击" in text:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def brief_news_summary(text: str | None, limit: int | None = None) -> str:
|
||||
"""企微新闻一句摘要:去 HTML、过滤占位文案、句读处截断。"""
|
||||
plain = _strip_summary_plain(text)
|
||||
if _is_junk_news_summary(plain):
|
||||
return ""
|
||||
lim = wecom_news_desc_limit() if limit is None else limit
|
||||
return trim_brief(plain, lim)
|
||||
|
||||
|
||||
def sync_wecom_news_rows(items: list[dict[str, Any]], flat: list[dict[str, Any]]) -> None:
|
||||
"""中文化后,用 flat 最新 summary 刷新企微 desc_short。"""
|
||||
by_link = {_normalize_link(str(i.get("link") or "")): i for i in flat if i.get("link")}
|
||||
for row in items:
|
||||
link = _normalize_link(str(row.get("link") or ""))
|
||||
src = by_link.get(link)
|
||||
if src:
|
||||
row["desc_short"] = brief_news_summary(src.get("summary"))
|
||||
|
||||
|
||||
def finalize_wecom_news_items(
|
||||
items: list[dict[str, Any]],
|
||||
*,
|
||||
force_chinese: bool = False,
|
||||
) -> None:
|
||||
"""企微新闻摘要:确保 desc_short 为中文(国际源 force_chinese=True)。"""
|
||||
from daily.localize import LocalizeJob, localize_brief_descriptions, needs_chinese
|
||||
from daily.news.sanitize import strip_relax_window_prefix
|
||||
|
||||
limit = wecom_news_desc_limit()
|
||||
jobs: list[LocalizeJob] = []
|
||||
keyed: list[tuple[str, dict[str, Any]]] = []
|
||||
for idx, item in enumerate(items):
|
||||
text = strip_relax_window_prefix(
|
||||
(item.get("desc_short") or item.get("summary_plain") or "").strip()
|
||||
)
|
||||
if text:
|
||||
item["desc_short"] = text
|
||||
if not text or _is_junk_news_summary(text):
|
||||
item["desc_short"] = ""
|
||||
continue
|
||||
if force_chinese or needs_chinese(text):
|
||||
key = f"wecom-news:{item.get('link') or idx}"
|
||||
jobs.append(LocalizeJob(key, text, limit))
|
||||
keyed.append((key, item))
|
||||
elif not item.get("desc_short"):
|
||||
item["desc_short"] = brief_news_summary(text, limit)
|
||||
|
||||
if not jobs:
|
||||
return
|
||||
zh_map = localize_brief_descriptions(jobs, archive=True)
|
||||
for key, item in keyed:
|
||||
if key in zh_map:
|
||||
item["desc_short"] = strip_relax_window_prefix(zh_map[key])
|
||||
|
||||
|
||||
def _normalize_link(link: str) -> str:
|
||||
@@ -317,10 +429,119 @@ def _dedupe_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
return result
|
||||
|
||||
|
||||
def _filter_flat_in_window(
|
||||
flat: list[dict[str, Any]],
|
||||
*,
|
||||
floor_today: bool,
|
||||
) -> list[dict[str, Any]]:
|
||||
cutoff = _cutoff_datetime(floor_today=floor_today)
|
||||
items = [i for i in _dedupe_items(flat) if _within_window(i, cutoff)]
|
||||
items.sort(key=_sort_key, reverse=True)
|
||||
return items
|
||||
|
||||
|
||||
def _pick_news_items(
|
||||
flat: list[dict[str, Any]],
|
||||
limit: int,
|
||||
preferred: tuple[str, ...],
|
||||
*,
|
||||
one_per_source: bool = False,
|
||||
) -> list[dict[str, Any]]:
|
||||
picked: list[dict[str, Any]] = []
|
||||
seen_links: set[str] = set()
|
||||
seen_sources: set[str] = set()
|
||||
|
||||
def _try_take(item: dict[str, Any]) -> bool:
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
if not link or link in seen_links:
|
||||
return False
|
||||
if one_per_source:
|
||||
source = item.get("source_name", "?")
|
||||
if source in seen_sources:
|
||||
return False
|
||||
seen_sources.add(source)
|
||||
seen_links.add(link)
|
||||
picked.append(item)
|
||||
return True
|
||||
|
||||
for cat in preferred:
|
||||
for item in flat:
|
||||
if item.get("category_id") != cat:
|
||||
continue
|
||||
if _try_take(item) and len(picked) >= limit:
|
||||
return picked[:limit]
|
||||
|
||||
for item in flat:
|
||||
if _try_take(item) and len(picked) >= limit:
|
||||
break
|
||||
return picked[:limit]
|
||||
|
||||
|
||||
def _fill_picked_to_limit(
|
||||
picked: list[dict[str, Any]],
|
||||
pools: list[list[dict[str, Any]]],
|
||||
limit: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
seen_links = {_normalize_link(i.get("link", "")) for i in picked}
|
||||
for pool in pools:
|
||||
for item in pool:
|
||||
if len(picked) >= limit:
|
||||
return picked[:limit]
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
if not link or link in seen_links:
|
||||
continue
|
||||
picked.append(item)
|
||||
seen_links.add(link)
|
||||
return picked[:limit]
|
||||
|
||||
|
||||
def _to_wecom_news_row(item: dict[str, Any]) -> dict[str, Any]:
|
||||
plain = _strip_summary_plain(item.get("summary", ""))
|
||||
return {
|
||||
"title": item.get("title", "?"),
|
||||
"link": item.get("link", ""),
|
||||
"source_name": item.get("source_name", "?"),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"desc_short": brief_news_summary(plain),
|
||||
"summary_plain": plain,
|
||||
}
|
||||
|
||||
|
||||
def _apply_pushed_dedup_with_backfill(
|
||||
items: list[dict[str, Any]],
|
||||
picked: list[dict[str, Any]],
|
||||
*,
|
||||
date_str: str | None,
|
||||
limit: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
if not date_str:
|
||||
return items[:limit]
|
||||
from daily.config import news_backfill_enabled
|
||||
from daily.news.pushed_links import filter_unpushed_items
|
||||
|
||||
fresh = filter_unpushed_items(items, date_str=date_str)
|
||||
if len(fresh) >= limit:
|
||||
return fresh[:limit]
|
||||
if not news_backfill_enabled():
|
||||
if len(fresh) < limit:
|
||||
logger.info("news_short:%s", len(fresh))
|
||||
return fresh[:limit]
|
||||
seen = {_normalize_link(i.get("link", "")) for i in fresh if i.get("link")}
|
||||
for item in picked:
|
||||
if len(fresh) >= limit:
|
||||
break
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
if not link or link in seen:
|
||||
continue
|
||||
fresh.append(_to_wecom_news_row(item))
|
||||
seen.add(link)
|
||||
return fresh[:limit]
|
||||
|
||||
|
||||
def _within_window(item: dict[str, Any], cutoff: datetime) -> bool:
|
||||
dt = _entry_datetime(item)
|
||||
if dt is None:
|
||||
return True
|
||||
return False
|
||||
return dt >= cutoff
|
||||
|
||||
|
||||
@@ -331,11 +552,11 @@ def _sort_key(item: dict[str, Any]) -> tuple[int, datetime]:
|
||||
return (0, dt)
|
||||
|
||||
|
||||
def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
|
||||
def _fetch_news(categories: tuple[NewsCategory, ...], *, floor_today: bool = False) -> dict[str, Any]:
|
||||
hours = _hours_window()
|
||||
per_feed = _per_feed_limit()
|
||||
per_category = _per_category_limit()
|
||||
cutoff = _now_utc() - timedelta(hours=hours)
|
||||
cutoff = _cutoff_datetime(floor_today=floor_today)
|
||||
|
||||
headers = {"User-Agent": USER_AGENT, "Accept": "application/rss+xml, application/atom+xml, application/xml, text/xml, */*"}
|
||||
tasks: list[tuple[NewsCategory, NewsFeed]] = []
|
||||
@@ -406,6 +627,7 @@ def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
|
||||
return {
|
||||
"enabled": True,
|
||||
"hours": hours,
|
||||
"floor_today": floor_today,
|
||||
"categories": categories_out,
|
||||
"flat": flat,
|
||||
"stats": stats,
|
||||
@@ -416,7 +638,7 @@ def fetch_ai_news() -> dict[str, Any]:
|
||||
"""按类别抓取国际 AI 时讯,返回 {enabled, hours, categories, flat, stats}。"""
|
||||
if not _enabled():
|
||||
return {"enabled": False, "categories": [], "flat": [], "stats": {}}
|
||||
return _fetch_news(NEWS_CATEGORIES)
|
||||
return _fetch_news(NEWS_CATEGORIES, floor_today=_floor_today_enabled())
|
||||
|
||||
|
||||
def fetch_cn_ai_news() -> dict[str, Any]:
|
||||
@@ -459,12 +681,13 @@ def _format_news_section(
|
||||
|
||||
categories = news.get("categories") or []
|
||||
hours = news.get("hours", 72)
|
||||
floor_note = " · 仅今日" if news.get("floor_today") else ""
|
||||
lines = [
|
||||
"---",
|
||||
"",
|
||||
f"## {section_no}、{title}",
|
||||
"",
|
||||
f"> 近 **{hours}h** · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用",
|
||||
f"> 近 **{hours}h**{floor_note} · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用",
|
||||
"",
|
||||
]
|
||||
|
||||
@@ -497,84 +720,27 @@ def _format_news_section(
|
||||
return lines
|
||||
|
||||
|
||||
def prepare_wecom_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
def prepare_wecom_news_items(news: dict[str, Any], *, date_str: str | None = None) -> list[dict[str, Any]]:
|
||||
if not news.get("enabled"):
|
||||
return []
|
||||
limit = _wecom_limit()
|
||||
flat = _dedupe_items(news.get("flat") or [])
|
||||
flat.sort(key=_sort_key, reverse=True)
|
||||
floor = bool(news.get("floor_today", _floor_today_enabled()))
|
||||
flat_strict = _filter_flat_in_window(news.get("flat") or [], floor_today=floor)
|
||||
flat_relaxed = _filter_flat_in_window(news.get("flat") or [], floor_today=False)
|
||||
preferred = ("media", "newsletter", "official", "community", "research", "developer")
|
||||
picked: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for cat in preferred:
|
||||
for item in flat:
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
if item.get("category_id") != cat or link in seen:
|
||||
continue
|
||||
picked.append(item)
|
||||
seen.add(link)
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
items: list[dict[str, Any]] = []
|
||||
for item in picked[:limit]:
|
||||
items.append(
|
||||
{
|
||||
"title": item.get("title", "?"),
|
||||
"link": item.get("link", ""),
|
||||
"source_name": item.get("source_name", "?"),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"desc_short": _clean_text(item.get("summary", ""), 36),
|
||||
}
|
||||
)
|
||||
return items
|
||||
picked = _pick_news_items(flat_strict, limit, preferred)
|
||||
picked = _fill_picked_to_limit(picked, [flat_relaxed, news.get("flat") or []], limit)
|
||||
items = [_to_wecom_news_row(item) for item in picked[:limit]]
|
||||
return _apply_pushed_dedup_with_backfill(items, picked, date_str=date_str, limit=limit)
|
||||
|
||||
|
||||
def prepare_wecom_cn_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
def prepare_wecom_cn_news_items(news: dict[str, Any], *, date_str: str | None = None) -> list[dict[str, Any]]:
|
||||
if not news.get("enabled"):
|
||||
return []
|
||||
limit = _wecom_cn_limit()
|
||||
flat = _dedupe_items(news.get("flat") or [])
|
||||
flat.sort(key=_sort_key, reverse=True)
|
||||
preferred = ("media", "tech", "dev")
|
||||
picked: list[dict[str, Any]] = []
|
||||
seen_links: set[str] = set()
|
||||
seen_sources: set[str] = set()
|
||||
|
||||
for cat in preferred:
|
||||
for item in flat:
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
source = item.get("source_name", "?")
|
||||
if item.get("category_id") != cat or not link or link in seen_links or source in seen_sources:
|
||||
continue
|
||||
picked.append(item)
|
||||
seen_links.add(link)
|
||||
seen_sources.add(source)
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
|
||||
if len(picked) < limit:
|
||||
for item in flat:
|
||||
link = _normalize_link(item.get("link", ""))
|
||||
if not link or link in seen_links:
|
||||
continue
|
||||
picked.append(item)
|
||||
seen_links.add(link)
|
||||
if len(picked) >= limit:
|
||||
break
|
||||
|
||||
items: list[dict[str, Any]] = []
|
||||
for item in picked[:limit]:
|
||||
items.append(
|
||||
{
|
||||
"title": item.get("title", "?"),
|
||||
"link": item.get("link", ""),
|
||||
"source_name": item.get("source_name", "?"),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"desc_short": _clean_text(item.get("summary", ""), 36),
|
||||
}
|
||||
)
|
||||
return items
|
||||
flat = _filter_flat_in_window(news.get("flat") or [], floor_today=False)
|
||||
preferred = ("media", "tech")
|
||||
picked = _pick_news_items(flat, limit, preferred, one_per_source=True)
|
||||
picked = _fill_picked_to_limit(picked, [news.get("flat") or []], limit)
|
||||
items = [_to_wecom_news_row(item) for item in picked[:limit]]
|
||||
return _apply_pushed_dedup_with_backfill(items, picked, date_str=date_str, limit=limit)
|
||||
|
||||
87
daily/news/pushed_links.py
Normal file
87
daily/news/pushed_links.py
Normal file
@@ -0,0 +1,87 @@
|
||||
"""已推送企微早报的新闻 link 去重缓存。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from daily.config import CACHE_DIR, news_dedup_days
|
||||
from daily.news.fetch import _normalize_link
|
||||
|
||||
|
||||
def _cache_path() -> Path:
|
||||
return CACHE_DIR / "pushed-news-links.json"
|
||||
|
||||
|
||||
def _load_raw() -> dict[str, Any]:
|
||||
path = _cache_path()
|
||||
if not path.exists():
|
||||
return {"dates": {}}
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return {"dates": {}}
|
||||
if not isinstance(data.get("dates"), dict):
|
||||
return {"dates": {}}
|
||||
return data
|
||||
|
||||
|
||||
def _save_raw(data: dict[str, Any]) -> None:
|
||||
path = _cache_path()
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def _prune(data: dict[str, Any], *, keep_days: int) -> None:
|
||||
dates: dict[str, list[str]] = data.setdefault("dates", {})
|
||||
try:
|
||||
anchor = max(datetime.strptime(d, "%Y-%m-%d") for d in dates)
|
||||
except ValueError:
|
||||
return
|
||||
cutoff = anchor - timedelta(days=keep_days)
|
||||
for key in list(dates.keys()):
|
||||
try:
|
||||
if datetime.strptime(key, "%Y-%m-%d") < cutoff:
|
||||
dates.pop(key, None)
|
||||
except ValueError:
|
||||
dates.pop(key, None)
|
||||
|
||||
|
||||
def load_pushed_link_set() -> set[str]:
|
||||
data = _load_raw()
|
||||
out: set[str] = set()
|
||||
for links in (data.get("dates") or {}).values():
|
||||
if isinstance(links, list):
|
||||
out.update(str(x) for x in links if x)
|
||||
return out
|
||||
|
||||
|
||||
def filter_unpushed_items(
|
||||
items: list[dict[str, Any]],
|
||||
*,
|
||||
date_str: str,
|
||||
) -> list[dict[str, Any]]:
|
||||
del date_str # reserved for per-day scoping if needed later
|
||||
seen = load_pushed_link_set()
|
||||
out: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
link = _normalize_link(str(item.get("link") or ""))
|
||||
if not link or link in seen:
|
||||
continue
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
def record_pushed_links(date_str: str, links: list[str]) -> None:
|
||||
data = _load_raw()
|
||||
dates: dict[str, list[str]] = data.setdefault("dates", {})
|
||||
normalized: list[str] = []
|
||||
for link in links:
|
||||
clean = _normalize_link(link)
|
||||
if clean:
|
||||
normalized.append(clean)
|
||||
dates[date_str] = sorted(set(normalized))
|
||||
_prune(data, keep_days=news_dedup_days())
|
||||
_save_raw(data)
|
||||
386
daily/news/research.py
Normal file
386
daily/news/research.py
Normal file
@@ -0,0 +1,386 @@
|
||||
"""Cursor SDK + deep-research 工作流:采集 AI 时讯(方案 A,内置 WebSearch)。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from daily.config import OUTPUT_DIR, ROOT, env, env_int, wecom_ai_news_tech_limit
|
||||
from daily.llm_client import cursor_agent_prompt, extract_json_object, has_cursor_configured
|
||||
from daily.news.fetch import brief_news_summary, _normalize_link
|
||||
from daily.news.pushed_links import filter_unpushed_items
|
||||
from daily.news.research_quality import post_process_research_news, research_cn_min
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SKILL_DIR = ROOT / "skills" / "daily-ai-news-research"
|
||||
_DEEP_RESEARCH_CANDIDATES = (
|
||||
ROOT / "skills" / "deep-research" / "SKILL.md",
|
||||
Path.home() / ".agents" / "skills" / "deep-research" / "SKILL.md",
|
||||
Path.home() / ".cursor" / "skills" / "deep-research" / "SKILL.md",
|
||||
)
|
||||
|
||||
|
||||
def ai_news_mode() -> str:
|
||||
return (env("DAILY_AI_NEWS_MODE") or "rss").strip().lower()
|
||||
|
||||
|
||||
def is_research_mode() -> bool:
|
||||
return ai_news_mode() == "research"
|
||||
|
||||
|
||||
def research_hours() -> int:
|
||||
return max(1, env_int("DAILY_AI_NEWS_HOURS", 24))
|
||||
|
||||
|
||||
def research_limit() -> int:
|
||||
return max(1, env_int("DAILY_WECOM_AI_NEWS", 10))
|
||||
|
||||
|
||||
def research_pool_limit(display_limit: int | None = None) -> int:
|
||||
"""Agent 原始候选条数(展示上限之上多拉,供可信/去重筛)。"""
|
||||
lim = display_limit if display_limit is not None else research_limit()
|
||||
explicit = env_int("DAILY_AI_NEWS_RESEARCH_POOL", 0)
|
||||
if explicit > 0:
|
||||
return max(lim, explicit)
|
||||
return max(lim * 2, lim + 8)
|
||||
|
||||
|
||||
def research_tech_pool_limit(display_limit: int | None = None) -> int:
|
||||
tech = display_limit if display_limit is not None else research_tech_limit()
|
||||
if tech <= 0:
|
||||
return 0
|
||||
explicit = env_int("DAILY_AI_NEWS_RESEARCH_TECH_POOL", 0)
|
||||
if explicit > 0:
|
||||
return max(tech, explicit)
|
||||
return max(tech * 2, tech + 4)
|
||||
|
||||
|
||||
def research_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.ai-news-research.json"
|
||||
|
||||
|
||||
def _save_research_json(path: Path, data: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def _load_skill() -> str:
|
||||
parts: list[str] = []
|
||||
for path in _DEEP_RESEARCH_CANDIDATES:
|
||||
if path.exists():
|
||||
parts.append(path.read_text(encoding="utf-8").strip())
|
||||
break
|
||||
local = _SKILL_DIR / "SKILL.md"
|
||||
if local.exists():
|
||||
parts.append(local.read_text(encoding="utf-8").strip())
|
||||
if not parts:
|
||||
return "你是 AI 时讯调研员,只输出 JSON。"
|
||||
return "\n\n---\n\n".join(parts)
|
||||
|
||||
|
||||
def _guess_source_name(link: str, explicit: str) -> str:
|
||||
name = (explicit or "").strip()
|
||||
if name:
|
||||
return name
|
||||
host = urlparse(link).netloc.lower().removeprefix("www.")
|
||||
mapping = {
|
||||
"techcrunch.com": "TechCrunch",
|
||||
"theverge.com": "The Verge",
|
||||
"openai.com": "OpenAI",
|
||||
"anthropic.com": "Anthropic",
|
||||
"arxiv.org": "arXiv",
|
||||
"qbitai.com": "量子位",
|
||||
"36kr.com": "36氪",
|
||||
"leiphone.com": "雷锋网",
|
||||
}
|
||||
for key, label in mapping.items():
|
||||
if host.endswith(key) or key in host:
|
||||
return label
|
||||
return host.split(".")[0].capitalize() if host else "?"
|
||||
|
||||
|
||||
def _normalize_research_item(raw: dict[str, Any]) -> dict[str, Any] | None:
|
||||
title = str(raw.get("title") or "").strip()
|
||||
link = _normalize_link(str(raw.get("link") or ""))
|
||||
if not title or not link or not link.startswith("http"):
|
||||
return None
|
||||
desc = brief_news_summary(str(raw.get("desc_short") or raw.get("summary") or ""))
|
||||
item: dict[str, Any] = {
|
||||
"title": title,
|
||||
"link": link,
|
||||
"source_name": _guess_source_name(link, str(raw.get("source_name") or "")),
|
||||
"published_fmt": str(raw.get("published_fmt") or "").strip(),
|
||||
"desc_short": desc,
|
||||
"summary_plain": desc,
|
||||
}
|
||||
region = str(raw.get("region") or "").strip().lower()
|
||||
if region:
|
||||
item["region"] = region
|
||||
return item
|
||||
|
||||
|
||||
def research_tech_limit() -> int:
|
||||
return wecom_ai_news_tech_limit()
|
||||
|
||||
|
||||
def _parse_items_array(
|
||||
items_raw: Any,
|
||||
*,
|
||||
limit: int,
|
||||
seen: set[str],
|
||||
) -> list[dict[str, Any]]:
|
||||
if not isinstance(items_raw, list):
|
||||
return []
|
||||
out: list[dict[str, Any]] = []
|
||||
for row in items_raw:
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
item = _normalize_research_item(row)
|
||||
if not item:
|
||||
continue
|
||||
if item["link"] in seen:
|
||||
continue
|
||||
seen.add(item["link"])
|
||||
out.append(item)
|
||||
if len(out) >= limit:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def parse_research_response(
|
||||
raw: str,
|
||||
*,
|
||||
limit: int,
|
||||
tech_limit: int = 0,
|
||||
pool_limit: int | None = None,
|
||||
tech_pool_limit: int | None = None,
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
parsed = extract_json_object(raw)
|
||||
item_cap = pool_limit if pool_limit is not None else limit
|
||||
tech_cap = tech_pool_limit if tech_pool_limit is not None else tech_limit
|
||||
seen: set[str] = set()
|
||||
items = _parse_items_array(parsed.get("items"), limit=item_cap, seen=seen)
|
||||
tech_items = (
|
||||
_parse_items_array(parsed.get("tech_items"), limit=tech_cap, seen=seen) if tech_cap else []
|
||||
)
|
||||
return items, tech_items
|
||||
|
||||
|
||||
def _apply_pushed_dedup(items: list[dict[str, Any]], *, date_str: str, limit: int) -> list[dict[str, Any]]:
|
||||
from daily.config import news_backfill_enabled
|
||||
from daily.news.sanitize import strip_relax_window_prefix
|
||||
|
||||
for item in items:
|
||||
if item.get("desc_short"):
|
||||
item["desc_short"] = strip_relax_window_prefix(str(item.get("desc_short") or ""))
|
||||
fresh = filter_unpushed_items(items, date_str=date_str)
|
||||
if len(fresh) >= limit:
|
||||
return fresh[:limit]
|
||||
if not news_backfill_enabled():
|
||||
if len(fresh) < limit:
|
||||
logger.info("news_short:%s", len(fresh))
|
||||
return fresh[:limit]
|
||||
seen = {i.get("link") for i in fresh}
|
||||
for item in items:
|
||||
if len(fresh) >= limit:
|
||||
break
|
||||
if item.get("link") not in seen:
|
||||
fresh.append(item)
|
||||
seen.add(item.get("link"))
|
||||
return fresh[:limit]
|
||||
|
||||
|
||||
def fetch_ai_news_research(
|
||||
*,
|
||||
date_str: str,
|
||||
hours: int | None = None,
|
||||
limit: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Cursor Agent 调研 AI 时讯;返回 {enabled, mode, hours, items, flat, stats}。"""
|
||||
h = hours if hours is not None else research_hours()
|
||||
lim = limit if limit is not None else research_limit()
|
||||
tech_lim = research_tech_limit()
|
||||
|
||||
if not has_cursor_configured():
|
||||
logger.warning("DAILY_AI_NEWS_MODE=research 但未配置 CURSOR_API_KEY")
|
||||
return {
|
||||
"enabled": False,
|
||||
"mode": "research",
|
||||
"items": [],
|
||||
"tech_items": [],
|
||||
"flat": [],
|
||||
"stats": {"error": "no_cursor_key"},
|
||||
}
|
||||
|
||||
skill = _load_skill()
|
||||
now_cst = datetime.now(timezone(timedelta(hours=8)))
|
||||
cn_min = research_cn_min(lim)
|
||||
pool = research_pool_limit(lim)
|
||||
tech_pool = research_tech_pool_limit(tech_lim)
|
||||
# 候选池内国内目标略高于展示配额,避免筛完国内不足
|
||||
cn_pool_target = max(cn_min * 2, cn_min + 2)
|
||||
# 解析多收一点原始行,输出前/后处理再压成「去重后候选池」
|
||||
raw_cap = max(pool + 10, (pool * 3) // 2)
|
||||
tech_raw_cap = max(tech_pool + 4, (tech_pool * 3) // 2) if tech_pool else 0
|
||||
tech_clause = ""
|
||||
if tech_pool:
|
||||
tech_clause = (
|
||||
f"\n另输出 **去重后** 约 **{tech_pool} 条** tech_items 候选(最终展示约 {tech_lim} 条),聚焦工程技术:"
|
||||
"模型/框架发布、开源项目、芯片算力、开发者工具、推理与工程实践。"
|
||||
"不得与 items 重复 link/同事件;输出前自行去重,候选池内每条应为独立事件。"
|
||||
)
|
||||
system = (
|
||||
f"{skill}\n\n"
|
||||
"当前执行 **早报 AI 时讯调研**。\n"
|
||||
f"时间窗口:近 **{h}** 小时(截至 {now_cst.strftime('%Y-%m-%d %H:%M')} UTC+8)。\n"
|
||||
f"输出 **同事件去重后** 约 **{pool} 条** items 候选(按重要性排序;最终展示约 {lim} 条)。\n"
|
||||
"候选池条数 = 独立事件数:同一事件多源报道只留一条最权威源,禁止用换源重复充数。\n"
|
||||
f"去重后的候选中国内可信源尽量不少于 **{cn_pool_target}** 条(展示侧至少 {cn_min} 条)。\n"
|
||||
f"禁止用低质源凑数;可信独立事件不足才少返回。{tech_clause}\n"
|
||||
"只采用官方博客/新闻稿、政府监管原文、一线权威媒体、学术官方;"
|
||||
"禁止二手搬运、标题党、营销号。使用 WebSearch 检索;不要读取本项目文档或 RSS 配置。"
|
||||
)
|
||||
user = (
|
||||
f"/deep-research 获取近 {h} 小时的 AI 人工智能新闻资讯,"
|
||||
f"国内与国际合并;items 去重后约 {pool} 条独立事件(国内可信尽量 ≥{cn_pool_target});"
|
||||
"输出前完成同事件去重;可信度不足则不写。"
|
||||
f"只输出 JSON,items 去重后目标约 {pool} 条"
|
||||
+ (f",tech_items 去重后目标约 {tech_pool} 条" if tech_pool else "")
|
||||
+ "。"
|
||||
)
|
||||
|
||||
try:
|
||||
raw = cursor_agent_prompt(system, user)
|
||||
except Exception as exc:
|
||||
logger.warning("AI 时讯 research 失败:%s", exc)
|
||||
return {
|
||||
"enabled": False,
|
||||
"mode": "research",
|
||||
"items": [],
|
||||
"tech_items": [],
|
||||
"flat": [],
|
||||
"stats": {"error": str(exc)},
|
||||
}
|
||||
|
||||
if not raw:
|
||||
return {
|
||||
"enabled": False,
|
||||
"mode": "research",
|
||||
"items": [],
|
||||
"tech_items": [],
|
||||
"flat": [],
|
||||
"stats": {"error": "empty_response"},
|
||||
}
|
||||
|
||||
items, tech_items = parse_research_response(
|
||||
raw,
|
||||
limit=lim,
|
||||
tech_limit=tech_lim,
|
||||
pool_limit=raw_cap,
|
||||
tech_pool_limit=tech_raw_cap,
|
||||
)
|
||||
payload = extract_json_object(raw)
|
||||
if payload:
|
||||
_save_research_json(research_json_path(date_str), payload)
|
||||
|
||||
if not items and not tech_items:
|
||||
logger.warning("AI 时讯 research JSON 无效或无条目")
|
||||
return {
|
||||
"enabled": False,
|
||||
"mode": "research",
|
||||
"items": [],
|
||||
"tech_items": [],
|
||||
"flat": [],
|
||||
"stats": {"error": "invalid_json"},
|
||||
}
|
||||
|
||||
items, tech_items = post_process_research_news(
|
||||
items,
|
||||
tech_items,
|
||||
limit=lim,
|
||||
tech_limit=tech_lim,
|
||||
min_cn=cn_min,
|
||||
)
|
||||
items = _apply_pushed_dedup(items, date_str=date_str, limit=lim)
|
||||
if tech_items:
|
||||
tech_items = _apply_pushed_dedup(tech_items, date_str=date_str, limit=tech_lim)
|
||||
logger.info(
|
||||
"AI 时讯 research 完成:%d 条 + %d 技术(候选池 %d/%d)",
|
||||
len(items),
|
||||
len(tech_items),
|
||||
pool,
|
||||
tech_pool,
|
||||
)
|
||||
|
||||
flat = [
|
||||
{
|
||||
"title": i["title"],
|
||||
"link": i["link"],
|
||||
"summary": i.get("summary_plain") or i.get("desc_short") or "",
|
||||
"source_name": i["source_name"],
|
||||
"published_fmt": i.get("published_fmt") or "",
|
||||
"category_id": "research",
|
||||
"category_name": "Deep Research",
|
||||
"category_icon": "🔍",
|
||||
}
|
||||
for i in items + tech_items
|
||||
]
|
||||
|
||||
return {
|
||||
"enabled": True,
|
||||
"mode": "research",
|
||||
"hours": h,
|
||||
"items": items,
|
||||
"tech_items": tech_items,
|
||||
"flat": flat,
|
||||
"stats": {"source": "cursor_research", "items": len(items), "tech_items": len(tech_items)},
|
||||
}
|
||||
|
||||
|
||||
def format_research_news_section(
|
||||
research: dict[str, Any],
|
||||
*,
|
||||
section_no: int,
|
||||
wecom_limit: int | None = None,
|
||||
) -> list[str]:
|
||||
if not research.get("enabled"):
|
||||
hint = research.get("stats", {}).get("error", "调研失败或未配置 CURSOR_API_KEY")
|
||||
return [
|
||||
"---",
|
||||
"",
|
||||
f"## {section_no}、AI 时讯精选(Deep Research)",
|
||||
"",
|
||||
f"*不可用:{hint}*",
|
||||
"",
|
||||
]
|
||||
|
||||
hours = research.get("hours", 24)
|
||||
items = (research.get("flat") or [])[: wecom_limit or research_limit()]
|
||||
lines = [
|
||||
"---",
|
||||
"",
|
||||
f"## {section_no}、AI 时讯精选(Deep Research)",
|
||||
"",
|
||||
f"> 近 **{hours}h** · Cursor Agent WebSearch · {len(items)} 条",
|
||||
"",
|
||||
]
|
||||
if not items:
|
||||
lines.append("*暂无可用条目。*")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
for i, item in enumerate(items, 1):
|
||||
pub = f" · {item['published_fmt']}" if item.get("published_fmt") else ""
|
||||
lines.append(
|
||||
f"{i}. **[{item['title']}]({item['link']})** · `{item['source_name']}`{pub}"
|
||||
)
|
||||
summary = item.get("summary") or ""
|
||||
if summary:
|
||||
lines.append(f" - {summary}")
|
||||
lines.append("")
|
||||
return lines
|
||||
442
daily/news/research_quality.py
Normal file
442
daily/news/research_quality.py
Normal file
@@ -0,0 +1,442 @@
|
||||
"""Research 时讯后处理:可信源、同事件去重、tech 主题过滤、国内配额。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from daily.config import env_int
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 官方域(同事件去重时优先保留)
|
||||
_OFFICIAL_HOST_SUFFIXES: tuple[str, ...] = (
|
||||
"openai.com",
|
||||
"anthropic.com",
|
||||
"deepmind.google",
|
||||
"blog.google",
|
||||
"ai.googleblog.com",
|
||||
"microsoft.com",
|
||||
"meta.com",
|
||||
"engineering.fb.com",
|
||||
"nvidia.com",
|
||||
"huggingface.co",
|
||||
"arxiv.org",
|
||||
"github.com",
|
||||
"github.blog",
|
||||
"modelcontextprotocol.io",
|
||||
"cursor.com",
|
||||
"vercel.com",
|
||||
"langchain.dev",
|
||||
"cohere.com",
|
||||
"moonshot.cn",
|
||||
"moonshot.ai",
|
||||
"sktelecom.com",
|
||||
"tether.io",
|
||||
)
|
||||
|
||||
# 权威媒体 / 可信站(国际 + 国内)
|
||||
_TRUSTED_HOST_SUFFIXES: tuple[str, ...] = _OFFICIAL_HOST_SUFFIXES + (
|
||||
"techcrunch.com",
|
||||
"theverge.com",
|
||||
"wired.com",
|
||||
"arstechnica.com",
|
||||
"venturebeat.com",
|
||||
"technologyreview.com",
|
||||
"engadget.com",
|
||||
"cnet.com",
|
||||
"zdnet.com",
|
||||
"axios.com",
|
||||
"reuters.com",
|
||||
"bloomberg.com",
|
||||
"bloomberglaw.com",
|
||||
"bbc.com",
|
||||
"bbc.co.uk",
|
||||
"nytimes.com",
|
||||
"wsj.com",
|
||||
"ft.com",
|
||||
"nbcnews.com",
|
||||
"time.com",
|
||||
"theguardian.com",
|
||||
"washingtonpost.com",
|
||||
"theregister.com",
|
||||
"nature.com",
|
||||
"science.org",
|
||||
"scmp.com",
|
||||
"caixinglobal.com",
|
||||
"caixin.com",
|
||||
"qbitai.com",
|
||||
"36kr.com",
|
||||
"leiphone.com",
|
||||
"jiqizhixin.com",
|
||||
"ithome.com",
|
||||
"tmtpost.com",
|
||||
"huxiu.com",
|
||||
"solidot.org",
|
||||
"synched.cn",
|
||||
"infoq.cn",
|
||||
"yicai.com",
|
||||
"news.cn",
|
||||
"xinhuanet.com",
|
||||
"people.com.cn",
|
||||
"cls.cn",
|
||||
"geekpark.net",
|
||||
"standard.com",
|
||||
"business-standard.com",
|
||||
"siliconvalley.com",
|
||||
)
|
||||
|
||||
_TRUSTED_SOURCE_NAMES: frozenset[str] = frozenset(
|
||||
{
|
||||
"techcrunch",
|
||||
"the verge",
|
||||
"wired",
|
||||
"ars technica",
|
||||
"engadget",
|
||||
"reuters",
|
||||
"bloomberg",
|
||||
"bloomberg law",
|
||||
"nbc news",
|
||||
"time",
|
||||
"the register",
|
||||
"openai",
|
||||
"anthropic",
|
||||
"arxiv",
|
||||
"hugging face",
|
||||
"mcp blog",
|
||||
"github",
|
||||
"sk telecom",
|
||||
"tether",
|
||||
"量子位",
|
||||
"36氪",
|
||||
"36kr",
|
||||
"雷锋网",
|
||||
"机器之心",
|
||||
"it之家",
|
||||
"财新",
|
||||
"caixin",
|
||||
"钛媒体",
|
||||
"虎嗅",
|
||||
"第一财经",
|
||||
"新华网",
|
||||
"新华社",
|
||||
"财联社",
|
||||
"极客公园",
|
||||
"人民日报",
|
||||
}
|
||||
)
|
||||
|
||||
_CN_HOST_SUFFIXES: tuple[str, ...] = (
|
||||
"qbitai.com",
|
||||
"36kr.com",
|
||||
"leiphone.com",
|
||||
"jiqizhixin.com",
|
||||
"ithome.com",
|
||||
"caixin.com",
|
||||
"caixinglobal.com",
|
||||
"tmtpost.com",
|
||||
"huxiu.com",
|
||||
"solidot.org",
|
||||
"synched.cn",
|
||||
"infoq.cn",
|
||||
"moonshot.cn",
|
||||
"yicai.com",
|
||||
"news.cn",
|
||||
"xinhuanet.com",
|
||||
"people.com.cn",
|
||||
"cls.cn",
|
||||
"geekpark.net",
|
||||
"zhihu.com",
|
||||
"sina.com.cn",
|
||||
"qq.com",
|
||||
"163.com",
|
||||
)
|
||||
|
||||
_CN_SOURCE_NAMES: frozenset[str] = frozenset(
|
||||
{
|
||||
"量子位",
|
||||
"36氪",
|
||||
"36kr",
|
||||
"雷锋网",
|
||||
"机器之心",
|
||||
"it之家",
|
||||
"财新",
|
||||
"caixin",
|
||||
"钛媒体",
|
||||
"虎嗅",
|
||||
"月之暗面",
|
||||
"第一财经",
|
||||
"新华网",
|
||||
"新华社",
|
||||
"财联社",
|
||||
"极客公园",
|
||||
"人民日报",
|
||||
}
|
||||
)
|
||||
|
||||
_ENTITIES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("openai", ("openai", "altman", "chatgpt", "奥特曼")),
|
||||
("anthropic", ("anthropic", "amodei", "claude")),
|
||||
("kimi", ("kimi", "moonshot", "月之暗面")),
|
||||
("mcp", ("mcp", "model context protocol", "modelcontextprotocol")),
|
||||
("nvidia", ("nvidia", "英伟达")),
|
||||
("amd", ("amd",)),
|
||||
("hugging_face", ("hugging face", "huggingface")),
|
||||
("google", ("google", "deepmind", "gemini")),
|
||||
("meta", ("meta", "llama")),
|
||||
("microsoft", ("microsoft", "copilot")),
|
||||
("huawei", ("huawei", "华为", "昇腾", "ascend")),
|
||||
("moore", ("摩尔线程", "moore threads", "musa")),
|
||||
)
|
||||
|
||||
_EVENT_CLUSTERS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("petition", ("petition", "联名", "decelerat", "pace ai", "控制", "减速")),
|
||||
("hack", ("hack", "入侵", "siege", "breach", "攻击", "逃逸")),
|
||||
("open_source", ("open-source", "opensource", "open sources", "开源", "open-sources")),
|
||||
("adapt", ("适配", "adapt", "day-0", "day0", "day 0", "推理部署", "训练适配")),
|
||||
("release", ("release", "发布", "specification", "规范", "v2.0", "changelog")),
|
||||
("chip", ("chip", "芯片", "data center", "数据中心", "mi455")),
|
||||
("regulate", ("framework", "监管", "voluntary", "审核", "ban", "禁止")),
|
||||
)
|
||||
|
||||
_RELEASE_FAMILY = frozenset({"open_source", "adapt", "release"})
|
||||
# 分发平台,不参与 tech↔items 主题冲突(避免 HF 上架与 HF 被黑误杀)
|
||||
_PLATFORM_ENTITIES = frozenset({"hugging_face"})
|
||||
|
||||
|
||||
def _host(link: str) -> str:
|
||||
return urlparse(link).netloc.lower().removeprefix("www.")
|
||||
|
||||
|
||||
def _ends_with_any(host: str, suffixes: tuple[str, ...]) -> bool:
|
||||
return any(host == s or host.endswith("." + s) for s in suffixes)
|
||||
|
||||
|
||||
def _norm_text(*parts: str) -> str:
|
||||
text = " ".join(p for p in parts if p).lower()
|
||||
text = re.sub(r"[\s\-_/|·,。、::()()【】\[\]]+", " ", text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _item_text(item: dict[str, Any]) -> str:
|
||||
return _norm_text(
|
||||
str(item.get("title") or ""),
|
||||
str(item.get("desc_short") or item.get("summary_plain") or ""),
|
||||
)
|
||||
|
||||
|
||||
def _match_labels(text: str, table: tuple[tuple[str, tuple[str, ...]], ...]) -> frozenset[str]:
|
||||
hit: set[str] = set()
|
||||
for label, kws in table:
|
||||
if any(kw in text for kw in kws):
|
||||
hit.add(label)
|
||||
return frozenset(hit)
|
||||
|
||||
|
||||
def entities_of(item: dict[str, Any]) -> frozenset[str]:
|
||||
return _match_labels(_item_text(item), _ENTITIES)
|
||||
|
||||
|
||||
def events_of(item: dict[str, Any]) -> frozenset[str]:
|
||||
return _match_labels(_item_text(item), _EVENT_CLUSTERS)
|
||||
|
||||
|
||||
def event_key(item: dict[str, Any]) -> tuple[frozenset[str], frozenset[str]] | None:
|
||||
ents = entities_of(item)
|
||||
evs = events_of(item)
|
||||
if not ents or not evs:
|
||||
return None
|
||||
return (ents, evs)
|
||||
|
||||
|
||||
def same_event(a: dict[str, Any], b: dict[str, Any]) -> bool:
|
||||
"""共享至少一实体且共享至少一事件簇 → 同事件(保守合并)。"""
|
||||
ea, eva = entities_of(a), events_of(a)
|
||||
eb, evb = entities_of(b), events_of(b)
|
||||
if not ea or not eb or not eva or not evb:
|
||||
return False
|
||||
return bool(ea & eb) and bool(eva & evb)
|
||||
|
||||
|
||||
def is_official_item(item: dict[str, Any]) -> bool:
|
||||
return _ends_with_any(_host(str(item.get("link") or "")), _OFFICIAL_HOST_SUFFIXES)
|
||||
|
||||
|
||||
def _is_institutional_cn_host(host: str) -> bool:
|
||||
"""新华社 / 政府站等机构域,默认可信。"""
|
||||
if host.endswith(".gov.cn") or host.endswith(".gov.cn."):
|
||||
return True
|
||||
if host == "news.cn" or host.endswith(".news.cn"):
|
||||
return True
|
||||
if host.endswith("xinhuanet.com") or host.endswith("people.com.cn"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def is_trusted_item(item: dict[str, Any]) -> bool:
|
||||
host = _host(str(item.get("link") or ""))
|
||||
if _is_institutional_cn_host(host):
|
||||
return True
|
||||
if _ends_with_any(host, _TRUSTED_HOST_SUFFIXES):
|
||||
return True
|
||||
name = str(item.get("source_name") or "").strip().lower()
|
||||
return name in _TRUSTED_SOURCE_NAMES
|
||||
|
||||
|
||||
def is_cn_item(item: dict[str, Any]) -> bool:
|
||||
region = str(item.get("region") or "").strip().lower()
|
||||
if region in {"cn", "china", "zh", "zh-cn"}:
|
||||
return True
|
||||
if region in {"intl", "international", "global", "en"}:
|
||||
return False
|
||||
host = _host(str(item.get("link") or ""))
|
||||
if host.endswith(".cn") or _ends_with_any(host, _CN_HOST_SUFFIXES):
|
||||
return True
|
||||
name = str(item.get("source_name") or "").strip().lower()
|
||||
return name in {n.lower() for n in _CN_SOURCE_NAMES}
|
||||
|
||||
|
||||
def research_cn_min(limit: int) -> int:
|
||||
explicit = env_int("DAILY_WECOM_AI_NEWS_CN_MIN", 0)
|
||||
if explicit > 0:
|
||||
return min(explicit, max(1, limit))
|
||||
return max(1, limit * 3 // 10)
|
||||
|
||||
|
||||
def _trust_rank(item: dict[str, Any]) -> int:
|
||||
if is_official_item(item):
|
||||
return 0
|
||||
if is_trusted_item(item):
|
||||
return 1
|
||||
return 2
|
||||
|
||||
|
||||
def filter_trusted(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
kept = [i for i in items if is_trusted_item(i)]
|
||||
dropped = len(items) - len(kept)
|
||||
if dropped:
|
||||
logger.info("news_dedup_drop:trusted=%s", dropped)
|
||||
return kept
|
||||
|
||||
|
||||
def dedupe_same_event(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""同事件只留一条;官方源优先,否则保留更靠前的。"""
|
||||
kept: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
replaced = False
|
||||
for idx, prev in enumerate(kept):
|
||||
if not same_event(item, prev):
|
||||
continue
|
||||
if _trust_rank(item) < _trust_rank(prev):
|
||||
kept[idx] = item
|
||||
replaced = True
|
||||
break
|
||||
if not replaced:
|
||||
kept.append(item)
|
||||
|
||||
dropped = len(items) - len(kept)
|
||||
if dropped:
|
||||
logger.info("news_dedup_drop:same_event=%s", dropped)
|
||||
return kept
|
||||
|
||||
|
||||
def filter_tech_against_items(
|
||||
tech_items: list[dict[str, Any]],
|
||||
items: list[dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
kept: list[dict[str, Any]] = []
|
||||
for tech in tech_items:
|
||||
t_ents = entities_of(tech)
|
||||
t_evs = events_of(tech)
|
||||
conflict = False
|
||||
for item in items:
|
||||
if same_event(tech, item):
|
||||
conflict = True
|
||||
break
|
||||
shared = (t_ents & entities_of(item)) - _PLATFORM_ENTITIES
|
||||
if shared and ((t_evs | events_of(item)) & _RELEASE_FAMILY):
|
||||
conflict = True
|
||||
break
|
||||
if not conflict:
|
||||
kept.append(tech)
|
||||
dropped = len(tech_items) - len(kept)
|
||||
if dropped:
|
||||
logger.info("news_dedup_drop:tech_topic=%s", dropped)
|
||||
return kept
|
||||
|
||||
|
||||
def pack_with_cn_quota(
|
||||
items: list[dict[str, Any]],
|
||||
*,
|
||||
limit: int,
|
||||
min_cn: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
if limit <= 0:
|
||||
return []
|
||||
min_cn = max(0, min(min_cn, limit))
|
||||
out: list[dict[str, Any]] = []
|
||||
used: set[str] = set()
|
||||
cn_got = 0
|
||||
|
||||
def _take(item: dict[str, Any]) -> None:
|
||||
nonlocal cn_got
|
||||
link = str(item.get("link") or "")
|
||||
if not link or link in used:
|
||||
return
|
||||
out.append(item)
|
||||
used.add(link)
|
||||
if is_cn_item(item):
|
||||
cn_got += 1
|
||||
|
||||
for item in items:
|
||||
if len(out) >= limit:
|
||||
break
|
||||
link = str(item.get("link") or "")
|
||||
if not link or link in used:
|
||||
continue
|
||||
slots_left = limit - len(out)
|
||||
need_cn = max(0, min_cn - cn_got)
|
||||
if not is_cn_item(item) and slots_left <= need_cn:
|
||||
continue
|
||||
_take(item)
|
||||
|
||||
if len(out) < limit:
|
||||
for item in items:
|
||||
if len(out) >= limit:
|
||||
break
|
||||
_take(item)
|
||||
|
||||
final_cn = sum(1 for i in out if is_cn_item(i))
|
||||
if final_cn < min_cn:
|
||||
logger.info("news_cn_short:%s", final_cn)
|
||||
return out[:limit]
|
||||
|
||||
|
||||
def build_deduped_candidate_pool(
|
||||
items: list[dict[str, Any]],
|
||||
tech_items: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""拉取后立即得到去重候选池:可信过滤 + 同事件去重 + tech 相对 items 主题过滤。"""
|
||||
items = dedupe_same_event(filter_trusted(items))
|
||||
tech_items = dedupe_same_event(filter_trusted(tech_items))
|
||||
tech_items = filter_tech_against_items(tech_items, items)
|
||||
logger.info("research_pool_deduped:items=%s tech=%s", len(items), len(tech_items))
|
||||
return items, tech_items
|
||||
|
||||
|
||||
def post_process_research_news(
|
||||
items: list[dict[str, Any]],
|
||||
tech_items: list[dict[str, Any]],
|
||||
*,
|
||||
limit: int,
|
||||
tech_limit: int,
|
||||
min_cn: int | None = None,
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
"""去重候选池 → 国内配额打包 → 截到展示上限。"""
|
||||
cn_min = research_cn_min(limit) if min_cn is None else max(0, min_cn)
|
||||
items, tech_items = build_deduped_candidate_pool(items, tech_items)
|
||||
items = pack_with_cn_quota(items, limit=limit, min_cn=cn_min)
|
||||
tech_items = filter_tech_against_items(tech_items, items)
|
||||
return items, tech_items[: max(0, tech_limit)]
|
||||
18
daily/news/sanitize.py
Normal file
18
daily/news/sanitize.py
Normal file
@@ -0,0 +1,18 @@
|
||||
"""新闻文案清洗:剥离「放宽窗口」类凑数前缀。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_RELAX_PREFIX = re.compile(
|
||||
r"^(?:放宽窗口|放宽至[^::]*)\s*[::]\s*",
|
||||
re.UNICODE,
|
||||
)
|
||||
|
||||
|
||||
def strip_relax_window_prefix(text: str) -> str:
|
||||
"""去掉开头的「放宽窗口:」/「放宽至…:」前缀。"""
|
||||
raw = (text or "").strip()
|
||||
if not raw:
|
||||
return ""
|
||||
return _RELAX_PREFIX.sub("", raw, count=1).strip()
|
||||
61
daily/push_gate.py
Normal file
61
daily/push_gate.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""企微早报推送闸门。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from daily.config import force_push, skip_push_when_silent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PushGateResult:
|
||||
should_push: bool
|
||||
reasons: list[str] = field(default_factory=list)
|
||||
silent: bool = False
|
||||
|
||||
|
||||
def _has_board_moves(movement: dict[str, Any]) -> bool:
|
||||
keys = (
|
||||
"skills_trending_moves",
|
||||
"skills_hot_moves",
|
||||
"github_trending_moves",
|
||||
"github_emerging_moves",
|
||||
"github_topic_moves",
|
||||
)
|
||||
return any(movement.get(k) for k in keys)
|
||||
|
||||
|
||||
def evaluate_push_gate(
|
||||
*,
|
||||
movement: dict[str, Any],
|
||||
ai_news_items: list[dict[str, Any]],
|
||||
cn_ai_news_items: list[dict[str, Any]],
|
||||
featured_pick: dict[str, Any] | None,
|
||||
) -> PushGateResult:
|
||||
if force_push():
|
||||
logger.info("push_gate: force_push=on, 强制推送")
|
||||
return PushGateResult(should_push=True, reasons=["force_push"], silent=False)
|
||||
|
||||
reasons: list[str] = []
|
||||
if _has_board_moves(movement):
|
||||
reasons.append("board_moves")
|
||||
if ai_news_items:
|
||||
reasons.append("ai_news")
|
||||
if cn_ai_news_items:
|
||||
reasons.append("cn_ai_news")
|
||||
if featured_pick:
|
||||
reasons.append("featured_pick")
|
||||
|
||||
should = bool(reasons)
|
||||
silent = not should and skip_push_when_silent()
|
||||
if should:
|
||||
logger.info("push_gate: 推送 (原因: %s)", ",".join(reasons))
|
||||
elif silent:
|
||||
logger.info("push_gate: 静默日, 跳过推送 (无任何更新信号)")
|
||||
else:
|
||||
logger.info("push_gate: 无更新但 skip_push_when_silent=off, 仍推送")
|
||||
return PushGateResult(should_push=should, reasons=reasons, silent=silent)
|
||||
@@ -6,10 +6,11 @@ import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from daily.config import OUTPUT_DIR, env_int
|
||||
from daily.delta import build_movement_baseline, build_movement_context, compare_depth
|
||||
from daily.config import OUTPUT_DIR, env_int, wecom_mode
|
||||
from daily.delta import build_movement_baseline, build_movement_context, compare_depth, effective_wecom_mode
|
||||
from daily.news.fetch import prepare_wecom_cn_news_items, prepare_wecom_news_items
|
||||
from daily.skills_group import group_skills_by_source
|
||||
from daily.text_utils import trim_brief
|
||||
|
||||
|
||||
def skill_id(item: dict[str, Any]) -> str:
|
||||
@@ -59,16 +60,20 @@ def _slim_news_items(
|
||||
limit: int,
|
||||
*,
|
||||
prepare=prepare_wecom_news_items,
|
||||
date_str: str | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
for item in prepare(ai_news):
|
||||
for item in prepare(ai_news, date_str=date_str):
|
||||
items.append(
|
||||
{
|
||||
"link": item.get("link", ""),
|
||||
"title": item.get("title", ""),
|
||||
"source_name": item.get("source_name", ""),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"summary": item.get("desc_short") or "",
|
||||
"summary": trim_brief(
|
||||
item.get("summary_plain") or item.get("desc_short") or "",
|
||||
120,
|
||||
),
|
||||
}
|
||||
)
|
||||
if len(items) >= limit:
|
||||
@@ -89,7 +94,7 @@ def _slim_news_items(
|
||||
|
||||
|
||||
def _wecom_skill_pool() -> int:
|
||||
return max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
|
||||
return max(10, env_int("DAILY_WECOM_SKILL_POOL", 400))
|
||||
|
||||
|
||||
def build_llm_input(
|
||||
@@ -105,10 +110,46 @@ def build_llm_input(
|
||||
ai_news: dict[str, Any],
|
||||
cn_ai_news: dict[str, Any],
|
||||
wecom_limits: dict[str, int],
|
||||
research_items: list[dict[str, Any]] | None = None,
|
||||
research_tech_items: list[dict[str, Any]] | None = None,
|
||||
boards_for_wecom: dict[str, list[dict[str, Any]]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""供 Cursor 编辑的精简 JSON(不含完整 markdown)。"""
|
||||
news_limit = wecom_limits.get("ai_news", 10)
|
||||
cn_news_limit = wecom_limits.get("cn_ai_news", 8)
|
||||
if research_items is not None:
|
||||
slim_research = [
|
||||
{
|
||||
"link": item.get("link", ""),
|
||||
"title": item.get("title", ""),
|
||||
"source_name": item.get("source_name", ""),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"summary": trim_brief(item.get("desc_short") or "", 120),
|
||||
}
|
||||
for item in research_items[:news_limit]
|
||||
]
|
||||
ai_news_payload = slim_research
|
||||
tech_news_payload = [
|
||||
{
|
||||
"link": item.get("link", ""),
|
||||
"title": item.get("title", ""),
|
||||
"source_name": item.get("source_name", ""),
|
||||
"published_fmt": item.get("published_fmt", ""),
|
||||
"summary": trim_brief(item.get("desc_short") or "", 120),
|
||||
}
|
||||
for item in (research_tech_items or [])
|
||||
]
|
||||
cn_news_payload: list[dict[str, Any]] = []
|
||||
ai_news_mode = "research"
|
||||
else:
|
||||
ai_news_payload = _slim_news_items(ai_news, news_limit, date_str=date_str) if ai_news.get("enabled") else []
|
||||
cn_news_payload = (
|
||||
_slim_news_items(cn_ai_news, cn_news_limit, prepare=prepare_wecom_cn_news_items, date_str=date_str)
|
||||
if cn_ai_news.get("enabled")
|
||||
else []
|
||||
)
|
||||
ai_news_mode = "rss"
|
||||
tech_news_payload: list[dict[str, Any]] = []
|
||||
depth = compare_depth()
|
||||
trend_cmp = trending[:depth]
|
||||
hot_cmp = hot[:depth]
|
||||
@@ -116,19 +157,26 @@ def build_llm_input(
|
||||
emerging_cmp = github_emerging[:depth]
|
||||
topic_cmp = github_topic[:depth]
|
||||
|
||||
trending_slice = group_skills_by_source(
|
||||
trending,
|
||||
limit=wecom_limits.get("trending", 10),
|
||||
pool_size=wecom_limits.get("trending_pool", _wecom_skill_pool()),
|
||||
)
|
||||
hot_slice = group_skills_by_source(
|
||||
hot,
|
||||
limit=wecom_limits.get("hot", 10),
|
||||
pool_size=wecom_limits.get("hot_pool", _wecom_skill_pool()),
|
||||
)
|
||||
github_slice = github_trending[: wecom_limits.get("github", 5)]
|
||||
emerging_slice = github_emerging[: wecom_limits.get("emerging", 3)]
|
||||
topic_slice = github_topic[: wecom_limits.get("topic", 3)]
|
||||
if boards_for_wecom:
|
||||
trending_slice = boards_for_wecom.get("skills_trending") or []
|
||||
hot_slice = boards_for_wecom.get("skills_hot") or []
|
||||
github_slice = boards_for_wecom.get("github_trending") or []
|
||||
emerging_slice = boards_for_wecom.get("github_emerging") or []
|
||||
topic_slice = boards_for_wecom.get("github_topic") or []
|
||||
else:
|
||||
trending_slice = group_skills_by_source(
|
||||
trending,
|
||||
limit=wecom_limits.get("trending", 10),
|
||||
pool_size=wecom_limits.get("trending_pool", _wecom_skill_pool()),
|
||||
)
|
||||
hot_slice = group_skills_by_source(
|
||||
hot,
|
||||
limit=wecom_limits.get("hot", 10),
|
||||
pool_size=wecom_limits.get("hot_pool", _wecom_skill_pool()),
|
||||
)
|
||||
github_slice = github_trending[: wecom_limits.get("github", 5)]
|
||||
emerging_slice = github_emerging[: wecom_limits.get("emerging", 3)]
|
||||
topic_slice = github_topic[: wecom_limits.get("topic", 3)]
|
||||
|
||||
movement = build_movement_context(
|
||||
date_str=date_str,
|
||||
@@ -147,10 +195,13 @@ def build_llm_input(
|
||||
github_topic=[_slim_github(x) for x in topic_cmp],
|
||||
depth=depth,
|
||||
)
|
||||
eff_mode = effective_wecom_mode(date_str=date_str)
|
||||
|
||||
return {
|
||||
"date": date_str,
|
||||
"data_updated": updated,
|
||||
"wecom_mode": wecom_mode(),
|
||||
"effective_wecom_mode": eff_mode,
|
||||
"skills_trending": [_slim_skill(x) for x in trending_slice],
|
||||
"skills_hot": [_slim_skill(x) for x in hot_slice],
|
||||
"github_trending": [_slim_github(x) for x in github_slice],
|
||||
@@ -159,12 +210,10 @@ def build_llm_input(
|
||||
"topic": topic_name,
|
||||
"repos": [_slim_github(x) for x in topic_slice],
|
||||
},
|
||||
"ai_news": _slim_news_items(ai_news, news_limit) if ai_news.get("enabled") else [],
|
||||
"cn_ai_news": _slim_news_items(
|
||||
cn_ai_news, cn_news_limit, prepare=prepare_wecom_cn_news_items
|
||||
)
|
||||
if cn_ai_news.get("enabled")
|
||||
else [],
|
||||
"ai_news": ai_news_payload,
|
||||
"tech_ai_news": tech_news_payload,
|
||||
"cn_ai_news": cn_news_payload,
|
||||
"ai_news_mode": ai_news_mode,
|
||||
"movement": movement,
|
||||
"movement_baseline": movement_baseline,
|
||||
}
|
||||
@@ -186,6 +235,10 @@ def editorial_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.editorial.json"
|
||||
|
||||
|
||||
def featured_json_path(date_str: str) -> Path:
|
||||
return OUTPUT_DIR / f"{date_str}.featured.json"
|
||||
|
||||
|
||||
def save_json(path: Path, data: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
307
daily/scheduler.py
Normal file
307
daily/scheduler.py
Normal file
@@ -0,0 +1,307 @@
|
||||
"""常驻调度:按配置时刻生成早报并推送企微。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, time as dt_time, timedelta
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from daily.config import (
|
||||
CACHE_DIR,
|
||||
LOG_DIR,
|
||||
OUTPUT_DIR,
|
||||
ROOT,
|
||||
schedule_generate_at,
|
||||
schedule_push_at,
|
||||
schedule_timezone_name,
|
||||
workday_only,
|
||||
)
|
||||
from daily.holiday import is_workday, load_holidays, workday_name
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_STATE_FILE = CACHE_DIR / "scheduler-state.json"
|
||||
_POLL_SECONDS = 15
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClockTime:
|
||||
hour: int
|
||||
minute: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchedulerState:
|
||||
last_generate_date: str | None = None
|
||||
last_push_date: str | None = None
|
||||
|
||||
@classmethod
|
||||
def load(cls) -> SchedulerState:
|
||||
if not _STATE_FILE.exists():
|
||||
return cls()
|
||||
try:
|
||||
raw = json.loads(_STATE_FILE.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return cls()
|
||||
if not isinstance(raw, dict):
|
||||
return cls()
|
||||
return cls(
|
||||
last_generate_date=raw.get("last_generate_date"),
|
||||
last_push_date=raw.get("last_push_date"),
|
||||
)
|
||||
|
||||
def save(self) -> None:
|
||||
_STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {
|
||||
"last_generate_date": self.last_generate_date,
|
||||
"last_push_date": self.last_push_date,
|
||||
}
|
||||
_STATE_FILE.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def parse_hhmm(value: str) -> ClockTime:
|
||||
raw = (value or "").strip()
|
||||
parts = raw.split(":", 1)
|
||||
if len(parts) != 2:
|
||||
raise ValueError(f"无效时间格式: {value!r},应为 HH:MM")
|
||||
hour = int(parts[0])
|
||||
minute = int(parts[1])
|
||||
if not (0 <= hour <= 23 and 0 <= minute <= 59):
|
||||
raise ValueError(f"无效时间: {value!r}")
|
||||
return ClockTime(hour=hour, minute=minute)
|
||||
|
||||
|
||||
def load_timezone() -> ZoneInfo:
|
||||
name = schedule_timezone_name()
|
||||
try:
|
||||
return ZoneInfo(name)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"无效时区 DAILY_SCHEDULE_TZ={name!r}") from exc
|
||||
|
||||
|
||||
def _localize(day: date, clock: ClockTime, tz: ZoneInfo) -> datetime:
|
||||
return datetime.combine(day, dt_time(clock.hour, clock.minute), tz)
|
||||
|
||||
|
||||
def next_occurrence_after(clock: ClockTime, tz: ZoneInfo, after: datetime) -> datetime:
|
||||
local = after.astimezone(tz)
|
||||
candidate = local.replace(hour=clock.hour, minute=clock.minute, second=0, microsecond=0)
|
||||
if candidate <= local:
|
||||
candidate += timedelta(days=1)
|
||||
return candidate
|
||||
|
||||
|
||||
def _today_slot(day: date, clock: ClockTime, tz: ZoneInfo) -> datetime:
|
||||
return _localize(day, clock, tz)
|
||||
|
||||
|
||||
def _run_daily_subcommand(subcmd: str, *extra: str) -> int:
|
||||
cmd = [sys.executable, "-m", "daily", subcmd, *extra]
|
||||
logger.info("执行: %s", " ".join(cmd))
|
||||
proc = subprocess.run(cmd, cwd=str(ROOT), check=False)
|
||||
return int(proc.returncode)
|
||||
|
||||
|
||||
def run_generate() -> int:
|
||||
return _run_daily_subcommand("generate")
|
||||
|
||||
|
||||
def run_push_for_date(date_str: str) -> int:
|
||||
report = OUTPUT_DIR / f"{date_str}.wecom.md"
|
||||
if not report.exists():
|
||||
logger.error("推送失败:报告不存在 %s", report)
|
||||
return 1
|
||||
return _run_daily_subcommand("push", str(report))
|
||||
|
||||
|
||||
def _setup_logging() -> Path:
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
log_path = LOG_DIR / "scheduler.log"
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
handlers=[
|
||||
logging.FileHandler(log_path, encoding="utf-8"),
|
||||
logging.StreamHandler(sys.stdout),
|
||||
],
|
||||
)
|
||||
return log_path
|
||||
|
||||
|
||||
def _sleep_until(target: datetime, tz: ZoneInfo) -> None:
|
||||
while True:
|
||||
now = datetime.now(tz)
|
||||
seconds = (target - now).total_seconds()
|
||||
if seconds <= 0:
|
||||
return
|
||||
time.sleep(min(seconds, _POLL_SECONDS))
|
||||
|
||||
|
||||
def plan_next_action(
|
||||
*,
|
||||
now: datetime,
|
||||
tz: ZoneInfo,
|
||||
state: SchedulerState,
|
||||
generate_at: ClockTime,
|
||||
push_at: ClockTime,
|
||||
) -> tuple[datetime, str] | None:
|
||||
"""返回下一次应执行的动作;若今日已全部完成则返回明日 generate。"""
|
||||
today = now.astimezone(tz).date()
|
||||
today_str = today.isoformat()
|
||||
gen_done = state.last_generate_date == today_str
|
||||
push_done = state.last_push_date == today_str
|
||||
gen_slot = _today_slot(today, generate_at, tz)
|
||||
push_slot = _today_slot(today, push_at, tz)
|
||||
|
||||
# 推送窗口内:generate 未做则立即补跑(须先于 push)
|
||||
if not gen_done and gen_slot <= now <= push_slot:
|
||||
return now, "generate"
|
||||
# 已过推送时刻:仅当 generate 已完成时补跑 push
|
||||
if not push_done and gen_done and now >= push_slot:
|
||||
return now, "push"
|
||||
|
||||
candidates: list[tuple[datetime, str]] = []
|
||||
if not gen_done and gen_slot > now:
|
||||
candidates.append((gen_slot, "generate"))
|
||||
if not push_done and push_slot > now:
|
||||
candidates.append((push_slot, "push"))
|
||||
if candidates:
|
||||
return min(candidates, key=lambda item: item[0])
|
||||
|
||||
tomorrow_gen = next_occurrence_after(generate_at, tz, now)
|
||||
return tomorrow_gen, "generate"
|
||||
|
||||
|
||||
def run_scheduled_action(action: str, *, today_str: str) -> int:
|
||||
if action == "generate":
|
||||
return run_generate()
|
||||
if action == "push":
|
||||
return run_push_for_date(today_str)
|
||||
raise ValueError(f"未知动作: {action}")
|
||||
|
||||
|
||||
def tick_once(
|
||||
*,
|
||||
now: datetime | None = None,
|
||||
tz: ZoneInfo | None = None,
|
||||
state: SchedulerState | None = None,
|
||||
generate_at: ClockTime | None = None,
|
||||
push_at: ClockTime | None = None,
|
||||
dry_run: bool = False,
|
||||
) -> SchedulerState:
|
||||
tz = tz or load_timezone()
|
||||
now = now or datetime.now(tz)
|
||||
state = state or SchedulerState.load()
|
||||
generate_at = generate_at or parse_hhmm(schedule_generate_at())
|
||||
push_at = push_at or parse_hhmm(schedule_push_at())
|
||||
|
||||
today_str = now.astimezone(tz).date().isoformat()
|
||||
|
||||
# 仅工作日运行:非工作日(法定节假日/周末,调休补班日除外)跳过当日 generate/push。
|
||||
# 将两者标记为已完成,避免 plan_next_action 在当天反复补跑。
|
||||
if workday_only() and state.last_generate_date != today_str:
|
||||
today = now.astimezone(tz).date()
|
||||
try:
|
||||
holidays = load_holidays(today.year)
|
||||
if not is_workday(today, holidays):
|
||||
name = workday_name(today, holidays) or "周末"
|
||||
logger.info("今日 %s 非工作日(%s),跳过生成与推送", today_str, name)
|
||||
state.last_generate_date = today_str
|
||||
state.last_push_date = today_str
|
||||
if not dry_run:
|
||||
state.save()
|
||||
if dry_run:
|
||||
logger.info("[dry-run] 非工作日,将跳过")
|
||||
return state
|
||||
except Exception as exc:
|
||||
# 节假日数据不可用时按常规工作日处理,避免因接口故障漏跑
|
||||
logger.warning("工作日判定失败(%s),按正常工作日处理", exc)
|
||||
|
||||
planned = plan_next_action(
|
||||
now=now,
|
||||
tz=tz,
|
||||
state=state,
|
||||
generate_at=generate_at,
|
||||
push_at=push_at,
|
||||
)
|
||||
if not planned:
|
||||
return state
|
||||
|
||||
run_at, action = planned
|
||||
if run_at > now:
|
||||
if not dry_run:
|
||||
logger.info("下次 %s @ %s (%s)", action, run_at.isoformat(), tz.key)
|
||||
_sleep_until(run_at, tz)
|
||||
elif not dry_run:
|
||||
slot = _today_slot(now.astimezone(tz).date(), generate_at if action == "generate" else push_at, tz)
|
||||
logger.info(
|
||||
"补跑 %s(计划 %02d:%02d,当前 %s)",
|
||||
action,
|
||||
slot.hour,
|
||||
slot.minute,
|
||||
now.astimezone(tz).strftime("%H:%M"),
|
||||
)
|
||||
|
||||
if dry_run:
|
||||
logger.info("[dry-run] 将执行 %s @ %s", action, run_at.isoformat())
|
||||
return state
|
||||
|
||||
logger.info("开始 %s(%s)", action, today_str)
|
||||
code = run_scheduled_action(action, today_str=today_str)
|
||||
if code != 0:
|
||||
logger.error("%s 失败,exit=%s", action, code)
|
||||
else:
|
||||
if action == "generate":
|
||||
state.last_generate_date = today_str
|
||||
elif action == "push":
|
||||
state.last_push_date = today_str
|
||||
state.save()
|
||||
logger.info("%s 完成", action)
|
||||
return state
|
||||
|
||||
|
||||
def main() -> int:
|
||||
dry_run = "--dry-run" in sys.argv[1:]
|
||||
once = "--once" in sys.argv[1:]
|
||||
|
||||
log_path = _setup_logging()
|
||||
tz = load_timezone()
|
||||
generate_at = parse_hhmm(schedule_generate_at())
|
||||
push_at = parse_hhmm(schedule_push_at())
|
||||
|
||||
logger.info(
|
||||
"调度器启动 tz=%s generate=%02d:%02d push=%02d:%02d log=%s",
|
||||
tz.key,
|
||||
generate_at.hour,
|
||||
generate_at.minute,
|
||||
push_at.hour,
|
||||
push_at.minute,
|
||||
log_path,
|
||||
)
|
||||
|
||||
state = SchedulerState.load()
|
||||
try:
|
||||
while True:
|
||||
state = tick_once(
|
||||
tz=tz,
|
||||
state=state,
|
||||
generate_at=generate_at,
|
||||
push_at=push_at,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
if once or dry_run:
|
||||
break
|
||||
except KeyboardInterrupt:
|
||||
logger.info("调度器已停止(KeyboardInterrupt)")
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -2,14 +2,16 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from typing import Any, Literal
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
from daily.config import env
|
||||
from daily.config import CACHE_DIR, env
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -17,6 +19,81 @@ Board = Literal["trending", "hot"]
|
||||
SKILLS_SITE = "https://www.skills.sh"
|
||||
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)"
|
||||
|
||||
FEED_URLS = [
|
||||
"https://cdn.jsdelivr.net/gh/NeverSight/skills.sh_feed@main/data/feed.json",
|
||||
"https://raw.githubusercontent.com/NeverSight/skills.sh_feed/main/data/feed.json",
|
||||
]
|
||||
FEED_CACHE_TTL = 600
|
||||
FEED_CACHE_FILE = CACHE_DIR / "feed.json"
|
||||
|
||||
_feed_cache: dict[str, Any] = {"data": None, "fetched_at": 0.0}
|
||||
|
||||
|
||||
def format_installs(n: int | float) -> str:
|
||||
if n >= 1_000_000:
|
||||
return f"{n / 1_000_000:.1f}M"
|
||||
if n >= 1_000:
|
||||
return f"{n / 1_000:.1f}K"
|
||||
return str(int(n))
|
||||
|
||||
|
||||
def _fetch_feed_json(url: str) -> dict[str, Any]:
|
||||
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
|
||||
with httpx.Client(
|
||||
timeout=httpx.Timeout(20.0, connect=10.0),
|
||||
verify=certifi.where(),
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
resp = client.get(url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
def _load_feed_disk_cache() -> dict[str, Any] | None:
|
||||
if not FEED_CACHE_FILE.exists():
|
||||
return None
|
||||
try:
|
||||
return json.loads(FEED_CACHE_FILE.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
logger.warning("读取 feed 本地缓存失败: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
def _save_feed_disk_cache(data: dict[str, Any]) -> None:
|
||||
FEED_CACHE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
FEED_CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
|
||||
def load_feed(force: bool = False) -> dict[str, Any]:
|
||||
now = time.time()
|
||||
if not force and _feed_cache["data"] and now - _feed_cache["fetched_at"] < FEED_CACHE_TTL:
|
||||
return _feed_cache["data"]
|
||||
|
||||
errors: list[str] = []
|
||||
for url in FEED_URLS:
|
||||
for attempt in range(3):
|
||||
try:
|
||||
data = _fetch_feed_json(url)
|
||||
_feed_cache["data"] = data
|
||||
_feed_cache["fetched_at"] = now
|
||||
_save_feed_disk_cache(data)
|
||||
logger.info("skills feed 已更新: %s", url)
|
||||
return data
|
||||
except Exception as exc:
|
||||
msg = f"{url} (#{attempt + 1}): {exc}"
|
||||
errors.append(msg)
|
||||
logger.debug("拉取失败 %s", msg)
|
||||
time.sleep(0.5 * (attempt + 1))
|
||||
|
||||
stale = _load_feed_disk_cache()
|
||||
if stale:
|
||||
logger.warning("网络不可用,回退到 feed 本地缓存")
|
||||
_feed_cache["data"] = stale
|
||||
_feed_cache["fetched_at"] = now
|
||||
return stale
|
||||
|
||||
raise RuntimeError(f"无法获取 skills 数据。最近错误: {errors[-1] if errors else 'unknown'}")
|
||||
|
||||
_SKILL_RE = re.compile(
|
||||
r'\{"source":"(?P<source>[^"]+)","skillId":"(?P<skill_id>[^"]+)",'
|
||||
r'"name":"(?P<name>[^"]+)","installs":(?P<installs>\d+)'
|
||||
|
||||
@@ -9,6 +9,28 @@ def skill_id(item: dict[str, Any]) -> str:
|
||||
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
|
||||
|
||||
|
||||
def source_from_skill_key(key: str) -> str:
|
||||
"""从 skill id(source/title…)还原 source:去掉最后一段 title。"""
|
||||
parts = [p for p in str(key or "").split("/") if p]
|
||||
if len(parts) >= 2:
|
||||
return "/".join(parts[:-1])
|
||||
return str(key or "").strip()
|
||||
|
||||
|
||||
def expand_skill_recent_keys(keys: set[str] | None) -> set[str]:
|
||||
"""周去重 blocklist:保留原始 key,并展开为 source。"""
|
||||
out: set[str] = set()
|
||||
for key in keys or set():
|
||||
k = str(key or "").strip()
|
||||
if not k:
|
||||
continue
|
||||
out.add(k)
|
||||
src = source_from_skill_key(k)
|
||||
if src:
|
||||
out.add(src)
|
||||
return out
|
||||
|
||||
|
||||
def format_installs(n: int | float) -> str:
|
||||
if n >= 1_000_000:
|
||||
return f"{n / 1_000_000:.1f}M"
|
||||
|
||||
@@ -15,7 +15,7 @@ def clip_text(text: str, limit: int) -> str:
|
||||
|
||||
|
||||
def trim_brief(text: str, limit: int) -> str:
|
||||
"""企微简要:控制在 limit 内,优先在句读处截断,不加省略号。"""
|
||||
"""企微简要:控制在 limit 内,优先在句读/词边界截断,不加省略号。"""
|
||||
text = _WS.sub(" ", (text or "").strip())
|
||||
if not text or limit <= 0 or len(text) <= limit:
|
||||
return text
|
||||
@@ -27,4 +27,12 @@ def trim_brief(text: str, limit: int) -> str:
|
||||
pos = text.find(sep)
|
||||
if pos != -1 and pos + 1 <= limit:
|
||||
return text[: pos + 1]
|
||||
return text[:limit].rstrip(",、;: ")
|
||||
for sep in (". ", "! ", "? ", "; "):
|
||||
pos = text.rfind(sep, 0, limit + 1)
|
||||
if pos != -1 and pos + 1 >= min(limit // 2, 20):
|
||||
return text[: pos + 1].rstrip()
|
||||
if len(text) > limit:
|
||||
space = text.rfind(" ", 0, limit + 1)
|
||||
if space >= min(limit // 2, 20):
|
||||
return text[:space].rstrip(",、;: ,.;")
|
||||
return text[:limit].rstrip(",、;: ,.;")
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
@@ -13,6 +15,7 @@ from daily.config import OUTPUT_DIR, ROOT, env_int, wecom_chunk_bytes
|
||||
from daily.wecom_split import split_wecom_messages
|
||||
|
||||
_PUSH_GAP_MS = 300
|
||||
_DATE_RE = re.compile(r"(\d{4}-\d{2}-\d{2})\.wecom\.md$")
|
||||
|
||||
|
||||
def _load_webhook_key() -> str:
|
||||
@@ -43,6 +46,31 @@ def _resolve_report_path(arg: str | None) -> Path:
|
||||
raise RuntimeError("未找到 .wecom.md 报告,请先运行 python -m daily")
|
||||
|
||||
|
||||
def _push_gate_for_report(path: Path) -> dict | None:
|
||||
match = _DATE_RE.search(path.name)
|
||||
if not match:
|
||||
return None
|
||||
data_path = path.parent / f"{match.group(1)}.data.json"
|
||||
if not data_path.exists():
|
||||
data_path = OUTPUT_DIR / f"{match.group(1)}.data.json"
|
||||
if not data_path.exists():
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(data_path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
meta = payload.get("meta") or {}
|
||||
gate = meta.get("push_gate")
|
||||
return gate if isinstance(gate, dict) else None
|
||||
|
||||
|
||||
def should_skip_push(report_path: Path) -> bool:
|
||||
gate = _push_gate_for_report(report_path)
|
||||
if not gate:
|
||||
return False
|
||||
return bool(gate.get("silent")) and not gate.get("should_push")
|
||||
|
||||
|
||||
def _post_markdown(client: httpx.Client, url: str, content: str) -> None:
|
||||
payload = {"msgtype": "markdown", "markdown": {"content": content}}
|
||||
resp = client.post(url, json=payload)
|
||||
@@ -54,6 +82,9 @@ def _post_markdown(client: httpx.Client, url: str, content: str) -> None:
|
||||
|
||||
def send_report(report_path: Path | None = None) -> None:
|
||||
path = _resolve_report_path(str(report_path) if report_path else None)
|
||||
if should_skip_push(path):
|
||||
print(f"[silent] no push gate matched for {path.name}")
|
||||
return
|
||||
if not path.exists():
|
||||
raise RuntimeError(f"报告文件不存在: {path}")
|
||||
|
||||
|
||||
273
docs/design-wecom-delta-mode.md
Normal file
273
docs/design-wecom-delta-mode.md
Normal file
@@ -0,0 +1,273 @@
|
||||
# Design: 企微早报 Delta 模式
|
||||
|
||||
Generated: 2026-07-09
|
||||
Repo: daily-robots
|
||||
Status: DRAFT
|
||||
Mode: Builder
|
||||
|
||||
## Problem Statement
|
||||
|
||||
企微早报每天推送五榜 Top 10 + 18 条新闻,内容与前几日高度重复(`find-skills`、`openclaw`、飞书集群等长期霸榜)。读者真实需求是「今天有什么新变化」,而非「再读一遍黄页」。
|
||||
|
||||
根因:
|
||||
|
||||
1. `daily-agent/SKILL.md` 要求即使较昨日无新增,仍须完整列出 Top 榜。
|
||||
2. `movement` 仅用于 opening / signals,列表区块仍全量渲染。
|
||||
3. Trending 与 Hot 独立展示,同一 skill 描述写两遍。
|
||||
4. 新闻 `DAILY_AI_NEWS_HOURS=72`,无已推送 link 去重,旧闻可连续出现。
|
||||
|
||||
## What Makes This Cool
|
||||
|
||||
把早报从「日报复印机」变成「变化通知」:只有新入榜、新新闻、编辑推荐时才占版面;榜全稳且无新新闻时静默不推。读者打开企微即知「今天值得扫一眼的是什么」。
|
||||
|
||||
## Explicit Non-Goals(已否决方案)
|
||||
|
||||
以下方案**不在本设计范围内**:
|
||||
|
||||
| 方案 | 状态 |
|
||||
|------|------|
|
||||
| 静态页 / 外链档案库 | ❌ 不做 |
|
||||
| 今日一装(每天一个 `npx skills add`) | ❌ 不做 |
|
||||
| 按星期轮换版面 | ❌ 不做 |
|
||||
| 榜首锚点(稳定日仍展示 #1) | ❌ 不做 |
|
||||
|
||||
## Premises
|
||||
|
||||
1. 重复感主要来自**列表区块全量复印**,而非 opening 里引用榜首数字。
|
||||
2. `output/*.data.json` 与 `daily/delta.py` 已具备新入榜对比能力,应上升为**列表渲染主数据源**。
|
||||
3. 企微消息仍在应用内读完,不依赖外部页面。
|
||||
4. 叙事层(opening、信号、首推、新闻)保持充实;缩短的是**榜单列表**,不是整报。
|
||||
|
||||
## Recommended Approach: Delta 模式
|
||||
|
||||
### 环境变量
|
||||
|
||||
```env
|
||||
# full = 现有行为(全量 Top 榜列表)
|
||||
# delta = 本设计(默认推荐)
|
||||
DAILY_WECOM_MODE=delta
|
||||
|
||||
# 无对比基准时(首日或缺历史 data.json)是否自动 full 一次
|
||||
DAILY_DELTA_BASELINE_FALLBACK=full # full | empty
|
||||
|
||||
# 推送闸门全不满足时是否跳过 webhook(仍写 output 文件)
|
||||
DAILY_SKIP_PUSH_WHEN_SILENT=1
|
||||
|
||||
# 强制推送(忽略静默)
|
||||
# DAILY_FORCE_PUSH=1
|
||||
|
||||
# 新闻:缩短窗口 + 去重天数
|
||||
DAILY_AI_NEWS_HOURS=24
|
||||
DAILY_NEWS_DEDUP_DAYS=7
|
||||
```
|
||||
|
||||
### 推送闸门(Push Gate)
|
||||
|
||||
满足**任一**条件则生成并推送企微早报:
|
||||
|
||||
| 条件 | 数据源 |
|
||||
|------|--------|
|
||||
| 任榜单有新入条目 | `movement.*_moves` 非空 |
|
||||
| 去重后仍有新新闻 | 国际或国内 AI 时讯 |
|
||||
| 存在 `featured_pick` | Step 0 编辑推荐 |
|
||||
| `DAILY_FORCE_PUSH=1` | 环境变量 |
|
||||
|
||||
**静默日**:以上皆不满足 → 不调用 webhook(`DAILY_SKIP_PUSH_WHEN_SILENT=1` 时)。
|
||||
仍执行 `daily generate`,写入 `output/{date}.md`、`output/{date}.wecom.md`、`output/{date}.data.json` 留档。
|
||||
|
||||
**注意**:仅新闻有新、榜单全稳时**仍推送**,但 Skills/GitHub 列表区块整块省略(不是全天静默)。
|
||||
|
||||
### 列表渲染(Delta 列表)
|
||||
|
||||
`DAILY_WECOM_MODE=delta` 时:
|
||||
|
||||
#### Skills
|
||||
|
||||
- **仅展示** `movement.skills_trending_moves` / `movement.skills_hot_moves` 中的新入榜条目。
|
||||
- **跨榜去重**:按 `skill_id`(`id` 或 `source/title`)合并;同一 skill 只出现一次,标注来源榜(如 `Trending #4 · Hot #2`)。
|
||||
- **无新入**:该榜区块**整块不出现**(不写多行「较昨日无新增」)。
|
||||
|
||||
#### GitHub
|
||||
|
||||
- 仅展示 `movement.github_trending_moves`、`github_emerging_moves`、`github_topic_moves`。
|
||||
- 无新入则区块省略。
|
||||
|
||||
#### 不包含
|
||||
|
||||
- 全量 Top N 列表
|
||||
- 榜首锚点
|
||||
- `(新入 … #n)` 括号标注(与现 `agent_workflow._strip_new_entry_notes` 一致,列表标题用 `[新入 #n]` 前缀即可)
|
||||
|
||||
### 固定骨架(不因 Delta 缩短)
|
||||
|
||||
Agent 模式(`DAILY_REPORT_MODE=agent`)下,以下区块**保持**:
|
||||
|
||||
- opening(2–3 句,首句含具体证据)
|
||||
- headline / 今日主题
|
||||
- 今日信号(3–5 条)
|
||||
- 今日首推
|
||||
- 国际 AI / 国内 AI 精选(条数仍由 `DAILY_WECOM_AI_NEWS` 等控制)
|
||||
|
||||
榜单变短;叙事与新闻不主动砍到 0。
|
||||
|
||||
### Full 模式逃生口
|
||||
|
||||
`DAILY_WECOM_MODE=full` 时行为与**当前生产一致**(`format_wecom.build_wecom_report` / `replace_wecom_skill_sections` 全量 Top N)。用于手动切回或对比测试。
|
||||
|
||||
### 首日 / 无历史基准
|
||||
|
||||
`find_previous_data(date)` 返回 `None` 时:
|
||||
|
||||
| `DAILY_DELTA_BASELINE_FALLBACK` | 行为 |
|
||||
|----------------------------------|------|
|
||||
| `full`(推荐) | 当日按 full 模式渲染列表一次;次日起 delta |
|
||||
| `empty` | 当日列表区块为空;opening 须说明「首日报,暂无对比基准」 |
|
||||
|
||||
实现时在 `generate_report` 或 `build_llm_input` 传入 `baseline_date` 供 Agent 引用。
|
||||
|
||||
## News Dedup
|
||||
|
||||
### P0(本阶段)
|
||||
|
||||
- 维护 `cache/pushed-news-links.json`(或写入 `output/` 旁 cache):最近 `DAILY_NEWS_DEDUP_DAYS` 天已出现在企微早报中的 `link` 集合。
|
||||
- `prepare_wecom_news_items` / `prepare_wecom_cn_news_items` 输出前过滤已见 link。
|
||||
- `DAILY_AI_NEWS_HOURS` 默认改为 `24`(`.env.example` 同步)。
|
||||
|
||||
### P1(可选后续)
|
||||
|
||||
- 标题归一化去重(同一事件多源报道)
|
||||
- 每 `source_name` 每日上限 N 条
|
||||
|
||||
## Agent Skill 变更
|
||||
|
||||
文件:`skills/daily-agent/SKILL.md`
|
||||
|
||||
### 删除 / 修改
|
||||
|
||||
- 删除规则:「即使某榜较昨日无新增,仍须完整列出 Top 榜条目」。
|
||||
- 删除:「禁止改用 movement 作为列表来源」(在 delta 模式下反转)。
|
||||
|
||||
### 新增
|
||||
|
||||
当 `DAILY_WECOM_MODE=delta`(或 llm_input 含 `wecom_mode: delta`):
|
||||
|
||||
1. Agent **不写** Skills Trending / Hot / GitHub 列表(仍由 Python 插入,与现流程一致)。
|
||||
2. opening / signals **可引用**榜首与 movement 摘要;禁止在 signals 重复列表已展示的同一事实。
|
||||
3. 榜全稳时,signals 聚焦新闻与首推,不必编造榜单变化。
|
||||
|
||||
当 `wecom_mode: full` 时保持现有 SKILL 规则。
|
||||
|
||||
## Python 模块变更
|
||||
|
||||
| 模块 | 变更 |
|
||||
|------|------|
|
||||
| `daily/config.py` | `wecom_mode()`, `news_dedup_days()`, `skip_push_when_silent()`, `delta_baseline_fallback()` |
|
||||
| `daily/delta.py` | 可选:`merge_skill_moves_for_wecom(trending_moves, hot_moves)` 跨榜去重 |
|
||||
| `daily/format_wecom.py` | `build_skills_delta_section()`, `build_github_delta_section()`;`replace_wecom_skill_sections` 支持 delta |
|
||||
| `daily/news/fetch.py` | `filter_pushed_news()` + cache 读写 |
|
||||
| `daily/generate.py` | 推送闸门;baseline fallback;静默 skip push |
|
||||
| `daily/report_data.py` | `llm_input` 增加 `wecom_mode`, `push_gate` 摘要 |
|
||||
| `daily/agent_workflow.py` | 无逻辑变更;依赖 Python 插入 delta 列表 |
|
||||
| `.env.example` | 新 env 文档 |
|
||||
|
||||
## 企微消息示例
|
||||
|
||||
### 有变化日
|
||||
|
||||
```markdown
|
||||
📰 **早报 · 2026-07-10**
|
||||
|
||||
[opening:今天最大变化,含数字/条目名]
|
||||
|
||||
🎯 **{headline}**
|
||||
|
||||
💡 **今日信号**
|
||||
> ...
|
||||
|
||||
📦 **今日首推**
|
||||
[...]
|
||||
|
||||
🌍 **国际 AI · 精选 N**
|
||||
...
|
||||
|
||||
📈 **Skills Trending 变化**
|
||||
1. [新入 #4] [**xxx**](...) · ...
|
||||
描述一行
|
||||
|
||||
🔥 **Skills Hot 变化**
|
||||
1. [新入 #2] [**yyy**](...) · ...
|
||||
|
||||
🐙 **GitHub Trending 变化**
|
||||
1. [新入 #4] [owner/repo](...) · ...
|
||||
```
|
||||
|
||||
### 仅新闻有新(榜稳)
|
||||
|
||||
- 无 📈/🔥/🐙 区块
|
||||
- opening 可一句:「榜单较昨日 Top15 无新入;以下为今日 AI 时讯。」
|
||||
|
||||
### 静默日
|
||||
|
||||
- 不推送企微
|
||||
- `output/` 仍落盘;日志:`[silent] no push gate matched for 2026-07-10`
|
||||
|
||||
## Approaches Considered
|
||||
|
||||
### Approach A: 配置瘦身(缩 Top N、24h 新闻)
|
||||
|
||||
- Effort: S | Risk: Low
|
||||
- 只减篇幅,榜头仍天天重复;未解决根因。
|
||||
|
||||
### Approach B: Delta 列表 + 推送闸门 + 新闻去重(本设计)
|
||||
|
||||
- Effort: M | Risk: Med
|
||||
- 复用 `delta.py`;改 format + skill + push 逻辑。
|
||||
|
||||
### Approach C: 仅改 Agent 文案
|
||||
|
||||
- Effort: S | Risk: High
|
||||
- Python 仍插入全量列表,规则冲突,不可持续。
|
||||
|
||||
**Recommendation: B** — 数据层与展示层一致,静默日与跨榜去重可测。
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. 连续 3 天对比 `output/*.data.json`:企微列表区块**重复 skill_id 占比**显著下降。
|
||||
2. 榜全稳且无新新闻日:`DAILY_SKIP_PUSH_WHEN_SILENT=1` 时不发 webhook。
|
||||
3. Trending/Hot 同一 skill 在列表中**最多出现 1 次**。
|
||||
4. `DAILY_WECOM_MODE=full` 与现网行为一致(回归用)。
|
||||
5. 首日 `DAILY_DELTA_BASELINE_FALLBACK=full` 不产生空列表投诉。
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. 静默日是否需要在企微发一行「今日无更新」?当前设计:**不发**。
|
||||
2. 新闻去重 cache 是否纳入 git?建议:**否**,放 `cache/`(已在 `.gitignore`)。
|
||||
3. Classic 模式(非 agent)是否同步 delta?建议:**是**,同一 `format_wecom` 路径。
|
||||
|
||||
## Implementation Tasks
|
||||
|
||||
| ID | Priority | Task | Files |
|
||||
|----|----------|------|-------|
|
||||
| T1 | P1 | 新增 config helpers + `.env.example` | `daily/config.py`, `.env.example` |
|
||||
| T2 | P1 | 新闻 link 去重 cache | `daily/news/fetch.py`, `daily/config.py` |
|
||||
| T3 | P1 | Delta 列表渲染 + 跨榜去重 | `daily/format_wecom.py`, `daily/delta.py` |
|
||||
| T4 | P1 | 推送闸门 + 静默 skip push | `daily/generate.py`, `daily/webhook.py` |
|
||||
| T5 | P1 | baseline fallback full 一次 | `daily/generate.py` |
|
||||
| T6 | P1 | 更新 `daily-agent/SKILL.md` | `skills/daily-agent/SKILL.md` |
|
||||
| T7 | P2 | `llm_input` 传 `wecom_mode` / push 摘要 | `daily/report_data.py` |
|
||||
| T8 | P2 | 单元测试:跨榜去重、推送闸门、新闻去重 | `tests/test_wecom_delta.py` |
|
||||
|
||||
## Test Plan
|
||||
|
||||
- [ ] 有 `2026-07-09.data.json` 时生成 `2026-07-10`:列表仅含新入项
|
||||
- [ ] 人造「全稳 + 无新新闻」:不 push
|
||||
- [ ] 人造「全稳 + 有新新闻」:push,无 Skills/GitHub 块
|
||||
- [ ] `DAILY_WECOM_MODE=full` 输出与改前 `2026-07-09.wecom.md` 结构一致
|
||||
- [ ] 无 baseline + `DAILY_DELTA_BASELINE_FALLBACK=full`:首日全量列表
|
||||
- [ ] 同一 skill 在 Trending/Hot moves 均出现:列表只 1 条
|
||||
|
||||
## What I Noticed
|
||||
|
||||
- 重复感是**产品形态**问题,不是 Agent 文笔问题。
|
||||
- 明确否决静态页、今日一装、轮换、锚点后,方案边界清晰,实现可分期。
|
||||
- 推送闸门必须**把新闻算进去**,否则静默日会被新闻绕过。
|
||||
1162
docs/superpowers/plans/2026-07-09-wecom-delta-mode.md
Normal file
1162
docs/superpowers/plans/2026-07-09-wecom-delta-mode.md
Normal file
File diff suppressed because it is too large
Load Diff
604
docs/superpowers/plans/2026-07-14-wecom-diversity-dedup.md
Normal file
604
docs/superpowers/plans/2026-07-14-wecom-diversity-dedup.md
Normal file
@@ -0,0 +1,604 @@
|
||||
# 企微早报多样性与去重 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 实现企微早报硬去重多样性:五榜周去重(深池补满)、首推与昨日相同则改推(月去重)、叙事轴代码互斥、取消新闻「放宽窗口」凑数;且 `movement_baseline` 与 `wecom_shown_keys` 严格分离。
|
||||
|
||||
**Architecture:** 在 `daily generate` 管线加代码选择器:`board_select` 为 full/delta 唯一列表主人;周历史只读写 `data.wecom_shown_keys`(post-render);`movement_baseline` 仍为 raw Top compare;`featured_resolve` 先定人再 research;`pick_narrative_axis` 代码选轴注入 Agent Step1;新闻关 backfill + 剥「放宽」前缀。
|
||||
|
||||
**Tech Stack:** Python 3.13+、现有 `unittest`/`pytest`、`daily/delta.py` / `format_wecom.py` / `featured_pick.py` / `news/fetch.py`、`output/*.data.json`
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md`(Status: APPROVED)
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- `movement_baseline` **禁止**被展示历史覆写;周去重只读 `wecom_shown_keys`
|
||||
- full/delta **唯一列表主人** = `board_select`(delta 的 pad 共用同一套 shown 历史)
|
||||
- `DAILY_BOARD_DEDUP_DAYS` 默认 `7`;与 pad lookback 对齐且数据源同一
|
||||
- `DAILY_FEATURED_DEDUP_DAYS` 默认 `30`
|
||||
- `DAILY_NARRATIVE_AXIS_DAYS` 默认 `3`;轴枚举固定 7 个(见 Task 5)
|
||||
- `DAILY_NEWS_BACKFILL` 默认 `0`(禁止旧闻凑数)
|
||||
- research 补新闻年龄上限 = `DAILY_AI_NEWS_HOURS`(不得变相 48h 放宽)
|
||||
- 不做:语义「同一类」、同日 Trending↔Hot 互斥、`FEATURED_FORCE`、关键短语硬匹配
|
||||
- Commit 信息正文用中文(若本任务含 Commit 步)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `daily/config.py` | 新 env:`board_dedup_days`、`board_pool_size`、`featured_dedup_days`、`theme_ban_days`、`narrative_axis_days`、`news_backfill_enabled` |
|
||||
| `daily/board_history.py` | **新建** — `load_recent_shown_keys` / `extract_shown_keys` / `attach_wecom_shown_keys`(读写 `data.wecom_shown_keys`) |
|
||||
| `daily/board_select.py` | **新建** — `board_select(...)` 周过滤+深池 |
|
||||
| `daily/delta.py` | `load_recent_board_keys` 改为委托 `load_recent_shown_keys`(保留函数名兼容);**不**改 `build_movement_baseline` |
|
||||
| `daily/format_wecom.py` | pad 使用 shown keys;可选返回最终展示 items 供写回 |
|
||||
| `daily/featured_pick.py` | `featured_identity_key`、`featured_resolve`、先定人再 research |
|
||||
| `daily/narrative_axis.py` | **新建** — `NARRATIVE_AXES`、`pick_narrative_axis`、`load_recent_axes` |
|
||||
| `daily/news/fetch.py` | `_apply_pushed_dedup_with_backfill` 尊重 `news_backfill_enabled()`;默认不塞回 |
|
||||
| `daily/news/research.py` + `skills/daily-ai-news-research/SKILL.md` | 禁放宽文案;补入不超时窗 |
|
||||
| `daily/text_utils.py` 或 `daily/news/sanitize.py` | `strip_news_relax_prefix(desc)` |
|
||||
| `daily/agent_workflow.py` | Step1 注入 axis + 近 7 日 theme 软禁;强制覆写冲突轴 |
|
||||
| `daily/generate.py` | 串联:select → featured → editorial → render → persist shown/key/axis |
|
||||
| `daily/report_data.py` | data.json 可携 `wecom_shown_keys` / `featured_pick_key` / `narrative_axis`(写回可由 generate 合并) |
|
||||
| `.env.example` | 文档化新变量 |
|
||||
| `skills/daily-agent/SKILL.md` | `narrative_axis` 必填且等于输入指定轴 |
|
||||
| `tests/test_board_select.py` | **新建** |
|
||||
| `tests/test_board_history.py` | **新建** |
|
||||
| `tests/test_featured_resolve.py` | **新建** |
|
||||
| `tests/test_narrative_axis.py` | **新建** |
|
||||
| `tests/test_news_relax.py` | **新建** |
|
||||
| `tests/test_wecom_delta.py` | 回归:pad 不读 movement 当展示史 |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Config + shown-keys 历史层
|
||||
|
||||
**Files:**
|
||||
- Modify: `daily/config.py`(文件末尾追加)
|
||||
- Create: `daily/board_history.py`
|
||||
- Modify: `daily/delta.py`(`load_recent_board_keys` 改委托)
|
||||
- Test: `tests/test_board_history.py`
|
||||
- Modify: `.env.example`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `OUTPUT_DIR`、现有 `delta.skill_id` / repo key 约定
|
||||
- Produces:
|
||||
- `board_dedup_days() -> int`(默认 7)
|
||||
- `board_pool_size() -> int`(默认 `max(50, env WECOM_SKILL_POOL)`)
|
||||
- `featured_dedup_days() -> int`(默认 30)
|
||||
- `theme_ban_days() -> int`(默认 7)
|
||||
- `narrative_axis_days() -> int`(默认 3)
|
||||
- `news_backfill_enabled() -> bool`(默认 False;env `DAILY_NEWS_BACKFILL`)
|
||||
- `extract_shown_keys(board: str, items: list[dict]) -> list[str]`
|
||||
- `load_recent_shown_keys(date_str: str, *, lookback_days: int | None = None) -> dict[str, set[str]]`
|
||||
- `merge_wecom_shown_into_data(data: dict, shown: dict[str, list[str]]) -> dict`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_board_history.py
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from daily.board_history import extract_shown_keys, load_recent_shown_keys, merge_wecom_shown_into_data
|
||||
from daily.config import board_dedup_days, news_backfill_enabled
|
||||
|
||||
|
||||
class ConfigDiversityTests(unittest.TestCase):
|
||||
def test_board_dedup_days_default(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertEqual(board_dedup_days(), 7)
|
||||
|
||||
def test_news_backfill_default_off(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertFalse(news_backfill_enabled())
|
||||
|
||||
|
||||
class ShownKeysTests(unittest.TestCase):
|
||||
def test_extract_github_repo_keys(self):
|
||||
items = [{"repo": "a/b"}, {"repo": "c/d"}]
|
||||
self.assertEqual(extract_shown_keys("github_trending", items), ["a/b", "c/d"])
|
||||
|
||||
def test_load_recent_reads_wecom_shown_not_baseline(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out = Path(tmp)
|
||||
# 前日:shown 只有 x/y;baseline raw 含 a/b —— 周去重只能看到 x/y
|
||||
payload = {
|
||||
"data": {
|
||||
"date": "2026-07-13",
|
||||
"movement_baseline": {
|
||||
"github_trending": [{"repo": "a/b"}, {"repo": "x/y"}],
|
||||
},
|
||||
"wecom_shown_keys": {"github_trending": ["x/y"]},
|
||||
}
|
||||
}
|
||||
(out / "2026-07-13.data.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
with patch("daily.board_history.OUTPUT_DIR", out):
|
||||
keys = load_recent_shown_keys("2026-07-14", lookback_days=7)
|
||||
self.assertEqual(keys["github_trending"], {"x/y"})
|
||||
self.assertNotIn("a/b", keys["github_trending"])
|
||||
|
||||
def test_merge_shown_does_not_touch_baseline(self):
|
||||
data = {
|
||||
"movement_baseline": {"github_trending": [{"repo": "raw/one"}]},
|
||||
}
|
||||
merged = merge_wecom_shown_into_data(
|
||||
data, {"github_trending": ["shown/one"]}
|
||||
)
|
||||
self.assertEqual(
|
||||
merged["movement_baseline"]["github_trending"][0]["repo"], "raw/one"
|
||||
)
|
||||
self.assertEqual(merged["wecom_shown_keys"]["github_trending"], ["shown/one"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_board_history.py -v`
|
||||
Expected: FAIL(模块/函数不存在)
|
||||
|
||||
- [ ] **Step 3: Implement config + board_history + delta 委托**
|
||||
|
||||
在 `daily/config.py` 追加:
|
||||
|
||||
```python
|
||||
def board_dedup_days() -> int:
|
||||
return max(1, env_int("DAILY_BOARD_DEDUP_DAYS", 7))
|
||||
|
||||
def board_pool_size() -> int:
|
||||
fallback = env_int("DAILY_WECOM_SKILL_POOL", 50)
|
||||
return max(1, env_int("DAILY_BOARD_POOL_SIZE", max(50, fallback)))
|
||||
|
||||
def featured_dedup_days() -> int:
|
||||
return max(1, env_int("DAILY_FEATURED_DEDUP_DAYS", 30))
|
||||
|
||||
def theme_ban_days() -> int:
|
||||
return max(1, env_int("DAILY_THEME_BAN_DAYS", 7))
|
||||
|
||||
def narrative_axis_days() -> int:
|
||||
return max(1, env_int("DAILY_NARRATIVE_AXIS_DAYS", 3))
|
||||
|
||||
def news_backfill_enabled() -> bool:
|
||||
return env_bool("DAILY_NEWS_BACKFILL", False)
|
||||
```
|
||||
|
||||
新建 `daily/board_history.py`:实现 `BOARD_KEYS` 与 `delta.RECENT_BOARD_KEYS` 同五榜;Skills 用 `delta.skill_id`;GitHub 用 `repo`;`load_recent_shown_keys` **只**读各日 `data.wecom_shown_keys`,缺省空集,读写失败打 log 后当空集。
|
||||
|
||||
修改 `daily/delta.py` 的 `load_recent_board_keys`:改为
|
||||
|
||||
```python
|
||||
def load_recent_board_keys(date_str: str, *, lookback_days: int | None = None) -> dict[str, set[str]]:
|
||||
from daily.board_history import load_recent_shown_keys
|
||||
from daily.config import board_dedup_days
|
||||
days = lookback_days if lookback_days is not None else board_dedup_days()
|
||||
return load_recent_shown_keys(date_str, lookback_days=days)
|
||||
```
|
||||
|
||||
删除(或不再走)原「从 movement_baseline 抽 keys」逻辑,避免 pad 继续把 raw Top 当展示史。
|
||||
|
||||
`.env.example` 追加注释块:
|
||||
|
||||
```env
|
||||
# 多样性 / 去重(见 docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md)
|
||||
# DAILY_BOARD_DEDUP_DAYS=7
|
||||
# DAILY_BOARD_POOL_SIZE=50
|
||||
# DAILY_FEATURED_DEDUP_DAYS=30
|
||||
# DAILY_THEME_BAN_DAYS=7
|
||||
# DAILY_NARRATIVE_AXIS_DAYS=3
|
||||
DAILY_NEWS_BACKFILL=0
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests**
|
||||
|
||||
Run: `pytest tests/test_board_history.py tests/test_wecom_delta.py -v`
|
||||
Expected: `test_board_history` PASS;既有 delta 测试若依赖「baseline 即 recent」行为,按 Task 1 语义改断言为 shown_keys(本 Task 内修回归,勿留红)。
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add daily/config.py daily/board_history.py daily/delta.py .env.example tests/test_board_history.py tests/test_wecom_delta.py
|
||||
git commit -m "feat: 拆分 wecom_shown_keys 与 movement_baseline 历史层"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: `board_select` 周去重 + 深池
|
||||
|
||||
**Files:**
|
||||
- Create: `daily/board_select.py`
|
||||
- Test: `tests/test_board_select.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `extract_shown_keys` / `skill_id`;`group_skills_by_source`(Skills 板)
|
||||
- Produces:
|
||||
- `board_select(*, board: str, items: list[dict], recent_keys: set[str], limit: int, pool_size: int, kind: Literal["skill","github"]) -> list[dict]`
|
||||
- 日志短榜:`board_short:{board}:{n}`(`logging.getLogger(__name__).info`)
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_board_select.py
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from daily.board_select import board_select
|
||||
|
||||
|
||||
def _gh(repo: str) -> dict:
|
||||
return {"repo": repo, "description": repo}
|
||||
|
||||
|
||||
class BoardSelectTests(unittest.TestCase):
|
||||
def test_filters_recent_and_keeps_order(self):
|
||||
pool = [_gh(f"o/r{i}") for i in range(20)]
|
||||
recent = {"o/r0", "o/r1", "o/r2"}
|
||||
out = board_select(
|
||||
board="github_trending",
|
||||
items=pool,
|
||||
recent_keys=recent,
|
||||
limit=5,
|
||||
pool_size=20,
|
||||
kind="github",
|
||||
)
|
||||
keys = [x["repo"] for x in out]
|
||||
self.assertEqual(keys, ["o/r3", "o/r4", "o/r5", "o/r6", "o/r7"])
|
||||
|
||||
def test_deep_pool_fills_after_filter(self):
|
||||
pool = [_gh(f"o/r{i}") for i in range(8)]
|
||||
recent = {f"o/r{i}" for i in range(6)} # 前 6 全封
|
||||
out = board_select(
|
||||
board="github_emerging",
|
||||
items=pool,
|
||||
recent_keys=recent,
|
||||
limit=5,
|
||||
pool_size=8,
|
||||
kind="github",
|
||||
)
|
||||
self.assertEqual([x["repo"] for x in out], ["o/r6", "o/r7"]) # 短榜
|
||||
|
||||
def test_skill_uses_skill_id(self):
|
||||
items = [
|
||||
{"id": "a/b/s1", "source": "a/b", "title": "s1", "installs": 10},
|
||||
{"id": "c/d/s2", "source": "c/d", "title": "s2", "installs": 9},
|
||||
]
|
||||
out = board_select(
|
||||
board="skills_trending",
|
||||
items=items,
|
||||
recent_keys={"a/b/s1"},
|
||||
limit=10,
|
||||
pool_size=50,
|
||||
kind="skill",
|
||||
)
|
||||
self.assertEqual([x["id"] for x in out], ["c/d/s2"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_board_select.py -v`
|
||||
Expected: FAIL
|
||||
|
||||
- [ ] **Step 3: Implement `board_select`**
|
||||
|
||||
```python
|
||||
# daily/board_select.py — 核心逻辑示意
|
||||
def board_select(*, board, items, recent_keys, limit, pool_size, kind):
|
||||
if kind == "skill":
|
||||
from daily.skills_group import group_skills_by_source
|
||||
pool = group_skills_by_source(items, limit=pool_size, pool_size=pool_size)
|
||||
def key_fn(x): return skill_id(x)
|
||||
else:
|
||||
pool = items[: max(pool_size, limit)]
|
||||
def key_fn(x): return str(x.get("repo") or "")
|
||||
out = []
|
||||
for item in pool:
|
||||
k = key_fn(item)
|
||||
if not k or k in recent_keys:
|
||||
continue
|
||||
out.append(item)
|
||||
if len(out) >= limit:
|
||||
break
|
||||
if len(out) < limit:
|
||||
logger.info("board_short:%s:%s", board, len(out))
|
||||
return out
|
||||
```
|
||||
|
||||
Skills:输入可为未 group 的 raw;函数内 group。GitHub:输入为 repo 列表。
|
||||
|
||||
- [ ] **Step 4: Run tests — expect PASS**
|
||||
|
||||
Run: `pytest tests/test_board_select.py -v`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add daily/board_select.py tests/test_board_select.py
|
||||
git commit -m "feat: 实现 board_select 周去重与深池补满"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: 接入 generate / format_wecom(唯一列表主人 + post-render 写回)
|
||||
|
||||
**Files:**
|
||||
- Modify: `daily/generate.py`(选榜、传入 pad、渲染后 merge shown)
|
||||
- Modify: `daily/format_wecom.py`(delta pad 已通过改写后的 `load_recent_board_keys` 读 shown;确保传入的 `*_pad` 池已是 `board_select` 深池结果)
|
||||
- Modify: `daily/report_data.py`(可选:llm_input 切片改为 board_select 后列表,避免 Agent 看见未去重 Top)
|
||||
- Test: `tests/test_board_history.py` 增补「shown ≠ baseline 推导」集成断言;`tests/test_wecom_delta.py` pad 用例
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task1–2
|
||||
- Produces: 每次成功 generate 后 `output/{date}.data.json` 含 `data.wecom_shown_keys`
|
||||
|
||||
- [ ] **Step 1: Write / extend failing integration test**
|
||||
|
||||
```python
|
||||
def test_persist_shown_keys_differs_from_baseline_keys(self):
|
||||
# 构造:raw trending 前 3 名本周已 shown;board_select 选出 3..;
|
||||
# movement_baseline 仍含 0..compare_depth
|
||||
# 断言 data["wecom_shown_keys"]["github_trending"] 与 baseline repos 集合不等
|
||||
...
|
||||
```
|
||||
|
||||
(可用临时 `OUTPUT_DIR` + 调用抽取出的 `persist` 辅助,或测 `merge_wecom_shown_into_data` + `board_select` 组合。)
|
||||
|
||||
- [ ] **Step 2: Run — expect FAIL(generate 尚未写 shown)**
|
||||
|
||||
- [ ] **Step 3: Wire generate**
|
||||
|
||||
在 `generate_report` 中,在组装 wecom 榜之前:
|
||||
|
||||
1. `recent = load_recent_shown_keys(date_str)`
|
||||
2. 对五榜分别 `board_select(...)` 得到 `selected_*`(limit=wecom_*,pool=`board_pool_size()`)
|
||||
3. full:渲染用 `selected_*`
|
||||
4. delta:`trending_pad`/`github_*_pad` = 同规则更大 pool 的 select 结果(或 raw 深池再 select);`replace_wecom_board_sections(..., pad=True)` 内部 recent 已是 shown
|
||||
5. 渲染后根据**最终写入正文的 items**(full=selected;delta=函数返回或并行计算最终列表)调用 `extract_shown_keys`,`merge_wecom_shown_into_data`,写回 data.json(在现有 `save_json` 路径合并字段)
|
||||
|
||||
注意:`movement_baseline` 仍用 **raw** compare 切片构建(`report_data.build_llm_input` 现逻辑保留)。
|
||||
|
||||
若 `build_llm_input` 当前把未过滤 Top 塞进 Agent:改为传入 `selected_*`(或另字段 `boards_for_wecom`),避免 opening 引用已周封杀的榜首。
|
||||
|
||||
辅助:在 `format_wecom` 增加 `resolve_wecom_board_items(...)` 返回最终 items dict,供写回与 featured 池 A 共用,避免正文与 history 分叉。
|
||||
|
||||
- [ ] **Step 4: Run tests**
|
||||
|
||||
Run: `pytest tests/test_board_select.py tests/test_board_history.py tests/test_wecom_delta.py -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add daily/generate.py daily/format_wecom.py daily/report_data.py tests/
|
||||
git commit -m "feat: generate 以 board_select 为唯一列表主人并写回 shown keys"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: `featured_resolve`(先定人再 research)
|
||||
|
||||
**Files:**
|
||||
- Modify: `daily/featured_pick.py`
|
||||
- Modify: `daily/generate.py`(调用顺序)
|
||||
- Test: `tests/test_featured_resolve.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: 最终展示 items(池 A)、raw 深池(池 B)、近 30 日 `featured_pick_key`
|
||||
- Produces:
|
||||
- `featured_identity_key(featured: dict) -> str`(`type==skill` → id;github → repo;兜底 url path)
|
||||
- `load_recent_featured_keys(date_str, days) -> set[str]`
|
||||
- `featured_resolve(*, date_str, candidate: dict | None, pool_a: list[dict], pool_b: list[dict], rng: random.Random | None) -> tuple[dict | None, str | None]`
|
||||
返回 `(resolved_seed_or_featured_stub, identity_key)`;**不含**完整 why(why 由后续 research 写)
|
||||
- 改 `apply_featured_pick`:先 resolve 身份(若与昨日冲突则换候选写入 query),再 `research_featured_pick`
|
||||
|
||||
- [ ] **Step 1: Failing tests**
|
||||
|
||||
```python
|
||||
# tests/test_featured_resolve.py
|
||||
def test_same_as_yesterday_picks_from_pool_a(self):
|
||||
yesterday_key = "headroomlabs-ai/headroom"
|
||||
pool_a = [
|
||||
{"repo": "headroomlabs-ai/headroom", "board": "github_topic"},
|
||||
{"repo": "ollama/ollama", "board": "github_trending"},
|
||||
]
|
||||
rng = random.Random(0)
|
||||
resolved, key = featured_resolve(
|
||||
date_str="2026-07-14",
|
||||
candidate={"type": "github", "url": "https://github.com/headroomlabs-ai/headroom", "title": "headroom"},
|
||||
pool_a=pool_a,
|
||||
pool_b=[],
|
||||
recent_featured={yesterday_key},
|
||||
yesterday_key=yesterday_key,
|
||||
rng=rng,
|
||||
)
|
||||
self.assertNotEqual(key, yesterday_key)
|
||||
self.assertEqual(key, "ollama/ollama")
|
||||
|
||||
def test_pool_a_before_pool_b(self):
|
||||
...
|
||||
|
||||
def test_exhausted_keeps_original(self):
|
||||
...
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run — FAIL**
|
||||
|
||||
- [ ] **Step 3: Implement**
|
||||
|
||||
`featured_resolve`:若无 candidate 或与 `yesterday_key` 不同 → 原样返回。
|
||||
冲突时:过滤 `recent_featured | {yesterday_key}`,先从 pool_a 建可选项(每项抽 identity),`rng.choice`;空则 pool_b;仍空 log `featured_fallback_exhausted` 并保留原 candidate。
|
||||
|
||||
`apply_featured_pick` / generate 流程:
|
||||
|
||||
1. 解析 env 得到初始 query/candidate
|
||||
2. `featured_resolve`(此时池 A 已是 board_select 结果)
|
||||
3. 若换人:用新 repo/skill 构造 config,再 `research_featured_pick`
|
||||
4. 写入 `llm_input["featured_pick"]` 与之后 data.`featured_pick_key`
|
||||
|
||||
随机默认:`random.Random(int(hashlib.sha256(f"{date_str}:featured".encode()).hexdigest()[:16], 16))`
|
||||
|
||||
- [ ] **Step 4: pytest PASS**
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add daily/featured_pick.py daily/generate.py tests/test_featured_resolve.py
|
||||
git commit -m "feat: 首推与昨日冲突时改推并保证一月不重复"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: `narrative_axis` 硬互斥 + Step1 软禁 theme
|
||||
|
||||
**Files:**
|
||||
- Create: `daily/narrative_axis.py`
|
||||
- Modify: `daily/agent_workflow.py`(`analyze_trends`)
|
||||
- Modify: `skills/daily-agent/SKILL.md`
|
||||
- Modify: `daily/generate.py`(落盘 `narrative_axis`;注入 llm_input)
|
||||
- Test: `tests/test_narrative_axis.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
- `NARRATIVE_AXES: tuple[str, ...] = ("政策监管", "模型发布", "工具链/Agent", "芯片算力", "开源生态", "应用落地", "安全/诉讼")`
|
||||
- `pick_narrative_axis(used: set[str], *, rng: random.Random | None = None) -> str`
|
||||
- `load_recent_axes(date_str, days) -> list[str]`(近 N 日 data.`narrative_axis`)
|
||||
- `enforce_narrative_axis(trends: dict, axis: str) -> dict`(强制 trends["narrative_axis"]=axis)
|
||||
|
||||
- [ ] **Step 1: Failing tests**
|
||||
|
||||
```python
|
||||
def test_pick_excludes_used(self):
|
||||
used = {"政策监管", "模型发布", "工具链/Agent"}
|
||||
for _ in range(20):
|
||||
axis = pick_narrative_axis(used, rng=random.Random(1))
|
||||
self.assertNotIn(axis, used)
|
||||
|
||||
def test_enforce_overwrites_llm(self):
|
||||
trends = {"narrative_axis": "开源生态", "opening": "..."}
|
||||
out = enforce_narrative_axis(trends, "芯片算力")
|
||||
self.assertEqual(out["narrative_axis"], "芯片算力")
|
||||
```
|
||||
|
||||
- [ ] **Step 2: FAIL → Step 3 implement**
|
||||
|
||||
`analyze_trends`:计算 `axis = pick_narrative_axis(set(load_recent_axes(...)))`;把 `required_narrative_axis` 与近 `theme_ban_days` 的 theme/opening 摘要列表注入 system prompt;要求 JSON 含 `narrative_axis` 且必须等于 required。解析后 `enforce_narrative_axis`。
|
||||
generate 将 axis 写入 data.json。
|
||||
|
||||
SKILL.md Step1 schema 增加 `narrative_axis` 字段说明。
|
||||
|
||||
- [ ] **Step 4–5: pytest + commit**
|
||||
|
||||
```bash
|
||||
git commit -m "feat: 代码选定叙事轴并注入 Agent 开场约束"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: 取消新闻「放宽窗口」
|
||||
|
||||
**Files:**
|
||||
- Modify: `daily/news/fetch.py`(`_apply_pushed_dedup_with_backfill`)
|
||||
- Modify: `daily/news/research.py`(确认只 filter_unpushed;不足不拉超窗)
|
||||
- Create: `daily/news/sanitize.py`(或放入 `text_utils`)— `strip_relax_window_prefix(text: str) -> str`
|
||||
- Modify: `skills/daily-ai-news-research/SKILL.md`(删除「放宽至 48h 并注明」;改为不足则少返回、禁止标注)
|
||||
- Modify: `daily/generate.py` / news finalize(对 desc_short 剥前缀)
|
||||
- Test: `tests/test_news_relax.py`;扩展 `tests/test_news_fetch_window.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `news_backfill_enabled() == False` 时:`_apply_pushed_dedup_with_backfill` 等价于只返回 `filter_unpushed_items(...)[:limit]`,**不**再从 `picked` 塞回
|
||||
- `strip_relax_window_prefix`:去掉开头的 `放宽窗口[::]?` / `放宽至[^::]*[::]`
|
||||
|
||||
- [ ] **Step 1: Failing tests**
|
||||
|
||||
```python
|
||||
def test_backfill_disabled_does_not_reinsert_pushed(self):
|
||||
# fresh 不足 limit;picked 含已推;BACKFILL=0 → 结果不含已推 link
|
||||
...
|
||||
|
||||
def test_strip_relax_prefix(self):
|
||||
self.assertEqual(
|
||||
strip_relax_window_prefix("放宽窗口:苹果起诉 OpenAI"),
|
||||
"苹果起诉 OpenAI",
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 2–4: 实现并跑 `pytest tests/test_news_relax.py tests/test_news_fetch_window.py -v`**
|
||||
|
||||
Research 路径:SKUILL 改完后,代码侧对 items 统一 `strip`;不足时 log `news_short:{n}`,接受短列表。
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git commit -m "fix: 关闭新闻放宽凑数并剥离放宽窗口文案"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: 端到端回归与文档对齐
|
||||
|
||||
**Files:**
|
||||
- Modify: 如有遗漏的 `.env.example` / SKILL
|
||||
- Test: 全量相关测试
|
||||
|
||||
- [ ] **Step 1: 跑全套**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
pytest tests/test_board_history.py tests/test_board_select.py tests/test_featured_resolve.py tests/test_narrative_axis.py tests/test_news_relax.py tests/test_news_fetch_window.py tests/test_wecom_delta.py tests/test_featured_pick.py -v
|
||||
```
|
||||
|
||||
Expected: 全部 PASS
|
||||
|
||||
- [ ] **Step 2: Spec 对照清单(人工)**
|
||||
|
||||
| Spec 要求 | 任务 |
|
||||
|-----------|------|
|
||||
| wecom_shown_keys ≠ movement_baseline | T1, T3 |
|
||||
| board_select 唯一主人 + delta pad 共用 shown | T2, T3 |
|
||||
| 首推月去重 A→B、先定人再 why | T4 |
|
||||
| narrative_axis 硬保证 | T5 |
|
||||
| 禁放宽 backfill + 剥前缀 + hours 窗 | T6 |
|
||||
| 成功标准可测 | 各测覆盖 |
|
||||
|
||||
- [ ] **Step 3: Commit(若有收尾文档)**
|
||||
|
||||
```bash
|
||||
git add -u
|
||||
git commit -m "test: 多样性去重全链路回归通过"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Spec Coverage Self-Review
|
||||
|
||||
| Spec 节 | 计划任务 |
|
||||
|---------|----------|
|
||||
| 1.1 双基准分离 | T1 |
|
||||
| 1.2 唯一列表主人 | T2–T3 |
|
||||
| 1.3 真相表 | T1, T3–T5 落盘字段 |
|
||||
| 2.1 board_select | T2 |
|
||||
| 2.2 featured_resolve | T4 |
|
||||
| 3.1 narrative_axis + 软 theme | T5 |
|
||||
| 3.2 取消放宽 | T6 |
|
||||
| 4.x 配置/降级/测试 | T1–T7 |
|
||||
|
||||
无 TBD;Commit 信息均为中文描述体。
|
||||
|
||||
## Execution Handoff
|
||||
|
||||
Plan complete and saved to `docs/superpowers/plans/2026-07-14-wecom-diversity-dedup.md`.
|
||||
|
||||
**两种执行方式:**
|
||||
|
||||
1. **Subagent-Driven(推荐)** — 每任务新开子代理,任务间审查
|
||||
2. **Inline Execution** — 本会话按 `executing-plans` 连续做完,设检查点
|
||||
|
||||
要哪个?
|
||||
@@ -0,0 +1,271 @@
|
||||
# Design: 企微早报多样性与去重
|
||||
|
||||
Generated: 2026-07-14
|
||||
Repo: daily-robots
|
||||
Status: APPROVED
|
||||
Mode: Builder
|
||||
Related: `docs/design-wecom-delta-mode.md`(Delta 列表模式)
|
||||
Revision: office-hours A —— 拆分展示历史、单一列表主人、历史真相表(2026-07-14)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
近两日企微早报(如 2026-07-13 / 07-14)骨架相同,且:
|
||||
|
||||
- **今日首推**连续两天同为 `headroom`
|
||||
- **开场主题**同属「上下文压缩 + 视频/Skills」腔调
|
||||
- **AI 时讯**出现「放宽窗口」旧闻凑数标注
|
||||
- **Skills / GitHub 各榜**(尤其新兴榜)周内大量重复展示
|
||||
|
||||
读者需要「今天新信息」,而不是换日期的复印机。
|
||||
|
||||
## Decisions(已确认)
|
||||
|
||||
| 决策点 | 选择 |
|
||||
|--------|------|
|
||||
| 实现路径 | **A:管线选择器**(代码硬保证去重;LLM 只写开场/理由/摘要;中文化仍走现有 `localize`) |
|
||||
| Skills「同一类」 | 暂不管;沿用现有 `group_skills_by_source`(**已知残留**:同 source 换 skill id 仍可能周内再出现) |
|
||||
| 周去重后不足 Top N | **深池补满**,仍保证周内未出现;池空则短榜,不破周约束 |
|
||||
| 首推改推候选 | **先展示榜(池 A),再 raw 深池(池 B)**;一月内首推不重复 |
|
||||
| 开场主题 | **近 7 天禁主题/句式(软)+ 叙事轴与近 3 天不同(硬,代码选轴)** |
|
||||
| 取消「放宽窗口」 | **禁止旧闻/已推凑数**;不够则深度检索补新闻;禁止任何「放宽」文案标注;仍不够则短列表 |
|
||||
| 展示历史 vs 异动基准 | **必须拆开**:`movement_baseline` ≠ `wecom_shown_keys` |
|
||||
| full/delta 列表主人 | **唯一主人** = `board_select`(含 full 与 delta 的 moves∪pad);禁止二次独立选榜 |
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
| 项 | 状态 |
|
||||
|----|------|
|
||||
| 语义级 Skill「同一类」分类 | ❌ 本期不做 |
|
||||
| 同日 Skills Trending ↔ Hot 互斥 | ❌ 本期不做 |
|
||||
| 独立 editorial 微服务 | ❌ 不做 |
|
||||
| 改写 `localize` 为脚本机翻 | ❌ 保持 LLM + 缓存 |
|
||||
| 编辑指定首推豁免改推(`FEATURED_FORCE`) | ❌ 本期不做 |
|
||||
| 「开场关键短语」硬匹配算法 | ❌ 本期不做(仅 prompt 软约束;不进硬成功标准) |
|
||||
|
||||
## Recommended Approach: 管线选择器(路径 A)
|
||||
|
||||
在现有 `daily generate` 内增加选条/裁决层,不新起进程:
|
||||
|
||||
```
|
||||
采集 raw 榜 + 新闻
|
||||
→ build movement_baseline(raw Top compare_depth —— 仅供次日「新入榜」,禁止写展示历史)
|
||||
→ board_select(读 wecom_shown_keys 周历史;周去重 + 深池;输出当日最终展示列表)
|
||||
· full:直接取 board_select 结果前 N
|
||||
· delta:在 board_select 候选池内做 moves∪pad(pad 也只从该池/同规则深池取,不再另起一套历史)
|
||||
→ featured_resolve(与昨日首推相同则改推;池 A = 本 run 最终展示 keys)
|
||||
→ research why(先定人,再写 why_today)
|
||||
→ news_select(禁放宽凑数;深检索补满;剥「放宽」前缀)
|
||||
→ editorial(代码选 narrative_axis;prompt 附近 7 天 theme 软禁)
|
||||
→ 渲染 wecom
|
||||
→ 写回 wecom_shown_keys = 最终进入企微正文的榜条目 keys(post-render)
|
||||
→ 其余 history(首推月、axis)写入 data.json 约定字段
|
||||
```
|
||||
|
||||
**LLM 负责**:`opening` / `theme_line`、首推 `why_today`、新闻与榜单项中文摘要(`localize`)。
|
||||
**代码负责**:谁上榜、首推换谁、周/月去重、`narrative_axis` 选取、是否允许旧闻。
|
||||
|
||||
### Approaches Considered
|
||||
|
||||
| | A 管线选择器(采用) | B 偏 LLM 约束 | C 独立 editorial 服务 |
|
||||
|--|--|--|--|
|
||||
| 优点 | 可测;与 pushed-links 模式一致 | 改 prompt 快 | 边界清晰 |
|
||||
| 缺点 | 需动 generate / featured / news / format | 易漏、难测 | 过重 |
|
||||
|
||||
---
|
||||
|
||||
## Section 1 — 总览、列表主人、历史真相表
|
||||
|
||||
### 1.1 两种「基准」禁止混用
|
||||
|
||||
| 字段 | 含义 | 写入时机 | 读者 |
|
||||
|------|------|----------|------|
|
||||
| `movement_baseline` | **Raw** 各榜 Top `compare_depth`(现网语义不变) | `build_llm_input` / 采集后尽早 | `build_movement_context`(新入榜) |
|
||||
| `wecom_shown_keys` | **读者实际见到**的各榜 key 集合(及可选 rank) | **wecom 渲染完成之后** | `board_select` 周去重;delta pad;测试 |
|
||||
|
||||
**禁止**:把 `wecom_shown_keys` 写入或覆写 `movement_baseline`。
|
||||
**禁止**:让 `load_recent_board_keys` 继续读 `movement_baseline` 充当「已展示」——应改为读近 N 日 `wecom_shown_keys`(可保留函数名,换数据源;或新建 `load_recent_shown_keys`)。
|
||||
|
||||
### 1.2 唯一列表主人
|
||||
|
||||
`board_select`(模块可挂在 `daily/board_select.py` 或扩 `delta.py`)是各榜**最终展示行**的唯一生产者:
|
||||
|
||||
| 模式 | 行为 |
|
||||
|------|------|
|
||||
| `full` | `board_select(raw, shown_history) →` 至多 N 条,直接渲染 |
|
||||
| `delta` | 先算相对 `movement_baseline` 的 moves;展示 = `moves`(已在候选内)∪ `pad`;**pad 候选必须来自同一周去重池**(与 full 同一套 `board_select` 规则),不得再读 raw baseline 当「已展示」 |
|
||||
|
||||
交互影响(非「完全正交」):周去重会减少可展示重复项 → delta 日可能更短、silent/gate 行为可能变化。`DAILY_WECOM_MODE` 枚举语义不变,但列表密度会变。
|
||||
|
||||
### 1.3 历史真相表(单一来源)
|
||||
|
||||
全部落在 `output/{date}.data.json`(新闻 pushed-links 例外,沿用现网 cache)。
|
||||
|
||||
| 字段路径 | 窗口 | Key 规则 | 写者 | 读者 |
|
||||
|----------|------|----------|------|------|
|
||||
| `data.movement_baseline` | 次日对比用 | raw 条目切片 | `build_movement_baseline` | movement |
|
||||
| `data.wecom_shown_keys.{board}` | 滚动 7 天(读近 7 日文件) | Skills:与现网 `_skill_keys_in_board_item` / `skill_id` 一致;GitHub:`owner/repo` | post-render persist | `board_select` / pad |
|
||||
| `data.featured_pick_key` | 滚动 30 天 | skill id 或 `owner/repo` | `featured_resolve` 成功后 | 月去重 |
|
||||
| `data.narrative_axis` | 滚动 3 天 | 枚举字符串 | 代码 `pick_narrative_axis` | Step 1 约束 / 校验 |
|
||||
| `data.theme_line` / trends opening | 近 7 日供 prompt | 原文 | editorial 落盘 | Step 1 软禁(不硬匹配) |
|
||||
| `CACHE_DIR/pushed-news-links.json` | `DAILY_NEWS_DEDUP_DAYS` | 规范化 URL | 推送成功后 | news filter |
|
||||
|
||||
不另建平行 CACHE「board-history.json」,避免双源漂移。冷启动:缺文件 = 空集合。
|
||||
|
||||
### 1.4 数据流挂点
|
||||
|
||||
| 逻辑 | 挂点 |
|
||||
|------|------|
|
||||
| `movement_baseline` | 现网:raw 榜入库时(不变) |
|
||||
| `board_select` | 渲染前;输出写入供 Agent/`llm_input` 与 wecom 共用的最终列表字段 |
|
||||
| delta pad | **调用同一周去重历史**(`wecom_shown_keys`),不再独立解释 `movement_baseline` 为展示史 |
|
||||
| `featured_resolve` | **先于** why 检索;池 A = 本 run `board_select`(delta 则为本 run 最终展示列表) |
|
||||
| 新闻 | 所有 prepare 路径关 backfill;research SKILL 改文案规则;后处理剥「放宽」 |
|
||||
| `narrative_axis` | 代码先选轴再注入 Step 1;LLM 不得另选冲突轴 |
|
||||
| `wecom_shown_keys` 写回 | `replace_wecom_*` / `build_wecom_report` 之后,与最终正文列表一致 |
|
||||
|
||||
---
|
||||
|
||||
## Section 2 — 各榜选条 + 今日首推改推
|
||||
|
||||
### 2.1 `board_select`(五榜共用)
|
||||
|
||||
适用:`skills_trending` / `skills_hot` / `github_trending` / `github_emerging` / `github_topic`。
|
||||
|
||||
```
|
||||
输入:当日 raw 池(深,pool ≥ DAILY_BOARD_POOL_SIZE)
|
||||
历史:近 DAILY_BOARD_DEDUP_DAYS 的 wecom_shown_keys[board]
|
||||
输出:至多 N 条(N = 现有 wecom Top 配置)
|
||||
|
||||
1. 现有整理(Skills:source 合并;GitHub:repo key)
|
||||
2. 滤掉近 7 天该榜 wecom_shown_keys
|
||||
3. 按原排名取前 N
|
||||
4. 不足 → 继续扫深池,仍排除周历史,直到满 N 或池空
|
||||
5. 池空仍不足 → 短榜;日志 board_short:{board}:{n};不回填周内已展示条目
|
||||
```
|
||||
|
||||
分榜独立历史:Trending 出过的 skill,Hot 仍可出。
|
||||
|
||||
Post-render:将**实际写入企微的** keys 写入当日 `wecom_shown_keys`(测试断言:history ⊆ / == 渲染列表,**≠** `movement_baseline`)。
|
||||
|
||||
### 2.2 `featured_resolve`
|
||||
|
||||
**触发**:本 run 拟用首推身份与**前一天** `featured_pick_key`(或等价 data 字段)相同。
|
||||
身份函数:skill → `skill_id`;github → `owner/repo`。
|
||||
含 `DAILY_FEATURED_PICK` 与自动首推;本期不豁免。无昨日文件 → 不改推。
|
||||
|
||||
**顺序(硬)**:定候选 → 再 `research`/`why_today`(禁止先写旧条目 why 再改人却不重写)。
|
||||
|
||||
**候选**:
|
||||
|
||||
1. **池 A**:本 run **最终会展示**的 Skills + GitHub 榜条目(与 `wecom_shown_keys` 同源结构)
|
||||
2. **池 B**:raw 深池中尚未进入本 run 展示者
|
||||
|
||||
**过滤**:近 30 天 `featured_pick_key`;排除冲突项自身。
|
||||
|
||||
**抽取**:`hash(date_str + "featured")` 可复现;测试可注入 RNG。先 A 后 B;仍空 → 保留原首推 + `featured_fallback_exhausted`。
|
||||
|
||||
**落盘**:`data.featured_pick_key`。
|
||||
|
||||
---
|
||||
|
||||
## Section 3 — 开场主题 + AI 时讯
|
||||
|
||||
### 3.1 开场主题
|
||||
|
||||
| 机制 | 强度 | 规则 |
|
||||
|------|------|------|
|
||||
| `narrative_axis` | **硬** | 代码 `pick_narrative_axis(used_last_N)` 从剩余枚举选取;注入 prompt;LLM 输出须等于该轴;冲突则重试 1 次,再失败则**强制覆写为代码所选轴**再落盘(保证成功标准可测) |
|
||||
| theme/opening 软禁 | **软** | prompt 附近 7 天 `theme_line`/opening 摘要;禁止复述;**无** n-gram 硬匹配;**不**列入硬成功标准 |
|
||||
|
||||
**叙事轴枚举**:
|
||||
|
||||
`政策监管` · `模型发布` · `工具链/Agent` · `芯片算力` · `开源生态` · `应用落地` · `安全/诉讼`
|
||||
|
||||
`opening` 首句证据须来自当日数据;首推改推后须跟新首推或当日主轴新闻。
|
||||
|
||||
### 3.2 AI 时讯:取消「放宽窗口」
|
||||
|
||||
目标条数 = 现网配置之和(如 `DAILY_WECOM_AI_NEWS` + tech/CN 等),文档不写死「15」。
|
||||
|
||||
1. **所有 prepare 路径**关闭「不够塞回已推/旧条」(`DAILY_NEWS_BACKFILL=0` 默认);`pushed-news-links` 过滤保留。
|
||||
2. 不够 → 深度检索补新闻(须:https link、未 pushed、可核实);**补入年龄上限** = `DAILY_AI_NEWS_HOURS`(与主窗一致),禁止借 research 变相放宽到任意旧闻。
|
||||
3. 改 research SKILL:删除「放宽至 48h 并注明」;后处理剥 `放宽窗口`/`放宽至` 前缀或丢弃。
|
||||
4. 仍不足 → 短列表 + `news_short:{n}`。
|
||||
|
||||
中文化:`daily/localize.py`(不变)。
|
||||
|
||||
---
|
||||
|
||||
## Section 4 — 配置、错误处理、测试
|
||||
|
||||
### 4.1 环境变量
|
||||
|
||||
| 变量 | 默认 | 含义 |
|
||||
|------|------|------|
|
||||
| `DAILY_BOARD_DEDUP_DAYS` | `7` | 读 `wecom_shown_keys` 的滚动天数 |
|
||||
| `DAILY_BOARD_POOL_SIZE` | ≥50 / 与现有 skill pool 对齐 | 深池扫描深度 |
|
||||
| `DAILY_FEATURED_DEDUP_DAYS` | `30` | 今日首推月去重 |
|
||||
| `DAILY_THEME_BAN_DAYS` | `7` | 软禁:注入 prompt 的 theme 天数 |
|
||||
| `DAILY_NARRATIVE_AXIS_DAYS` | `3` | 叙事轴互斥窗 |
|
||||
| `DAILY_NEWS_BACKFILL` | `0` | `0`=禁止旧闻凑数 |
|
||||
| `DAILY_NEWS_DEDUP_DAYS` | 已有 `7` | pushed-links |
|
||||
|
||||
`DAILY_DELTA_PAD_LOOKBACK_DAYS` 应与 `DAILY_BOARD_DEDUP_DAYS` 对齐,且 **pad 与 board_select 共用 `wecom_shown_keys`**(窗口对齐不够,数据源必须同一)。
|
||||
|
||||
### 4.2 错误与降级
|
||||
|
||||
| 情况 | 行为 |
|
||||
|------|------|
|
||||
| 无 `wecom_shown_keys` 历史 | 空集合,正常满榜 |
|
||||
| 周去重后深池不足 | 短榜 + `board_short` |
|
||||
| 首推冲突且 A/B 空 | 保留原首推 + `featured_fallback_exhausted` |
|
||||
| LLM 轴与代码轴冲突 | 覆写为代码轴 |
|
||||
| 深检索仍不足时讯 | 短列表;禁止 backfill |
|
||||
| history 读写失败 | 当次按空历史 + error 日志 |
|
||||
|
||||
### 4.3 测试(pytest)
|
||||
|
||||
1. `board_select`:假 `wecom_shown_keys` + 深池 → 无周交集;深池补满;不足短榜
|
||||
2. **回归钉死**:写回后 `wecom_shown_keys` ≠ 用 `movement_baseline` 推导的集合(构造 raw Top 与展示 Top 故意不同)
|
||||
3. delta:pad 不引入近 7 日 `wecom_shown_keys` 内 key
|
||||
4. `featured_resolve`:先定人再 why;A 优先 B;月未见;可注入 RNG
|
||||
5. news:`BACKFILL=0`;剥「放宽*」;research 补入不超 hours 窗
|
||||
6. `pick_narrative_axis`:近 3 天互斥;落盘轴 == 代码轴
|
||||
7. 既有 delta / pushed_links / wecom 回归不挂
|
||||
|
||||
### 4.4 成功标准(硬)
|
||||
|
||||
- 连续两天:**首推 key 不同**(除非 `featured_fallback_exhausted`)
|
||||
- 同一榜近 7 日 `wecom_shown_keys`:**无重复 key**(池足够时)
|
||||
- 时讯:无「放宽*」标注;无 backfill 已推 link
|
||||
- 近 3 天 `narrative_axis`:**两两不同**(代码保证)
|
||||
|
||||
软标准(不闸门):opening 读感不像连续复印。
|
||||
|
||||
---
|
||||
|
||||
## Implementation Sketch(非计划明细)
|
||||
|
||||
1. `data.json` 增加 `wecom_shown_keys`;改 `load_recent_*` 数据源
|
||||
2. `board_select` + 让 delta pad 共用
|
||||
3. post-render persist shown keys
|
||||
4. `featured_resolve` 时序修正
|
||||
5. news backfill off + SKILL + 剥前缀
|
||||
6. `pick_narrative_axis` + prompt 注入
|
||||
7. 测试如上
|
||||
|
||||
正式任务拆解 → `writing-plans`。
|
||||
|
||||
## Office-hours Review Notes
|
||||
|
||||
- 对抗审阅质量约 4/10 → 本修订处理三大硬伤(存储拆分、列表主人、真相表)。
|
||||
- 未纳入本期(原选项 B):首推质量加权、关键短语硬匹配。
|
||||
- 已知残留:source 级「同类」周内可再现。
|
||||
|
||||
## Spec Self-Review
|
||||
|
||||
- [x] `movement_baseline` 与 `wecom_shown_keys` 职责分离写死
|
||||
- [x] 单一列表主人 + full/delta 交互说明
|
||||
- [x] 历史真相表无「与/或」双源
|
||||
- [x] 轴硬 / 短语软;成功标准不含无法验证的短语匹配
|
||||
- [x] 周不足=深池、首推=A→B、新闻禁放宽 与访谈一致
|
||||
@@ -0,0 +1,141 @@
|
||||
# Design: Research 模式 AI 时讯质量(去重 · 国内配额 · 可信源)
|
||||
|
||||
Generated: 2026-07-29
|
||||
Repo: daily-robots
|
||||
Status: APPROVED
|
||||
Related: `skills/daily-ai-news-research/SKILL.md`, `daily/news/research.py`, `docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md`
|
||||
|
||||
## Problem Statement
|
||||
|
||||
当前 `DAILY_AI_NEWS_MODE=research` 下:
|
||||
|
||||
1. **内容重复高**:同事件多源报道并列入榜(如 OpenAI 联名 / Altman 减速 / HF 入侵同簇;Kimi K3 开源与多条适配);`items` 与 `tech_items` 只按 link 去重,skill 还允许主题重叠。
|
||||
2. **几乎无国内资讯**:prompt 写「不区分国内国外,合并精选」,模型偏国际源;research 分支关闭国内 RSS,企微也不再出国内独立区块。
|
||||
3. **可信度不稳**:低质搬运 / 营销号可进榜(如证券之星类),与「以官方/权威为准」不符。
|
||||
|
||||
样例:`output/2026-07-29.ai-news-research.json`。
|
||||
|
||||
## Decisions(已确认)
|
||||
|
||||
| 决策点 | 选择 |
|
||||
|--------|------|
|
||||
| 国内资讯路径 | **A:继续 research 合并展示**;不恢复国内独立区块;不引入 RSS 回填 |
|
||||
| 国内配额 | 主列表至少约 30%(10 条 → **≥3 条国内**);`tech_items` **不强制**国内 |
|
||||
| 去重范围 | **C**:同事件多源只留 1 条 + `tech_items` 相对 `items` 主题不得复述 |
|
||||
| 可信度 | 国内/国际均以 **官方 + 权威媒体** 为准;不可信则不写,宁缺毋滥 |
|
||||
| 实现路径 | **Prompt/skill 约束 + `research.py` 解析后硬校验**(推荐方案 2) |
|
||||
| 条数不足 | **少返回**;禁止为凑满 limit / 国内配额灌低质或同事件条目 |
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
| 项 | 状态 |
|
||||
|----|------|
|
||||
| 恢复「国内 AI 时讯」独立企微区块 | ❌ 本期不做 |
|
||||
| research 不足时用国内 RSS 补齐 | ❌ 本期不做(偏离路径 A) |
|
||||
| 切回纯 RSS | ❌ 本期不做 |
|
||||
| LLM 语义嵌入聚类 | ❌ 本期不做(规则 + 实体/关键词即可) |
|
||||
| 改 `generate.py` research/rss 分支结构 | ❌ 不动 |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Cursor Agent (WebSearch)
|
||||
│
|
||||
▼
|
||||
skills/daily-ai-news-research/SKILL.md + research prompt
|
||||
│ JSON: items / tech_items
|
||||
▼
|
||||
parse_research_response (link 去重)
|
||||
│
|
||||
▼
|
||||
post_process_research_items
|
||||
1. trusted-source filter
|
||||
2. same-event dedupe (items)
|
||||
3. tech vs items topic filter
|
||||
4. CN quota packing (min_cn)
|
||||
│
|
||||
▼
|
||||
_apply_pushed_dedup → generate / wecom
|
||||
```
|
||||
|
||||
仍走现有 `fetch_ai_news_research` → `generate.py` research 分支;企微保持一块合并时讯。
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Skill / prompt(`skills/daily-ai-news-research/SKILL.md` + `fetch_ai_news_research` 文案)
|
||||
|
||||
更新要点:
|
||||
|
||||
- 中英文检索都要做;主列表国内 ≥ `min_cn`(默认 30%),但**只收可信国内源**。
|
||||
- 删除「主题可重叠」「必须恰好 N 条硬凑」;改为「不足少返回」。
|
||||
- 同事件只保留一条最权威源;`tech_items` 不得复述 `items` 已覆盖的事件/产品发布。
|
||||
- 来源优先级:官方 blog/changelog/新闻稿、政府/监管原文、一线权威媒体、学术官方;禁止二手搬运、标题党、不明自媒体、证券营销号(无权威交叉验证则不写)。
|
||||
- 用户 prompt 从「不区分国内国外,合并精选」改为「合并展示,国内配额硬性,可信度优先」。
|
||||
|
||||
### 2. 后处理(`daily/news/research.py`)
|
||||
|
||||
#### 2.1 国内判定(任一即国内)
|
||||
|
||||
1. 可选字段 `region: "cn"|"intl"`(有则优先)。
|
||||
2. `source_name` / host 命中国内白名单(量子位、36氪、雷锋网、IT之家、财新、机器之心、钛媒体、虎嗅等)。
|
||||
3. 常见中文域(`.cn` 及已知国内 host)。
|
||||
|
||||
#### 2.2 可信源过滤
|
||||
|
||||
- 维护 `TRUSTED_HOSTS` / `TRUSTED_SOURCE_NAMES`(国际 + 国内权威;含官方域如 `openai.com`、`anthropic.com`、`blog.google` 等)。
|
||||
- 不在白名单 → **剔除**(保守:宁可短列表)。
|
||||
- 官方域优先于二手报道;同事件去重时官方源胜出。
|
||||
|
||||
#### 2.3 同事件去重(items)
|
||||
|
||||
- 在 link 去重之后。
|
||||
- 归一化 title+desc;抽取主实体(公司/产品)+ 事件动词簇(开源、入侵、联名、适配…)。
|
||||
- 高置信重叠 → 只留排序更靠前且更可信的一条;保守策略避免误杀明显不同事件。
|
||||
|
||||
#### 2.4 tech vs items
|
||||
|
||||
- tech 与任一 item 同事件,或同主实体且同属「发布/适配/SDK」簇 → 丢弃 tech。
|
||||
- 例:items 已有 Kimi K3 开源 → tech 中 HF / 昇腾 / 摩尔线程适配全部剔除。
|
||||
|
||||
#### 2.5 国内配额
|
||||
|
||||
- `min_cn = env DAILY_WECOM_AI_NEWS_CN_MIN`;若未设或 ≤0,则 `max(1, lim * 3 // 10)`(10 → 3)。
|
||||
- 去重后尽量保证最终 `items` 中国内 ≥ `min_cn`:国内可信条优先占位,再用国际可信条补满。
|
||||
- 可信国内不足 → 短列表 + 日志 `news_cn_short:N`;**不用低质源灌满**。
|
||||
|
||||
### 3. 配置
|
||||
|
||||
| 变量 | 默认 | 含义 |
|
||||
|------|------|------|
|
||||
| `DAILY_WECOM_AI_NEWS_CN_MIN` | `0`(= 按 30% 推算) | 主列表国内最少条数;正整数覆盖 |
|
||||
|
||||
在 `.env.example` 注明;`.env` 可不改(沿用推算)。
|
||||
|
||||
### 4. 测试(`tests/test_ai_news_research.py`)
|
||||
|
||||
- 国内判定(白名单 / `.cn` / `region`)。
|
||||
- 可信过滤:证券之星类 host 剔除;官方/权威保留。
|
||||
- 同事件:Altman 减速与员工联名若同属「减速/治理请愿」簇 → 只留 1 条;与 HF 入侵若实体+事件动词不同则可分簇保留(保守,避免误杀)。
|
||||
- tech:items 已有 Kimi K3 → 多条 Kimi 适配 tech 全剔除。
|
||||
- 配额:混合列表打包后国内 ≥ `min_cn`(在可信国内足够时)。
|
||||
|
||||
验收可对 `2026-07-29.ai-news-research.json` 跑后处理做人工对照。
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
| 情况 | 行为 |
|
||||
|------|------|
|
||||
| 无 `CURSOR_API_KEY` / Agent 失败 | 与现网一致:`enabled: False` |
|
||||
| 去重/可信过滤后变短 | 接受短列表;日志 `news_dedup_drop` / `news_cn_short` |
|
||||
| 可信国内为 0 | 不编造、不 RSS 回填;国际可信条照常(若有) |
|
||||
|
||||
## Files Touched
|
||||
|
||||
- `skills/daily-ai-news-research/SKILL.md`
|
||||
- `daily/news/research.py`
|
||||
- `tests/test_ai_news_research.py`
|
||||
- `.env.example`(可选配置说明)
|
||||
|
||||
## Out of Scope Reminder
|
||||
|
||||
不改 RSS 抓取路径;不改 research/rss 模式切换结构;不恢复国内独立企微区块。
|
||||
43
run-scheduler.ps1
Normal file
43
run-scheduler.ps1
Normal file
@@ -0,0 +1,43 @@
|
||||
# Start the daily report scheduler (generate @ 08:50, push @ 09:00 by default)
|
||||
# Usage:
|
||||
# .\run-scheduler.ps1
|
||||
# .\run-scheduler.ps1 -DryRun
|
||||
# .\run-scheduler.ps1 -Once
|
||||
|
||||
param(
|
||||
[switch]$DryRun,
|
||||
[switch]$Once
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
Set-Location $Root
|
||||
|
||||
function Import-DotEnvFile {
|
||||
param([string]$Path)
|
||||
if (-not (Test-Path $Path)) { return }
|
||||
Get-Content $Path -Encoding UTF8 | ForEach-Object {
|
||||
if ($_ -match '^\s*#' -or $_ -notmatch '=') { return }
|
||||
$pair = $_ -split '=', 2
|
||||
if ($pair.Count -eq 2) {
|
||||
$name = $pair[0].Trim()
|
||||
$value = $pair[1].Trim().Trim('"').Trim("'")
|
||||
if ($name -and $value) {
|
||||
Set-Item -Path "Env:$name" -Value $value
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Import-DotEnvFile (Join-Path $Root ".env")
|
||||
Import-DotEnvFile (Join-Path $Root ".env.local")
|
||||
|
||||
$args = @("python", "-m", "daily", "schedule")
|
||||
if ($DryRun) { $args += "--dry-run" }
|
||||
if ($Once) { $args += "--once" }
|
||||
|
||||
Write-Host "Starting scheduler: $($args -join ' ')"
|
||||
& $args[0] $args[1..($args.Length - 1)]
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "scheduler failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
@@ -7,6 +7,48 @@
|
||||
"skillPath": "skills/brand-voice/SKILL.md",
|
||||
"computedHash": "f07173b4a886e800df150f9824535c0bfceed289f00a27ad4a3b1df5226dfa4d"
|
||||
},
|
||||
"cavecrew": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/cavecrew/SKILL.md",
|
||||
"computedHash": "9633c1391fa246091ce68ea522c0e424b2bc93aeb69fc44221a30b53e8a2c23d"
|
||||
},
|
||||
"caveman": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/caveman/SKILL.md",
|
||||
"computedHash": "723fb2a8bec1156c0f0b5bf020cc739ed09702b7726ec6377480038871339f6e"
|
||||
},
|
||||
"caveman-commit": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/caveman-commit/SKILL.md",
|
||||
"computedHash": "f028652defd5fdeddcce2994083cb1a7b201ee827bba8e2495546ee159fca3de"
|
||||
},
|
||||
"caveman-compress": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/caveman-compress/SKILL.md",
|
||||
"computedHash": "1055abaf7cb2f8c0ca78b64101b84dfc910d9819733ed9f0277b661441797aeb"
|
||||
},
|
||||
"caveman-help": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/caveman-help/SKILL.md",
|
||||
"computedHash": "4dba39eea07a050108d47940b39600bc8f45489201ecff0ccf03627180fd8e50"
|
||||
},
|
||||
"caveman-review": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/caveman-review/SKILL.md",
|
||||
"computedHash": "b9091dbc51de0f3710ea818fd4d638539f8c1784f8fda931eb159c44861e702e"
|
||||
},
|
||||
"caveman-stats": {
|
||||
"source": "JuliusBrussee/caveman",
|
||||
"sourceType": "github",
|
||||
"skillPath": "skills/caveman-stats/SKILL.md",
|
||||
"computedHash": "331f720e2fa97b68cacdae44384878071e8cac6013479edea68f4c8eca308852"
|
||||
},
|
||||
"python-patterns": {
|
||||
"source": "affaan-m/everything-claude-code",
|
||||
"sourceType": "github",
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
# 早报 Agent 工作流
|
||||
|
||||
你是 **Skills / GitHub / AI 时讯早报** 的主编 Agent。Python 已完成数据抓取;你分 **两步** 产出可读性强的企微早报。
|
||||
你是 **Skills / GitHub / AI 时讯早报** 的主编 Agent。Python 已完成数据抓取;若配置了 `DAILY_FEATURED_PICK`,Python 会先完成 **Step 0 首推检索** 并注入 `featured_pick`。你再分 **两步** 产出可读性强的企微早报。
|
||||
|
||||
## 工作流
|
||||
|
||||
```
|
||||
Step 0 首推检索(Python + daily-featured-pick,输出 featured_pick)
|
||||
Step 1 读话题 → 识别热门趋势(输出 trends JSON)
|
||||
Step 2 基于趋势 + 原始数据 → 写企微 Markdown 早报
|
||||
```
|
||||
@@ -27,21 +28,31 @@ Step 2 基于趋势 + 原始数据 → 写企微 Markdown 早报
|
||||
| `github_emerging` | GitHub **新兴** 当前 Top N |
|
||||
| `github_topic.repos` | GitHub **Topic** 当前 Top N |
|
||||
| `github_topic.topic` | Topic 名称,用于区块标题(如 `llm`) |
|
||||
| `movement.*_moves` | 较昨日新增(**仅**用于 opening / signals,**不**用于列表区块) |
|
||||
| `movement.*_moves` | 较昨日新增(**仅**用于 opening / signals;`effective_wecom_mode=delta` 时列表由 Python 插入) |
|
||||
| `movement.*_summary` | 新增摘要(可选写入 signals) |
|
||||
| `ai_news` | 国际 AI 时讯 Top N |
|
||||
| `cn_ai_news` | 国内 AI 时讯 Top N |
|
||||
| `ai_news` | 国际 AI 时讯 Top N(RSS 模式) |
|
||||
| `cn_ai_news` | 国内 AI 时讯 Top N(RSS 模式) |
|
||||
| `ai_news_mode` | `research` 时仅 `ai_news` 有 10 条合并精选,`cn_ai_news` 为空 |
|
||||
| `featured_pick` | **可选**,编辑指定今日首推(Step 0 产出;含 command / why_today / evidence) |
|
||||
|
||||
**禁止**使用已合并的 `skills_moves` / `github_moves` 自行扩写;**禁止**排名变化、安装涨跌。
|
||||
|
||||
Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目、movement 新增与 AI 时讯**。
|
||||
|
||||
### 当输入含 `featured_pick` 时(Step 1)
|
||||
|
||||
- `top_picks.skill` **必须**使用 `featured_pick`:`id`/`title`/`command`/`url` 来自 featured,`why` 来自 `featured_pick.why_today`
|
||||
- `opening` 首句优先引用 `featured_pick.evidence` 中的数字或条目名
|
||||
- `signals` 至少 1 条与 featured 相关
|
||||
- featured 不在 Top 榜时仍可作为首推,`why` 须来自 `featured_pick.evidence`,禁止编造榜单排名
|
||||
|
||||
**只输出 JSON:**
|
||||
|
||||
```json
|
||||
{
|
||||
"headline": "8–16字焦点标题",
|
||||
"opening": "2–3句中文导语;首句必须是具体证据(榜首 skill+安装量 / 头条新闻 / GitHub #1),再解释为什么值得看",
|
||||
"narrative_axis": "必须等于输入 required_narrative_axis(政策监管|模型发布|工具链/Agent|芯片算力|开源生态|应用落地|安全/诉讼)",
|
||||
"themes": [
|
||||
{
|
||||
"title": "主题名",
|
||||
@@ -50,7 +61,7 @@ Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目
|
||||
}
|
||||
],
|
||||
"top_picks": {
|
||||
"skill": { "id": "owner/repo/skill", "title": "...", "why": "中文,为什么今天首推" },
|
||||
"skill": { "id": "owner/repo/skill", "title": "...", "command": "npx skills add ...", "why": "中文,为什么今天首推" },
|
||||
"github": { "repo": "owner/repo", "why": "中文" },
|
||||
"news": { "link": "完整URL", "title_zh": "中文标题", "why": "中文一句话" }
|
||||
},
|
||||
@@ -64,16 +75,17 @@ Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目
|
||||
要求:
|
||||
|
||||
- 所有结论必须能在输入 JSON 中找到依据,禁止编造
|
||||
- `narrative_axis` **必填**,且必须等于输入中的 `required_narrative_axis`(代码已选定;勿自选其它轴)
|
||||
- `opening` 遵循 **article-writing Newsletter** 规则:首句用数字/条目名/新闻标题开头,不用「今天有三条线」「值得关注」等空框架
|
||||
- `signals` 3–5 条,每条单行,可含 emoji 前缀;与 `opening` 不重复同一句信息
|
||||
- `top_picks.why` 用「事实/数字 + 一句判断」,不用空泛形容词
|
||||
- `top_picks` 必须引用输入中真实存在的 id/repo/link
|
||||
- `top_picks` 必须引用输入中真实存在的 id/repo/link;有 `featured_pick` 时 skill 首推以 featured 为准
|
||||
|
||||
---
|
||||
|
||||
## Step 2:撰写企微早报
|
||||
|
||||
你会收到 **原始数据 JSON** + **Step 1 的 trends JSON**。
|
||||
你会收到 **原始数据 JSON**(含可选 `featured_pick`)+ **Step 1 的 trends JSON**。
|
||||
|
||||
**只输出企微 Markdown 正文**(不要代码块包裹,不要解释)。
|
||||
|
||||
@@ -93,21 +105,33 @@ Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目
|
||||
> {signal 3}
|
||||
|
||||
📦 **今日首推**
|
||||
`npx skills add {source}/{skill}`
|
||||
> {top_picks.skill.why}
|
||||
[{featured_pick.title 或 repo 或 skill 名}]({featured_pick.url 或 command 或 skills.sh 链接})
|
||||
> {featured_pick.why_today 或 top_picks.skill.why}
|
||||
|
||||
### 当输入含 `featured_pick` 时(Step 2)
|
||||
|
||||
- **今日首推**链接行用 Markdown `[标题](URL)`,与新闻条目同格式;**不要**用反引号裸 URL 或裸 `npx` 命令
|
||||
- GitHub 首推:标题用 `owner/repo`(如 `[garrytan/gstack](https://github.com/...)`)
|
||||
- Skill 首推:标题用 skill 名,链接用 `featured_pick.url` 或 `https://skills.sh/{id}`
|
||||
- 说明用 `featured_pick.why_today`;**不要**改回 Trending 榜首
|
||||
- 读者可见文案(opening / 今日首推说明 / why)**不得**写「编辑指定首推」等元信息,只陈述项目事实与判断
|
||||
|
||||
🌍 **国际 AI · 精选 10**
|
||||
1. [{title_zh}]({link}) — {why 或摘要}
|
||||
2. ...(**必须 10 条**,来自 `ai_news`,按重要性排序)
|
||||
|
||||
🇨🇳 **国内 AI · 精选 8**
|
||||
🇨🇳 **国内 AI · 精选 10**
|
||||
1. [{title}]({link}) — {why 或摘要}
|
||||
2. ...(**必须 8 条**,来自 `cn_ai_news`,按重要性排序;标题已是中文,可微调润色)
|
||||
2. ...(**必须 10 条**,来自 `cn_ai_news`;RSS 模式)
|
||||
|
||||
<!-- **不要写** Skills Trending / Skills Hot 区块,Python 会在推送前按 source 合并后自动插入 -->
|
||||
📰 **AI 时讯精选 · 15**(当 `ai_news_mode=research`)
|
||||
1. [{title}]({link}) — {why 或摘要}
|
||||
2. ...(**必须 15 条**:前 10 条综合精选 + 后 5 条偏工程技术,来自 `ai_news` 与 `tech_ai_news`;**不要**再写 🌍/🇨🇳/🔧 分块)
|
||||
|
||||
<!-- delta 模式(effective_wecom_mode=delta):不要写任何 Skills / GitHub 榜单区块,Python 会插入变化列表 -->
|
||||
|
||||
🐙 **GitHub Trending Top {N}**
|
||||
<!-- 只列 data.github_trending -->
|
||||
<!-- full 模式:只列 data.github_trending -->
|
||||
1. [{repo}]({url}) · {lang} · ⭐{stars} — {中文一句话}
|
||||
|
||||
🌱 **GitHub 新兴 Top {N}**
|
||||
@@ -121,14 +145,16 @@ Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目
|
||||
|
||||
### 榜单选取规则(top_n)
|
||||
|
||||
1. **五个 GitHub 区块分开写**:GitHub Trending / 新兴 / Topic,**禁止合并**
|
||||
1. **五个 GitHub 区块分开写**(仅 `effective_wecom_mode=full`):GitHub Trending / 新兴 / Topic,**禁止合并**
|
||||
2. **Skills Trending / Hot 由 Python 自动插入**,Agent 不要写这两段
|
||||
3. **禁止**改用 `movement.*_moves` 作为列表来源;movement 仅用于 opening / signals 描述「今日新增」
|
||||
3. 当 `data.effective_wecom_mode` 为 `delta` 时:**禁止写** Skills Trending / Hot / GitHub 列表区块(Python 插入变化列表);movement 可用于 opening / signals。`full` 模式保持原 GitHub 列表规则
|
||||
4. **禁止**在条目后写 `(新入 … #n)` 类括号标注
|
||||
5. **即使某榜较昨日无新增,仍须完整列出 Top 榜条目**
|
||||
5. `effective_wecom_mode=full` 时:即使某榜较昨日无新增,仍须完整列出 Top 榜条目
|
||||
6. **国际 AI 必须 10 条**(来自 `ai_news`)
|
||||
7. **国内 AI 必须 8 条**(来自 `cn_ai_news`;无数据时写「暂无可用条目」)
|
||||
8. 禁止排名变化、安装涨跌、连霸描述
|
||||
7. **国内 AI 必须 10 条**(来自 `cn_ai_news`;无数据时写「暂无可用条目」)
|
||||
8. **`ai_news_mode=research` 时**:只写 **📰 AI 时讯精选 15 条**(`ai_news` 10 条 + `tech_ai_news` 5 条合并展示),不写 🌍/🇨🇳/🔧 分块;Python 会用调研结果覆盖该区块
|
||||
9. 禁止排名变化、安装涨跌、连霸描述
|
||||
10. 榜全稳(movement 各 `*_stable` 为 true)时,signals 聚焦新闻与首推,不编造榜单变化
|
||||
|
||||
```markdown
|
||||
📈 **Skills Trending Top 10**
|
||||
@@ -161,6 +187,7 @@ Step 1 的 `signals` 与 `top_picks` **优先引用 Top 榜榜首/前列条目
|
||||
**禁止(Banned Patterns):**
|
||||
|
||||
- 「据悉」「值得关注」「快速演进」「In today's rapidly evolving landscape」
|
||||
- 「编辑指定首推」「编辑今日首推」等内部流程用语(读者不应感知编辑配置)
|
||||
- 「今天有三条线叠在一起」这类无证据的空框架开场
|
||||
- 无证据的「为什么这很重要」「 here's why this matters」
|
||||
- 结尾硬塞互动问句(如「你怎么看?」「值得花十分钟扫一眼」)
|
||||
|
||||
64
skills/daily-ai-news-research/SKILL.md
Normal file
64
skills/daily-ai-news-research/SKILL.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# AI 时讯 Deep Research(早报专用)
|
||||
|
||||
你是 **AI 时讯调研员**。使用 **WebSearch** 与网页抓取工具,收集近 N 小时 AI 新闻(国内 + 国际合并展示),输出供企微早报使用的结构化 JSON。
|
||||
|
||||
## 工作流
|
||||
|
||||
1. 将任务拆成 3–5 个子问题(模型发布、监管政策、大厂动态、芯片算力、研究突破等);**中英文检索都要做**
|
||||
2. 每个子问题用 WebSearch 检索 2–3 组关键词(中英文混合)
|
||||
3. 交叉验证:只采用 **官方博客 / 新闻稿、政府或监管原文、一线权威媒体、学术官方**
|
||||
4. 按请求输出 **已去重候选池**(通常多于最终展示,如展示 10 → 去重后约 20):**条数 = 独立事件数**。输出前必须完成同事件/同 link 去重;多源只留最权威一条。国内可信独立事件也要明显多于展示配额;禁止换源重复充数或用低质源灌满
|
||||
5. 另输出已去重的 `tech_items` 候选;不得与 `items` 重复 link/同事件;技术区不强制国内
|
||||
6. **候选池内同事件只允许一条**(官方 > 一线媒体)
|
||||
7. **只输出 JSON**,不要 Markdown 报告,不要代码块
|
||||
|
||||
## 质量规则
|
||||
|
||||
1. 每条必须有可访问的 `link`(https://)
|
||||
2. 禁止编造未在搜索结果中出现的事实
|
||||
3. 优先近 N 小时内的新闻;若不足目标条数,**少返回**即可,禁止放宽至 48 小时凑数,禁止在 `desc_short` 标注「放宽窗口」
|
||||
4. `desc_short` 用中文一句话摘要(≤72 字)
|
||||
5. `title` 保留原文标题;中文源可用中文标题
|
||||
6. `source_name` 为媒体/站点简称(如 TechCrunch、量子位、OpenAI Blog)
|
||||
7. 可选 `region`: `"cn"` 或 `"intl"`(国内源标 `cn`)
|
||||
8. **禁止**二手搬运、标题党、不明自媒体、证券营销号;无权威源交叉验证则 **不写**
|
||||
|
||||
## 输出格式(严格 JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"title": "Apple sues OpenAI over trade secret theft",
|
||||
"link": "https://techcrunch.com/...",
|
||||
"source_name": "TechCrunch",
|
||||
"region": "intl",
|
||||
"desc_short": "苹果起诉 OpenAI 涉嫌窃取硬件商业机密",
|
||||
"published_fmt": "07-11 05:00"
|
||||
}
|
||||
],
|
||||
"tech_items": [
|
||||
{
|
||||
"title": "Meta Iris AI chip enters production",
|
||||
"link": "https://techcrunch.com/...",
|
||||
"source_name": "TechCrunch",
|
||||
"region": "intl",
|
||||
"desc_short": "Meta 自研 Iris 芯片 9 月量产",
|
||||
"published_fmt": ""
|
||||
}
|
||||
],
|
||||
"methodology": "检索 6 组 query,分析 12 源,子问题:诉讼、模型安全、监管"
|
||||
}
|
||||
```
|
||||
|
||||
- `items` / `tech_items`:条数以请求的**去重后候选目标**为准(独立事件数;可略少,不可灌重复或低质源)
|
||||
- `tech_items` 聚焦工程技术;可与 `items` 领域相近,但 **事件与产品不得重复**
|
||||
- `published_fmt` 格式 `MM-DD HH:MM`(UTC+8),无法确定则留空字符串
|
||||
- 不要输出 `items` 以外的长文;`methodology` 可选,一行即可
|
||||
|
||||
## 禁止
|
||||
|
||||
- 不要输出 ```json 代码块包裹(直接输出 JSON 对象)
|
||||
- 不要输出 Executive Summary / Key Takeaways 等报告章节
|
||||
- 不要使用本项目 RSS 或本地文档作为来源
|
||||
- 不要为凑国内配额或条数而写入低可信来源
|
||||
87
skills/daily-featured-pick/SKILL.md
Normal file
87
skills/daily-featured-pick/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# 早报今日首推检索
|
||||
|
||||
你是 **Skills / GitHub / AI 时讯早报** 的编辑研究员。Python 已完成榜单抓取;你负责为 **编辑指定的今日首推** 收集可核实信息,供后续趋势分析与写稿使用。
|
||||
|
||||
## 场景
|
||||
|
||||
- 触发:环境变量 `DAILY_FEATURED_PICK` 有值(如 `gstack` 或 `gstack|https://github.com/you/gstack`)
|
||||
- 输出:严格 JSON,写入 `output/YYYY-MM-DD.featured.json`
|
||||
- 你 **不** 写整篇早报、 **不** 改榜单顺序、 **不** 推送
|
||||
|
||||
## 输入
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "gstack",
|
||||
"url_hint": "https://github.com/you/gstack",
|
||||
"cwd": "D:\\path\\to\\workspace",
|
||||
"data_matches": {
|
||||
"skills": [],
|
||||
"github": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `query` | 编辑指定的关键词(skill 名 / repo 名片段) |
|
||||
| `url_hint` | 可选,项目主页或仓库 URL |
|
||||
| `cwd` | Cursor 工作目录,可在此检索 README / SKILL.md |
|
||||
| `data_matches` | Python 已在今日 Top 榜中预匹配的条目(**优先使用其数字**) |
|
||||
|
||||
## 检索顺序
|
||||
|
||||
1. **读 `data_matches`**:若 skills/github 有匹配, installs / star / repo / link **必须来自此处**,不得改写
|
||||
2. **读 `cwd` 本地仓库**:搜索 README、SKILL.md、package.json 描述,提炼「做什么 + 技术栈」
|
||||
3. **用 `url_hint`**:作为项目主页;无本地文件时可仅基于 URL 与 query 写 summary(须标注 evidence 来源)
|
||||
4. **禁止编造**:未在 data_matches / 本地文件 / url_hint 出现的数字、功能、版本一律不写
|
||||
|
||||
## 输出
|
||||
|
||||
**只输出一个 JSON 对象**,不要 markdown 围栏,不要解释。
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "gstack",
|
||||
"type": "skill|github|other",
|
||||
"command": "npx skills add owner/repo/skill",
|
||||
"url": "https://...",
|
||||
"summary": "2–3 句中文:做什么 + 技术栈/场景",
|
||||
"why_today": "中文,为什么今天主推(事实 + 一句判断)",
|
||||
"evidence": ["Skills Trending 匹配 · remotion-render · 22.3K", "README: Agent 工作流 CLI"],
|
||||
"tags": ["agent", "workflow"]
|
||||
}
|
||||
```
|
||||
|
||||
### 字段要求
|
||||
|
||||
| 字段 | 要求 |
|
||||
|------|------|
|
||||
| `title` | 展示名,通常与 query 或匹配条目 title/repo 短名一致 |
|
||||
| `type` | `skill` = Skills 条目;`github` = 仓库;`other` = 仅关键词/URL |
|
||||
| `command` | Skill:`npx skills add {source}/{title}`;GitHub:仓库 URL;other:url_hint 或 query |
|
||||
| `url` | 可点击链接,来自 data_matches.link / repo url / url_hint |
|
||||
| `summary` | 36–80 字中文,动词开头,说清用途 |
|
||||
| `why_today` | 40–80 字,「事实/数字 + 判断」,不用空泛形容词;**不得**出现「编辑指定」「编辑首推」等内部流程用语 |
|
||||
| `evidence` | 2–4 条短字符串,标明信息来源 |
|
||||
| `tags` | 0–4 个英文或中文关键词 |
|
||||
|
||||
### type 与 command 示例
|
||||
|
||||
- Skill 匹配:`type=skill`,`command=npx skills add vercel-labs/skills/find-skills`
|
||||
- GitHub 匹配:`type=github`,`command=https://github.com/openclaw/openclaw`
|
||||
- 仅关键词:`type=other`,`command` 用 url_hint
|
||||
|
||||
## 写作原则
|
||||
|
||||
1. **事实优先**:why_today 每条 claim 能在 evidence 或 data_matches 中找到
|
||||
2. **数字必真**:installs、star 与 data_matches 完全一致
|
||||
3. **中文叙述**:summary / why_today 全中文;skill/repo 名保留英文
|
||||
4. **克制**:不写「值得关注」「game-changer」等空话
|
||||
|
||||
## 输出前自检
|
||||
|
||||
- [ ] 仅有 JSON,无围栏、无前后说明
|
||||
- [ ] data_matches 有数字时,summary/why_today 已引用
|
||||
- [ ] command / url 与 type 一致
|
||||
- [ ] 未编造未检索到的事实
|
||||
134
tests/test_ai_news_research.py
Normal file
134
tests/test_ai_news_research.py
Normal file
@@ -0,0 +1,134 @@
|
||||
"""AI 时讯 deep-research 解析与企微格式。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from daily.format_wecom import _ai_news_lines, replace_wecom_news_sections
|
||||
from daily.news.research import (
|
||||
parse_research_response,
|
||||
research_pool_limit,
|
||||
research_tech_pool_limit,
|
||||
)
|
||||
|
||||
|
||||
class TestAiNewsResearchPool(unittest.TestCase):
|
||||
def test_pool_defaults_above_display(self):
|
||||
self.assertEqual(research_pool_limit(10), 20)
|
||||
self.assertEqual(research_tech_pool_limit(5), 10)
|
||||
|
||||
def test_pool_env_override(self):
|
||||
import os
|
||||
from unittest import mock
|
||||
|
||||
with mock.patch.dict(os.environ, {"DAILY_AI_NEWS_RESEARCH_POOL": "24"}, clear=False):
|
||||
self.assertEqual(research_pool_limit(10), 24)
|
||||
|
||||
|
||||
class TestAiNewsResearchParse(unittest.TestCase):
|
||||
def test_parse_items(self):
|
||||
raw = """
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"title": "Apple sues OpenAI",
|
||||
"link": "https://techcrunch.com/2026/07/10/apple/",
|
||||
"source_name": "TechCrunch",
|
||||
"desc_short": "苹果起诉 OpenAI 涉嫌窃取商业机密",
|
||||
"published_fmt": "07-11 05:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
items, tech = parse_research_response(raw, limit=10)
|
||||
self.assertEqual(len(items), 1)
|
||||
self.assertEqual(items[0]["source_name"], "TechCrunch")
|
||||
self.assertIn("苹果", items[0]["desc_short"])
|
||||
self.assertEqual(tech, [])
|
||||
|
||||
def test_dedupe_links(self):
|
||||
raw = """{"items": [
|
||||
{"title": "A", "link": "https://example.com/a", "source_name": "Ex", "desc_short": "一"},
|
||||
{"title": "B", "link": "https://example.com/a", "source_name": "Ex", "desc_short": "二"}
|
||||
]}"""
|
||||
items, _ = parse_research_response(raw, limit=10)
|
||||
self.assertEqual(len(items), 1)
|
||||
|
||||
def test_parse_tech_items(self):
|
||||
raw = """{"items": [
|
||||
{"title": "A", "link": "https://example.com/a", "source_name": "Ex", "desc_short": "一"}
|
||||
], "tech_items": [
|
||||
{"title": "B", "link": "https://example.com/b", "source_name": "Ex", "desc_short": "二"}
|
||||
]}"""
|
||||
items, tech = parse_research_response(raw, limit=10, tech_limit=5)
|
||||
self.assertEqual(len(items), 1)
|
||||
self.assertEqual(len(tech), 1)
|
||||
self.assertEqual(tech[0]["title"], "B")
|
||||
|
||||
|
||||
class TestMergedWecomNews(unittest.TestCase):
|
||||
def test_merged_block_format(self):
|
||||
items = [
|
||||
{
|
||||
"title": "Apple sues OpenAI",
|
||||
"link": "https://techcrunch.com/x",
|
||||
"source_name": "TechCrunch",
|
||||
"desc_short": "苹果起诉 OpenAI",
|
||||
"published_fmt": "07-11 05:00",
|
||||
}
|
||||
]
|
||||
lines = _ai_news_lines(items, merged=True)
|
||||
self.assertIn("TechCrunch - Apple sues OpenAI", lines[0])
|
||||
self.assertIn("— 苹果起诉 OpenAI", lines[0])
|
||||
self.assertNotIn("07-11", lines[0])
|
||||
|
||||
def test_merged_with_tech_block(self):
|
||||
md = """📰 **早报**
|
||||
|
||||
📈 **Skills Trending Top 1**
|
||||
1. skill
|
||||
"""
|
||||
main = [
|
||||
{"title": "Main", "link": "https://example.com/m", "source_name": "Src", "desc_short": "主条", "published_fmt": "07-11"}
|
||||
]
|
||||
tech = [
|
||||
{"title": "Tech", "link": "https://example.com/t", "source_name": "Src2", "desc_short": "技术条", "published_fmt": "07-12"}
|
||||
]
|
||||
out = replace_wecom_news_sections(md, ai_news=main, tech_ai_news=tech, merged=True)
|
||||
self.assertIn("📰 **AI 时讯精选 Top 2**", out)
|
||||
self.assertNotIn("技术类时讯", out)
|
||||
self.assertNotIn("🔧", out)
|
||||
self.assertNotIn("07-11", out)
|
||||
self.assertNotIn("07-12", out)
|
||||
self.assertIn("2. [Src2 - Tech]", out)
|
||||
|
||||
def test_replace_merged_removes_split_blocks(self):
|
||||
md = """📰 **早报**
|
||||
|
||||
🌍 **国际 AI 时讯 Top 1**
|
||||
1. [old](https://example.com/old) · `X`
|
||||
|
||||
🇨🇳 **国内 AI 时讯 Top 1**
|
||||
1. [old2](https://example.com/old2) · `Y`
|
||||
|
||||
📈 **Skills Trending Top 1**
|
||||
1. skill
|
||||
"""
|
||||
items = [
|
||||
{
|
||||
"title": "New story",
|
||||
"link": "https://example.com/new",
|
||||
"source_name": "Fortune",
|
||||
"desc_short": "新故事",
|
||||
"published_fmt": "",
|
||||
}
|
||||
]
|
||||
out = replace_wecom_news_sections(md, ai_news=items, merged=True)
|
||||
self.assertIn("📰 **AI 时讯精选 Top 1**", out)
|
||||
self.assertNotIn("国际 AI 时讯", out)
|
||||
self.assertNotIn("国内 AI 时讯", out)
|
||||
self.assertIn("📈 **Skills Trending Top 1**", out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
293
tests/test_ai_news_research_quality.py
Normal file
293
tests/test_ai_news_research_quality.py
Normal file
@@ -0,0 +1,293 @@
|
||||
"""Research 时讯质量:可信源、同事件去重、tech 主题过滤、国内配额。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from daily.news.research_quality import (
|
||||
build_deduped_candidate_pool,
|
||||
dedupe_same_event,
|
||||
filter_tech_against_items,
|
||||
is_cn_item,
|
||||
is_trusted_item,
|
||||
pack_with_cn_quota,
|
||||
post_process_research_news,
|
||||
research_cn_min,
|
||||
)
|
||||
|
||||
|
||||
def _item(
|
||||
title: str,
|
||||
link: str,
|
||||
*,
|
||||
source_name: str = "",
|
||||
desc_short: str = "",
|
||||
region: str | None = None,
|
||||
) -> dict:
|
||||
row = {
|
||||
"title": title,
|
||||
"link": link,
|
||||
"source_name": source_name,
|
||||
"desc_short": desc_short,
|
||||
"summary_plain": desc_short,
|
||||
"published_fmt": "",
|
||||
}
|
||||
if region is not None:
|
||||
row["region"] = region
|
||||
return row
|
||||
|
||||
|
||||
class TestTrustedAndCn(unittest.TestCase):
|
||||
def test_trusted_official_and_authority(self):
|
||||
self.assertTrue(
|
||||
is_trusted_item(_item("x", "https://openai.com/blog/x", source_name="OpenAI"))
|
||||
)
|
||||
self.assertTrue(
|
||||
is_trusted_item(_item("x", "https://techcrunch.com/a", source_name="TechCrunch"))
|
||||
)
|
||||
self.assertTrue(
|
||||
is_trusted_item(_item("x", "https://www.qbitai.com/a", source_name="量子位"))
|
||||
)
|
||||
|
||||
def test_rejects_low_quality(self):
|
||||
self.assertFalse(
|
||||
is_trusted_item(
|
||||
_item("x", "https://wap.stockstar.com/detail/IG1", source_name="证券之星")
|
||||
)
|
||||
)
|
||||
self.assertFalse(
|
||||
is_trusted_item(_item("x", "https://random-blog.xyz/a", source_name="Unknown"))
|
||||
)
|
||||
self.assertFalse(
|
||||
is_trusted_item(
|
||||
_item("x", "https://www.techtimes.com/articles/1.htm", source_name="TechTimes")
|
||||
)
|
||||
)
|
||||
|
||||
def test_trusted_cn_majors_and_engadget(self):
|
||||
self.assertTrue(
|
||||
is_trusted_item(_item("x", "https://www.yicai.com/news/1.html", source_name="第一财经"))
|
||||
)
|
||||
self.assertTrue(
|
||||
is_trusted_item(
|
||||
_item("x", "https://www.news.cn/world/20260729/a/c.html", source_name="新华网")
|
||||
)
|
||||
)
|
||||
self.assertTrue(
|
||||
is_trusted_item(
|
||||
_item("x", "https://www.engadget.com/2225849/google/", source_name="Engadget")
|
||||
)
|
||||
)
|
||||
|
||||
def test_cn_by_whitelist_region_and_cn_tld(self):
|
||||
self.assertTrue(is_cn_item(_item("x", "https://www.qbitai.com/a", source_name="量子位")))
|
||||
self.assertTrue(
|
||||
is_cn_item(_item("x", "https://techcrunch.com/a", source_name="TechCrunch", region="cn"))
|
||||
)
|
||||
self.assertTrue(is_cn_item(_item("x", "https://news.example.cn/a", source_name="X")))
|
||||
self.assertFalse(is_cn_item(_item("x", "https://techcrunch.com/a", source_name="TechCrunch")))
|
||||
|
||||
|
||||
class TestSameEventAndTech(unittest.TestCase):
|
||||
def test_petition_cluster_keeps_one(self):
|
||||
items = [
|
||||
_item(
|
||||
"OpenAI, Anthropic scientists ask U.S. for tools to pace AI development",
|
||||
"https://www.nbcnews.com/tech/a",
|
||||
source_name="NBC News",
|
||||
desc_short="超千名前沿实验室员工联名,吁美政府支持控制 AI 研发节奏",
|
||||
),
|
||||
_item(
|
||||
"Sam Altman is ready to decelerate",
|
||||
"https://techcrunch.com/2026/07/28/sam-altman-is-ready-to-decelerate/",
|
||||
source_name="TechCrunch",
|
||||
desc_short="奥特曼称或需控制 AI 发展速度,并支持员工联名请愿",
|
||||
),
|
||||
]
|
||||
out = dedupe_same_event(items)
|
||||
self.assertEqual(len(out), 1)
|
||||
|
||||
def test_distinct_clusters_kept(self):
|
||||
items = [
|
||||
_item(
|
||||
"Sam Altman is ready to decelerate",
|
||||
"https://techcrunch.com/a",
|
||||
source_name="TechCrunch",
|
||||
desc_short="奥特曼称或需控制 AI 发展速度",
|
||||
),
|
||||
_item(
|
||||
"OpenAI’s agent siege forced rebuild at Hugging Face",
|
||||
"https://www.theregister.com/ai/a",
|
||||
source_name="The Register",
|
||||
desc_short="Hugging Face 因 OpenAI 智能体入侵重建基础设施",
|
||||
),
|
||||
]
|
||||
out = dedupe_same_event(items)
|
||||
self.assertEqual(len(out), 2)
|
||||
|
||||
def test_tech_drops_kimi_adapt_when_items_have_kimi_open_source(self):
|
||||
items = [
|
||||
_item(
|
||||
"Moonshot Open-Sources Kimi K3",
|
||||
"https://www.caixinglobal.com/a",
|
||||
source_name="Caixin",
|
||||
desc_short="月之暗面开放 Kimi K3 权重与技术报告",
|
||||
)
|
||||
]
|
||||
tech = [
|
||||
_item(
|
||||
"moonshotai/Kimi-K3 · Hugging Face",
|
||||
"https://huggingface.co/moonshotai/Kimi-K3",
|
||||
source_name="Hugging Face",
|
||||
desc_short="Kimi K3 开源权重上线",
|
||||
),
|
||||
_item(
|
||||
"华为官宣昇腾 Day0 支持 Kimi K3",
|
||||
"https://www.ithome.com/0/982/615.htm",
|
||||
source_name="IT之家",
|
||||
desc_short="昇腾宣布适配 Kimi K3 训练与推理",
|
||||
),
|
||||
_item(
|
||||
"MCP Specification 2026-07-28",
|
||||
"https://blog.modelcontextprotocol.io/posts/2026-07-28/",
|
||||
source_name="MCP Blog",
|
||||
desc_short="MCP 正式发布新规范",
|
||||
),
|
||||
]
|
||||
kept = filter_tech_against_items(tech, items)
|
||||
self.assertEqual(len(kept), 1)
|
||||
self.assertIn("MCP", kept[0]["title"])
|
||||
|
||||
|
||||
class TestDedupedCandidatePool(unittest.TestCase):
|
||||
def test_pool_is_unique_events_after_fetch(self):
|
||||
items = [
|
||||
_item(
|
||||
"OpenAI, Anthropic scientists ask U.S. for tools to pace AI development",
|
||||
"https://www.nbcnews.com/tech/a",
|
||||
source_name="NBC News",
|
||||
desc_short="超千名前沿实验室员工联名,吁美政府支持控制 AI 研发节奏",
|
||||
),
|
||||
_item(
|
||||
"Sam Altman is ready to decelerate",
|
||||
"https://techcrunch.com/2026/07/28/sam-altman-is-ready-to-decelerate/",
|
||||
source_name="TechCrunch",
|
||||
desc_short="奥特曼称或需控制 AI 发展速度,并支持员工联名请愿",
|
||||
),
|
||||
_item(
|
||||
"junk",
|
||||
"https://wap.stockstar.com/detail/1",
|
||||
source_name="证券之星",
|
||||
desc_short="营销稿",
|
||||
),
|
||||
]
|
||||
pool, tech = build_deduped_candidate_pool(items, [])
|
||||
self.assertEqual(tech, [])
|
||||
self.assertEqual(len(pool), 1)
|
||||
self.assertTrue(all(is_trusted_item(i) for i in pool))
|
||||
|
||||
|
||||
class TestCnQuota(unittest.TestCase):
|
||||
def test_cn_min_default_30_percent(self):
|
||||
with mock.patch.dict("os.environ", {"DAILY_WECOM_AI_NEWS_CN_MIN": "0"}, clear=False):
|
||||
self.assertEqual(research_cn_min(10), 3)
|
||||
with mock.patch.dict("os.environ", {"DAILY_WECOM_AI_NEWS_CN_MIN": "4"}, clear=False):
|
||||
self.assertEqual(research_cn_min(10), 4)
|
||||
|
||||
def test_pack_reserves_cn_slots(self):
|
||||
items = [
|
||||
_item("I1", "https://techcrunch.com/1", source_name="TechCrunch", desc_short="国际1"),
|
||||
_item("I2", "https://techcrunch.com/2", source_name="TechCrunch", desc_short="国际2"),
|
||||
_item("I3", "https://techcrunch.com/3", source_name="TechCrunch", desc_short="国际3"),
|
||||
_item("C1", "https://www.qbitai.com/1", source_name="量子位", desc_short="国内1"),
|
||||
_item("C2", "https://www.36kr.com/1", source_name="36氪", desc_short="国内2"),
|
||||
_item("C3", "https://www.jiqizhixin.com/1", source_name="机器之心", desc_short="国内3"),
|
||||
]
|
||||
out = pack_with_cn_quota(items, limit=5, min_cn=3)
|
||||
self.assertEqual(len(out), 5)
|
||||
self.assertGreaterEqual(sum(1 for i in out if is_cn_item(i)), 3)
|
||||
|
||||
|
||||
class TestPostProcessIntegration(unittest.TestCase):
|
||||
def test_sample_day_filters_stockstar_and_dedupes(self):
|
||||
sample = Path(__file__).resolve().parents[1] / "output" / "2026-07-29.ai-news-research.json"
|
||||
if not sample.exists():
|
||||
self.skipTest("sample research json missing")
|
||||
raw = json.loads(sample.read_text(encoding="utf-8"))
|
||||
items = [
|
||||
_item(
|
||||
r["title"],
|
||||
r["link"],
|
||||
source_name=r.get("source_name", ""),
|
||||
desc_short=r.get("desc_short", ""),
|
||||
)
|
||||
for r in raw["items"]
|
||||
]
|
||||
tech = [
|
||||
_item(
|
||||
r["title"],
|
||||
r["link"],
|
||||
source_name=r.get("source_name", ""),
|
||||
desc_short=r.get("desc_short", ""),
|
||||
)
|
||||
for r in raw["tech_items"]
|
||||
]
|
||||
out_items, out_tech = post_process_research_news(
|
||||
items, tech, limit=10, tech_limit=5, min_cn=3
|
||||
)
|
||||
links = {i["link"] for i in out_items + out_tech}
|
||||
self.assertTrue(all("stockstar" not in link for link in links))
|
||||
# 联名/减速同簇只留一条
|
||||
petitionish = [
|
||||
i
|
||||
for i in out_items
|
||||
if "decelerat" in i["title"].lower()
|
||||
or "联名" in (i.get("desc_short") or "")
|
||||
or "pace AI" in i["title"]
|
||||
]
|
||||
self.assertLessEqual(len(petitionish), 1)
|
||||
# Kimi 适配不应再堆在 tech
|
||||
kimi_tech = [t for t in out_tech if "kimi" in (t["title"] + t.get("desc_short", "")).lower()]
|
||||
self.assertEqual(kimi_tech, [])
|
||||
|
||||
def test_sample_2026_07_30_keeps_cn_majors(self):
|
||||
sample = Path(__file__).resolve().parents[1] / "output" / "2026-07-30.ai-news-research.json"
|
||||
if not sample.exists():
|
||||
self.skipTest("sample research json missing")
|
||||
raw = json.loads(sample.read_text(encoding="utf-8"))
|
||||
items = [
|
||||
_item(
|
||||
r["title"],
|
||||
r["link"],
|
||||
source_name=r.get("source_name", ""),
|
||||
desc_short=r.get("desc_short", ""),
|
||||
region=r.get("region"),
|
||||
)
|
||||
for r in raw["items"]
|
||||
]
|
||||
tech = [
|
||||
_item(
|
||||
r["title"],
|
||||
r["link"],
|
||||
source_name=r.get("source_name", ""),
|
||||
desc_short=r.get("desc_short", ""),
|
||||
region=r.get("region"),
|
||||
)
|
||||
for r in raw["tech_items"]
|
||||
]
|
||||
out_items, out_tech = post_process_research_news(
|
||||
items, tech, limit=10, tech_limit=5, min_cn=3
|
||||
)
|
||||
# 不应再被白名单误杀成只剩 1 条
|
||||
self.assertGreaterEqual(len(out_items) + len(out_tech), 4)
|
||||
self.assertGreaterEqual(sum(1 for i in out_items if is_cn_item(i)), 2)
|
||||
hosts = " ".join(i["link"] for i in out_items + out_tech)
|
||||
self.assertNotIn("techtimes.com", hosts)
|
||||
self.assertTrue("yicai.com" in hosts or "news.cn" in hosts)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
140
tests/test_board_history.py
Normal file
140
tests/test_board_history.py
Normal file
@@ -0,0 +1,140 @@
|
||||
# tests/test_board_history.py
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from daily.board_history import extract_shown_keys, load_recent_shown_keys, merge_wecom_shown_into_data
|
||||
from daily.config import board_dedup_days, news_backfill_enabled
|
||||
|
||||
|
||||
class ConfigDiversityTests(unittest.TestCase):
|
||||
def test_board_dedup_days_default(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertEqual(board_dedup_days(), 7)
|
||||
|
||||
def test_news_backfill_default_off(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertFalse(news_backfill_enabled())
|
||||
|
||||
|
||||
class ShownKeysTests(unittest.TestCase):
|
||||
def test_extract_github_repo_keys(self):
|
||||
items = [{"repo": "a/b"}, {"repo": "c/d"}]
|
||||
self.assertEqual(extract_shown_keys("github_trending", items), ["a/b", "c/d"])
|
||||
|
||||
def test_extract_skill_keys_include_source(self):
|
||||
items = [
|
||||
{
|
||||
"id": "open.feishu.cn/lark-drive",
|
||||
"source": "open.feishu.cn",
|
||||
"title": "lark-drive",
|
||||
}
|
||||
]
|
||||
self.assertEqual(
|
||||
extract_shown_keys("skills_trending", items),
|
||||
["open.feishu.cn/lark-drive", "open.feishu.cn"],
|
||||
)
|
||||
|
||||
def test_load_recent_reads_wecom_shown_not_baseline(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out = Path(tmp)
|
||||
# 前日:shown 只有 x/y;baseline raw 含 a/b —— 周去重只能看到 x/y
|
||||
payload = {
|
||||
"data": {
|
||||
"date": "2026-07-13",
|
||||
"movement_baseline": {
|
||||
"github_trending": [{"repo": "a/b"}, {"repo": "x/y"}],
|
||||
},
|
||||
"wecom_shown_keys": {"github_trending": ["x/y"]},
|
||||
}
|
||||
}
|
||||
(out / "2026-07-13.data.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
with patch("daily.board_history.OUTPUT_DIR", out):
|
||||
keys = load_recent_shown_keys("2026-07-14", lookback_days=7)
|
||||
self.assertEqual(keys["github_trending"], {"x/y"})
|
||||
self.assertNotIn("a/b", keys["github_trending"])
|
||||
|
||||
def test_load_recent_falls_back_to_wecom_md_when_shown_missing(self):
|
||||
"""旧日 data 无 wecom_shown_keys 时,从同日 wecom.md 解析实际展示 keys。"""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out = Path(tmp)
|
||||
payload = {
|
||||
"data": {
|
||||
"date": "2026-07-13",
|
||||
"github_trending": [{"repo": "other/top"}],
|
||||
}
|
||||
}
|
||||
(out / "2026-07-13.data.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
(out / "2026-07-13.wecom.md").write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"🐙 **GitHub Trending Top 2**",
|
||||
"1. [vinta/awesome-python](https://github.com/vinta/awesome-python)",
|
||||
"2. [react/react](https://github.com/react/react)",
|
||||
"",
|
||||
"🌱 **GitHub 新兴 Top 1**",
|
||||
"1. [elder-plinius/T3MP3ST](https://github.com/elder-plinius/T3MP3ST)",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
with patch("daily.board_history.OUTPUT_DIR", out):
|
||||
keys = load_recent_shown_keys("2026-07-14", lookback_days=7)
|
||||
self.assertEqual(
|
||||
keys["github_trending"],
|
||||
{"vinta/awesome-python", "react/react"},
|
||||
)
|
||||
self.assertEqual(keys["github_emerging"], {"elder-plinius/T3MP3ST"})
|
||||
self.assertNotIn("other/top", keys["github_trending"])
|
||||
|
||||
def test_merge_shown_does_not_touch_baseline(self):
|
||||
data = {
|
||||
"movement_baseline": {"github_trending": [{"repo": "raw/one"}]},
|
||||
}
|
||||
merged = merge_wecom_shown_into_data(
|
||||
data, {"github_trending": ["shown/one"]}
|
||||
)
|
||||
self.assertEqual(
|
||||
merged["movement_baseline"]["github_trending"][0]["repo"], "raw/one"
|
||||
)
|
||||
self.assertEqual(merged["wecom_shown_keys"]["github_trending"], ["shown/one"])
|
||||
|
||||
def test_persist_shown_keys_differs_from_baseline_keys(self):
|
||||
from daily.board_select import board_select
|
||||
|
||||
raw = [{"repo": f"o/r{i}"} for i in range(10)]
|
||||
recent = {f"o/r{i}" for i in range(3)}
|
||||
selected = board_select(
|
||||
board="github_trending",
|
||||
items=raw,
|
||||
recent_keys=recent,
|
||||
limit=5,
|
||||
pool_size=50,
|
||||
kind="github",
|
||||
)
|
||||
baseline_keys = [x["repo"] for x in raw[:5]]
|
||||
shown = extract_shown_keys("github_trending", selected)
|
||||
data = {
|
||||
"movement_baseline": {
|
||||
"github_trending": [{"repo": k} for k in baseline_keys],
|
||||
},
|
||||
}
|
||||
merged = merge_wecom_shown_into_data(data, {"github_trending": shown})
|
||||
self.assertNotEqual(
|
||||
set(merged["wecom_shown_keys"]["github_trending"]),
|
||||
{x["repo"] for x in merged["movement_baseline"]["github_trending"]},
|
||||
)
|
||||
self.assertEqual(shown, ["o/r3", "o/r4", "o/r5", "o/r6", "o/r7"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
91
tests/test_board_select.py
Normal file
91
tests/test_board_select.py
Normal file
@@ -0,0 +1,91 @@
|
||||
# tests/test_board_select.py
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
|
||||
from daily.board_select import board_select
|
||||
|
||||
|
||||
def _gh(repo: str) -> dict:
|
||||
return {"repo": repo, "description": repo}
|
||||
|
||||
|
||||
class BoardSelectTests(unittest.TestCase):
|
||||
def test_filters_recent_and_keeps_order(self):
|
||||
pool = [_gh(f"o/r{i}") for i in range(20)]
|
||||
recent = {"o/r0", "o/r1", "o/r2"}
|
||||
out = board_select(
|
||||
board="github_trending",
|
||||
items=pool,
|
||||
recent_keys=recent,
|
||||
limit=5,
|
||||
pool_size=20,
|
||||
kind="github",
|
||||
)
|
||||
keys = [x["repo"] for x in out]
|
||||
self.assertEqual(keys, ["o/r3", "o/r4", "o/r5", "o/r6", "o/r7"])
|
||||
|
||||
def test_deep_pool_fills_after_filter(self):
|
||||
pool = [_gh(f"o/r{i}") for i in range(8)]
|
||||
recent = {f"o/r{i}" for i in range(6)} # 前 6 全封
|
||||
out = board_select(
|
||||
board="github_emerging",
|
||||
items=pool,
|
||||
recent_keys=recent,
|
||||
limit=5,
|
||||
pool_size=8,
|
||||
kind="github",
|
||||
)
|
||||
self.assertEqual([x["repo"] for x in out], ["o/r6", "o/r7"]) # 短榜
|
||||
|
||||
def test_skill_uses_skill_id(self):
|
||||
items = [
|
||||
{"id": "a/b/s1", "source": "a/b", "title": "s1", "installs": 10},
|
||||
{"id": "c/d/s2", "source": "c/d", "title": "s2", "installs": 9},
|
||||
]
|
||||
out = board_select(
|
||||
board="skills_trending",
|
||||
items=items,
|
||||
recent_keys={"a/b/s1"},
|
||||
limit=10,
|
||||
pool_size=50,
|
||||
kind="skill",
|
||||
)
|
||||
self.assertEqual([x["id"] for x in out], ["c/d/s2"])
|
||||
|
||||
def test_skill_filters_recent_by_source(self):
|
||||
items = [
|
||||
{"id": "a/b/s-new", "source": "a/b", "title": "s-new", "installs": 10},
|
||||
{"id": "c/d/s2", "source": "c/d", "title": "s2", "installs": 9},
|
||||
]
|
||||
out = board_select(
|
||||
board="skills_hot",
|
||||
items=items,
|
||||
recent_keys={"a/b"}, # source-level history
|
||||
limit=10,
|
||||
pool_size=50,
|
||||
kind="skill",
|
||||
)
|
||||
self.assertEqual([x["id"] for x in out], ["c/d/s2"])
|
||||
|
||||
def test_skill_filters_recent_skill_id_as_same_source(self):
|
||||
from daily.format_wecom import expand_skill_recent_keys
|
||||
|
||||
items = [
|
||||
{"id": "open.feishu.cn/lark-drive", "source": "open.feishu.cn", "title": "lark-drive", "installs": 10},
|
||||
{"id": "fresh/src/s", "source": "fresh/src", "title": "s", "installs": 9},
|
||||
]
|
||||
recent = expand_skill_recent_keys({"open.feishu.cn/lark-doc"})
|
||||
out = board_select(
|
||||
board="skills_trending",
|
||||
items=items,
|
||||
recent_keys=recent,
|
||||
limit=10,
|
||||
pool_size=50,
|
||||
kind="skill",
|
||||
)
|
||||
self.assertEqual([x["id"] for x in out], ["fresh/src/s"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
95
tests/test_featured_pick.py
Normal file
95
tests/test_featured_pick.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""Tests for daily.featured_pick."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from daily.featured_pick import (
|
||||
apply_featured_pick,
|
||||
match_in_data,
|
||||
parse_featured_pick,
|
||||
pick_command_from_featured,
|
||||
pick_why_from_featured,
|
||||
research_featured_pick,
|
||||
)
|
||||
|
||||
|
||||
SAMPLE_INPUT = {
|
||||
"skills_trending": [
|
||||
{
|
||||
"id": "foo/bar/gstack-cli",
|
||||
"title": "gstack-cli",
|
||||
"source": "foo/bar",
|
||||
"installs": 1200,
|
||||
"installs_fmt": "1.2K",
|
||||
"link": "https://skills.sh/foo/bar/gstack-cli",
|
||||
"description": "Agent workflow CLI",
|
||||
}
|
||||
],
|
||||
"skills_hot": [],
|
||||
"github_trending": [
|
||||
{
|
||||
"repo": "acme/gstack",
|
||||
"url": "https://github.com/acme/gstack",
|
||||
"total_stars_fmt": "3.2K",
|
||||
"description": "GStack toolkit",
|
||||
}
|
||||
],
|
||||
"github_emerging": [],
|
||||
"github_topic": {"topic": "llm", "repos": []},
|
||||
}
|
||||
|
||||
|
||||
class ParseFeaturedPickTests(unittest.TestCase):
|
||||
def test_empty(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertIsNone(parse_featured_pick())
|
||||
|
||||
def test_query_only(self):
|
||||
with patch.dict(os.environ, {"DAILY_FEATURED_PICK": "gstack"}, clear=True):
|
||||
self.assertEqual(parse_featured_pick(), {"query": "gstack"})
|
||||
|
||||
def test_query_with_url(self):
|
||||
with patch.dict(os.environ, {"DAILY_FEATURED_PICK": "gstack|https://example.com"}, clear=True):
|
||||
self.assertEqual(
|
||||
parse_featured_pick(),
|
||||
{"query": "gstack", "url_hint": "https://example.com"},
|
||||
)
|
||||
|
||||
|
||||
class MatchInDataTests(unittest.TestCase):
|
||||
def test_matches_skill_and_github(self):
|
||||
matches = match_in_data(SAMPLE_INPUT, "gstack")
|
||||
self.assertEqual(len(matches["skills"]), 1)
|
||||
self.assertEqual(matches["skills"][0]["title"], "gstack-cli")
|
||||
self.assertEqual(len(matches["github"]), 1)
|
||||
self.assertEqual(matches["github"][0]["repo"], "acme/gstack")
|
||||
|
||||
|
||||
class FeaturedPickWorkflowTests(unittest.TestCase):
|
||||
def test_fallback_without_llm(self):
|
||||
llm_input = dict(SAMPLE_INPUT)
|
||||
with patch.dict(os.environ, {"DAILY_FEATURED_PICK": "gstack"}, clear=True):
|
||||
with patch("daily.featured_pick.has_llm_configured", return_value=False):
|
||||
featured = research_featured_pick(llm_input, date_str="2026-07-03")
|
||||
self.assertIsNotNone(featured)
|
||||
assert featured is not None
|
||||
self.assertEqual(featured["type"], "skill")
|
||||
self.assertIn("npx skills add foo/bar/gstack-cli", featured["command"])
|
||||
self.assertTrue(featured["why_today"])
|
||||
|
||||
def test_apply_featured_pick_mutates_input(self):
|
||||
llm_input = dict(SAMPLE_INPUT)
|
||||
with patch.dict(os.environ, {"DAILY_FEATURED_PICK": "gstack"}, clear=True):
|
||||
with patch("daily.featured_pick.has_llm_configured", return_value=False):
|
||||
featured = apply_featured_pick(llm_input, date_str="2026-07-03")
|
||||
self.assertIsNotNone(featured)
|
||||
self.assertIn("featured_pick", llm_input)
|
||||
self.assertEqual(pick_command_from_featured(featured), llm_input["featured_pick"]["command"])
|
||||
self.assertEqual(pick_why_from_featured(featured), llm_input["featured_pick"]["why_today"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
55
tests/test_featured_reason.py
Normal file
55
tests/test_featured_reason.py
Normal file
@@ -0,0 +1,55 @@
|
||||
# tests/test_featured_reason.py
|
||||
"""T6: 首推理由行(pick_why + DAILY_FEATURED_REASON 开关)测试。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import daily.format_wecom as fw
|
||||
|
||||
|
||||
def _build(pick_why: str) -> str:
|
||||
return fw.build_wecom_report(
|
||||
date_str="2026-07-18",
|
||||
time_str="08:50 (UTC+8)",
|
||||
updated="2026-07-18",
|
||||
highlights=[],
|
||||
theme_line="**今日主题**:测试",
|
||||
ai_news=None,
|
||||
cn_ai_news=None,
|
||||
merged_ai_news=None,
|
||||
merged_tech_ai_news=None,
|
||||
trending=[],
|
||||
hot=[],
|
||||
repos=[],
|
||||
emerging=[],
|
||||
topic_name="t",
|
||||
topic_repos=[],
|
||||
pick_command="npx skills add src/alpha",
|
||||
pick_why=pick_why,
|
||||
pick_title="alpha",
|
||||
pick_url="https://x/a",
|
||||
include_boards=False,
|
||||
)
|
||||
|
||||
|
||||
class FeaturedReasonTests(unittest.TestCase):
|
||||
def test_reason_shown_when_present_and_enabled(self):
|
||||
with mock.patch.object(fw, "env_bool", return_value=True):
|
||||
md = _build("昨日 star 增速第一")
|
||||
self.assertIn("> 昨日 star 增速第一", md)
|
||||
|
||||
def test_reason_hidden_when_switch_off(self):
|
||||
with mock.patch.object(fw, "env_bool", return_value=False):
|
||||
md = _build("昨日 star 增速第一")
|
||||
self.assertNotIn("> 昨日 star 增速第一", md)
|
||||
|
||||
def test_reason_omitted_when_empty(self):
|
||||
with mock.patch.object(fw, "env_bool", return_value=True):
|
||||
md = _build("")
|
||||
self.assertIn("今日首推", md)
|
||||
self.assertNotIn("> \n", md)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
132
tests/test_featured_resolve.py
Normal file
132
tests/test_featured_resolve.py
Normal file
@@ -0,0 +1,132 @@
|
||||
# tests/test_featured_resolve.py
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from daily.featured_pick import (
|
||||
featured_identity_key,
|
||||
featured_resolve,
|
||||
load_recent_featured_keys,
|
||||
load_yesterday_featured_key,
|
||||
)
|
||||
|
||||
|
||||
class FeaturedResolveTests(unittest.TestCase):
|
||||
def test_same_as_yesterday_picks_from_pool_a(self):
|
||||
yesterday_key = "headroomlabs-ai/headroom"
|
||||
pool_a = [
|
||||
{"repo": "headroomlabs-ai/headroom", "board": "github_topic"},
|
||||
{"repo": "ollama/ollama", "board": "github_trending"},
|
||||
]
|
||||
rng = random.Random(0)
|
||||
resolved, key = featured_resolve(
|
||||
date_str="2026-07-14",
|
||||
candidate={
|
||||
"type": "github",
|
||||
"url": "https://github.com/headroomlabs-ai/headroom",
|
||||
"title": "headroom",
|
||||
},
|
||||
pool_a=pool_a,
|
||||
pool_b=[],
|
||||
recent_featured={yesterday_key},
|
||||
yesterday_key=yesterday_key,
|
||||
rng=rng,
|
||||
)
|
||||
self.assertNotEqual(key, yesterday_key)
|
||||
self.assertEqual(key, "ollama/ollama")
|
||||
self.assertIsNotNone(resolved)
|
||||
assert resolved is not None
|
||||
self.assertEqual(resolved.get("repo"), "ollama/ollama")
|
||||
|
||||
def test_pool_a_before_pool_b(self):
|
||||
yesterday_key = "blocked/one"
|
||||
pool_a = [{"repo": "pool-a/repo", "board": "github_trending"}]
|
||||
pool_b = [{"repo": "pool-b/repo", "board": "github_emerging"}]
|
||||
resolved, key = featured_resolve(
|
||||
date_str="2026-07-14",
|
||||
candidate={"type": "github", "repo": "blocked/one", "title": "one"},
|
||||
pool_a=pool_a,
|
||||
pool_b=pool_b,
|
||||
recent_featured={yesterday_key},
|
||||
yesterday_key=yesterday_key,
|
||||
rng=random.Random(1),
|
||||
)
|
||||
self.assertEqual(key, "pool-a/repo")
|
||||
assert resolved is not None
|
||||
self.assertEqual(resolved.get("repo"), "pool-a/repo")
|
||||
|
||||
def test_exhausted_keeps_original(self):
|
||||
yesterday_key = "only/one"
|
||||
candidate = {"type": "github", "repo": "only/one", "title": "one"}
|
||||
resolved, key = featured_resolve(
|
||||
date_str="2026-07-14",
|
||||
candidate=candidate,
|
||||
pool_a=[{"repo": "only/one", "board": "github_trending"}],
|
||||
pool_b=[],
|
||||
recent_featured={yesterday_key},
|
||||
yesterday_key=yesterday_key,
|
||||
rng=random.Random(2),
|
||||
)
|
||||
self.assertEqual(key, yesterday_key)
|
||||
self.assertEqual(resolved, candidate)
|
||||
|
||||
def test_identity_key_skill_and_github(self):
|
||||
self.assertEqual(
|
||||
featured_identity_key({"type": "skill", "id": "a/b/c"}),
|
||||
"a/b/c",
|
||||
)
|
||||
self.assertEqual(
|
||||
featured_identity_key(
|
||||
{"type": "github", "url": "https://github.com/foo/bar"}
|
||||
),
|
||||
"foo/bar",
|
||||
)
|
||||
|
||||
def test_load_yesterday_falls_back_to_featured_pick(self):
|
||||
"""缺 featured_pick_key 时从 featured_pick.url 推导身份,避免连日重复首推。"""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out = Path(tmp)
|
||||
payload = {
|
||||
"data": {
|
||||
"date": "2026-07-13",
|
||||
"featured_pick": {
|
||||
"type": "github",
|
||||
"title": "headroom",
|
||||
"url": "https://github.com/headroomlabs-ai/headroom",
|
||||
},
|
||||
}
|
||||
}
|
||||
(out / "2026-07-13.data.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
with patch("daily.featured_pick.OUTPUT_DIR", out):
|
||||
key = load_yesterday_featured_key("2026-07-14")
|
||||
self.assertEqual(key, "headroomlabs-ai/headroom")
|
||||
|
||||
def test_load_recent_falls_back_to_featured_pick(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out = Path(tmp)
|
||||
payload = {
|
||||
"data": {
|
||||
"date": "2026-07-13",
|
||||
"featured_pick": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/headroomlabs-ai/headroom",
|
||||
},
|
||||
}
|
||||
}
|
||||
(out / "2026-07-13.data.json").write_text(
|
||||
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
with patch("daily.featured_pick.OUTPUT_DIR", out):
|
||||
keys = load_recent_featured_keys("2026-07-14", days=7)
|
||||
self.assertIn("headroomlabs-ai/headroom", keys)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
103
tests/test_generate_golden.py
Normal file
103
tests/test_generate_golden.py
Normal file
@@ -0,0 +1,103 @@
|
||||
# tests/test_generate_golden.py
|
||||
"""黄金文件/确定性回归: 冻结时间、mock 网络与 LLM, 验证 generate_report
|
||||
在同一输入下产出逐字节一致(拆分 _collect/_select/_render 不得改变行为)。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import daily.generate as g
|
||||
|
||||
FIXED_NOW = datetime(2026, 7, 18, 8, 50, tzinfo=timezone(timedelta(hours=8)))
|
||||
|
||||
|
||||
def _feed() -> dict:
|
||||
return {
|
||||
"updatedAt": "2026-07-18T08:00:00",
|
||||
"topTrending": [
|
||||
{"id": "a", "source": "src", "title": "alpha", "description": "desc a",
|
||||
"installs": 100, "link": "https://x/a"},
|
||||
{"id": "b", "source": "src", "title": "beta", "description": "desc b",
|
||||
"installs": 90, "link": "https://x/b"},
|
||||
],
|
||||
"topHot": [
|
||||
{"id": "c", "source": "src", "title": "gamma", "description": "desc c",
|
||||
"installs": 80, "link": "https://x/c"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _patches(tmp: Path):
|
||||
"""集中 mock 所有外部边界: 网络/文件/时间/LLM。返回 patcher 列表。"""
|
||||
return [
|
||||
mock.patch.object(g, "_now_cst", return_value=FIXED_NOW),
|
||||
mock.patch.object(g, "load_feed", return_value=_feed()),
|
||||
mock.patch.object(g, "_load_snapshot", return_value=set()),
|
||||
mock.patch.object(g, "_save_snapshot", lambda *a, **k: None),
|
||||
mock.patch.object(g, "load_boards", return_value=(
|
||||
_feed()["topTrending"], _feed()["topHot"])),
|
||||
mock.patch.object(g, "fetch_github_trending", return_value=[]),
|
||||
mock.patch.object(g, "fetch_emerging_repos", return_value=[]),
|
||||
mock.patch.object(g, "fetch_topic_hot_repos", return_value=("topic", [])),
|
||||
mock.patch.object(g, "fetch_ai_news", return_value={
|
||||
"enabled": False, "categories": [], "flat": [], "stats": {}}),
|
||||
mock.patch.object(g, "fetch_cn_ai_news", return_value={
|
||||
"enabled": False, "categories": [], "flat": [], "stats": {}}),
|
||||
mock.patch.object(g, "is_research_mode", return_value=False),
|
||||
mock.patch.object(g, "is_agent_mode", return_value=False),
|
||||
mock.patch.object(g, "run_editorial", return_value=None),
|
||||
mock.patch.object(g, "cursor_editor_enabled", return_value=False),
|
||||
mock.patch.object(g, "record_pushed_links", lambda *a, **k: None),
|
||||
mock.patch.object(g, "save_json", lambda *a, **k: None),
|
||||
mock.patch.object(g, "load_recent_shown_keys", return_value={
|
||||
"skills_trending": set(), "skills_hot": set(),
|
||||
"github_trending": set(), "github_emerging": set(),
|
||||
"github_topic": set()}),
|
||||
mock.patch.object(g, "OUTPUT_DIR", tmp),
|
||||
mock.patch.object(g, "_localize_descriptions_in_place", lambda *a, **k: None),
|
||||
# featured_pick / push_gate 有独立测试;此处冻结以免触网(LLM)与读盘。
|
||||
mock.patch.object(g, "apply_featured_pick", return_value={
|
||||
"title": "alpha", "url": "https://x/a",
|
||||
"command": "npx skills add src/alpha", "why": "昨日 star 增速第一"}),
|
||||
mock.patch.object(g, "evaluate_push_gate", return_value=mock.Mock(
|
||||
should_push=False, silent=True, reasons=["golden-mock"])),
|
||||
]
|
||||
|
||||
|
||||
class GenerateGoldenTests(unittest.TestCase):
|
||||
def _run(self, tmp: Path) -> tuple[str, str]:
|
||||
patches = _patches(tmp)
|
||||
for p in patches:
|
||||
p.start()
|
||||
try:
|
||||
markdown, wecom_md, _md, _we = g.generate_report()
|
||||
return markdown, wecom_md
|
||||
finally:
|
||||
for p in patches:
|
||||
try:
|
||||
p.stop()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def test_deterministic_same_input_same_output(self):
|
||||
import tempfile
|
||||
with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
|
||||
md1, we1 = self._run(Path(d1))
|
||||
md2, we2 = self._run(Path(d2))
|
||||
self.assertEqual(md1, md2, "完整版 markdown 在相同输入下必须逐字节一致")
|
||||
self.assertEqual(we1, we2, "企微 wecom_md 在相同输入下必须逐字节一致")
|
||||
|
||||
def test_output_contains_core_sections(self):
|
||||
import tempfile
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
markdown, wecom_md = self._run(Path(d))
|
||||
self.assertIn("# 早报 · 2026-07-18", markdown)
|
||||
self.assertIn("主题聚类", markdown)
|
||||
self.assertIsInstance(wecom_md, str)
|
||||
self.assertTrue(len(wecom_md) > 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
71
tests/test_github_search.py
Normal file
71
tests/test_github_search.py
Normal file
@@ -0,0 +1,71 @@
|
||||
"""Tests for GitHub Search pagination / deep pool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class SearchGithubReposPaginationTests(unittest.TestCase):
|
||||
def test_search_paginates_beyond_first_page_of_30(self):
|
||||
from daily.github.search import search_github_repos
|
||||
|
||||
def make_items(start: int, n: int) -> list[dict]:
|
||||
return [
|
||||
{
|
||||
"full_name": f"org/repo{i}",
|
||||
"html_url": f"https://github.com/org/repo{i}",
|
||||
"description": f"desc {i}",
|
||||
"language": "Python",
|
||||
"stargazers_count": 1000 - i,
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
for i in range(start, start + n)
|
||||
]
|
||||
|
||||
responses = [
|
||||
MagicMock(status_code=200, json=lambda: {"items": make_items(1, 100)}),
|
||||
MagicMock(status_code=200, json=lambda: {"items": make_items(101, 50)}),
|
||||
]
|
||||
client = MagicMock()
|
||||
client.__enter__.return_value = client
|
||||
client.__exit__.return_value = False
|
||||
client.get.side_effect = responses
|
||||
|
||||
with patch.dict(os.environ, {"GITHUB_TOKEN": "test-token"}, clear=False):
|
||||
with patch("daily.github.search.httpx.Client", return_value=client):
|
||||
with patch("daily.github.search.github_token", return_value="test-token"):
|
||||
repos = search_github_repos("stars:>50", 120, require_token=True)
|
||||
|
||||
self.assertEqual(len(repos), 120)
|
||||
self.assertEqual(repos[0]["repo"], "org/repo1")
|
||||
self.assertEqual(repos[119]["repo"], "org/repo120")
|
||||
self.assertEqual(client.get.call_count, 2)
|
||||
first_params = client.get.call_args_list[0].kwargs["params"]
|
||||
self.assertEqual(first_params["per_page"], 100)
|
||||
self.assertEqual(first_params["page"], 1)
|
||||
|
||||
|
||||
class GithubBoardDeepPoolTests(unittest.TestCase):
|
||||
def test_board_select_fills_ten_when_deep_pool_has_fresh_repos(self):
|
||||
from daily.board_select import board_select
|
||||
|
||||
recent = {f"old/r{i}" for i in range(1, 33)}
|
||||
items = [{"repo": f"old/r{i}"} for i in range(1, 31)] + [
|
||||
{"repo": f"fresh/r{i}"} for i in range(1, 20)
|
||||
]
|
||||
selected = board_select(
|
||||
board="github_trending",
|
||||
items=items,
|
||||
recent_keys=recent,
|
||||
limit=10,
|
||||
pool_size=100,
|
||||
kind="github",
|
||||
)
|
||||
self.assertEqual(len(selected), 10)
|
||||
self.assertTrue(all(r["repo"].startswith("fresh/") for r in selected))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
57
tests/test_holiday.py
Normal file
57
tests/test_holiday.py
Normal file
@@ -0,0 +1,57 @@
|
||||
"""Tests for daily.holiday."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import date
|
||||
|
||||
from daily.holiday import is_workday, workday_name
|
||||
|
||||
# 模拟 2026 节假日表:1/1 元旦(休),1/4 元旦调休(补班)
|
||||
HOLIDAYS = {
|
||||
"2026-01-01": {"rest": True, "name": "元旦节"},
|
||||
"2026-01-02": {"rest": True, "name": "元旦节"},
|
||||
"2026-01-03": {"rest": True, "name": "元旦节"},
|
||||
"2026-01-04": {"rest": False, "name": "元旦节调休"},
|
||||
"2026-02-16": {"rest": True, "name": "春节"},
|
||||
"2026-02-14": {"rest": False, "name": "春节调休"},
|
||||
}
|
||||
|
||||
|
||||
class IsWorkdayTests(unittest.TestCase):
|
||||
def test_legal_holiday_is_rest(self):
|
||||
# 2026-01-01 周四,元旦 → 休息
|
||||
self.assertFalse(is_workday(date(2026, 1, 1), HOLIDAYS))
|
||||
|
||||
def test_tiaoxiu_makeup_day_is_workday(self):
|
||||
# 2026-01-04 周日,调休补班 → 上班
|
||||
self.assertEqual(date(2026, 1, 4).weekday(), 6)
|
||||
self.assertTrue(is_workday(date(2026, 1, 4), HOLIDAYS))
|
||||
|
||||
def test_spring_festival_holiday(self):
|
||||
self.assertFalse(is_workday(date(2026, 2, 16), HOLIDAYS))
|
||||
|
||||
def test_spring_festival_makeup_saturday(self):
|
||||
# 2026-02-14 周六,调休 → 上班
|
||||
self.assertEqual(date(2026, 2, 14).weekday(), 5)
|
||||
self.assertTrue(is_workday(date(2026, 2, 14), HOLIDAYS))
|
||||
|
||||
def test_normal_weekday(self):
|
||||
# 2026-07-23 周四,非节假日 → 上班
|
||||
self.assertTrue(is_workday(date(2026, 7, 23), HOLIDAYS))
|
||||
|
||||
def test_normal_weekend(self):
|
||||
# 2026-07-25 周六,非节假日 → 休息
|
||||
self.assertFalse(is_workday(date(2026, 7, 25), HOLIDAYS))
|
||||
|
||||
|
||||
class WorkdayNameTests(unittest.TestCase):
|
||||
def test_holiday_name(self):
|
||||
self.assertEqual(workday_name(date(2026, 1, 1), HOLIDAYS), "元旦节")
|
||||
|
||||
def test_normal_day_has_no_name(self):
|
||||
self.assertIsNone(workday_name(date(2026, 7, 23), HOLIDAYS))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
34
tests/test_narrative_axis.py
Normal file
34
tests/test_narrative_axis.py
Normal file
@@ -0,0 +1,34 @@
|
||||
# tests/test_narrative_axis.py
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
import unittest
|
||||
|
||||
from daily.narrative_axis import (
|
||||
NARRATIVE_AXES,
|
||||
enforce_narrative_axis,
|
||||
pick_narrative_axis,
|
||||
)
|
||||
|
||||
|
||||
class NarrativeAxisTests(unittest.TestCase):
|
||||
def test_pick_excludes_used(self):
|
||||
used = {"政策监管", "模型发布", "工具链/Agent"}
|
||||
for _ in range(20):
|
||||
axis = pick_narrative_axis(used, rng=random.Random(1))
|
||||
self.assertNotIn(axis, used)
|
||||
self.assertIn(axis, NARRATIVE_AXES)
|
||||
|
||||
def test_enforce_overwrites_llm(self):
|
||||
trends = {"narrative_axis": "开源生态", "opening": "..."}
|
||||
out = enforce_narrative_axis(trends, "芯片算力")
|
||||
self.assertEqual(out["narrative_axis"], "芯片算力")
|
||||
|
||||
def test_pick_when_all_used_falls_back(self):
|
||||
used = set(NARRATIVE_AXES)
|
||||
axis = pick_narrative_axis(used, rng=random.Random(0))
|
||||
self.assertIn(axis, NARRATIVE_AXES)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
156
tests/test_news_fetch_window.py
Normal file
156
tests/test_news_fetch_window.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""Tests for news time window filtering."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from unittest.mock import patch
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from daily.news.fetch import _cutoff_datetime, _parse_datetime, _within_window
|
||||
|
||||
|
||||
class NewsWindowTests(unittest.TestCase):
|
||||
def test_cutoff_floor_today_excludes_yesterday_even_within_24h(self):
|
||||
tz = ZoneInfo("Asia/Shanghai")
|
||||
# 2026-07-09 09:00 CST = 2026-07-09 01:00 UTC
|
||||
fixed = datetime(2026, 7, 9, 1, 0, tzinfo=timezone.utc)
|
||||
with patch("daily.news.fetch._now_utc", return_value=fixed):
|
||||
with patch.dict(os.environ, {"DAILY_AI_NEWS_HOURS": "24"}, clear=False):
|
||||
cutoff = _cutoff_datetime(floor_today=True)
|
||||
start_today_cst = datetime(2026, 7, 9, 0, 0, tzinfo=tz).astimezone(timezone.utc)
|
||||
self.assertEqual(cutoff, start_today_cst)
|
||||
yesterday = datetime(2026, 7, 8, 20, 0, tzinfo=tz).astimezone(timezone.utc)
|
||||
self.assertFalse(_within_window({"published": yesterday.isoformat()}, cutoff))
|
||||
|
||||
def test_cutoff_rolling_only_includes_last_24h(self):
|
||||
fixed = datetime(2026, 7, 9, 12, 0, tzinfo=timezone.utc)
|
||||
with patch("daily.news.fetch._now_utc", return_value=fixed):
|
||||
with patch.dict(os.environ, {"DAILY_AI_NEWS_HOURS": "24"}, clear=False):
|
||||
cutoff = _cutoff_datetime(floor_today=False)
|
||||
self.assertEqual(cutoff, fixed - timedelta(hours=24))
|
||||
|
||||
def test_within_window_rejects_missing_datetime(self):
|
||||
cutoff = datetime(2026, 7, 9, 0, 0, tzinfo=timezone.utc)
|
||||
self.assertFalse(_within_window({"title": "x", "link": "https://a.com"}, cutoff))
|
||||
|
||||
def test_parse_date_only_uses_local_noon(self):
|
||||
with patch.dict(os.environ, {"DAILY_AI_NEWS_TZ": "Asia/Shanghai"}, clear=False):
|
||||
dt = _parse_datetime("2026-07-09")
|
||||
self.assertIsNotNone(dt)
|
||||
assert dt is not None
|
||||
local = dt.astimezone(ZoneInfo("Asia/Shanghai"))
|
||||
self.assertEqual(local.hour, 12)
|
||||
|
||||
|
||||
class NewsFormatTests(unittest.TestCase):
|
||||
def test_ai_news_lines_use_desc_as_link_text(self):
|
||||
from daily.format_wecom import _ai_news_lines
|
||||
|
||||
lines = _ai_news_lines(
|
||||
[
|
||||
{
|
||||
"title": "English Title",
|
||||
"link": "https://example.com/a",
|
||||
"source_name": "Src",
|
||||
"published_fmt": "07-11",
|
||||
"desc_short": "中文摘要一句",
|
||||
}
|
||||
]
|
||||
)
|
||||
self.assertEqual(len(lines), 1)
|
||||
self.assertIn("[中文摘要一句](https://example.com/a)", lines[0])
|
||||
self.assertNotIn("English Title", lines[0])
|
||||
self.assertNotIn("> ", lines[0])
|
||||
|
||||
def test_ai_news_lines_fallback_to_title(self):
|
||||
from daily.format_wecom import _ai_news_lines
|
||||
|
||||
lines = _ai_news_lines(
|
||||
[
|
||||
{
|
||||
"title": "仅标题",
|
||||
"link": "https://example.com/b",
|
||||
"source_name": "Src",
|
||||
"published_fmt": "",
|
||||
"desc_short": "",
|
||||
}
|
||||
]
|
||||
)
|
||||
self.assertIn("[仅标题](https://example.com/b)", lines[0])
|
||||
|
||||
|
||||
class NewsSummaryTests(unittest.TestCase):
|
||||
def test_brief_news_summary_no_ellipsis(self):
|
||||
from daily.news.fetch import brief_news_summary
|
||||
|
||||
text = (
|
||||
"Meta told Dylan Byers, of Puck News, that the company removed "
|
||||
"the controversial AI feature after user backlash on Instagram."
|
||||
)
|
||||
out = brief_news_summary(text, limit=72)
|
||||
self.assertNotIn("...", out)
|
||||
self.assertLessEqual(len(out), 72)
|
||||
self.assertTrue(out.startswith("Meta told"))
|
||||
|
||||
def test_brief_news_summary_filters_junk(self):
|
||||
from daily.news.fetch import brief_news_summary
|
||||
|
||||
self.assertEqual(brief_news_summary("点击查看原文>"), "")
|
||||
self.assertEqual(brief_news_summary("Article URL: https://example.com"), "")
|
||||
|
||||
def test_sync_wecom_news_rows_after_localize(self):
|
||||
from daily.news.fetch import _to_wecom_news_row, sync_wecom_news_rows
|
||||
|
||||
row = _to_wecom_news_row(
|
||||
{
|
||||
"title": "t",
|
||||
"link": "https://a.com/x",
|
||||
"source_name": "s",
|
||||
"published_fmt": "07-11",
|
||||
"summary": "Short english stub that was truncated early...",
|
||||
}
|
||||
)
|
||||
flat = [
|
||||
{
|
||||
"link": "https://a.com/x",
|
||||
"summary": "苹果指控 OpenAI 窃取硬件商业机密,诉讼称 misconduct 涉及多名前员工。",
|
||||
}
|
||||
]
|
||||
sync_wecom_news_rows([row], flat)
|
||||
self.assertNotIn("...", row["desc_short"])
|
||||
self.assertIn("苹果", row["desc_short"])
|
||||
|
||||
|
||||
def test_finalize_wecom_news_forces_chinese(self):
|
||||
from daily.news.fetch import finalize_wecom_news_items
|
||||
|
||||
items = [
|
||||
{
|
||||
"link": "https://a.com/1",
|
||||
"desc_short": "Meta removed the feature after backlash.",
|
||||
"summary_plain": "Meta removed the feature after backlash.",
|
||||
}
|
||||
]
|
||||
with patch(
|
||||
"daily.localize.localize_brief_descriptions",
|
||||
return_value={"wecom-news:https://a.com/1": "Meta 在舆论压力下移除了该功能"},
|
||||
):
|
||||
finalize_wecom_news_items(items, force_chinese=True)
|
||||
self.assertIn("Meta", items[0]["desc_short"])
|
||||
self.assertNotIn("backlash", items[0]["desc_short"])
|
||||
|
||||
|
||||
class NewsPickTests(unittest.TestCase):
|
||||
def test_pick_and_backfill_to_limit(self):
|
||||
from daily.news.fetch import _fill_picked_to_limit, _pick_news_items
|
||||
|
||||
flat = [
|
||||
{"link": f"https://a.com/{i}", "title": f"t{i}", "category_id": "media", "summary": "s"}
|
||||
for i in range(12)
|
||||
]
|
||||
picked = _pick_news_items(flat, 10, ("media",))
|
||||
self.assertEqual(len(picked), 10)
|
||||
picked = _fill_picked_to_limit(picked[:3], [flat], 10)
|
||||
self.assertEqual(len(picked), 10)
|
||||
62
tests/test_news_relax.py
Normal file
62
tests/test_news_relax.py
Normal file
@@ -0,0 +1,62 @@
|
||||
# tests/test_news_relax.py
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
class NewsRelaxTests(unittest.TestCase):
|
||||
def test_strip_relax_prefix(self):
|
||||
from daily.news.sanitize import strip_relax_window_prefix
|
||||
|
||||
self.assertEqual(
|
||||
strip_relax_window_prefix("放宽窗口:苹果起诉 OpenAI"),
|
||||
"苹果起诉 OpenAI",
|
||||
)
|
||||
self.assertEqual(
|
||||
strip_relax_window_prefix("放宽至48小时:某新闻"),
|
||||
"某新闻",
|
||||
)
|
||||
self.assertEqual(
|
||||
strip_relax_window_prefix("正常摘要无前缀"),
|
||||
"正常摘要无前缀",
|
||||
)
|
||||
|
||||
def test_backfill_disabled_does_not_reinsert_pushed(self):
|
||||
from daily.news.fetch import _apply_pushed_dedup_with_backfill
|
||||
|
||||
fresh_only = [
|
||||
{
|
||||
"link": "https://example.com/fresh",
|
||||
"title": "fresh",
|
||||
"source_name": "S",
|
||||
"published_fmt": "07-14",
|
||||
"desc_short": "新",
|
||||
"summary_plain": "新",
|
||||
}
|
||||
]
|
||||
picked = [
|
||||
{
|
||||
"link": "https://example.com/old",
|
||||
"title": "old",
|
||||
"source_name": "S",
|
||||
"published": "2026-07-13T10:00:00+00:00",
|
||||
"summary": "旧闻",
|
||||
}
|
||||
]
|
||||
with patch("daily.news.pushed_links.filter_unpushed_items", return_value=list(fresh_only)):
|
||||
with patch.dict(os.environ, {"DAILY_NEWS_BACKFILL": "0"}, clear=False):
|
||||
out = _apply_pushed_dedup_with_backfill(
|
||||
fresh_only + [{"link": "https://example.com/old", "title": "old"}],
|
||||
picked,
|
||||
date_str="2026-07-14",
|
||||
limit=5,
|
||||
)
|
||||
links = [x.get("link") for x in out]
|
||||
self.assertIn("https://example.com/fresh", links)
|
||||
self.assertNotIn("https://example.com/old", links)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
156
tests/test_scheduler.py
Normal file
156
tests/test_scheduler.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""Tests for daily.scheduler."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from datetime import datetime
|
||||
from unittest import mock
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from daily.scheduler import (
|
||||
ClockTime,
|
||||
SchedulerState,
|
||||
next_occurrence_after,
|
||||
parse_hhmm,
|
||||
plan_next_action,
|
||||
tick_once,
|
||||
)
|
||||
|
||||
|
||||
class ParseHhmmTests(unittest.TestCase):
|
||||
def test_parse(self):
|
||||
t = parse_hhmm("08:50")
|
||||
self.assertEqual((t.hour, t.minute), (8, 50))
|
||||
|
||||
def test_invalid(self):
|
||||
with self.assertRaises(ValueError):
|
||||
parse_hhmm("25:00")
|
||||
|
||||
|
||||
class PlanNextActionTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.tz = ZoneInfo("Asia/Shanghai")
|
||||
self.gen = ClockTime(8, 50)
|
||||
self.push = ClockTime(9, 0)
|
||||
|
||||
def test_before_generate_waits_for_generate(self):
|
||||
now = datetime(2026, 7, 9, 8, 30, tzinfo=self.tz)
|
||||
run_at, action = plan_next_action(
|
||||
now=now,
|
||||
tz=self.tz,
|
||||
state=SchedulerState(),
|
||||
generate_at=self.gen,
|
||||
push_at=self.push,
|
||||
)
|
||||
self.assertEqual(action, "generate")
|
||||
self.assertEqual(run_at.hour, 8)
|
||||
self.assertEqual(run_at.minute, 50)
|
||||
|
||||
def test_after_generate_before_push_waits_for_push(self):
|
||||
now = datetime(2026, 7, 9, 8, 55, tzinfo=self.tz)
|
||||
state = SchedulerState(last_generate_date="2026-07-09")
|
||||
run_at, action = plan_next_action(
|
||||
now=now,
|
||||
tz=self.tz,
|
||||
state=state,
|
||||
generate_at=self.gen,
|
||||
push_at=self.push,
|
||||
)
|
||||
self.assertEqual(action, "push")
|
||||
self.assertEqual(run_at.hour, 9)
|
||||
|
||||
def test_catch_up_generate_when_started_late(self):
|
||||
now = datetime(2026, 7, 9, 8, 55, tzinfo=self.tz)
|
||||
run_at, action = plan_next_action(
|
||||
now=now,
|
||||
tz=self.tz,
|
||||
state=SchedulerState(),
|
||||
generate_at=self.gen,
|
||||
push_at=self.push,
|
||||
)
|
||||
self.assertEqual(action, "generate")
|
||||
self.assertEqual(run_at, now)
|
||||
|
||||
def test_next_day_after_both_done(self):
|
||||
now = datetime(2026, 7, 9, 10, 0, tzinfo=self.tz)
|
||||
state = SchedulerState(last_generate_date="2026-07-09", last_push_date="2026-07-09")
|
||||
run_at, action = plan_next_action(
|
||||
now=now,
|
||||
tz=self.tz,
|
||||
state=state,
|
||||
generate_at=self.gen,
|
||||
push_at=self.push,
|
||||
)
|
||||
self.assertEqual(action, "generate")
|
||||
self.assertEqual(run_at.date().isoformat(), "2026-07-10")
|
||||
|
||||
def test_evening_start_waits_for_tomorrow_generate(self):
|
||||
now = datetime(2026, 7, 9, 20, 35, tzinfo=self.tz)
|
||||
run_at, action = plan_next_action(
|
||||
now=now,
|
||||
tz=self.tz,
|
||||
state=SchedulerState(),
|
||||
generate_at=self.gen,
|
||||
push_at=self.push,
|
||||
)
|
||||
self.assertEqual(action, "generate")
|
||||
self.assertEqual(run_at.date().isoformat(), "2026-07-10")
|
||||
self.assertEqual((run_at.hour, run_at.minute), (8, 50))
|
||||
|
||||
def test_catch_up_push_when_generate_done(self):
|
||||
now = datetime(2026, 7, 9, 20, 35, tzinfo=self.tz)
|
||||
state = SchedulerState(last_generate_date="2026-07-09")
|
||||
run_at, action = plan_next_action(
|
||||
now=now,
|
||||
tz=self.tz,
|
||||
state=state,
|
||||
generate_at=self.gen,
|
||||
push_at=self.push,
|
||||
)
|
||||
self.assertEqual(action, "push")
|
||||
self.assertEqual(run_at, now)
|
||||
|
||||
|
||||
class NextOccurrenceTests(unittest.TestCase):
|
||||
def test_tomorrow_when_past(self):
|
||||
tz = ZoneInfo("Asia/Shanghai")
|
||||
now = datetime(2026, 7, 9, 10, 0, tzinfo=tz)
|
||||
nxt = next_occurrence_after(ClockTime(8, 50), tz, now)
|
||||
self.assertEqual(nxt.date().isoformat(), "2026-07-10")
|
||||
self.assertEqual((nxt.hour, nxt.minute), (8, 50))
|
||||
|
||||
|
||||
class WorkdayGateTests(unittest.TestCase):
|
||||
"""非工作日 tick_once 直接标记完成并跳过,不触发 generate。"""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self.tz = ZoneInfo("Asia/Shanghai")
|
||||
self.gen = ClockTime(8, 50)
|
||||
self.push = ClockTime(9, 0)
|
||||
|
||||
def _tick(self, day, workday):
|
||||
now = datetime(day.year, day.month, day.day, 8, 30, tzinfo=self.tz)
|
||||
holidays = {} if workday else {day.isoformat(): {"rest": True, "name": "测试假"}}
|
||||
with mock.patch("daily.scheduler.load_holidays", return_value=holidays), \
|
||||
mock.patch("daily.scheduler.run_scheduled_action") as run:
|
||||
state = tick_once(
|
||||
now=now, tz=self.tz, state=SchedulerState(),
|
||||
generate_at=self.gen, push_at=self.push, dry_run=True,
|
||||
)
|
||||
return state, run
|
||||
|
||||
def test_holiday_skips_and_marks_done(self):
|
||||
# 2026-07-25 是周六;强制为非工作日
|
||||
from datetime import date
|
||||
day = date(2026, 7, 25)
|
||||
state, run = self._tick(day, workday=False)
|
||||
self.assertFalse(run.called)
|
||||
self.assertEqual(state.last_generate_date, "2026-07-25")
|
||||
self.assertEqual(state.last_push_date, "2026-07-25")
|
||||
|
||||
def test_workday_proceeds(self):
|
||||
from datetime import date
|
||||
day = date(2026, 7, 23) # 周四,工作日
|
||||
state, run = self._tick(day, workday=True)
|
||||
# 工作日不提前标记完成(dry-run 不执行动作,故 last_generate_date 仍为 None)
|
||||
self.assertIsNone(state.last_generate_date)
|
||||
60
tests/test_top_line.py
Normal file
60
tests/test_top_line.py
Normal file
@@ -0,0 +1,60 @@
|
||||
# tests/test_top_line.py
|
||||
"""T5: _top_line 看点行(theme_line 取数升级)测试。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import daily.generate as g
|
||||
|
||||
FEED_HIT = {
|
||||
"topTrending": [
|
||||
{"title": "remotion-video", "source": "src", "description": "video tool"},
|
||||
],
|
||||
"topHot": [],
|
||||
}
|
||||
FEED_MISS = {
|
||||
"topTrending": [
|
||||
{"title": "zzz-nomatch", "source": "src", "description": "nothing"},
|
||||
],
|
||||
"topHot": [],
|
||||
}
|
||||
|
||||
|
||||
class TopLineTests(unittest.TestCase):
|
||||
def test_scores_hit_returns_theme(self):
|
||||
# 命中 THEME_RULES(video) -> 评分最高主题
|
||||
with mock.patch.object(g, "env_bool", return_value=True):
|
||||
line = g._top_line(FEED_HIT)
|
||||
self.assertIn("今日主题", line)
|
||||
self.assertIn("AI 多媒体", line)
|
||||
|
||||
def test_fallback_to_theme_names_when_no_score(self):
|
||||
# 无评分命中但 theme_clusters 能聚类 -> 用主题名
|
||||
feed = {
|
||||
"topTrending": [
|
||||
{"id": "x", "title": "runcomfy-x", "source": "s", "description": "video x"},
|
||||
],
|
||||
"topHot": [],
|
||||
}
|
||||
with mock.patch.object(g, "env_bool", return_value=True):
|
||||
# 让评分落空(前 10 无命中)但 clusters(前 20)命中
|
||||
line = g._top_line(feed)
|
||||
self.assertIn("今日主题", line)
|
||||
self.assertIn("AI 多媒体", line)
|
||||
|
||||
def test_switch_off_uses_legacy_detect(self):
|
||||
# DAILY_WECOM_TOP_LINE=0 -> 退回 _detect_theme_line
|
||||
with mock.patch.object(g, "env_bool", return_value=False):
|
||||
line = g._top_line(FEED_MISS)
|
||||
self.assertEqual(line, g._detect_theme_line(FEED_MISS))
|
||||
|
||||
def test_empty_feed_hardcoded_fallback(self):
|
||||
# 完全无数据 -> 硬编码兜底, 不抛异常
|
||||
with mock.patch.object(g, "env_bool", return_value=True):
|
||||
line = g._top_line({"topTrending": [], "topHot": []})
|
||||
self.assertIn("今日主题", line)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
804
tests/test_wecom_delta.py
Normal file
804
tests/test_wecom_delta.py
Normal file
@@ -0,0 +1,804 @@
|
||||
"""Tests for WeCom delta mode."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from daily.config import (
|
||||
delta_baseline_fallback,
|
||||
env_bool,
|
||||
force_push,
|
||||
news_dedup_days,
|
||||
skip_push_when_silent,
|
||||
wecom_mode,
|
||||
)
|
||||
from daily.news.pushed_links import filter_unpushed_items, record_pushed_links
|
||||
|
||||
|
||||
class ConfigHelpersTests(unittest.TestCase):
|
||||
def test_wecom_mode_defaults_delta(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertEqual(wecom_mode(), "delta")
|
||||
|
||||
def test_wecom_mode_full(self):
|
||||
with patch.dict(os.environ, {"DAILY_WECOM_MODE": "full"}, clear=True):
|
||||
self.assertEqual(wecom_mode(), "full")
|
||||
|
||||
def test_env_bool_truthy(self):
|
||||
with patch.dict(os.environ, {"DAILY_FORCE_PUSH": "1"}, clear=True):
|
||||
self.assertTrue(env_bool("DAILY_FORCE_PUSH", False))
|
||||
|
||||
def test_skip_push_when_silent_default(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertTrue(skip_push_when_silent())
|
||||
|
||||
def test_news_dedup_days_default(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertEqual(news_dedup_days(), 7)
|
||||
|
||||
def test_delta_baseline_fallback_default(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertEqual(delta_baseline_fallback(), "full")
|
||||
|
||||
def test_wecom_delta_pad_default_true(self):
|
||||
from daily.config import wecom_delta_pad
|
||||
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertTrue(wecom_delta_pad())
|
||||
|
||||
|
||||
class NewsPushedLinksTests(unittest.TestCase):
|
||||
def test_filter_and_record_roundtrip(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
cache = Path(tmp) / "pushed-news-links.json"
|
||||
items = [
|
||||
{"title": "A", "link": "https://example.com/a?utm_source=x"},
|
||||
{"title": "B", "link": "https://example.com/b"},
|
||||
]
|
||||
with patch("daily.news.pushed_links._cache_path", return_value=cache):
|
||||
with patch("daily.news.pushed_links.news_dedup_days", return_value=7):
|
||||
record_pushed_links("2026-07-08", ["https://example.com/a"])
|
||||
out = filter_unpushed_items(items, date_str="2026-07-09")
|
||||
self.assertEqual(len(out), 1)
|
||||
self.assertEqual(out[0]["link"], "https://example.com/b")
|
||||
|
||||
|
||||
class SkillMovePartitionTests(unittest.TestCase):
|
||||
def test_partition_dedupes_across_boards(self):
|
||||
trending = [
|
||||
{
|
||||
"id": "a/b/foo",
|
||||
"rank": 4,
|
||||
"title": "foo",
|
||||
"source": "a/b",
|
||||
"installs": 1,
|
||||
"link": "",
|
||||
"description": "",
|
||||
}
|
||||
]
|
||||
hot = [
|
||||
{
|
||||
"id": "a/b/foo",
|
||||
"rank": 2,
|
||||
"title": "foo",
|
||||
"source": "a/b",
|
||||
"installs": 1,
|
||||
"link": "",
|
||||
"description": "",
|
||||
}
|
||||
]
|
||||
from daily.delta import partition_skill_moves_for_wecom
|
||||
|
||||
t_out, h_out = partition_skill_moves_for_wecom(trending, hot)
|
||||
self.assertEqual(len(t_out), 1)
|
||||
self.assertEqual(len(h_out), 0)
|
||||
self.assertIn("Trending #4", t_out[0]["badge"])
|
||||
self.assertIn("Hot #2", t_out[0]["badge"])
|
||||
|
||||
def test_effective_mode_fallback_full_without_baseline(self):
|
||||
from daily.delta import effective_wecom_mode
|
||||
|
||||
with patch("daily.delta.find_previous_data", return_value=None):
|
||||
with patch("daily.delta.wecom_mode", return_value="delta"):
|
||||
with patch("daily.delta.delta_baseline_fallback", return_value="full"):
|
||||
self.assertEqual(effective_wecom_mode(date_str="2026-07-10"), "full")
|
||||
|
||||
|
||||
class DeltaFormatTests(unittest.TestCase):
|
||||
def test_skills_delta_omits_empty_board(self):
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
moves = [
|
||||
{
|
||||
"id": "x/y/z",
|
||||
"rank": 3,
|
||||
"title": "z",
|
||||
"source": "x/y",
|
||||
"installs": 10,
|
||||
"installs_fmt": "10",
|
||||
"link": "https://skills.sh/x/y/z",
|
||||
"description": "d",
|
||||
"badge": "Trending #3",
|
||||
}
|
||||
]
|
||||
text = build_skills_delta_sections(moves, [])
|
||||
self.assertIn("Skills Trending 变化", text)
|
||||
self.assertNotIn("[新入 #", text)
|
||||
self.assertNotIn("Skills Hot 变化", text)
|
||||
|
||||
def test_github_delta_omits_stable_board(self):
|
||||
from daily.format_wecom import build_github_delta_sections
|
||||
|
||||
movement = {
|
||||
"github_trending_moves": [
|
||||
{
|
||||
"repo": "a/b",
|
||||
"url": "https://github.com/a/b",
|
||||
"rank": 1,
|
||||
"language": "Go",
|
||||
"description": "open-source codebase and curriculum",
|
||||
}
|
||||
],
|
||||
"github_emerging_moves": [],
|
||||
"github_topic_moves": [],
|
||||
}
|
||||
with patch(
|
||||
"daily.format_wecom.localize_brief_descriptions",
|
||||
return_value={"github:a/b": "开源代码库与课程体系"},
|
||||
):
|
||||
text = build_github_delta_sections(movement, topic_name="llm")
|
||||
self.assertIn("开源代码库", text)
|
||||
self.assertIn("GitHub Trending 变化", text)
|
||||
self.assertNotIn("新兴", text)
|
||||
self.assertNotIn("[新入 #", text)
|
||||
self.assertNotIn("\n > ", text)
|
||||
|
||||
def test_delta_pad_groups_same_source_moves(self):
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
moves = [
|
||||
{
|
||||
"id": f"lllllllama/rigorpilot-skills/s{i}",
|
||||
"title": f"s{i}",
|
||||
"source": "lllllllama/rigorpilot-skills",
|
||||
"installs": 250 - i,
|
||||
"installs_fmt": str(250 - i),
|
||||
"link": f"https://www.skills.sh/lllllllama/rigorpilot-skills/s{i}",
|
||||
"description": f"skill {i}",
|
||||
}
|
||||
for i in range(1, 11)
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(moves, [], trending_limit=10, pad=True)
|
||||
self.assertIn("10 skills", text)
|
||||
self.assertNotIn("[**s2**]", text)
|
||||
|
||||
def test_delta_pad_fills_skills_to_limit(self):
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
moves = [
|
||||
{
|
||||
"id": "a/b/new",
|
||||
"rank": 3,
|
||||
"title": "new",
|
||||
"source": "a/b",
|
||||
"installs": 99,
|
||||
"installs_fmt": "99",
|
||||
"link": "https://skills.sh/a/b/new",
|
||||
"description": "new skill",
|
||||
}
|
||||
]
|
||||
full = [
|
||||
{
|
||||
"id": "a/b/new",
|
||||
"title": "new",
|
||||
"source": "a/b",
|
||||
"installs": 99,
|
||||
"installs_fmt": "99",
|
||||
"link": "https://skills.sh/a/b/new",
|
||||
"description": "new skill",
|
||||
},
|
||||
*[
|
||||
{
|
||||
"id": f"src{i}/skill",
|
||||
"title": "skill",
|
||||
"source": f"src{i}/pkg",
|
||||
"installs": 100 - i,
|
||||
"installs_fmt": str(100 - i),
|
||||
"link": f"https://skills.sh/src{i}/pkg/skill",
|
||||
"description": f"skill from src{i}",
|
||||
}
|
||||
for i in range(1, 12)
|
||||
],
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(
|
||||
moves,
|
||||
[],
|
||||
trending_full=full,
|
||||
hot_full=[],
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
)
|
||||
self.assertIn("Skills Trending Top 10", text)
|
||||
self.assertNotIn("Skills Trending 变化", text)
|
||||
|
||||
def test_delta_pad_keeps_large_clusters_merged_and_fills_limit(self):
|
||||
"""补榜按 source 合并态取条,大 cluster 不得撑爆 flat 预算导致短榜。"""
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
def cluster(source: str, n: int, installs: int) -> dict:
|
||||
titles = [f"t{i}" for i in range(n)]
|
||||
return {
|
||||
"id": f"{source}/{titles[0]}",
|
||||
"title": titles[0],
|
||||
"source": source,
|
||||
"installs": installs,
|
||||
"installs_fmt": str(installs),
|
||||
"cluster": True,
|
||||
"cluster_count": n,
|
||||
"cluster_skills": titles,
|
||||
"cluster_titles": ", ".join(titles[:4]) + "…",
|
||||
"link": f"https://skills.sh/{source}/{titles[0]}",
|
||||
"description": f"{source} cluster",
|
||||
}
|
||||
|
||||
full = [cluster(f"big{i}/pkg", 20, 1000 - i) for i in range(1, 5)] + [
|
||||
{
|
||||
"id": f"other{n}/pkg/skill",
|
||||
"title": "skill",
|
||||
"source": f"other{n}/pkg",
|
||||
"installs": 50 - n,
|
||||
"installs_fmt": str(50 - n),
|
||||
"link": f"https://skills.sh/other{n}/pkg/skill",
|
||||
"description": f"other {n}",
|
||||
}
|
||||
for n in range(1, 12)
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=full,
|
||||
hot_full=[],
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
)
|
||||
self.assertIn("Skills Trending Top 10", text)
|
||||
self.assertIn("20 skills", text)
|
||||
self.assertIn("other6/pkg", text)
|
||||
|
||||
def test_delta_pad_recent_blocks_same_source_not_just_primary_id(self):
|
||||
"""周去重按 source:换同仓另一个 skill id 不得再上榜。"""
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
titles = [f"t{i}" for i in range(20)]
|
||||
full = [
|
||||
{
|
||||
"id": f"big/pkg/{titles[0]}",
|
||||
"title": titles[0],
|
||||
"source": "big/pkg",
|
||||
"installs": 999,
|
||||
"installs_fmt": "999",
|
||||
"cluster": True,
|
||||
"cluster_count": 20,
|
||||
"cluster_skills": titles,
|
||||
"cluster_titles": ", ".join(titles[:4]) + "…",
|
||||
"link": f"https://skills.sh/big/pkg/{titles[0]}",
|
||||
"description": "big cluster",
|
||||
},
|
||||
*[
|
||||
{
|
||||
"id": f"other{n}/pkg/skill",
|
||||
"title": "skill",
|
||||
"source": f"other{n}/pkg",
|
||||
"installs": 50 - n,
|
||||
"installs_fmt": str(50 - n),
|
||||
"link": f"https://skills.sh/other{n}/pkg/skill",
|
||||
"description": f"other {n}",
|
||||
}
|
||||
for n in range(1, 12)
|
||||
],
|
||||
]
|
||||
# 昨日展示的是同 source 另一 skill id(非今日 primary)
|
||||
recent = {f"big/pkg/{titles[5]}"}
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=full,
|
||||
hot_full=[],
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
recent_trending=recent,
|
||||
)
|
||||
self.assertIn("Skills Trending Top 10", text)
|
||||
self.assertNotIn("big/pkg", text)
|
||||
self.assertIn("other1/pkg", text)
|
||||
|
||||
def test_delta_pad_hot_recent_unions_trending_history_by_source(self):
|
||||
"""Skills Hot 周去重合并 Trending 历史:隔日换榜也不能同 source 再出现。"""
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
hot_full = [
|
||||
{
|
||||
"id": "101-skills/skills/ai-music",
|
||||
"title": "ai-music",
|
||||
"source": "101-skills/skills",
|
||||
"installs": 200,
|
||||
"installs_fmt": "200",
|
||||
"link": "https://skills.sh/101-skills/skills/ai-music",
|
||||
"description": "hot candidate",
|
||||
},
|
||||
{
|
||||
"id": "fresh/src/skill",
|
||||
"title": "skill",
|
||||
"source": "fresh/src",
|
||||
"installs": 100,
|
||||
"installs_fmt": "100",
|
||||
"link": "https://skills.sh/fresh/src/skill",
|
||||
"description": "fresh",
|
||||
},
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=[],
|
||||
hot_full=hot_full,
|
||||
trending_limit=10,
|
||||
hot_limit=10,
|
||||
pad=True,
|
||||
recent_trending={"101-skills/skills/ai-video-generation"},
|
||||
recent_hot=set(),
|
||||
)
|
||||
self.assertIn("Skills Hot Top 1", text)
|
||||
self.assertIn("fresh/src", text)
|
||||
self.assertNotIn("101-skills", text)
|
||||
|
||||
def test_delta_pad_hot_excludes_trending_by_source(self):
|
||||
"""同日 Hot 补榜按 source 避开 Trending,而非展开全部 cluster skill id。"""
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
trending_full = [
|
||||
{
|
||||
"id": "same/src/a",
|
||||
"title": "a",
|
||||
"source": "same/src",
|
||||
"installs": 100,
|
||||
"installs_fmt": "100",
|
||||
"link": "https://skills.sh/same/src/a",
|
||||
"description": "trending item",
|
||||
}
|
||||
]
|
||||
hot_full = [
|
||||
{
|
||||
"id": "same/src/b",
|
||||
"title": "b",
|
||||
"source": "same/src",
|
||||
"installs": 90,
|
||||
"installs_fmt": "90",
|
||||
"link": "https://skills.sh/same/src/b",
|
||||
"description": "hot twin",
|
||||
},
|
||||
{
|
||||
"id": "fresh/src/skill",
|
||||
"title": "skill",
|
||||
"source": "fresh/src",
|
||||
"installs": 80,
|
||||
"installs_fmt": "80",
|
||||
"link": "https://skills.sh/fresh/src/skill",
|
||||
"description": "fresh hot",
|
||||
},
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=trending_full,
|
||||
hot_full=hot_full,
|
||||
trending_limit=10,
|
||||
hot_limit=10,
|
||||
pad=True,
|
||||
)
|
||||
self.assertIn("Skills Hot Top 1", text)
|
||||
self.assertIn("fresh/src", text)
|
||||
self.assertNotIn("same/src/b", text)
|
||||
|
||||
def test_delta_pad_uses_large_pool_when_recent_excludes_top(self):
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
full_small = [
|
||||
{
|
||||
"id": f"seen/src/s{i}",
|
||||
"title": f"s{i}",
|
||||
"source": "seen/src",
|
||||
"installs": 100 - i,
|
||||
"installs_fmt": str(100 - i),
|
||||
"link": f"https://skills.sh/seen/src/s{i}",
|
||||
"description": f"seen {i}",
|
||||
}
|
||||
for i in range(1, 11)
|
||||
]
|
||||
full_large = [
|
||||
{
|
||||
"id": f"fresh/src{n}/skill",
|
||||
"title": "skill",
|
||||
"source": f"fresh/src{n}",
|
||||
"installs": 50 - n,
|
||||
"installs_fmt": str(50 - n),
|
||||
"link": f"https://skills.sh/fresh/src{n}/skill",
|
||||
"description": f"fresh {n}",
|
||||
}
|
||||
for n in range(1, 11)
|
||||
]
|
||||
recent = {f"seen/src/s{i}" for i in range(1, 11)}
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
small = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=full_small,
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
recent_trending=recent,
|
||||
)
|
||||
large = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=full_large,
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
recent_trending=recent,
|
||||
)
|
||||
self.assertNotIn("Skills Trending Top 10", small)
|
||||
self.assertIn("Skills Trending Top 10", large)
|
||||
self.assertIn("fresh/src1", large)
|
||||
|
||||
def test_delta_pad_fills_github_to_limit(self):
|
||||
from daily.format_wecom import build_github_delta_sections
|
||||
|
||||
movement = {"github_trending_moves": [], "github_emerging_moves": [], "github_topic_moves": []}
|
||||
full = [
|
||||
{
|
||||
"repo": f"org/r{i}",
|
||||
"url": f"https://github.com/org/r{i}",
|
||||
"language": "Go",
|
||||
"stars_today_fmt": "100",
|
||||
"total_stars_fmt": "1K",
|
||||
"description": f"repo {i}",
|
||||
"desc_short": f"repo {i}",
|
||||
}
|
||||
for i in range(1, 12)
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
text = build_github_delta_sections(
|
||||
movement,
|
||||
topic_name="llm",
|
||||
github_trending=full,
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
)
|
||||
self.assertIn("GitHub Trending Top 10", text)
|
||||
self.assertNotIn("GitHub Trending 变化", text)
|
||||
|
||||
def test_delta_pad_github_unions_recent_across_boards(self):
|
||||
"""GitHub 三榜共用周去重:Trending 出过的 repo,新兴/Topic 不得再出。"""
|
||||
from daily.format_wecom import build_github_delta_sections
|
||||
|
||||
movement = {"github_trending_moves": [], "github_emerging_moves": [], "github_topic_moves": []}
|
||||
shared = {
|
||||
"repo": "seen/repo",
|
||||
"url": "https://github.com/seen/repo",
|
||||
"language": "Go",
|
||||
"stars_today_fmt": "100",
|
||||
"total_stars_fmt": "1K",
|
||||
"created_at": "2026-07-01",
|
||||
"description": "already shown",
|
||||
"desc_short": "already shown",
|
||||
}
|
||||
fresh = {
|
||||
"repo": "fresh/repo",
|
||||
"url": "https://github.com/fresh/repo",
|
||||
"language": "Go",
|
||||
"stars_today_fmt": "90",
|
||||
"total_stars_fmt": "900",
|
||||
"created_at": "2026-07-02",
|
||||
"description": "fresh",
|
||||
"desc_short": "fresh",
|
||||
}
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
text = build_github_delta_sections(
|
||||
movement,
|
||||
topic_name="llm",
|
||||
github_trending=[],
|
||||
github_emerging=[shared, fresh],
|
||||
github_topic=[shared],
|
||||
emerging_limit=5,
|
||||
topic_limit=5,
|
||||
pad=True,
|
||||
recent_board_keys={"github_trending": {"seen/repo"}},
|
||||
)
|
||||
self.assertIn("fresh/repo", text)
|
||||
self.assertNotIn("seen/repo", text)
|
||||
|
||||
def test_delta_pad_skips_recent_skills(self):
|
||||
from daily.format_wecom import build_skills_delta_sections
|
||||
|
||||
full = [
|
||||
{
|
||||
"id": f"x/y/s{i}",
|
||||
"title": f"s{i}",
|
||||
"source": "x/y",
|
||||
"installs": 100 - i,
|
||||
"installs_fmt": str(100 - i),
|
||||
"link": f"https://skills.sh/x/y/s{i}",
|
||||
"description": f"skill {i}",
|
||||
}
|
||||
for i in range(4, 6)
|
||||
] + [
|
||||
{
|
||||
"id": "fresh/src/skill",
|
||||
"title": "skill",
|
||||
"source": "fresh/src",
|
||||
"installs": 50,
|
||||
"installs_fmt": "50",
|
||||
"link": "https://skills.sh/fresh/src/skill",
|
||||
"description": "fresh skill",
|
||||
}
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
text = build_skills_delta_sections(
|
||||
[],
|
||||
[],
|
||||
trending_full=full,
|
||||
hot_full=[],
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
# 同仓历史 skill id → 整仓 source 去重;仅保留其它 source
|
||||
recent_trending={f"x/y/s{i}" for i in range(1, 4)},
|
||||
)
|
||||
self.assertIn("Skills Trending Top 1", text)
|
||||
self.assertIn("fresh/src", text)
|
||||
self.assertNotIn("x/y", text)
|
||||
|
||||
def test_delta_pad_skips_recent_github(self):
|
||||
from daily.format_wecom import build_github_delta_sections
|
||||
|
||||
movement = {"github_trending_moves": [], "github_emerging_moves": [], "github_topic_moves": []}
|
||||
full = [
|
||||
{
|
||||
"repo": f"org/r{i}",
|
||||
"url": f"https://github.com/org/r{i}",
|
||||
"language": "Go",
|
||||
"total_stars_fmt": "1K",
|
||||
"description": f"repo {i}",
|
||||
"desc_short": f"repo {i}",
|
||||
}
|
||||
for i in range(1, 6)
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
text = build_github_delta_sections(
|
||||
movement,
|
||||
topic_name="llm",
|
||||
github_trending=full,
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
recent_board_keys={"github_trending": {f"org/r{i}" for i in range(1, 4)}},
|
||||
)
|
||||
self.assertIn("GitHub Trending Top 2", text)
|
||||
self.assertIn("org/r4", text)
|
||||
self.assertNotIn("org/r1", text)
|
||||
|
||||
def test_delta_pad_skips_recent_github_moves(self):
|
||||
"""异动新入榜若昨日企微已展示,pad 时仍应排除(不只滤补榜)。"""
|
||||
from daily.format_wecom import build_github_delta_sections
|
||||
|
||||
movement = {
|
||||
"github_trending_moves": [
|
||||
{
|
||||
"repo": "vinta/awesome-python",
|
||||
"url": "https://github.com/vinta/awesome-python",
|
||||
"language": "Python",
|
||||
"total_stars_fmt": "308K",
|
||||
"description": "list",
|
||||
}
|
||||
],
|
||||
"github_emerging_moves": [],
|
||||
"github_topic_moves": [],
|
||||
}
|
||||
full = [
|
||||
{
|
||||
"repo": "fresh/repo",
|
||||
"url": "https://github.com/fresh/repo",
|
||||
"language": "Go",
|
||||
"total_stars_fmt": "1K",
|
||||
"description": "fresh",
|
||||
"desc_short": "fresh",
|
||||
}
|
||||
]
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
text = build_github_delta_sections(
|
||||
movement,
|
||||
topic_name="llm",
|
||||
github_trending=full,
|
||||
trending_limit=10,
|
||||
pad=True,
|
||||
recent_board_keys={"github_trending": {"vinta/awesome-python"}},
|
||||
)
|
||||
self.assertIn("fresh/repo", text)
|
||||
self.assertNotIn("vinta/awesome-python", text)
|
||||
|
||||
def test_load_recent_board_keys_from_data_json(self):
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from daily.delta import load_recent_board_keys
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out = Path(tmp)
|
||||
payload = {
|
||||
"data": {
|
||||
"date": "2026-07-09",
|
||||
"movement_baseline": {
|
||||
"skills_trending": [
|
||||
{"id": "a/b/raw", "title": "raw", "source": "a/b"},
|
||||
],
|
||||
"skills_hot": [],
|
||||
"github_trending": [{"repo": "org/raw"}],
|
||||
"github_emerging": [],
|
||||
"github_topic": [],
|
||||
},
|
||||
"wecom_shown_keys": {
|
||||
"skills_trending": ["a/b/foo"],
|
||||
"github_trending": ["org/bar"],
|
||||
},
|
||||
}
|
||||
}
|
||||
(out / "2026-07-09.data.json").write_text(json.dumps(payload), encoding="utf-8")
|
||||
with patch("daily.board_history.OUTPUT_DIR", out):
|
||||
recent = load_recent_board_keys("2026-07-10", lookback_days=7)
|
||||
self.assertIn("a/b/foo", recent["skills_trending"])
|
||||
self.assertIn("org/bar", recent["github_trending"])
|
||||
self.assertNotIn("a/b/raw", recent["skills_trending"])
|
||||
self.assertNotIn("org/raw", recent["github_trending"])
|
||||
|
||||
|
||||
class PushGateTests(unittest.TestCase):
|
||||
def test_push_when_board_has_moves(self):
|
||||
from daily.push_gate import evaluate_push_gate
|
||||
|
||||
gate = evaluate_push_gate(
|
||||
movement={"skills_trending_moves": [{"id": "a/b/c"}], "skills_hot_moves": [], "github_trending_moves": [], "github_emerging_moves": [], "github_topic_moves": []},
|
||||
ai_news_items=[],
|
||||
cn_ai_news_items=[],
|
||||
featured_pick=None,
|
||||
)
|
||||
self.assertTrue(gate.should_push)
|
||||
self.assertIn("board_moves", gate.reasons)
|
||||
|
||||
def test_silent_when_all_empty(self):
|
||||
from daily.push_gate import evaluate_push_gate
|
||||
|
||||
gate = evaluate_push_gate(
|
||||
movement={
|
||||
"skills_trending_moves": [],
|
||||
"skills_hot_moves": [],
|
||||
"github_trending_moves": [],
|
||||
"github_emerging_moves": [],
|
||||
"github_topic_moves": [],
|
||||
},
|
||||
ai_news_items=[],
|
||||
cn_ai_news_items=[],
|
||||
featured_pick=None,
|
||||
)
|
||||
self.assertFalse(gate.should_push)
|
||||
self.assertTrue(gate.silent)
|
||||
|
||||
def test_force_push_overrides_silent(self):
|
||||
from daily.push_gate import evaluate_push_gate
|
||||
|
||||
with patch("daily.push_gate.force_push", return_value=True):
|
||||
gate = evaluate_push_gate(
|
||||
movement={
|
||||
"skills_trending_moves": [],
|
||||
"skills_hot_moves": [],
|
||||
"github_trending_moves": [],
|
||||
"github_emerging_moves": [],
|
||||
"github_topic_moves": [],
|
||||
},
|
||||
ai_news_items=[],
|
||||
cn_ai_news_items=[],
|
||||
featured_pick=None,
|
||||
)
|
||||
self.assertTrue(gate.should_push)
|
||||
self.assertIn("force_push", gate.reasons)
|
||||
|
||||
|
||||
class WebhookSilentTests(unittest.TestCase):
|
||||
def test_skip_when_silent(self):
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from daily.webhook import should_skip_push
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
data = root / "2026-07-10.data.json"
|
||||
data.write_text(
|
||||
json.dumps(
|
||||
{"meta": {"push_gate": {"should_push": False, "silent": True, "reasons": []}}}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
report = root / "2026-07-10.wecom.md"
|
||||
report.write_text("📰 test", encoding="utf-8")
|
||||
self.assertTrue(should_skip_push(report))
|
||||
|
||||
|
||||
class E2ESmokeTests(unittest.TestCase):
|
||||
def test_delta_mode_no_full_top_label(self):
|
||||
from daily.format_wecom import replace_wecom_board_sections
|
||||
|
||||
md = "📰 **早报 · 2026-07-10**\n\n🌍 **国际 AI · 精选 1**\n1. [x](https://a.com)\n"
|
||||
movement = {
|
||||
"skills_trending_moves": [
|
||||
{
|
||||
"id": "a/b/c",
|
||||
"rank": 2,
|
||||
"title": "c",
|
||||
"source": "a/b",
|
||||
"installs": 1,
|
||||
"link": "https://skills.sh/a/b/c",
|
||||
"description": "d",
|
||||
}
|
||||
],
|
||||
"skills_hot_moves": [],
|
||||
"github_trending_moves": [],
|
||||
"github_emerging_moves": [],
|
||||
"github_topic_moves": [],
|
||||
}
|
||||
with patch("daily.format_wecom.localize_brief_descriptions", return_value={}):
|
||||
with patch("daily.format_wecom.needs_chinese", return_value=False):
|
||||
out = replace_wecom_board_sections(
|
||||
md,
|
||||
mode="delta",
|
||||
movement=movement,
|
||||
trending=[],
|
||||
hot=[],
|
||||
topic_name="llm",
|
||||
pad=False,
|
||||
)
|
||||
self.assertIn("Skills Trending 变化", out)
|
||||
self.assertNotIn("Skills Trending Top", out)
|
||||
|
||||
|
||||
class SyncMovementGithubTests(unittest.TestCase):
|
||||
def test_sync_copies_localized_description(self):
|
||||
from daily.generate import _sync_movement_github_descriptions
|
||||
|
||||
movement = {
|
||||
"github_trending_moves": [{"repo": "a/b", "description": "english"}],
|
||||
"github_emerging_moves": [],
|
||||
"github_topic_moves": [],
|
||||
}
|
||||
_sync_movement_github_descriptions(
|
||||
movement,
|
||||
github_trending=[{"repo": "a/b", "description": "中文描述"}],
|
||||
github_emerging=[],
|
||||
github_topic=[],
|
||||
)
|
||||
self.assertEqual(movement["github_trending_moves"][0]["description"], "中文描述")
|
||||
@@ -1,14 +0,0 @@
|
||||
import re
|
||||
|
||||
import certifi
|
||||
import httpx
|
||||
|
||||
url = "https://skills.sh/vercel-labs/skills/find-skills"
|
||||
r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20, verify=certifi.where())
|
||||
chunks = re.findall(r'self\.__next_f\.push\(\[1,"(.*?)"\]\)', r.text, re.DOTALL)
|
||||
blob = "\n".join(chunks).encode("utf-8").decode("unicode_escape", errors="ignore")
|
||||
for needle in ("description", "Helps users", "SKILL.md", "summary"):
|
||||
print(needle, blob.count(needle))
|
||||
idx = blob.find("Helps users")
|
||||
if idx >= 0:
|
||||
print(blob[idx : idx + 300])
|
||||
Binary file not shown.
Reference in New Issue
Block a user