Compare commits

..

29 Commits

Author SHA1 Message Date
85614f5921 chore(skills): 接入 caveman 系列 agent skills
同步 skills-lock,便于压缩沟通、提交与评审等快捷模式。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-31 15:48:00 +08:00
e2f609a027 feat(news): research 时讯加可信过滤、去重与国内配额
候选池按独立事件拉取,白名单过滤低质源,同事件与 tech 主题去重后按展示上限打包。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-31 15:47:50 +08:00
0384e2c7a9 docs: research AI 时讯质量设计(去重、国内配额、可信源)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-29 16:19:55 +08:00
f166d1e504 fix: LLM 客户端自建大超时 Client,绕开 SDK 默认 60s unary 超时
CURSOR_MODEL=auto 时后端首 token 常超过 SDK 默认 unary_timeout,
自建带 DAILY_CURSOR_UNARY_TIMEOUT(300s)/DAILY_CURSOR_STREAM_TIMEOUT(900s)
的 Client 并显式传入 Agent.prompt,finally 中关闭释放资源。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 17:58:09 +08:00
36085f107d feat: 新增节假日缓存,仅工作日生成与推送
- daily/holiday.py: 抓取 xiaoai.me 全年节假日并缓存到 .cache/holidays-<year>.json,
  is_workday() 判定(法定节假日/周末休息,调休补班日上班),联网失败回退本地缓存
- scheduler: tick_once 前置工作日闸门,非工作日标记完成并跳过,判定失败降级为正常工作日避免漏跑
- config/.env.example: 新增 DAILY_WORKDAY_ONLY 开关(默认开)
- tests: test_holiday 8 例 + scheduler 非工作日闸门 2 例

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 17:51:09 +08:00
d66f2c716c docs: env 示例补充 DAILY_WECOM_TOP_LINE / DAILY_FEATURED_REASON
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 17:08:35 +08:00
d9aef4b340 chore: 删 tmp 调试文件,裸 except 改命名异常+日志
- 删除被跟踪的 tmp_check_desc.py / tmp_desc_snip.txt
- bridge_manager discovery、generate release 抓取的裸
  except Exception 改命名异常 + logger.warning(T8)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 17:01:20 +08:00
d7992e7a7a feat: push_gate 判定分支加结构化日志
force/推送/静默日/无更新仍推 四分支各记一条 logger.info,
无人值守时可定位「早报没推是静默日还是别的原因」(T7)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:55:47 +08:00
3321a307a0 feat: 首推理由行加 DAILY_FEATURED_REASON 开关
pick_why 本已渲染,现加开关默认开,空则省略(T6)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:53:19 +08:00
33bfed0e79 feat: theme_line 取数升级,无评分命中时回退 theme_names
新增 _top_line: 优先评分最高主题,无命中时回退 _theme_clusters 的
主题名(非 markdown 示例),DAILY_WECOM_TOP_LINE=0 退回原逻辑。
复用现有 theme_line 行,不新增头部行(T5)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:44:15 +08:00
ea8de9fe61 test: 新增 generate_report 黄金/确定性回归
冻结 _now_cst、mock 网络/文件/LLM(featured_pick 与 push_gate
有独立测试,此处冻结),验证同一输入下完整版与企微版产物逐字节
一致——拆分 _collect/_select/_render 不得改变行为(T1+T4)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:35:42 +08:00
cd92644be4 refactor: 拆 generate_report 为 _collect/_select/_render 三段
所有选择逻辑(9 处 board_select + featured_pick + _localize + pad)
全归 _select,_render 纯拼装写盘,薄壳 generate_report 仅编排。
结构化 bundle 传参;副作用(record_pushed_links/_save_snapshot/
save_json)保持原时序在 _render 末尾。行为不变,89 测试绿(T2)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 14:02:32 +08:00
92e0ed9a27 refactor: _theme_clusters 迁入 narrative_axis 并以注入消除反向依赖
theme_clusters/theme_names 经参数注入 THEME_RULES 与 skill_id_fn,
避免 generate.py 的循环 import(T3,为拆分 generate_report 铺路)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 13:49:37 +08:00
6ea2a4e4c6 feat: 早报系统重构与功能增强
- 新增常驻调度器 daily/scheduler.py + run-scheduler.ps1(定时生成/推送)
- 新增 daily/bridge_manager.py:Windows 兼容的 Cursor SDK 桥接
- 新增 skills/daily-featured-pick 首推 Skill 与叙事轴/去重逻辑
- 新闻抓取窗口、GitHub 搜索、企微 delta 模式等多项改进
- 补充设计文档与 superpowers 计划/规范
- 新增对应测试(scheduler、featured_pick、github_search、news_fetch_window 等)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 18:12:00 +08:00
6192dd4e2a chore: 移除独立企微 bot/浏览器桥接系统
两套系统并存,bot/(browser_service、preview_service、wecom_media、
router、xiaobao 场景等)与早报生成是独立的一条线,不再维护,整体删除。
同时将 __pycache__/*.pyc/.cursor 加入 .gitignore。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 18:11:11 +08:00
4a128b0fa6 fix: 关闭新闻放宽凑数并剥离放宽窗口文案
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 10:42:00 +08:00
dcd0608b5d feat: 代码选定叙事轴并注入 Agent 开场约束
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 10:40:14 +08:00
54f164cdbf feat: 首推与昨日冲突时改推并保证一月不重复
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 10:39:12 +08:00
d448002e7a feat: generate 以 board_select 为唯一列表主人并写回 shown keys
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 10:36:53 +08:00
0c324f9ace feat: 实现 board_select 周去重与深池补满
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 10:32:39 +08:00
f563239e0b feat: 拆分 wecom_shown_keys 与 movement_baseline 历史层
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 10:31:49 +08:00
02e97e057f docs: 更新 Agent 技能与 env 示例以支持 Delta 模式
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:37:55 +08:00
3ad2e7c090 feat: 静默日跳过企微 webhook 推送
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:37:55 +08:00
588def8eb2 feat: 在 generate 流程中接入 Delta 模式与推送闸门
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:37:54 +08:00
b2dd8721d0 feat: 新增企微早报推送闸门判定逻辑
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:35:49 +08:00
e3b5623860 feat: 新增企微 Delta 榜单区块渲染与替换逻辑
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:35:49 +08:00
2f1fac9308 feat: 实现 Skills 跨榜去重与首日 baseline 模式判定
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:33:36 +08:00
ba8631c867 feat: 新增企微已推送新闻 link 去重缓存
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:33:26 +08:00
aff75141cb feat: 新增企微 Delta 模式相关配置读取函数
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 11:32:38 +08:00
132 changed files with 12242 additions and 4213 deletions

View File

@@ -0,0 +1,61 @@
# cavecrew
Decision guide. When to delegate to caveman subagents instead of doing the work inline.
## What it does
Tells the main thread when to spawn a caveman-style subagent versus the vanilla equivalent. The win: subagent tool-results inject back into main context verbatim, and caveman output is roughly 1/3 the size of vanilla prose. Across 20 delegations in one session, that is the difference between context exhaustion and finishing the task.
Three subagents:
| Subagent | Job | Use when |
|----------|-----|----------|
| `cavecrew-investigator` | Locate code (read-only) | "Where is X defined / what calls Y / list uses of Z" |
| `cavecrew-builder` | Surgical edit, 1-2 files | Scope is obvious, ≤2 files. Refuses 3+ file scope. |
| `cavecrew-reviewer` | Diff/file review | One-line findings with severity emoji |
Use vanilla `Explore` or `Code Reviewer` when you want prose, architecture commentary, or rationale. Use main thread directly for one-line answers and 3+ file refactors.
This skill is a decision guide, not a slash command. It activates when the conversation mentions delegation.
## How to invoke
Triggers on phrases like "delegate to subagent", "use cavecrew", "spawn investigator", "save context", "compressed agent output".
## Example chaining
Locate → fix → verify (most common):
1. `cavecrew-investigator` returns site list (`path:line — symbol — note`)
2. Main thread picks 1-2 sites, hands paths to `cavecrew-builder`
3. `cavecrew-reviewer` audits the resulting diff
Parallel scout: spawn 2-3 `cavecrew-investigator` calls in one message with different angles (defs, callers, tests). Aggregate in main.
## Model overrides
By default, `cavecrew-reviewer` and `cavecrew-investigator` pin `model: haiku` in their frontmatter; `cavecrew-builder` has no `model:` line (uses the API session default). Set env vars in your shell before launching Claude Code to override per-agent:
| Env var | Agent |
|---|---|
| `CAVECREW_REVIEWER_MODEL` | `cavecrew-reviewer` |
| `CAVECREW_BUILDER_MODEL` | `cavecrew-builder` |
| `CAVECREW_INVESTIGATOR_MODEL` | `cavecrew-investigator` |
Example — run reviewer on sonnet, keep others on default:
```sh
export CAVECREW_REVIEWER_MODEL=sonnet
```
Use the same model name strings you'd use in any Claude Code agent frontmatter (e.g. `haiku`, `sonnet`, `opus`).
Overrides patch only the `model:` line in the installed agent's frontmatter; the prompt body is untouched and keeps receiving upstream updates. Plugin installs only — standalone hook installs have no local agent files to patch. Unset or blank = no change. The patch persists in the installed file until the plugin is updated or reinstalled.
## See also
- [`SKILL.md`](./SKILL.md) — full decision matrix and output contracts
- [`agents/cavecrew-investigator.md`](../../agents/cavecrew-investigator.md)
- [`agents/cavecrew-builder.md`](../../agents/cavecrew-builder.md)
- [`agents/cavecrew-reviewer.md`](../../agents/cavecrew-reviewer.md)
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,82 @@
---
name: cavecrew
description: >
Decision guide for delegating to caveman-style subagents. Tells the main
thread WHEN to spawn `cavecrew-investigator` (locate code), `cavecrew-builder`
(1-2 file edit), or `cavecrew-reviewer` (diff review) instead of doing the
work inline or using vanilla `Explore`. Subagent output is caveman-compressed
so the tool-result injected back into main context is ~60% smaller — main
context lasts longer across long sessions.
Trigger: "delegate to subagent", "use cavecrew", "spawn investigator/builder/reviewer",
"save context", "compressed agent output".
---
Cavecrew = three subagent presets that emit caveman output. Same job as Anthropic defaults (`Explore`, edit-style agents, reviewer); difference is the tool-result they return is compressed, so main context shrinks per delegation.
## When to use cavecrew vs alternatives
| Task | Use |
|---|---|
| "Where is X defined / what calls Y / list uses of Z" | `cavecrew-investigator` |
| Same but you also want suggestions/architecture commentary | `Explore` (vanilla) |
| Surgical edit, ≤2 files, scope obvious | `cavecrew-builder` |
| New feature / 3+ files / cross-cutting refactor | Main thread or `feature-dev:code-architect` |
| Review diff, branch, or file for bugs | `cavecrew-reviewer` |
| Deep code review with rationale + alternatives | `Code Reviewer` (vanilla) |
| One-line answer you already know | Main thread, no subagent |
Rule of thumb: **if you'd want the subagent's output in 1/3 the tokens, pick cavecrew. If you'd want prose, pick vanilla.**
## Why this exists (the real win)
Subagent tool results get injected into main context verbatim. A vanilla `Explore` that returns 2k tokens of prose costs 2k tokens of main-context budget every time. The same finding from `cavecrew-investigator` returns ~700 tokens. Across 20 delegations in one session that's the difference between context exhaustion and finishing the task.
## Output contracts
What main thread can rely on per agent:
**`cavecrew-investigator`**
```
<Header>:
- path:line — `symbol` — short note
totals: <counts>.
```
Or `No match.` Always file-path-first, line-number-attached, backticked symbols. Safe to grep with `path:\d+`.
**`cavecrew-builder`**
```
<path:line-range> — <change ≤10 words>.
verified: <re-read OK | mismatch @ path:line>.
```
Or one of: `too-big.` / `needs-confirm.` / `ambiguous.` / `regressed.` (terminal first token).
**`cavecrew-reviewer`**
```
path:line: <emoji> <severity>: <problem>. <fix>.
totals: N🔴 N🟡 N🔵 N❓
```
Or `No issues.` Findings sorted file → line ascending.
## Chaining patterns
**Locate → fix → verify** (most common):
1. `cavecrew-investigator` returns site list.
2. Main thread picks 1-2 sites, hands paths to `cavecrew-builder`.
3. `cavecrew-reviewer` audits the diff.
**Parallel scout** (when investigation is broad):
Spawn 2-3 `cavecrew-investigator` calls in one message (different angles: defs vs callers vs tests). Aggregate in main thread.
**Single-shot edit** (when site is already known):
Skip investigator. Hand exact path:line to `cavecrew-builder` directly.
## What NOT to do
- Don't use `cavecrew-builder` when you don't already know the file. Spawn investigator first or main thread will eat tokens passing context.
- Don't chain `cavecrew-investigator → cavecrew-builder` for a 5-file refactor. Builder will return `too-big.` and you'll have wasted a turn.
- Don't ask `cavecrew-reviewer` for "general feedback" — it returns findings only, no architecture opinions. Use `Code Reviewer` for that.
- Don't expect prose. Cavecrew output is structured, sometimes terse to the point of cryptic. If a human will read it directly, paraphrase.
## Auto-clarity (inherited)
Subagents drop caveman → normal English for security warnings, irreversible-action confirmations, and any output where fragment ambiguity could be misread. Resume caveman after.

View File

@@ -0,0 +1,44 @@
# caveman-commit
Terse Conventional Commits. Why over what.
## What it does
Generates commit messages in Conventional Commits format. Subject ≤50 chars, hard cap 72. Imperative mood. Body only when the *why* is non-obvious or there are breaking changes. No AI attribution, no "this commit does X", no emoji unless the project uses them. Body always required for breaking changes, security fixes, data migrations, and reverts — future debuggers need the context.
Outputs only the message. Does not stage, commit, or amend.
## How to invoke
```
/caveman-commit
```
Also triggers on phrases like "write a commit", "commit message", "generate commit".
## Example output
Diff: new endpoint for user profile.
```
feat(api): add GET /users/:id/profile
Mobile client needs profile data without the full user payload
to reduce LTE bandwidth on cold-launch screens.
Closes #128
```
Diff: breaking API rename.
```
feat(api)!: rename /v1/orders to /v1/checkout
BREAKING CHANGE: clients on /v1/orders must migrate to /v1/checkout
before 2026-06-01. Old route returns 410 after that date.
```
## See also
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,65 @@
---
name: caveman-commit
description: >
Ultra-compressed commit message generator. Cuts noise from commit messages while preserving
intent and reasoning. Conventional Commits format. Subject ≤50 chars, body only when "why"
isn't obvious. Use when user says "write a commit", "commit message", "generate commit",
"/commit", or invokes /caveman-commit. Auto-triggers when staging changes.
---
Write commit messages terse and exact. Conventional Commits format. No fluff. Why over what.
## Rules
**Subject line:**
- `<type>(<scope>): <imperative summary>``<scope>` optional
- Types: `feat`, `fix`, `refactor`, `perf`, `docs`, `test`, `chore`, `build`, `ci`, `style`, `revert`
- Imperative mood: "add", "fix", "remove" — not "added", "adds", "adding"
- ≤50 chars when possible, hard cap 72
- No trailing period
- Match project convention for capitalization after the colon
**Body (only if needed):**
- Skip entirely when subject is self-explanatory
- Add body only for: non-obvious *why*, breaking changes, migration notes, linked issues
- Wrap at 72 chars
- Bullets `-` not `*`
- Reference issues/PRs at end: `Closes #42`, `Refs #17`
**What NEVER goes in:**
- "This commit does X", "I", "we", "now", "currently" — the diff says what
- "As requested by..." — use Co-authored-by trailer
- "Generated with Claude Code" or any AI attribution — unless the user's own rule requires an `Assisted-by`/AI-attribution trailer, then add it as a trailer
- Emoji (unless project convention requires)
- Restating the file name when scope already says it
## Examples
Diff: new endpoint for user profile with body explaining the why
- ❌ "feat: add a new endpoint to get user profile information from the database"
-
```
feat(api): add GET /users/:id/profile
Mobile client needs profile data without the full user payload
to reduce LTE bandwidth on cold-launch screens.
Closes #128
```
Diff: breaking API change
- ✅
```
feat(api)!: rename /v1/orders to /v1/checkout
BREAKING CHANGE: clients on /v1/orders must migrate to /v1/checkout
before 2026-06-01. Old route returns 410 after that date.
```
## Auto-Clarity
Always include body for: breaking changes, security fixes, data migrations, anything reverting a prior commit. Never compress these into subject-only — future debuggers need the context.
## Boundaries
Only generates the commit message. Does not run `git commit`, does not stage files, does not amend. Output the message as a code block ready to paste. "stop caveman-commit" or "normal mode": revert to verbose commit style.

View File

@@ -0,0 +1,163 @@
<p align="center">
<img src="https://em-content.zobj.net/source/apple/391/rock_1faa8.png" width="80" />
</p>
<h1 align="center">caveman-compress</h1>
<p align="center">
<strong>shrink memory file. save token every session.</strong>
</p>
---
A Claude Code skill that compresses your project memory files (`CLAUDE.md`, todos, preferences) into caveman format — so every session loads fewer tokens automatically.
Claude read `CLAUDE.md` on every session start. If file big, cost big. Caveman make file small. Cost go down forever.
## What It Do
```
/caveman-compress CLAUDE.md
```
```
CLAUDE.md ← compressed (Claude reads this — fewer tokens every session)
CLAUDE.original.md ← human-readable backup (you edit this)
```
Original never lost. You can read and edit `.original.md`. Run skill again to re-compress after edits.
## Benchmarks
Real results on real project files:
| File | Original | Compressed | Saved |
|------|----------:|----------:|------:|
| `claude-md-preferences.md` | 706 | 285 | **59.6%** |
| `project-notes.md` | 1145 | 535 | **53.3%** |
| `claude-md-project.md` | 1122 | 636 | **43.3%** |
| `todo-list.md` | 627 | 388 | **38.1%** |
| `mixed-with-code.md` | 888 | 560 | **36.9%** |
| **Average** | **898** | **481** | **46%** |
All validations passed ✅ — headings, code blocks, URLs, file paths preserved exactly.
## Before / After
<table>
<tr>
<td width="50%">
### 📄 Original (706 tokens)
> "I strongly prefer TypeScript with strict mode enabled for all new code. Please don't use `any` type unless there's genuinely no way around it, and if you do, leave a comment explaining the reasoning. I find that taking the time to properly type things catches a lot of bugs before they ever make it to runtime."
</td>
<td width="50%">
### <img src="../../docs/assets/dancing-rock.svg" width="20" height="20" alt="rock"/> Caveman (285 tokens)
> "Prefer TypeScript strict mode always. No `any` unless unavoidable — comment why if used. Proper types catch bugs early."
</td>
</tr>
</table>
**Same instructions. 60% fewer tokens. Every. Single. Session.**
## Security
`caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O patterns detected by static analysis. This is a false positive — see [SECURITY.md](./SECURITY.md) for a full explanation of what the skill does and does not do.
## Install
Compress is built in with the `caveman` plugin. Install `caveman` once, then use `/caveman-compress`.
If you need local files, the compress skill lives at:
```bash
caveman-compress/
```
**Requires:** Python 3.10+
## Usage
```
/caveman-compress <filepath>
```
Examples:
```
/caveman-compress CLAUDE.md
/caveman-compress docs/preferences.md
/caveman-compress todos.md
```
### What files work
| Type | Compress? |
|------|-----------|
| `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` | ✅ Yes |
| Extensionless natural language | ✅ Yes |
| `.py`, `.js`, `.ts`, `.json`, `.yaml` | ❌ Skip (code/config) |
| `*.original.md` | ❌ Skip (backup files) |
## How It Work
```
/caveman-compress CLAUDE.md
detect file type (no tokens)
Claude compresses (tokens — one call)
validate output (no tokens)
checks: headings, code blocks, URLs, file paths, bullets
if errors: Claude fixes cherry-picked issues only (tokens — targeted fix)
does NOT recompress — only patches broken parts
retry up to 2 times
write compressed → CLAUDE.md
write original → CLAUDE.original.md
```
Only two things use tokens: initial compression + targeted fix if validation fails. Everything else is local Python.
## What Is Preserved
Caveman compress natural language. It never touch:
- Code blocks (` ``` ` fenced or indented)
- Inline code (`` `backtick content` ``)
- URLs and links
- File paths (`/src/components/...`)
- Commands (`npm install`, `git commit`)
- Technical terms, library names, API names
- Headings (exact text preserved)
- Tables (structure preserved, cell text compressed)
- Dates, version numbers, numeric values
## Why This Matter
`CLAUDE.md` loads on **every session start**. A 1000-token project memory file costs tokens every single time you open a project. Over 100 sessions that's 100,000 tokens of overhead — just for context you already wrote.
Caveman cut that by ~46% on average. Same instructions. Same accuracy. Less waste.
```
┌────────────────────────────────────────────┐
│ TOKEN SAVINGS PER FILE █████ 46% │
│ SESSIONS THAT BENEFIT ██████████ 100% │
│ INFORMATION PRESERVED ██████████ 100% │
│ SETUP TIME █ 1x │
└────────────────────────────────────────────┘
```
## Part of Caveman
This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit — making Claude use fewer tokens without losing accuracy.
- **caveman** — make Claude *speak* like caveman (cuts response tokens ~65%)
- **caveman-compress** — make Claude *read* less (cuts context tokens ~46%)

View File

@@ -0,0 +1,31 @@
# Security
## Snyk High Risk Rating
`caveman-compress` receives a Snyk High Risk rating due to static analysis heuristics. This document explains what the skill does and does not do.
### What triggers the rating
1. **subprocess usage**: The skill calls the `claude` CLI via `subprocess.run()` as a fallback when `ANTHROPIC_API_KEY` is not set. The subprocess call uses a fixed argument list — no shell interpolation occurs. User file content is passed via stdin, not as a shell argument.
2. **File read/write**: The skill reads the file the user explicitly points it at, compresses it, and writes the result back to the same path. A `.original.md` backup is saved alongside it. No files outside the user-specified path are read or written.
### What the skill does NOT do
- Does not execute user file content as code
- Does not make network requests except to Anthropic's API (via SDK or CLI)
- Does not access files outside the path the user provides
- Does not use shell=True or string interpolation in subprocess calls
- Does not collect or transmit any data beyond the file being compressed
### Auth behavior
If `ANTHROPIC_API_KEY` is set, the skill uses the Anthropic Python SDK directly (no subprocess). If not set, it falls back to the `claude` CLI, which uses the user's existing Claude desktop authentication.
### File size limit
Files larger than 500KB are rejected before any API call is made.
### Reporting a vulnerability
If you believe you've found a genuine security issue, please open a GitHub issue with the label `security`.

View File

@@ -0,0 +1,111 @@
---
name: caveman-compress
description: >
Compress natural language memory files (CLAUDE.md, todos, preferences) into caveman format
to save input tokens. Preserves all technical substance, code, URLs, and structure.
Compressed version overwrites the original file. Human-readable backup saved as FILE.original.md.
Trigger: /caveman-compress FILEPATH or "compress memory file"
---
# Caveman Compress
## Purpose
Compress natural language files (CLAUDE.md, todos, preferences) into caveman-speak to reduce input tokens. Compressed version overwrites original. Human-readable backup saved as `<filename>.original.md`.
## Trigger
`/caveman-compress <filepath>` or when user asks to compress a memory file.
## Process
1. The compression scripts live in `scripts/` (adjacent to this SKILL.md). If the path is not immediately available, search for `scripts/__main__.py` next to this SKILL.md.
2. From the directory containing this SKILL.md, run:
python3 -m scripts <absolute_filepath>
3. The CLI will:
- detect file type (no tokens)
- call Claude to compress
- validate output (no tokens)
- if errors: cherry-pick fix with Claude (targeted fixes only, no recompression)
- retry up to 2 times
- if still failing after 2 retries: report error to user, leave original file untouched
4. Return result to user
## Compression Rules
### Remove
- Articles: a, an, the
- Filler: just, really, basically, actually, simply, essentially, generally
- Pleasantries: "sure", "certainly", "of course", "happy to", "I'd recommend"
- Hedging: "it might be worth", "you could consider", "it would be good to"
- Redundant phrasing: "in order to" → "to", "make sure to" → "ensure", "the reason is because" → "because"
- Connective fluff: "however", "furthermore", "additionally", "in addition"
### Preserve EXACTLY (never modify)
- Code blocks (fenced ``` and indented)
- Inline code (`backtick content`)
- URLs and links (full URLs, markdown links)
- File paths (`/src/components/...`, `./config.yaml`)
- Commands (`npm install`, `git commit`, `docker build`)
- Technical terms (library names, API names, protocols, algorithms)
- Proper nouns (project names, people, companies)
- Dates, version numbers, numeric values
- Environment variables (`$HOME`, `NODE_ENV`)
### Preserve Structure
- All markdown headings (keep exact heading text, compress body below)
- Bullet point hierarchy (keep nesting level)
- Numbered lists (keep numbering)
- Tables (compress cell text, keep structure)
- Frontmatter/YAML headers in markdown files
### Compress
- Use short synonyms: "big" not "extensive", "fix" not "implement a solution for", "use" not "utilize"
- Fragments OK: "Run tests before commit" not "You should always run tests before committing"
- Drop "you should", "make sure to", "remember to" — just state the action
- Merge redundant bullets that say the same thing differently
- Keep one example where multiple examples show the same pattern
CRITICAL RULE:
Anything inside ``` ... ``` must be copied EXACTLY.
Do not:
- remove comments
- remove spacing
- reorder lines
- shorten commands
- simplify anything
Inline code (`...`) must be preserved EXACTLY.
Do not modify anything inside backticks.
If file contains code blocks:
- Treat code blocks as read-only regions
- Only compress text outside them
- Do not merge sections around code
## Pattern
Original:
> You should always make sure to run the test suite before pushing any changes to the main branch. This is important because it helps catch bugs early and prevents broken builds from being deployed to production.
Compressed:
> Run tests before push to main. Catch bugs early, prevent broken prod deploys.
Original:
> The application uses a microservices architecture with the following components. The API gateway handles all incoming requests and routes them to the appropriate service. The authentication service is responsible for managing user sessions and JWT tokens.
Compressed:
> Microservices architecture. API gateway route all requests to services. Auth service manage user sessions + JWT tokens.
## Boundaries
- ONLY compress natural language files (.md, .txt, .typ, .typst, .tex, extensionless)
- NEVER modify: .py, .js, .ts, .json, .yaml, .yml, .toml, .env, .lock, .css, .html, .xml, .sql, .sh
- If file has mixed content (prose + code), compress ONLY the prose sections
- If unsure whether something is code or prose, leave it unchanged
- Original file is backed up as FILE.original.md before overwriting
- Never compress FILE.original.md (skip it)

View File

@@ -0,0 +1,9 @@
"""Caveman compress scripts.
This package provides tools to compress natural language markdown files
into caveman format to save input tokens.
"""
__all__ = ["cli", "compress", "detect", "validate"]
__version__ = "1.0.0"

View File

@@ -0,0 +1,3 @@
from .cli import main
main()

View File

@@ -0,0 +1,80 @@
#!/usr/bin/env python3
from pathlib import Path
import sys
# Support both direct execution and module import
try:
from .validate import validate
except ImportError:
sys.path.insert(0, str(Path(__file__).parent))
from validate import validate
try:
import tiktoken
_enc = tiktoken.get_encoding("o200k_base")
except ImportError:
_enc = None
def count_tokens(text):
if _enc is None:
return len(text.split()) # fallback: word count
return len(_enc.encode(text))
def benchmark_pair(orig_path: Path, comp_path: Path):
orig_text = orig_path.read_text()
comp_text = comp_path.read_text()
orig_tokens = count_tokens(orig_text)
comp_tokens = count_tokens(comp_text)
saved = 100 * (orig_tokens - comp_tokens) / orig_tokens if orig_tokens > 0 else 0.0
result = validate(orig_path, comp_path)
return (comp_path.name, orig_tokens, comp_tokens, saved, result.is_valid)
def print_table(rows):
print("\n| File | Original | Compressed | Saved % | Valid |")
print("|------|----------|------------|---------|-------|")
for r in rows:
print(f"| {r[0]} | {r[1]} | {r[2]} | {r[3]:.1f}% | {'' if r[4] else ''} |")
def main():
# Direct file pair: python3 benchmark.py original.md compressed.md
if len(sys.argv) == 3:
orig = Path(sys.argv[1]).resolve()
comp = Path(sys.argv[2]).resolve()
if not orig.exists():
print(f"❌ Not found: {orig}")
sys.exit(1)
if not comp.exists():
print(f"❌ Not found: {comp}")
sys.exit(1)
print_table([benchmark_pair(orig, comp)])
return
# Glob mode: repo_root/tests/caveman-compress/
# __file__ lives at <repo_root>/skills/caveman-compress/scripts/benchmark.py
# Walk up four dirs: scripts → caveman-compress → skills → repo_root.
tests_dir = Path(__file__).resolve().parents[3] / "tests" / "caveman-compress"
if not tests_dir.exists():
print(f"❌ Tests dir not found: {tests_dir}")
sys.exit(1)
rows = []
for orig in sorted(tests_dir.glob("*.original.md")):
comp = orig.with_name(orig.stem.removesuffix(".original") + ".md")
if comp.exists():
rows.append(benchmark_pair(orig, comp))
if not rows:
print("No compressed file pairs found.")
return
print_table(rows)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,85 @@
#!/usr/bin/env python3
"""
Caveman Compress CLI
Usage:
caveman <filepath>
"""
import sys
# Force UTF-8 on stdout/stderr before any code can print. Windows consoles
# default to cp1252 and crash on the ❌ glyphs in error/validation branches,
# masking the real error and leaving the user with a half-compressed file.
for _stream in (sys.stdout, sys.stderr):
reconfigure = getattr(_stream, "reconfigure", None)
if callable(reconfigure):
try:
reconfigure(encoding="utf-8", errors="replace")
except Exception:
pass
from pathlib import Path
from .compress import backup_dir_for, compress_file
from .detect import detect_file_type, should_compress
def print_usage():
print("Usage: caveman <filepath>")
def main():
if len(sys.argv) != 2:
print_usage()
sys.exit(1)
filepath = Path(sys.argv[1])
# Check file exists
if not filepath.exists():
print(f"❌ File not found: {filepath}")
sys.exit(1)
if not filepath.is_file():
print(f"❌ Not a file: {filepath}")
sys.exit(1)
filepath = filepath.resolve()
# Detect file type
file_type = detect_file_type(filepath)
print(f"Detected: {file_type}")
# Check if compressible
if not should_compress(filepath):
print("Skipping: file is not natural language (code/config)")
sys.exit(0)
print("Starting caveman compression...\n")
try:
success = compress_file(filepath)
if success:
print("\nCompression completed successfully")
backup_path = backup_dir_for(filepath) / (filepath.stem + ".original.md")
print(f"Compressed: {filepath}")
print(f"Original: {backup_path}")
sys.exit(0)
else:
print("\n❌ Compression failed after retries")
sys.exit(2)
except KeyboardInterrupt:
print("\nInterrupted by user")
sys.exit(130)
except Exception as e:
print(f"\n❌ Error: {e}")
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,342 @@
#!/usr/bin/env python3
"""
Caveman Memory Compression Orchestrator
Usage:
python scripts/compress.py <filepath>
"""
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
from typing import List
OUTER_FENCE_REGEX = re.compile(
r"\A\s*(`{3,}|~{3,})[^\n]*\n(.*)\n\1\s*\Z", re.DOTALL
)
# YAML frontmatter: starts at file start with --- on its own line, ends with --- on its own line.
# Captures the entire block (including delimiters and trailing newline) and the body after.
FRONTMATTER_REGEX = re.compile(
r"\A(---\r?\n.*?\r?\n---\r?\n)(.*)", re.DOTALL
)
def split_frontmatter(text: str):
"""Split YAML frontmatter from body. Returns (frontmatter, body).
Memory files (and many other markdown docs) start with a YAML frontmatter
block delimited by `---` lines. The compression LLM has a habit of stripping
or rewriting these despite preserve-structure rules in the prompt — so we
surgically remove the frontmatter before compression and prepend it back
verbatim to the output. Files without frontmatter pass through unchanged.
"""
m = FRONTMATTER_REGEX.match(text)
if m:
return m.group(1), m.group(2)
return "", text
# Filenames and paths that almost certainly hold secrets or PII. Compressing
# them ships raw bytes to the Anthropic API — a third-party data boundary that
# developers on sensitive codebases cannot cross. detect.py already skips .env
# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would
# slip through the natural-language filter. This is a hard refuse before read.
SENSITIVE_BASENAME_REGEX = re.compile(
r"(?ix)^("
r"\.env(\..+)?"
r"|\.netrc"
r"|credentials(\..+)?"
r"|secrets?(\..+)?"
r"|passwords?(\..+)?"
r"|id_(rsa|dsa|ecdsa|ed25519)(\.pub)?"
r"|authorized_keys"
r"|known_hosts"
r"|.*\.(pem|key|p12|pfx|crt|cer|jks|keystore|asc|gpg)"
r")$"
)
SENSITIVE_PATH_COMPONENTS = frozenset({".ssh", ".aws", ".gnupg", ".kube", ".docker"})
SENSITIVE_NAME_TOKENS = (
"secret", "credential", "password", "passwd",
"apikey", "accesskey", "token", "privatekey",
)
def backup_dir_for(filepath: Path) -> Path:
"""Resolve the out-of-tree backup directory for a given source file.
Backups must live OUTSIDE the source directory so skill auto-loaders
(Claude Code rules/, opencode instructions/, etc.) stop re-ingesting the
`.original.md` copies as live files. Base dir is platform-aware:
- Windows: %LOCALAPPDATA%\\caveman-compress\\backups
- else: $XDG_DATA_HOME/caveman-compress/backups if set,
else ~/.local/share/caveman-compress/backups
The source file's parent-dir name is mirrored under the base to reduce
cross-project collisions (e.g. two `task.md` files in different repos).
"""
if os.name == "nt" or sys.platform == "win32":
local_appdata = os.environ.get("LOCALAPPDATA")
base = Path(local_appdata) if local_appdata else Path.home() / "AppData" / "Local"
base = base / "caveman-compress" / "backups"
else:
xdg = os.environ.get("XDG_DATA_HOME")
base = Path(xdg) if xdg else Path.home() / ".local" / "share"
base = base / "caveman-compress" / "backups"
return base / filepath.parent.name
def is_sensitive_path(filepath: Path) -> bool:
"""Heuristic denylist for files that must never be shipped to a third-party API."""
name = filepath.name
if SENSITIVE_BASENAME_REGEX.match(name):
return True
lowered_parts = {p.lower() for p in filepath.parts}
if lowered_parts & SENSITIVE_PATH_COMPONENTS:
return True
# Normalize separators so "api-key" and "api_key" both match "apikey".
lower = re.sub(r"[_\-\s.]", "", name.lower())
return any(tok in lower for tok in SENSITIVE_NAME_TOKENS)
def strip_llm_wrapper(text: str) -> str:
"""Strip outer ```markdown ... ``` fence when it wraps the entire output."""
m = OUTER_FENCE_REGEX.match(text)
if m:
return m.group(2)
return text
from .detect import should_compress
from .validate import validate
MAX_RETRIES = 2
# ---------- Claude Calls ----------
def call_claude(prompt: str) -> str:
"""Send a prompt to Claude.
Prefers the Anthropic SDK when ANTHROPIC_API_KEY is set; otherwise falls
back to the ``claude --print`` CLI (which handles desktop auth).
On Windows the CLI subprocess decoding defaults to the system codepage
(cp1251 / cp1252) and crashes on UTF-8 output — see issue #152. Pinning
``encoding="utf-8"`` with ``errors="replace"`` matches the CLI's actual
native I/O and prevents the UnicodeDecodeError before validation can
report. Windows users with non-ASCII content can also set
``ANTHROPIC_API_KEY`` to route through the SDK and skip the subprocess.
"""
api_key = os.environ.get("ANTHROPIC_API_KEY")
if api_key:
try:
import anthropic
client = anthropic.Anthropic(api_key=api_key)
msg = client.messages.create(
model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),
max_tokens=8192,
messages=[{"role": "user", "content": prompt}],
)
return strip_llm_wrapper(msg.content[0].text.strip())
except ImportError:
pass # anthropic not installed, fall back to CLI
# Fallback: use claude CLI (handles desktop auth).
# Resolve binary via shutil.which so Windows .cmd/.bat shims (e.g.
# %APPDATA%\npm\claude.CMD) work without shell=True. On POSIX,
# shutil.which returns the same absolute path as the implicit lookup,
# so this is a no-op there. Falls back to bare "claude" if not found
# on PATH so subprocess raises a clear FileNotFoundError.
claude_bin = shutil.which("claude") or "claude"
try:
result = subprocess.run(
[claude_bin, "--print"],
input=prompt,
text=True,
capture_output=True,
check=True,
encoding="utf-8",
errors="replace",
)
return strip_llm_wrapper(result.stdout.strip())
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Claude call failed:\n{e.stderr}")
def build_compress_prompt(original: str) -> str:
return f"""
Compress this markdown into caveman format.
STRICT RULES:
- Do NOT modify anything inside ``` code blocks
- Do NOT modify anything inside inline backticks
- Preserve ALL URLs exactly
- Preserve ALL headings exactly
- Preserve file paths and commands
- Return ONLY the compressed markdown body — do NOT wrap the entire output in a ```markdown fence or any other fence. Inner code blocks from the original stay as-is; do not add a new outer fence around the whole file.
Only compress natural language.
TEXT:
{original}
"""
def build_fix_prompt(original: str, compressed: str, errors: List[str]) -> str:
errors_str = "\n".join(f"- {e}" for e in errors)
return f"""You are fixing a caveman-compressed markdown file. Specific validation errors were found.
CRITICAL RULES:
- DO NOT recompress or rephrase the file
- ONLY fix the listed errors — leave everything else exactly as-is
- The ORIGINAL is provided as reference only (to restore missing content)
- Preserve caveman style in all untouched sections
ERRORS TO FIX:
{errors_str}
HOW TO FIX:
- Missing URL: find it in ORIGINAL, restore it exactly where it belongs in COMPRESSED
- Code block mismatch: find the exact code block in ORIGINAL, restore it in COMPRESSED
- Heading mismatch: restore the exact heading text from ORIGINAL into COMPRESSED
- Do not touch any section not mentioned in the errors
ORIGINAL (reference only):
{original}
COMPRESSED (fix this):
{compressed}
Return ONLY the fixed compressed file. No explanation.
"""
# ---------- Core Logic ----------
def compress_file(filepath: Path) -> bool:
# Resolve and validate path
filepath = filepath.resolve()
MAX_FILE_SIZE = 500_000 # 500KB
if not filepath.exists():
raise FileNotFoundError(f"File not found: {filepath}")
if filepath.stat().st_size > MAX_FILE_SIZE:
raise ValueError(f"File too large to compress safely (max 500KB): {filepath}")
# Refuse files that look like they contain secrets or PII. Compressing ships
# the raw bytes to the Anthropic API — a third-party boundary — so we fail
# loudly rather than silently exfiltrate credentials or keys. Override is
# intentional: the user must rename the file if the heuristic is wrong.
if is_sensitive_path(filepath):
raise ValueError(
f"Refusing to compress {filepath}: filename looks sensitive "
"(credentials, keys, secrets, or known private paths). "
"Compression sends file contents to the Anthropic API. "
"Rename the file if this is a false positive."
)
print(f"Processing: {filepath}")
if not should_compress(filepath):
print("Skipping (not natural language)")
return False
original_text = filepath.read_text(errors="ignore")
# Store backup outside the source directory so skill auto-loaders don't
# re-ingest the `.original.md` copy as a live file. Mirror the source's
# parent-dir name + stem under a platform-aware base to reduce collisions.
backup_dir = backup_dir_for(filepath)
backup_dir.mkdir(parents=True, exist_ok=True)
backup_path = backup_dir / (filepath.stem + ".original.md")
if not original_text.strip():
print("❌ Refusing to compress: file is empty or whitespace-only.")
return False
# Check if backup already exists to prevent accidental overwriting
if backup_path.exists():
print(f"⚠️ Backup file already exists: {backup_path}")
print("The original backup may contain important content.")
print("Aborting to prevent data loss. Please remove or rename the backup file if you want to proceed.")
return False
# Split YAML frontmatter off before compression. Claude tends to strip or
# rewrite frontmatter despite preserve-structure rules; we keep it verbatim
# by removing it from the input and re-prepending it to the output.
frontmatter, body = split_frontmatter(original_text)
if frontmatter:
print(f"Detected YAML frontmatter ({len(frontmatter)} chars) — preserving verbatim")
if not body.strip():
print("❌ Refusing to compress: body is empty after frontmatter removal.")
return False
# Step 1: Compress (body only, frontmatter excluded)
print("Compressing with Claude...")
compressed_body = call_claude(build_compress_prompt(body))
if compressed_body is None or not compressed_body.strip():
print("❌ Compression aborted: Claude returned an empty response.")
print(" Original file is untouched (no backup created).")
return False
# Compare the BODY (not the whole file) — frontmatter is preserved verbatim
# and would never change, so identity must be judged on the compressible part.
if compressed_body.strip() == body.strip():
print("❌ Compression aborted: output is identical to input.")
print(" Likely causes: Claude refused, returned the prompt verbatim, or the file is")
print(" already in caveman form. Original file is untouched (no backup created).")
return False
# Reassemble: frontmatter (verbatim) + compressed body
compressed = frontmatter + compressed_body
# Save original as backup, then verify the backup readback before
# touching the input file. If the filesystem dropped bytes (encoding,
# antivirus, disk full), unlink the bad backup and abort instead of
# leaving the user with a corrupt backup + compressed primary.
backup_path.write_text(original_text)
backup_readback = backup_path.read_text(errors="ignore")
if backup_readback != original_text:
print(f"❌ Backup write verification failed: {backup_path}")
print(" In-memory original differs from on-disk backup. Aborting before touching the input file.")
try:
backup_path.unlink()
except OSError:
pass
return False
filepath.write_text(compressed)
# Step 2: Validate + Retry
for attempt in range(MAX_RETRIES):
print(f"\nValidation attempt {attempt + 1}")
result = validate(backup_path, filepath)
if result.is_valid:
print("Validation passed")
break
print("❌ Validation failed:")
for err in result.errors:
print(f" - {err}")
if attempt == MAX_RETRIES - 1:
# Restore original on failure
filepath.write_text(original_text)
backup_path.unlink(missing_ok=True)
print("❌ Failed after retries — original restored")
return False
print("Fixing with Claude...")
compressed = call_claude(
build_fix_prompt(original_text, compressed, result.errors)
)
filepath.write_text(compressed)
return True

View File

@@ -0,0 +1,139 @@
#!/usr/bin/env python3
"""Detect whether a file is natural language (compressible) or code/config (skip)."""
import json
import re
from pathlib import Path
# Extensions that are natural language and compressible
COMPRESSIBLE_EXTENSIONS = {".md", ".txt", ".markdown", ".rst", ".typ", ".typst", ".tex"}
# Extensions that are code/config and should be skipped
SKIP_EXTENSIONS = {
".py", ".js", ".ts", ".tsx", ".jsx", ".json", ".yaml", ".yml",
".toml", ".env", ".lock", ".css", ".scss", ".html", ".xml",
".sql", ".sh", ".bash", ".zsh", ".go", ".rs", ".java", ".c",
".cpp", ".h", ".hpp", ".rb", ".php", ".swift", ".kt", ".lua",
".dockerfile", ".makefile", ".csv", ".ini", ".cfg",
}
# Well-known build/config files that carry no (or a misleading) extension —
# `Dockerfile` has no suffix so `.dockerfile` above never matches it, and
# `CMakeLists.txt` would ride the compressible `.txt` rule. Checked by
# basename before any extension rule.
KNOWN_CODE_FILENAMES = {
"dockerfile", "makefile", "gnumakefile", "jenkinsfile", "vagrantfile",
"rakefile", "gemfile", "justfile", "procfile", "brewfile",
"cmakelists.txt",
}
# Patterns that indicate a line is code
CODE_PATTERNS = [
re.compile(r"^\s*(import |from .+ import |require\(|const |let |var )"),
re.compile(r"^\s*(def |class |function |async function |export )"),
re.compile(r"^\s*(if\s*\(|for\s*\(|while\s*\(|switch\s*\(|try\s*\{)"),
re.compile(r"^\s*[\}\]\);]+\s*$"), # closing braces/brackets
re.compile(r"^\s*@\w+"), # decorators/annotations
re.compile(r'^\s*"[^"]+"\s*:\s*'), # JSON-like key-value
re.compile(r"^\s*\w+\s*=\s*[{\[\(\"']"), # assignment with literal
]
def _is_code_line(line: str) -> bool:
"""Check if a line looks like code."""
return any(p.match(line) for p in CODE_PATTERNS)
def _is_json_content(text: str) -> bool:
"""Check if content is valid JSON."""
try:
json.loads(text)
return True
except (json.JSONDecodeError, ValueError):
return False
def _is_yaml_content(lines: list[str]) -> bool:
"""Heuristic: check if content looks like YAML."""
yaml_indicators = 0
for line in lines[:30]:
stripped = line.strip()
if stripped.startswith("---"):
yaml_indicators += 1
elif re.match(r"^\w[\w\s]*:\s", stripped):
yaml_indicators += 1
elif stripped.startswith("- ") and ":" in stripped:
yaml_indicators += 1
# If most non-empty lines look like YAML
non_empty = sum(1 for l in lines[:30] if l.strip())
return non_empty > 0 and yaml_indicators / non_empty > 0.6
def detect_file_type(filepath: Path) -> str:
"""Classify a file as 'natural_language', 'code', 'config', or 'unknown'.
Returns:
One of: 'natural_language', 'code', 'config', 'unknown'
"""
ext = filepath.suffix.lower()
# Known code filenames win over any extension rule
if filepath.name.lower() in KNOWN_CODE_FILENAMES:
return "code"
# Extension-based classification
if ext in COMPRESSIBLE_EXTENSIONS:
return "natural_language"
if ext in SKIP_EXTENSIONS:
return "code" if ext not in {".json", ".yaml", ".yml", ".toml", ".ini", ".cfg", ".env"} else "config"
# Extensionless files (like CLAUDE.md, TODO) — check content
if not ext:
try:
text = filepath.read_text(errors="ignore")
except (OSError, PermissionError):
return "unknown"
lines = text.splitlines()[:50]
# Shebang means executable script, never prose
if text.startswith("#!"):
return "code"
if _is_json_content(text[:10000]):
return "config"
if _is_yaml_content(lines):
return "config"
code_lines = sum(1 for l in lines if l.strip() and _is_code_line(l))
non_empty = sum(1 for l in lines if l.strip())
if non_empty > 0 and code_lines / non_empty > 0.4:
return "code"
return "natural_language"
return "unknown"
def should_compress(filepath: Path) -> bool:
"""Return True if the file is natural language and should be compressed."""
if not filepath.is_file():
return False
# Skip backup files
if filepath.name.endswith(".original.md"):
return False
return detect_file_type(filepath) == "natural_language"
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("Usage: python detect.py <file1> [file2] ...")
sys.exit(1)
for path_str in sys.argv[1:]:
p = Path(path_str).resolve()
file_type = detect_file_type(p)
compress = should_compress(p)
print(f" {p.name:30s} type={file_type:20s} compress={compress}")

View File

@@ -0,0 +1,213 @@
#!/usr/bin/env python3
import re
from collections import Counter
from pathlib import Path
URL_REGEX = re.compile(r"https?://[^\s)]+")
FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
# crude but effective path detection
# Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
class ValidationResult:
def __init__(self):
self.is_valid = True
self.errors = []
self.warnings = []
def add_error(self, msg):
self.is_valid = False
self.errors.append(msg)
def add_warning(self, msg):
self.warnings.append(msg)
def read_file(path: Path) -> str:
return path.read_text(errors="ignore")
# ---------- Extractors ----------
def extract_headings(text):
return [(level, title.strip()) for level, title in HEADING_REGEX.findall(text)]
def extract_code_blocks(text):
"""Line-based fenced code block extractor.
Handles ``` and ~~~ fences with variable length (CommonMark: closing
fence must use same char and be at least as long as opening). Supports
nested fences (e.g. an outer 4-backtick block wrapping inner 3-backtick
content).
"""
blocks = []
lines = text.split("\n")
i = 0
n = len(lines)
while i < n:
m = FENCE_OPEN_REGEX.match(lines[i])
if not m:
i += 1
continue
fence_char = m.group(2)[0]
fence_len = len(m.group(2))
open_line = lines[i]
block_lines = [open_line]
i += 1
closed = False
while i < n:
close_m = FENCE_OPEN_REGEX.match(lines[i])
if (
close_m
and close_m.group(2)[0] == fence_char
and len(close_m.group(2)) >= fence_len
and close_m.group(3).strip() == ""
):
block_lines.append(lines[i])
closed = True
i += 1
break
block_lines.append(lines[i])
i += 1
if closed:
blocks.append("\n".join(block_lines))
# Unclosed fences are silently skipped — they indicate malformed markdown
# and including them would cause false-positive validation failures.
return blocks
def extract_urls(text):
return set(URL_REGEX.findall(text))
def extract_paths(text):
return set(PATH_REGEX.findall(text))
def count_bullets(text):
return len(BULLET_REGEX.findall(text))
def extract_inline_codes(text):
text_without_fences = re.sub(r"^```[\s\S]*?^```", "", text, flags=re.MULTILINE)
text_without_fences = re.sub(r"^~~~[\s\S]*?^~~~", "", text_without_fences, flags=re.MULTILINE)
return re.findall(r"`([^`]+)`", text_without_fences)
# ---------- Validators ----------
def validate_headings(orig, comp, result):
h1 = extract_headings(orig)
h2 = extract_headings(comp)
if len(h1) != len(h2):
result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
if h1 != h2:
result.add_warning("Heading text/order changed")
def validate_code_blocks(orig, comp, result):
c1 = extract_code_blocks(orig)
c2 = extract_code_blocks(comp)
if c1 != c2:
result.add_error("Code blocks not preserved exactly")
def validate_urls(orig, comp, result):
u1 = extract_urls(orig)
u2 = extract_urls(comp)
if u1 != u2:
result.add_error(f"URL mismatch: lost={u1 - u2}, added={u2 - u1}")
def validate_paths(orig, comp, result):
p1 = extract_paths(orig)
p2 = extract_paths(comp)
if p1 != p2:
result.add_warning(f"Path mismatch: lost={p1 - p2}, added={p2 - p1}")
def validate_bullets(orig, comp, result):
b1 = count_bullets(orig)
b2 = count_bullets(comp)
if b1 == 0:
return
diff = abs(b1 - b2) / b1
if diff > 0.15:
result.add_warning(f"Bullet count changed too much: {b1} -> {b2}")
def validate_inline_codes(orig, comp, result):
c1 = Counter(extract_inline_codes(orig))
c2 = Counter(extract_inline_codes(comp))
if c1 != c2:
lost = set(c1.keys()) - set(c2.keys())
added = set(c2.keys()) - set(c1.keys())
for code, count in c1.items():
if code in c2 and c2[code] < count:
lost.add(f"{code} (lost {count - c2[code]} of {count} occurrences)")
if lost:
result.add_error(f"Inline code lost: {lost}")
if added:
result.add_warning(f"Inline code added: {added}")
# ---------- Main ----------
def validate(original_path: Path, compressed_path: Path) -> ValidationResult:
result = ValidationResult()
orig = read_file(original_path)
comp = read_file(compressed_path)
validate_headings(orig, comp, result)
validate_code_blocks(orig, comp, result)
validate_urls(orig, comp, result)
validate_paths(orig, comp, result)
validate_bullets(orig, comp, result)
validate_inline_codes(orig, comp, result)
return result
# ---------- CLI ----------
if __name__ == "__main__":
import sys
if len(sys.argv) != 3:
print("Usage: python validate.py <original> <compressed>")
sys.exit(1)
orig = Path(sys.argv[1]).resolve()
comp = Path(sys.argv[2]).resolve()
res = validate(orig, comp)
print(f"\nValid: {res.is_valid}")
if res.errors:
print("\nErrors:")
for e in res.errors:
print(f" - {e}")
if res.warnings:
print("\nWarnings:")
for w in res.warnings:
print(f" - {w}")

View File

@@ -0,0 +1,38 @@
# caveman-help
Quick-reference card. One shot, no mode change.
## What it does
Prints a cheat sheet of all caveman modes, sibling skills, deactivation triggers, and how to set the default mode via env var or config file. One-shot display — does not flip the active mode, write flag files, or persist anything. Use when you forget the slash commands.
## How to invoke
```
/caveman-help
```
Also triggers on "caveman help", "what caveman commands", "how do I use caveman".
## Example output
```
Modes:
/caveman full (default)
/caveman lite lighter
/caveman ultra extreme
/caveman wenyan classical Chinese
Skills:
/caveman-commit terse Conventional Commits
/caveman-review one-line PR comments
/caveman-stats session token savings
Deactivate:
"stop caveman" or "normal mode"
```
## See also
- [`SKILL.md`](./SKILL.md) — full reference card
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,63 @@
---
name: caveman-help
description: >
Quick-reference card for all caveman modes, skills, and commands.
One-shot display, not a persistent mode. Trigger: /caveman-help,
"caveman help", "what caveman commands", "how do I use caveman".
---
# Caveman Help
Display this reference card when invoked. One-shot — do NOT change mode, write flag files, or persist anything. Output in caveman style.
## Modes
| Mode | Trigger | What change |
|------|---------|-------------|
| **Lite** | `/caveman lite` | Drop filler. Keep sentence structure. |
| **Full** | `/caveman` | Drop articles, filler, pleasantries, hedging. Fragments OK. Default. |
| **Ultra** | `/caveman ultra` | Extreme compression. Bare fragments. Tables over prose. |
| **Wenyan-Lite** | `/caveman wenyan-lite` | Classical Chinese style, light compression. |
| **Wenyan-Full** | `/caveman wenyan` | Full 文言文. Maximum classical terseness. |
| **Wenyan-Ultra** | `/caveman wenyan-ultra` | Extreme. Ancient scholar on a budget. |
Mode stick until changed or session end.
## Skills
| Skill | Trigger | What it do |
|-------|---------|-----------|
| **caveman-commit** | `/caveman-commit` | Terse commit messages. Conventional Commits. ≤50 char subject. |
| **caveman-review** | `/caveman-review` | One-line PR comments: `L42: bug: user null. Add guard.` |
| **caveman-compress** | `/caveman-compress <file>` | Compress .md files to caveman prose. Saves ~46% input tokens. |
| **caveman-help** | `/caveman-help` | This card. |
## Deactivate
Say "stop caveman" or "normal mode". Resume anytime with `/caveman`.
## Language
Keep user's language by default. User write Portuguese → reply Portuguese caveman. Compress the style, not the language. Technical terms, code, commands, commit types, and exact error strings stay verbatim unless user ask for translation.
## Configure Default Mode
Default mode = `full`. Change it:
**Environment variable** (highest priority):
```bash
export CAVEMAN_DEFAULT_MODE=ultra
```
**Config file** (`~/.config/caveman/config.json`):
```json
{ "defaultMode": "lite" }
```
Set `"off"` to disable auto-activation on session start. User can still activate manually with `/caveman`.
Resolution: env var > config file > `full`.
## More
Full docs: https://github.com/JuliusBrussee/caveman

View File

@@ -0,0 +1,33 @@
# caveman-review
One-line PR comments. Location, problem, fix. No throat-clearing.
## What it does
Generates code review comments in `L<line>: <severity> <problem>. <fix>.` format. One line per finding. Severity emoji: 🔴 bug, 🟡 risk, 🔵 nit, ❓ question. Drops "I noticed that...", hedging, and restating what the diff already shows. Keeps exact line numbers, backticked symbols, and concrete fixes.
Auto-clarity: drops terse mode for CVE-class security findings, architectural disagreements, and onboarding contexts where the author needs the *why*. Resumes terse for the rest.
Output only — does not approve, request changes, or run linters.
## How to invoke
```
/caveman-review
```
Also triggers on "review this PR", "code review", "review the diff".
## Example output
```
L42: 🔴 bug: user can be null after .find(). Add guard before .email.
L88-140: 🔵 nit: 50-line fn does 4 things. Extract validate/normalize/persist.
L23: 🟡 risk: no retry on 429. Wrap in withBackoff(3).
L107: ❓ q: why drop the cache here? Reads on next request will miss.
```
## See also
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,55 @@
---
name: caveman-review
description: >
Ultra-compressed code review comments. Cuts noise from PR feedback while preserving
the actionable signal. Each comment is one line: location, problem, fix. Use when user
says "review this PR", "code review", "review the diff", "/review", or invokes
/caveman-review. Auto-triggers when reviewing pull requests.
---
Write code review comments terse and actionable. One line per finding. Location, problem, fix. No throat-clearing.
## Rules
**Format:** `L<line>: <problem>. <fix>.` — or `<file>:L<line>: ...` when reviewing multi-file diffs.
**Severity prefix (optional, when mixed):**
- `🔴 bug:` — broken behavior, will cause incident
- `🟡 risk:` — works but fragile (race, missing null check, swallowed error)
- `🔵 nit:` — style, naming, micro-optim. Author can ignore
- `❓ q:` — genuine question, not a suggestion
**Drop:**
- "I noticed that...", "It seems like...", "You might want to consider..."
- "This is just a suggestion but..." — use `nit:` instead
- "Great work!", "Looks good overall but..." — say it once at the top, not per comment
- Restating what the line does — the reviewer can read the diff
- Hedging ("perhaps", "maybe", "I think") — if unsure use `q:`
**Keep:**
- Exact line numbers
- Exact symbol/function/variable names in backticks
- Concrete fix, not "consider refactoring this"
- The *why* if the fix isn't obvious from the problem statement
## Examples
❌ "I noticed that on line 42 you're not checking if the user object is null before accessing the email property. This could potentially cause a crash if the user is not found in the database. You might want to add a null check here."
`L42: 🔴 bug: user can be null after .find(). Add guard before .email.`
❌ "It looks like this function is doing a lot of things and might benefit from being broken up into smaller functions for readability."
`L88-140: 🔵 nit: 50-line fn does 4 things. Extract validate/normalize/persist.`
❌ "Have you considered what happens if the API returns a 429? I think we should probably handle that case."
`L23: 🟡 risk: no retry on 429. Wrap in withBackoff(3).`
## Auto-Clarity
Drop terse mode for: security findings (CVE-class bugs need full explanation + reference), architectural disagreements (need rationale, not just a one-liner), and onboarding contexts where the author is new and needs the "why". In those cases write a normal paragraph, then resume terse for the rest.
## Boundaries
Reviews only — does not write the code fix, does not approve/request-changes, does not run linters. Output the comment(s) ready to paste into the PR. "stop caveman-review" or "normal mode": revert to verbose review style.

View File

@@ -0,0 +1,30 @@
# caveman-stats
Real session token receipts. No AI estimation.
## What it does
Reads the current Claude Code session log directly and reports actual input/output token usage plus estimated savings versus a non-caveman baseline. Numbers come from the JSONL session log on disk — the model itself does not compute or estimate them. Output is injected by the `caveman-mode-tracker` hook, which intercepts `/caveman-stats` and returns the formatted stats as a blocked-decision reason.
Each run also writes a lifetime-savings suffix file used by the statusline badge (`⛏ 12.4k`).
## How to invoke
```
/caveman-stats
```
## Example output
```
Session: 47 turns
Input: 12,304 tokens
Output: 3,891 tokens (caveman)
Baseline: 11,247 tokens (estimated without caveman)
Saved: 7,356 tokens (~65%)
```
## See also
- [`SKILL.md`](./SKILL.md) — hook contract and mechanics
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,10 @@
---
name: caveman-stats
description: >
Show real token usage and estimated savings for the current session.
Reads directly from the Claude Code session log — no AI estimation.
Triggers on /caveman-stats. Output is injected by the mode-tracker hook;
the model itself does not compute the numbers.
---
This skill is delivered by `hooks/caveman-stats.js` (read by `hooks/caveman-mode-tracker.js` on `/caveman-stats`). The model does not need to do anything when this skill fires — the hook returns `decision: "block"` with the formatted stats as the reason. The user sees the numbers immediately.

View File

@@ -0,0 +1,48 @@
# caveman
Talk like smart caveman. Same brain, fewer tokens.
## What it does
Compress every model response to caveman-style prose. Drops articles, filler, pleasantries, and hedging. Keeps every technical detail, code block, error string, and symbol exact. Cuts 65% of output tokens (measured) with full accuracy preserved. Mode persists for the whole session until changed or stopped.
Six intensity levels:
| Level | What change |
|-------|-------------|
| `lite` | Drop filler/hedging. Sentences stay full. Professional but tight. |
| `full` | Default. Drop articles, fragments OK, short synonyms. |
| `ultra` | Bare fragments. Abbreviations (DB, auth, fn). Arrows for causality. |
| `wenyan-lite` | Classical Chinese register, light compression. |
| `wenyan-full` | Maximum 文言文. 80-90% character reduction. |
| `wenyan-ultra` | Extreme classical compression. |
Auto-clarity rule: caveman drops to normal prose for security warnings, irreversible-action confirmations, multi-step sequences where fragment ambiguity risks misread, and when user repeats a question. Resumes after the clear part.
## How to invoke
```
/caveman # full mode (default)
/caveman lite # lighter compression
/caveman ultra # extreme compression
/caveman wenyan # classical Chinese
stop caveman # back to normal prose
```
## Example output
Question: "Why does my React component re-render?"
Normal prose:
> Your component re-renders because you create a new object reference each render. Wrapping it in `useMemo` will fix the issue.
Caveman (full):
> New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`.
Caveman (ultra):
> Inline obj prop → new ref → re-render. `useMemo`.
## See also
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
- [Caveman README](../../README.md) — repo overview, install, benchmarks

View File

@@ -0,0 +1,78 @@
---
name: caveman
description: >
Ultra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman
while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra,
wenyan-lite, wenyan-full, wenyan-ultra.
Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens",
"be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested.
---
Respond terse like smart caveman. All technical substance stay. Only fluff die.
## Persistence
ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode".
Default: **full**. Switch: `/caveman lite|full|ultra`.
## Rules
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). No tool-call narration, no decorative tables/emoji, no dumping long raw error logs unless asked — quote shortest decisive line. Standard well-known tech acronyms OK (DB/API/HTTP); never invent new abbreviations (cfg/impl/req/res/fn) — tokenizer split them same as full word: zero token saved, reader still decode. Full word cheaper AND clearer. No causal arrows (→) either — own token, save nothing. Technical terms exact. Code blocks unchanged. Errors quoted exact.
Preserve user's dominant language. User write Portuguese → reply Portuguese caveman. User write Spanish → reply Spanish caveman. Compress the style, not the language. No forced English openings or status phrases. ALWAYS keep technical terms, code, API names, CLI commands, commit-type keywords (feat/fix/...), and exact error strings verbatim — unless user explicitly ask for translation.
No self-reference. Never name or announce the style. No "caveman mode on", "me caveman think", no third-person caveman tags. Output caveman-only — never normal answer plus "Caveman:" recap. Exception: user explicitly ask what the mode is.
Pattern: `[thing] [action] [reason]. [next step].`
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
## Intensity
| Level | What change |
|-------|------------|
| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |
| **full** | Drop articles, fragments OK, short synonyms. Classic caveman. No tool-call narration, no decorative tables/emoji, no long raw error-log dumps unless asked. Standard acronyms OK; no invented abbreviations |
| **ultra** | Strip conjunctions when cause-then-effect stay unambiguous. One word when one word enough. State each fact once. NO prose abbreviations (cfg/impl/req/res/fn/auth), NO arrows (X → Y) — measured zero token saving under tokenizer, cost decode clarity. Code symbols, function names, API names, error strings: never touch |
| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |
| **wenyan-full** | Maximum classical terseness. Fully 文言文. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (之/乃/為/其) |
| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |
Example — "Why React component re-render?"
- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`."
- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`."
- ultra: "Inline obj prop, new ref, re-render. `useMemo`."
- wenyan-lite: "組件頻重繪,以每繪新生對象參照故。以 useMemo 包之。"
- wenyan-full: "每繪新生對象參照,故重繪;以 useMemo 包之則免。"
- wenyan-ultra: "新參照則重繪。useMemo 包之。"
Example — "Explain database connection pooling."
- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead."
- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead."
- ultra: "Pool reuse open DB connections. No per-request handshake."
- wenyan-full: "池蓄已開之連,不逐請而新開,省握手之費。"
- wenyan-ultra: "池蓄連,免逐請新開,省握手。"
## Auto-Clarity
Drop caveman when:
- Security warnings
- Irreversible action confirmations
- Multi-step sequences where fragment order or omitted conjunctions risk misread
- Compression itself creates technical ambiguity (e.g., `"migrate table drop column backup first"` — order unclear without articles/conjunctions)
- User asks to clarify or repeats question
Resume caveman after clear part done.
Example — destructive op:
> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.
> ```sql
> DROP TABLE users;
> ```
> Caveman resume. Verify backup exist first.
## Boundaries
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.

View File

@@ -1,18 +0,0 @@
.git
.github
.env
.env.local
output
.cache
logs
bot/.env
bot/.venv
bot/.cache
__pycache__
*.pyc
.idea
tmp_*
tests
.pytest_cache
*.md
!skills/**/*.md

View File

@@ -1,9 +1,6 @@
# 企微群机器人 webhook早报推送,与 bot API 模式凭证不同
# 企微群机器人 webhook早报推送
WECOM_WEBHOOK_KEY=your-webhook-key
# GitHub Actions将上述 key 与下方可选项写入 repo Secrets / Variables
# 详见 README「部署GitHub Actions / Docker
# 早报内容
DAILY_TRENDING_LIMIT=150
DAILY_HOT_LIMIT=150
@@ -11,6 +8,10 @@ DAILY_GITHUB_TRENDING_LIMIT=10
DAILY_GITHUB_EMERGING_LIMIT=10
DAILY_GITHUB_TOPIC_LIMIT=10
# 企微版可选行(默认均开)
# DAILY_WECOM_TOP_LINE=1 # 顶部「今日主题」行,无评分命中时回退主题名
# DAILY_FEATURED_REASON=1 # 首推下「为什么值得点开」理由行
# GitHub Trending
GITHUB_TRENDING_SINCE=daily
# GITHUB_TRENDING_LANGUAGE=python
@@ -28,14 +29,23 @@ GITHUB_TRENDING_SINCE=daily
# GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxx
# GITHUB_API_ENRICH=1
# 企微短版(各榜 Top N默认 10
DAILY_WECOM_TRENDING=10
DAILY_WECOM_HOT=10
DAILY_WECOM_GITHUB_TRENDING=10
DAILY_WECOM_GITHUB_EMERGING=10
DAILY_WECOM_GITHUB_TOPIC=10
# 企微短版(各榜 Top N默认 5周内不重复见 DAILY_BOARD_DEDUP_DAYS
DAILY_WECOM_TRENDING=5
DAILY_WECOM_HOT=5
DAILY_WECOM_GITHUB_TRENDING=5
DAILY_WECOM_GITHUB_EMERGING=5
DAILY_WECOM_GITHUB_TOPIC=5
DAILY_WECOM_AI_NEWS=10
DAILY_WECOM_CN_AI_NEWS=8
# research 模式额外技术类时讯条数(叠加在 AI 时讯精选之上)
DAILY_WECOM_AI_NEWS_TECH=5
# research 主列表国内最少条数0=按约 30% 推算10→3
# DAILY_WECOM_AI_NEWS_CN_MIN=3
# research 去重后候选池独立事件数默认展示×20=自动)
# DAILY_AI_NEWS_RESEARCH_POOL=20
# DAILY_AI_NEWS_RESEARCH_TECH_POOL=10
DAILY_WECOM_CN_AI_NEWS=10
# 企微新闻摘要字数(句读/词边界截断,不加省略号)
# DAILY_WECOM_NEWS_DESC_LIMIT=72
# 企微 Skills 合并前扫描池大小(同 source 合并后仍凑满 Top N
# DAILY_WECOM_SKILL_POOL=200
@@ -45,10 +55,40 @@ DAILY_WECOM_CN_AI_NEWS=8
DAILY_WECOM_CHUNK_BYTES=4096
# DAILY_WECOM_MAX_PARTS=5
# 企微列表模式delta=仅展示新入榜 | full=全量 Top 榜(回退)
DAILY_WECOM_MODE=delta
# Delta 模式下新入榜优先,不足时用当日 Top 榜补满各区块条数0=仅展示变化)
# 补榜时会排除近 N 天 baseline 已出现过的条目,避免周内重复(默认 7 天)
DAILY_WECOM_DELTA_PAD=1
# 补榜时从更大候选池选取默认展示条数×5至少 50
# DAILY_WECOM_PAD_POOL=50
# 无历史 data.json 时full=首日全量一次 | empty=列表为空
DAILY_DELTA_BASELINE_FALLBACK=full
# 推送闸门不满足时跳过 webhook仍写 output
DAILY_SKIP_PUSH_WHEN_SILENT=1
# DAILY_FORCE_PUSH=1
# 已推送新闻 link 去重天数
DAILY_NEWS_DEDUP_DAYS=7
# 常驻调度python -m daily schedule
DAILY_SCHEDULE_TZ=Asia/Shanghai
DAILY_SCHEDULE_GENERATE_AT=08:50
DAILY_SCHEDULE_PUSH_AT=09:00
# 仅工作日生成/推送(法定节假日、周末跳过;调休补班日照常)
# 节假日数据取自 xiaoai.me缓存于 .cache/holidays-<year>.json每年首次自动获取一次
DAILY_WORKDAY_ONLY=1
# 编辑指定今日首推(可选):关键词,或 关键词|URL
# Python Step 0 检索 → featured.jsonAgent / classic 企微「今日首推」优先使用
# DAILY_FEATURED_PICK=gstack
# DAILY_FEATURED_PICK=gstack|https://github.com/you/gstack
# 国际 AI 时讯RSS见 daily/news/feeds.py
DAILY_AI_NEWS=1
# 国内 AI 时讯RSS见 daily/news/feeds_cn.py
# 国内 AI 时讯RSS见 daily/news/feeds_cn.pyresearch 模式下忽略
DAILY_CN_AI_NEWS=1
# AI 时讯来源rss=RSS 抓取 | research=Cursor SDK + deep-researchWebSearch
# DAILY_AI_NEWS_MODE=research
# 英文描述 → 简短中文DAILY_CURSOR_EDITOR=0 时生效)
# DAILY_ZH_DESC=1
# DAILY_ZH_DESC_BATCH=20
@@ -64,7 +104,7 @@ DAILY_CN_AI_NEWS=1
# DAILY_REPORT_MODE=agent
# CURSOR_API_KEY=cursor_...
# CURSOR_MODEL=composer-2.5
# DAILY_CURSOR_CWD=.
# DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily
# 新增榜对比(较昨日 Top15供 Agent 导语/signals列表展示 Top N
# DAILY_DELTA_COMPARE_DEPTH=15
@@ -72,16 +112,20 @@ DAILY_CN_AI_NEWS=1
# DAILY_DELTA_LOOKBACK_DAYS=7
# DAILY_FULL_DESC_LIMIT=0
# DAILY_FULL_NEWS_SUMMARY_LIMIT=0
DAILY_AI_NEWS_HOURS=72
DAILY_AI_NEWS_HOURS=24
DAILY_AI_NEWS_PER_FEED=3
DAILY_AI_NEWS_PER_CATEGORY=5
# 新闻排序 / 去重(Phase 2
# 标题 Jaccard 相似度阈值0-95默认 55 表示 0.55
# DAILY_NEWS_TITLE_SIM=55
# Agent 模式 LLM 输入池classic 仍用 DAILY_WECOM_* 条数)
# DAILY_AGENT_NEWS_POOL=40
# DAILY_AGENT_CN_NEWS_POOL=30
# 多样性 / 去重(见 docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md
DAILY_BOARD_DEDUP_DAYS=7
# DAILY_BOARD_POOL_SIZE=50
# DAILY_FEATURED_DEDUP_DAYS=30
# DAILY_THEME_BAN_DAYS=7
# DAILY_NARRATIVE_AXIS_DAYS=3
DAILY_NEWS_BACKFILL=0
# 国际时讯:在 24h 滚动窗口基础上,不早于今日 0 点DAILY_AI_NEWS_TZ
DAILY_AI_NEWS_FLOOR_TODAY=1
# DAILY_AI_NEWS_TZ=Asia/Shanghai
# Reddit RSS403/429 时在 Reddit 偏好设置 → RSS feeds 复制 user / feed 参数)
# REDDIT_RSS_USER=your_username

View File

@@ -1,64 +0,0 @@
name: daily
on:
schedule:
# 08:30 CST (UTC+8) = 00:30 UTC
- cron: "30 0 * * *"
workflow_dispatch:
inputs:
skip_push:
description: Skip WeCom push (generate only)
type: boolean
default: false
concurrency:
group: daily-report
cancel-in-progress: false
jobs:
report:
runs-on: ubuntu-latest
timeout-minutes: 30
env:
TZ: Asia/Shanghai
WECOM_WEBHOOK_KEY: ${{ secrets.WECOM_WEBHOOK_KEY }}
GITHUB_TOKEN: ${{ secrets.GH_PAT || github.token }}
CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }}
DAILY_LLM_API_KEY: ${{ secrets.DAILY_LLM_API_KEY }}
DAILY_LLM_API_BASE: ${{ vars.DAILY_LLM_API_BASE }}
DAILY_LLM_MODEL: ${{ vars.DAILY_LLM_MODEL }}
DAILY_REPORT_MODE: ${{ vars.DAILY_REPORT_MODE }}
CURSOR_MODEL: ${{ vars.CURSOR_MODEL }}
DAILY_AI_NEWS: ${{ vars.DAILY_AI_NEWS }}
DAILY_CN_AI_NEWS: ${{ vars.DAILY_CN_AI_NEWS }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: pip install -r requirements.txt
- name: Generate and push daily report
run: |
ARGS=()
if [ "${{ inputs.skip_push }}" = "true" ]; then
ARGS+=(--skip-push)
fi
chmod +x ./run-daily.sh
./run-daily.sh --force "${ARGS[@]}"
- name: Upload report artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: daily-report-${{ github.run_id }}
path: |
output/*.md
output/*.json
if-no-files-found: ignore
retention-days: 14

View File

@@ -1,20 +0,0 @@
name: test
on:
push:
branches: [main]
pull_request:
jobs:
pytest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: pip install -r requirements-dev.txt
- name: Run tests
run: pytest -q

6
.gitignore vendored
View File

@@ -2,11 +2,7 @@
.env.local
logs/
.cache/
bot/.env
bot/.venv/
bot/.cache/
output
__pycache__/
*.pyc
.idea/
tmp_*
.cursor/

3
.idea/.gitignore generated vendored Normal file
View File

@@ -0,0 +1,3 @@
# Default ignored files
/shelf/
/workspace.xml

9
.idea/daily-robots.iml generated Normal file
View File

@@ -0,0 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$" />
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>

2
.idea/misc.xml generated
View File

@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectRootManager" version="2" default="true">
<component name="ProjectRootManager" version="2" languageLevel="JDK_21" default="true" project-jdk-name="21" project-jdk-type="JavaSDK">
<output url="file://$PROJECT_DIR$/out" />
</component>
</project>

8
.idea/modules.xml generated Normal file
View File

@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/daily-robots.iml" filepath="$PROJECT_DIR$/.idea/daily-robots.iml" />
</modules>
</component>
</project>

2
.idea/vcs.xml generated
View File

@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="VcsDirectoryMappings">
<mapping directory="$PROJECT_DIR$" vcs="Git" />
<mapping directory="" vcs="Git" />
</component>
</project>

View File

@@ -1,27 +0,0 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
TZ=Asia/Shanghai
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends tzdata \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY daily/ daily/
COPY shared/ shared/
COPY skills/ skills/
COPY bot/ bot/
COPY run-daily.sh .
RUN chmod +x run-daily.sh \
&& mkdir -p output .cache logs
VOLUME ["/app/output", "/app/.cache", "/app/logs"]
ENTRYPOINT ["./run-daily.sh"]

261
README.md
View File

@@ -1,169 +1,164 @@
# daily-robots
# skills-hot-daily
> **Daily Briefing** · **WeCom Push** · **Agent Mode**
Skills / GitHub 早报推送(企微 Webhook
给 AI 工程小团队推送**每日 curated 情报**skills.sh 榜单、GitHub AI 趋势、国际/国内 RSS生成叙事化早报并推到企业微信群。
## 项目结构
与 TLDR AI / Ben's Bites 不同,这里专注 **Agent Skills 生态** + **可 fork 的自托管流水线**(不是又一个 email newsletter
**产出示例**[docs/sample-output/wecom-agent-sample.md](docs/sample-output/wecom-agent-sample.md)Agent 模式企微版节选)
```
skills-hot-daily/
├── README.md
├── .env.example # 早报 webhook、GitHub 等
├── requirements.txt # Python 依赖
├── run-daily.ps1 # 生成 + 推送一条龙
├── send-wecom.ps1 # 仅推送
├── daily/ # 早报 Python 包
│ ├── __main__.py # python -m daily [generate|push]
│ ├── config.py
│ ├── generate.py
│ ├── report_data.py # JSON 中间层
│ ├── cursor_editor.py # Cursor 编辑层
│ ├── llm_client.py
│ ├── localize.py # 仅中文化Tier A
│ ├── format_wecom.py
│ ├── webhook.py
│ ├── news/ # 国际 AI 时讯 RSS
│ │ ├── feeds.py
│ │ └── fetch.py
│ └── github/
│ ├── auth.py
│ ├── search.py
│ └── trending.py
├── output/ # YYYY-MM-DD.md / .wecom.md / .data.json / .editorial.json
├── skills/daily-editor/ # 早报 Cursor 编辑 Skill
│ └── SKILL.md
├── logs/
└── .cache/
```
---
## Quick Start
## 早报推送
1. **Clone** 本仓库
2. **安装依赖**`pip install -r requirements.txt`
3. **配置环境**`copy .env.example .env`
4. **填写 webhook**:企微群 → 群机器人 → 添加 → 把 `key=` 写入 `.env`
```env
WECOM_WEBHOOK_KEY=your-webhook-key
```
5. **生成并推送**`.\run-daily.ps1`
```powershell
cd d:\LY\diy\daily-robots
pip install -r requirements.txt
copy .env.example .env
.\run-daily.ps1
```
生成文件在 `output/`
- 生成:`output/YYYY-MM-DD.md`(完整版)、`output/YYYY-MM-DD.wecom.md`(企微短版)
- 仅生成:`.\run-daily.ps1 -SkipPush`
- 仅推送:`python -m daily push output\2026-06-25.wecom.md`
**Webhook 配置**:企微群 → 群机器人 → 添加,将 `key=` 后的值写入项目根 `.env`
```env
WECOM_WEBHOOK_KEY=your-key
```
**GitHub 数据源**(见 `.env.example`
| 来源 | 说明 |
|------|------|
| GitHub Trending | `GITHUB_TRENDING_MODE=scrape``api` |
| 新兴 / Topic | Search API`GITHUB_TOKEN` |
| Release | 可选 `GITHUB_REPOS=owner/repo` |
**国际 AI 时讯**RSS`daily/news/feeds.py`
| 类别 | 覆盖 |
|------|------|
| 厂商官方 | Anthropic、OpenAI、Google、Meta、Microsoft、Mistral、Cursor 等 |
| Agent / LLM 开发者 | LangChain、LlamaIndex、Hugging Face、Copilot 等 |
| 综合科技媒体 | The Verge、TechCrunch、Ars、Wired、MIT TR 等 |
| Newsletter | Ben's Bites、Latent Space、Simon Willison、TLDR AI 等 |
| 研究 / 论文 | arXiv cs.CL/AI/LG、HF Papers |
| 社区讨论 | HN、Reddit r/LocalLLaMA / ClaudeAI / ML 等 |
环境变量:`DAILY_AI_NEWS=1` · `DAILY_CN_AI_NEWS=1` · `DAILY_AI_NEWS_HOURS=24` · `DAILY_WECOM_AI_NEWS=10` · `DAILY_WECOM_CN_AI_NEWS=10`
**国内 AI 时讯**RSS`daily/news/feeds_cn.py`
| 类别 | 覆盖 |
|------|------|
| AI 专业媒体 | 量子位 |
| 综合科技 | 36氪、雷锋网、Google News 中文 |
### 生成架构Tier B · Cursor 编辑层)
早报默认走 **Python 抓取 + 模板渲染**;可选开启 Cursor 做「编辑」:
```
抓取数据 → output/日期.data.json → Cursor 读 Skill 写 editorial → 模板填字 → .md / .wecom.md
```
| 文件 | 说明 |
|------|------|
| `YYYY-MM-DD.wecom.md` | 企微推送版(主产物 |
| `YYYY-MM-DD.md` | 完整归档版 |
| `YYYY-MM-DD.data.json` | 结构化数据(供 LLM / 调试 |
| `output/YYYY-MM-DD.data.json` | 结构化榜单(供 LLM 输入 |
| `output/YYYY-MM-DD.editorial.json` | Cursor 输出的主题、速览、中文描述 |
| `skills/daily-editor/SKILL.md` | 编辑规范语气、JSON 格式 |
常用变体:
```powershell
.\run-daily.ps1 -SkipPush # 只生成
python -m daily push output\2026-07-03.wecom.md # 只推送
```env
# 开启 Tier B需 CURSOR_API_KEY 或 DAILY_LLM_API_KEY
DAILY_CURSOR_EDITOR=1
```
Linux / macOS / CI 等价脚本:
- 开启后:**一次 LLM 调用** 生成 `theme_line` + `highlights` + 全部中文描述
- 关闭时(默认):规则主题 + `DAILY_ZH_DESC` 仅中文化描述
- LLM 失败自动回退规则模式,不影响推送
```bash
chmod +x run-daily.sh
./run-daily.sh # 生成 + 推送
./run-daily.sh --skip-push # 只生成
./run-daily.sh --force # 忽略 30 分钟内重复运行锁
### 生成架构Agent 工作流 · 推荐)
若觉得模板版「榜单堆砌」不友好,可改用 **Agent 三步流水线**
```
Python 抓取 → Step1 趋势分析 → Step2 叙事写稿 → Python 分条推送企微
(.trends.json) (.wecom.md)
```
**前置条件**Python 3.10+、企业微信群机器人 webhook。无需 Bot API 凭证。
---
## 部署GitHub Actions / Docker
### GitHub Actions零服务器定时推送
仓库已包含 [`.github/workflows/daily.yml`](.github/workflows/daily.yml),默认每天 **08:30北京时间** 生成并推送。
1. Fork 或启用本仓库的 Actions
2. **Settings → Secrets and variables → Actions** 添加 Secrets
| Secret | 必需 | 说明 |
|--------|------|------|
| `WECOM_WEBHOOK_KEY` | 是 | 企微群机器人 webhook key |
| `GH_PAT` | 否 | 个人 GitHub PAT提高 API 限额;不设则用 Actions 内置 token |
| `CURSOR_API_KEY` | 否 | Agent 模式Cursor SDK |
| `DAILY_LLM_API_KEY` | 否 | Agent / 编辑层OpenAI 兼容 API |
3. 可选 **Variables**(不设则用代码默认值):`DAILY_REPORT_MODE`、`DAILY_LLM_MODEL`、`DAILY_AI_NEWS` 等
4. **Actions → daily → Run workflow** 可手动触发;勾选 *Skip WeCom push* 则只生成不上报
运行产物可在 workflow 的 **Artifacts** 中下载(`.md` / `.json`,保留 14 天)。
### Docker
```bash
cp .env.example .env # 填入 WECOM_WEBHOOK_KEY 等
docker compose build
docker compose run --rm daily # 生成 + 推送
docker compose run --rm daily --skip-push # 只生成
```
`output/`、`.cache/`、`logs/` 挂载到宿主机,与本地 `run-daily.ps1` 行为一致。
---
## Agent 模式(推荐)
默认 `classic` 是分区榜单模板;**推荐 Agent 模式**——导语 + 今日信号 + 精选新闻 + 榜单,全中文叙述。
| 模式 | 环境变量 | 企微版风格 |
|------|----------|------------|
| `classic`(默认) | — | 分区榜单 + 模板 |
| `editor` | `DAILY_CURSOR_EDITOR=1` | 模板 + LLM 中文化 |
| `agent` | `DAILY_REPORT_MODE=agent` | **导语 + 信号 + 精选**,全中文叙述 |
```env
DAILY_REPORT_MODE=agent
CURSOR_API_KEY=cursor_...
CURSOR_MODEL=composer-2.5
DAILY_CURSOR_CWD=.
DAILY_CURSOR_CWD=d:\LY\diy\daily-robots
```
| 模式 | 环境变量 | 风格 |
|------|----------|------|
| `classic` | — | 分区榜单 + 模板 |
| `editor` | `DAILY_CURSOR_EDITOR=1` | 模板 + LLM 中文化 |
| **`agent`** | `DAILY_REPORT_MODE=agent` | **叙事化早报(推荐)** |
| 文件 | 说明 |
|------|------|
| `output/YYYY-MM-DD.trends.json` | Step1 趋势分析结果 |
| `skills/daily-agent/SKILL.md` | Agent 工作流规范 |
流水线:
- 完整版 `YYYY-MM-DD.md` 仍为数据表格归档;企微版由 Agent 直接写 Markdown
- Agent 失败自动回退 `classic`,不影响 `run-daily.ps1`
```
Python 抓取 → Step1 趋势分析 (.trends.json) → Step2 写企微稿 (.wecom.md) → webhook 分条推送
```
定时推送:
- 规范见 `skills/daily-agent/SKILL.md`
- LLM 失败自动回退 `classic`,不影响 `run-daily.ps1`
- 无 `CURSOR_API_KEY` 时请用 `classic`,或配置 `DAILY_LLM_API_KEY`OpenAI 兼容 API
定时推送Windows 任务计划程序每日执行 `run-daily.ps1`,或 cron / `schtasks`。
- **常驻调度(推荐)**`python -m daily schedule``.\run-scheduler.ps1`(默认 08:50 生成、09:00 推送,见 `DAILY_SCHEDULE_*`
- Windows 任务计划:`.\register-daily-task.ps1`
- Cursor`/loop 1d`(时间会漂移,仅临时用
---
## 自定义 RSS 源
## 其他推送方案
国际 RSS 列表:`daily/news/feeds.py`
国内 RSS 列表:`daily/news/feeds_cn.py`
在对应文件的 `FEEDS` 列表中增删 URL 即可。常用开关(见 `.env.example`
```env
DAILY_AI_NEWS=1
DAILY_CN_AI_NEWS=1
DAILY_AI_NEWS_HOURS=72
DAILY_WECOM_AI_NEWS=10
DAILY_WECOM_CN_AI_NEWS=8
```
Skills 榜单默认爬 skills.sh 官网600+ 条);可改为第三方 feed
```env
# SKILLS_BOARD_SOURCE=feed
```
GitHub 新兴/Topic 榜需 `GITHUB_TOKEN`(见 `.env.example` 注释)。
---
## Experimental`bot/` 企微对话机器人
同仓库内的 **API 模式智能机器人**@ 机器人快查 skills、Cursor 任务、Playwright 截图)**不在 v1 开源支持范围内**,需独立 venv 与 `WECOM_BOT_ID` / `SECRET`。
详见 **[bot/README.md](bot/README.md)**。
---
## 项目结构
```
daily-robots/
├── run-daily.ps1 # 生成 + 推送
├── daily/ # 早报主包
├── shared/skills_data.py # skills feed 共用数据层
├── skills/daily-agent/ # Agent 工作流规范
├── output/ # 生成产物gitignore
└── bot/ # experimental
```
| 方案 | 适用场景 | 复杂度 |
|------|----------|--------|
| **群机器人 webhook** | 推送到固定群 | 低 |
| **应用消息 API** | 推送给指定成员/部门 | 中(需 corp_id、secret、agent_id |
| **邮件 + 企业微信邮箱** | 已有 SMTP | 中 |
| **PushPlus / Server酱** | 个人微信中转 | 低(第三方) |
---
## 注意事项
- Webhook / API Key **勿提交 Git**,只用 `.env`
- 企微 markdown 为子集;超长报告**自动分条推送**(默认 4096 bytes/条
- 完整配置项见 [`.env.example`](.env.example)
- 开发测试:`pip install -r requirements-dev.txt` → `pytest`
- Webhook **不要提交 Git**,只用环境变量
- 企业微信 markdown 为**子集**(不支持完整 GitHub 表格语法时可改为文本列表
- 单条消息约 **4096 字节** 上限,`send-wecom.ps1` 已做截断

View File

@@ -1,29 +0,0 @@
# 企业微信智能机器人API 模式 · 长连接)
# 管理后台 → 安全与管理 → 管理工具 → 智能机器人 → 创建 → API 模式 → 使用长连接
WECOM_BOT_ID=your-bot-id
WECOM_BOT_SECRET=your-bot-secret
# Cursor SDK@ 机器人后的通用任务由 Cursor 执行)
CURSOR_API_KEY=cursor_...
CURSOR_CWD=.
CURSOR_MODEL=composer-2.5
CURSOR_TIMEOUT=600
# 前端截图预览(基于 CURSOR_CWD
PREVIEW_PORT=5173
PREVIEW_URL=http://127.0.0.1:5173/
# PREVIEW_DEV_COMMAND=npm run dev
# PREVIEW_STARTUP_TIMEOUT=120
# 登录后截图(账号密码只放 .env切勿发到企微群
# PREVIEW_LOGIN_USER=your_account_or_phone
# PREVIEW_LOGIN_PASSWORD=your_password
# PREVIEW_AFTER_LOGIN_URL=/app/dashboard
# PREVIEW_AUTO_LOGIN=true
# 网页操作场景目录(可选,默认 bot/scenarios 与 CURSOR_CWD/.browser-scenarios
# BROWSER_SCENARIOS_DIR=d:\path\to\scenarios
# BROWSER_DEFAULT_SCENARIO=xiaobao-agent-manage
# hybrid=快查走本地 / 其余走 Cursor | cursor=全部 Cursor | skills=仅本地
ROUTING_MODE=hybrid

4
bot/.gitignore vendored
View File

@@ -1,4 +0,0 @@
.cache/
.env
.venv/
.cache/screenshots/

View File

@@ -1,143 +0,0 @@
# bot/ — 企微 API 模式智能机器人
> **EXPERIMENTAL** — 本模块不在 daily-robots v1 开源支持范围内。
> 主产品为根目录 **早报推送**webhook见 [README](../README.md)。
> Bot 需独立 venv、企微 API 凭证、常驻进程;问题请自行排查或提 issue 标注 `bot`。
在企微里 @ 机器人即可:
- **快查**`trending 10``hot 10``搜索 react`(本地 skills 数据,秒回)
- **截图预览**`preview` / `截图`(基于 `CURSOR_CWD` 启动前端并发图)
- **通用任务**:任意自然语言需求,由 **Cursor Agent** 执行并回传结果
数据层与早报共用 [`shared/skills_data.py`](../shared/skills_data.py)。
---
## 1. 创建 API 模式机器人
1. [企业微信管理后台](https://work.weixin.qq.com/) → **安全与管理****管理工具****智能机器人****创建机器人**
2. 选择 **API 模式创建****使用长连接**
3. 记录 **Bot ID****Secret**Secret 只显示一次,请立即保存)
4. 设置可见范围,将机器人 **添加到目标群** 或允许成员单聊
普通成员路径:工作台 → 智能机器人 → 手动创建 → API 模式 → 长连接
---
## 2. 启动本地服务
```powershell
cd path\to\daily-robots\bot
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
playwright install chromium
copy .env.example .env
# 编辑 .envWECOM_BOT_ID / WECOM_BOT_SECRET / CURSOR_API_KEY
python main.py
```
服务需 **常驻运行**(本机、服务器或 Docker。长连接模式下机器人进程须在线才能收消息。
---
## 3. 路由模式ROUTING_MODE
| 模式 | 行为 |
|------|------|
| `hybrid`(默认) | `trending`/`hot`/`搜索`/`详情` 走本地快查;其余 @ 消息交给 Cursor |
| `cursor` | 所有消息都交给 Cursor 执行 |
| `skills` | 仅本地 skills 快查 |
**Cursor 任务示例**(群里发送):
```
@test 总结 trending top10并推荐 3 个适合前端团队的 skill
@test 对比 mattpocock/skills 和 obra/superpowers 各有哪些热门 skill
@test 帮我写一段 npx skills add 的安装说明
```
Cursor 在本机 `CURSOR_CWD` 目录下运行(见 `bot/.env`)。复杂任务可能需要 110 分钟流式消息会显示「Cursor 正在执行任务…」。
---
## 4. 前端截图预览API 模式发图)
项目路径读取 `.env` 中的 **`CURSOR_CWD`**。机器人会:
1.`CURSOR_CWD` 检测 `package.json`,若有 `dev` 脚本则执行 `PREVIEW_DEV_COMMAND`(默认 `npm run dev`
2. 等待 `PREVIEW_PORT`(默认 `5173`)就绪,或用 `PREVIEW_URL` 直接访问
3. Playwright 打开页面并截图
4. 通过 API 模式 **上传图片 + 回复 image 消息** 到群
| 命令 | 说明 |
|------|------|
| `preview` / `截图` / `预览` | 访问 `http://127.0.0.1:5173/` 并截图 |
| `preview /login` | 指定路径 |
| `preview / 3000` | 指定端口 |
| `preview http://127.0.0.1:8080/` | 指定完整 URL |
`.env` 可选配置:
```env
CURSOR_CWD=.
PREVIEW_PORT=5173
PREVIEW_URL=http://127.0.0.1:5173/
PREVIEW_DEV_COMMAND=npm run dev
PREVIEW_STARTUP_TIMEOUT=120
```
---
## 4b. 网页操作Playwright 步骤引擎)
支持三种方式定义操作流程,**无需改 Python 代码**
**1. 自然语言(企微里直接说)**
```
@test 访问登录页,输入账号密码,点击登录后进入主页,点击智能体管理菜单然后截图
```
账号密码从 `.env` 读取(`{{PREVIEW_LOGIN_USER}}` / `{{PREVIEW_LOGIN_PASSWORD}}`),勿在群里发密码。
**2. 场景文件 YAML** — 见 `scenarios/`,触发:`@test browser <场景名>`
**3. 消息内 DSL** — 以 `browser:` 开头的多行步骤
支持的步骤:`goto` · `fill` · `click` · `wait` · `screenshot` · `press`
---
## 5. 快查命令
| 命令 | 说明 |
|------|------|
| `trending 10` / `趋势 10` | 近期增长榜 Top N默认 10最大 30 |
| `hot 10` / `实时 10` | 实时热度榜 |
| `all 10` / `总榜 10` | 历史总安装榜 |
| `搜索 react` / `search tdd` | 关键词搜索 |
| `详情 find-skills` | 单个 skill 详情 + 安装命令 |
| `preview` / `截图` | 启动 CURSOR_CWD 前端并截图发群 |
| `帮助` | 命令列表 |
---
## 6. 本地测试(无需企微凭证)
```powershell
cd path\to\daily-robots\bot
python -c "from skills_service import handle_command; print(handle_command('trending 5'))"
```
---
## 配置
| 文件 | 说明 |
|------|------|
| `bot/.env` | `WECOM_BOT_ID``WECOM_BOT_SECRET``CURSOR_API_KEY` 等 |
| `.env.example` | 模板 |
与根目录 `.env`(早报 webhook**相互独立**,勿混用。

View File

@@ -1,12 +0,0 @@
"""Bot 内部数据结构。"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class RouteResult:
source: str
text: str
image_path: str | None = None

View File

@@ -1,49 +0,0 @@
"""浏览器场景变量替换与 base URL 解析。"""
from __future__ import annotations
import re
from urllib.parse import urlparse
import env_config
_VAR_PATTERN = re.compile(r"\{\{([A-Z0-9_]+)\}\}")
def interpolate(value: str) -> str:
def repl(match: re.Match[str]) -> str:
key = match.group(1)
resolved = env_config.env(key)
if resolved is None:
raise RuntimeError(f"场景变量未配置:{key}")
return resolved
return _VAR_PATTERN.sub(repl, value)
def default_base_url() -> str:
explicit = (env_config.env("PREVIEW_BASE_URL") or "").strip()
if explicit:
return interpolate(explicit.rstrip("/"))
preview = (env_config.env("PREVIEW_URL") or "").strip()
if preview:
parsed = urlparse(preview)
scheme = parsed.scheme or "http"
host = parsed.hostname or "127.0.0.1"
port = parsed.port
if port and port not in (80, 443):
return f"{scheme}://{host}:{port}"
return f"{scheme}://{host}"
port = env_config.env("PREVIEW_PORT", "5173") or "5173"
return f"http://127.0.0.1:{port}"
def resolve_url(base_url: str, target: str) -> str:
target = interpolate(target.strip())
if target.startswith("http://") or target.startswith("https://"):
return target
if not target.startswith("/"):
target = "/" + target
return base_url.rstrip("/") + target

View File

@@ -1,269 +0,0 @@
"""通用 Playwright 步骤执行器(不写死业务页面)。"""
from __future__ import annotations
import logging
import re
import time
from datetime import datetime
from pathlib import Path
from typing import Any
from browser_env import interpolate, resolve_url
from browser_models import BrowserResult, BrowserScenario
logger = logging.getLogger(__name__)
SCREENSHOT_DIR = Path(__file__).resolve().parent / ".cache" / "screenshots"
FIELD_HINTS: dict[str, list[str]] = {
"账号": [
"#login-username",
"input#login-username",
"input[autocomplete='username']",
"username",
"account",
"phone",
"账号",
"手机号",
"企业账号",
],
"密码": [
"#login-password input",
"#login-password",
"input#login-password",
"input[type='password']",
"password",
"密码",
],
"用户名": ["#login-username", "input#login-username", "username", "account", "账号"],
}
def _step_label(step: dict[str, Any], index: int) -> str:
action = step.get("action", "?")
target = step.get("target") or step.get("field") or step.get("url") or ""
return f"{index + 1}. {action} {target}".strip()
def _resolve_fill_locator(page, field: str, step: dict[str, Any]):
if step.get("selector"):
return page.locator(interpolate(str(step["selector"])))
field_key = interpolate(str(field))
if step.get("label"):
return page.get_by_label(interpolate(str(step["label"])), exact=False)
if step.get("placeholder"):
return page.get_by_placeholder(interpolate(str(step["placeholder"])), exact=False)
hints = FIELD_HINTS.get(field_key, [field_key])
for hint in hints:
if hint.startswith("#") or hint.startswith(".") or hint.startswith("["):
locator = page.locator(hint)
if locator.count() > 0:
return locator.first
for getter in (
lambda h=hint: page.get_by_label(h, exact=False),
lambda h=hint: page.get_by_placeholder(h, exact=False),
):
locator = getter()
if locator.count() > 0:
return locator.first
return page.locator("input, textarea").filter(has_text=field_key).first
def _fill_field(page, field: str, step: dict[str, Any]) -> None:
value = interpolate(str(step.get("value", "")))
locator = _resolve_fill_locator(page, field, step)
locator.click(timeout=10_000)
locator.fill("", timeout=5_000)
locator.fill(value, timeout=10_000)
def _page_error_text(page) -> str | None:
for selector in (
".ant-message-error",
".ant-form-item-explain-error",
".ant-alert-error",
):
try:
locator = page.locator(selector).first
if locator.is_visible(timeout=300):
text = locator.inner_text(timeout=1_000).strip()
if text:
return text
except Exception:
continue
return None
def _pathname_matches(pattern: str, pathname: str) -> bool:
pattern = pattern.strip()
if pattern in {"**/app/**", "**/app/*", "/app/**"}:
return pathname.startswith("/app")
if pattern.endswith("/**"):
prefix = pattern[:-3].rstrip("/")
if prefix.startswith("**/"):
prefix = prefix[3:]
if not prefix.startswith("/"):
prefix = "/" + prefix
return pathname.startswith(prefix)
if "**" in pattern or "*" in pattern:
regex = "^" + re.escape(pattern).replace(r"\*\*", ".*").replace(r"\*", "[^/]*") + "$"
return re.search(regex, pathname) is not None
return pathname == pattern or pathname.startswith(pattern)
def _wait_for_url_pattern(page, pattern: str, timeout: int = 60_000) -> None:
"""SPA 路由用 pathname 轮询glob 模式不依赖 navigation 事件。"""
deadline = time.monotonic() + timeout / 1000
last_error: str | None = None
while time.monotonic() < deadline:
pathname = page.evaluate("() => window.location.pathname")
if _pathname_matches(pattern, pathname):
try:
page.wait_for_load_state("networkidle", timeout=8_000)
except Exception:
page.wait_for_timeout(800)
return
err = _page_error_text(page)
if err and err != last_error:
last_error = err
logger.warning("页面提示:%s", err)
if "/login" in pathname:
raise RuntimeError(f"登录失败:{err}")
page.wait_for_timeout(400)
err = _page_error_text(page)
hint_parts = [f"当前 URL`{page.url}`"]
if err:
hint_parts.append(f"页面错误:{err}")
elif last_error:
hint_parts.append(f"页面错误:{last_error}")
hint_parts.append("请确认 PREVIEW_LOGIN_USER/PASSWORD 正确,且登录 API内网网关可达。")
raise RuntimeError(f"等待 URL 匹配 `{pattern}` 超时({timeout}ms{' '.join(hint_parts)}")
def _click_target(page, target: str) -> None:
target = interpolate(target.strip())
if target.lower() in {"登录", "login"}:
for selector in ("button.login-submit", "button[type='submit']"):
locator = page.locator(selector)
if locator.count() > 0:
locator.first.click(timeout=10_000)
return
candidates = [
page.get_by_role("menuitem", name=target, exact=True),
page.get_by_role("button", name=target, exact=True),
page.get_by_role("link", name=target, exact=True),
page.get_by_text(target, exact=True),
page.get_by_text(target, exact=False),
]
for locator in candidates:
if locator.count() > 0:
locator.first.click(timeout=10_000)
return
raise RuntimeError(f"未找到可点击元素:{target}")
def _execute_step(page, base_url: str, step: dict[str, Any]) -> None:
action = str(step.get("action", "")).lower()
if action == "goto":
target = step.get("target") or step.get("url") or "/"
url = resolve_url(base_url, str(target))
page.goto(url, wait_until="networkidle", timeout=60_000)
return
if action == "fill":
field = str(step.get("field") or step.get("target") or "账号")
_fill_field(page, field, step)
return
if action == "click":
target = step.get("target") or step.get("text")
if not target:
raise RuntimeError("click 步骤缺少 target")
_click_target(page, str(target))
page.wait_for_timeout(800)
return
if action == "wait":
timeout = int(step.get("timeout") or 60_000)
if step.get("url"):
_wait_for_url_pattern(page, str(step["url"]), timeout=timeout)
return
if step.get("selector"):
page.locator(interpolate(str(step["selector"]))).wait_for(timeout=30_000)
return
if step.get("text"):
page.get_by_text(interpolate(str(step["text"])), exact=False).wait_for(timeout=30_000)
return
ms = int(step.get("ms") or 1500)
page.wait_for_timeout(ms)
return
if action == "press":
key = str(step.get("key") or step.get("target") or "Enter")
page.keyboard.press(key)
return
if action == "screenshot":
return
raise RuntimeError(f"未知步骤 action={action}")
def run_browser_scenario_sync(scenario: BrowserScenario) -> BrowserResult:
from playwright.sync_api import sync_playwright
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
slug = (scenario.name or "browser").replace(" ", "-")
output = SCREENSHOT_DIR / f"{slug}-{stamp}.png"
output.parent.mkdir(parents=True, exist_ok=True)
step_log: list[str] = []
final_url = scenario.base_url
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=True)
page = browser.new_page(viewport={"width": 1280, "height": 720})
steps = list(scenario.steps)
if steps and steps[-1].get("action") != "screenshot" and not any(
s.get("action") == "screenshot" for s in steps
):
steps.append({"action": "screenshot"})
for index, step in enumerate(steps):
label = _step_label(step, index)
logger.info("执行步骤 %s", label)
action = str(step.get("action", "")).lower()
if action == "screenshot":
page.wait_for_timeout(int(step.get("ms") or 1500))
page.screenshot(path=str(output), full_page=False, type="png")
final_url = page.url
step_log.append(label + "")
continue
try:
_execute_step(page, scenario.base_url, step)
final_url = page.url
step_log.append(label + "")
except Exception as exc:
err = _page_error_text(page)
detail = f"{err}" if err else ""
raise RuntimeError(f"步骤失败:{label} @ {page.url}{detail}") from exc
browser.close()
return BrowserResult(
scenario_name=scenario.name,
base_url=scenario.base_url,
final_url=final_url,
screenshot_path=output,
step_count=len(steps),
step_log=step_log,
)

View File

@@ -1,26 +0,0 @@
"""浏览器自动化步骤模型。"""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
@dataclass
class BrowserScenario:
name: str | None
base_url: str
steps: list[dict[str, Any]]
source: str = "natural"
@dataclass
class BrowserResult:
scenario_name: str | None
base_url: str
final_url: str
screenshot_path: Path
step_count: int
started_dev_server: bool = False
step_log: list[str] = field(default_factory=list)

View File

@@ -1,307 +0,0 @@
"""解析自然语言 / YAML / 场景名 → 浏览器步骤。"""
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
import yaml
import env_config
from browser_env import default_base_url, interpolate
from browser_models import BrowserScenario
SCENARIO_DIRS = [
Path(__file__).resolve().parent / "scenarios",
Path(__file__).resolve().parent.parent / "scenarios",
]
def _project_cwd() -> Path:
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
return Path(raw).resolve()
def _strip_mention(text: str) -> str:
return re.sub(r"@\S+\s*", "", text).strip()
def _scenario_search_dirs() -> list[Path]:
dirs = list(SCENARIO_DIRS)
dirs.append(_project_cwd() / ".browser-scenarios")
custom = (env_config.env("BROWSER_SCENARIOS_DIR") or "").strip()
if custom:
dirs.append(Path(custom).resolve())
return dirs
def is_browser_intent(text: str) -> bool:
raw = _strip_mention(text)
if not raw:
return False
if re.match(r"^(browser|网页|网页操作|操作)\b", raw, re.IGNORECASE):
return True
if re.search(r"```(?:yaml|yml)", raw, re.IGNORECASE):
return True
if re.search(r"(?m)^browser\s*:", raw, re.IGNORECASE):
return True
if re.match(r"^(preview|截图|预览|截屏)\s", raw, re.IGNORECASE):
if not re.search(r"[,。;;]|然后|输入|点击|填写|访问|打开|登录", raw):
return False
if len(_split_segments(text)) >= 2:
return True
verbs = 0
for pattern in (r"访问", r"打开", r"输入", r"填写", r"点击", r"点选", r"选择", r"登录"):
if re.search(pattern, raw):
verbs += 1
return verbs >= 2
def _load_yaml_scenario(path: Path) -> BrowserScenario:
data = yaml.safe_load(path.read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise RuntimeError(f"场景文件格式错误:{path}")
base_url = interpolate(str(data.get("base_url") or default_base_url()))
steps = data.get("steps")
if not isinstance(steps, list) or not steps:
raise RuntimeError(f"场景缺少 steps{path}")
return BrowserScenario(
name=data.get("name") or path.stem,
base_url=base_url,
steps=_normalize_steps(steps),
source=f"file:{path.name}",
)
def _find_scenario_file(name: str) -> Path | None:
slug = name.strip().replace(" ", "-")
for directory in _scenario_search_dirs():
for candidate in (directory / f"{slug}.yaml", directory / f"{slug}.yml"):
if candidate.exists():
return candidate
return None
def _normalize_steps(raw_steps: list[Any]) -> list[dict[str, Any]]:
normalized: list[dict[str, Any]] = []
for item in raw_steps:
if isinstance(item, str):
normalized.append({"action": item})
continue
if not isinstance(item, dict) or not item:
raise RuntimeError(f"无效步骤:{item!r}")
if "action" in item:
normalized.append(dict(item))
continue
if len(item) == 1:
action, payload = next(iter(item.items()))
step = {"action": action}
if payload is not None:
if isinstance(payload, dict):
step.update(payload)
elif action == "wait" and isinstance(payload, int):
step["ms"] = payload
elif action == "wait" and isinstance(payload, str) and payload.isdigit():
step["ms"] = int(payload)
else:
step["target"] = payload
normalized.append(step)
continue
raise RuntimeError(f"无效步骤:{item!r}")
return normalized
def _parse_inline_dsl(text: str) -> BrowserScenario | None:
raw = _strip_mention(text)
match = re.search(r"(?ms)^browser\s*:\s*\n(.+)$", raw, re.IGNORECASE)
if not match:
return None
steps: list[dict[str, Any]] = []
for line in match.group(1).splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
line = re.sub(r"^[-*]\s*", "", line)
if not line:
continue
steps.append(_parse_dsl_line(line))
if not steps:
return None
return BrowserScenario(
name="inline",
base_url=default_base_url(),
steps=steps,
source="inline-dsl",
)
def _parse_dsl_line(line: str) -> dict[str, Any]:
parts = line.split(None, 2)
action = parts[0].lower()
if action == "goto":
return {"action": "goto", "target": parts[1] if len(parts) > 1 else "/"}
if action == "click":
return {"action": "click", "target": " ".join(parts[1:])}
if action == "fill":
if len(parts) < 3:
raise RuntimeError(f"fill 语法fill 字段 值({line}")
return {"action": "fill", "field": parts[1], "value": parts[2]}
if action == "wait":
payload = parts[1] if len(parts) > 1 else "1500"
if payload.isdigit():
return {"action": "wait", "ms": int(payload)}
return {"action": "wait", "url": payload}
if action in {"screenshot", "shot"}:
return {"action": "screenshot"}
raise RuntimeError(f"未知 DSL 步骤:{line}")
def _split_segments(text: str) -> list[str]:
raw = _strip_mention(text)
raw = re.sub(r"^(browser|网页|网页操作|操作)\s*[:]?\s*", "", raw, flags=re.IGNORECASE)
raw = re.sub(r"然后截图|再截图|最后截图", "截图", raw)
chunks = re.split(r"[,。;;]\s*|\s+然后\s+|\s+接着\s+|\s+并\s*", raw)
expanded: list[str] = []
for chunk in chunks:
chunk = chunk.strip()
if not chunk:
continue
subchunks = re.split(r"\s+然后\s+", chunk)
if "" in chunk and len(subchunks) == 1:
subchunks = re.split(r"(?<=[登录页表单])后(?=[进入打开等待点击访问])", chunk)
for part in subchunks:
part = part.strip()
if part:
expanded.append(part)
return expanded
def _parse_segment(segment: str) -> list[dict[str, Any]]:
seg = segment.strip()
if not seg or seg.lower() in {"browser", "网页操作"}:
return []
if re.fullmatch(r"截图|截屏", seg, re.IGNORECASE):
return [{"action": "screenshot"}]
match = re.search(r"访问登录页|打开登录页|进入登录页|运行登录页", seg, re.IGNORECASE)
if match:
return [{"action": "goto", "target": "/login"}]
match = re.search(r"输入账号密码|填写账号密码|输入账号和密码", seg, re.IGNORECASE)
if match:
return [
{"action": "fill", "field": "账号", "value": "{{PREVIEW_LOGIN_USER}}"},
{"action": "fill", "field": "密码", "value": "{{PREVIEW_LOGIN_PASSWORD}}"},
]
match = re.search(r"输入账号|填写账号|输入用户名|填写用户名", seg, re.IGNORECASE)
if match:
return [{"action": "fill", "field": "账号", "value": "{{PREVIEW_LOGIN_USER}}"}]
match = re.search(r"输入密码|填写密码", seg, re.IGNORECASE)
if match:
return [{"action": "fill", "field": "密码", "value": "{{PREVIEW_LOGIN_PASSWORD}}"}]
match = re.search(r"进入主页|进入首页|打开主页|打开首页|等待主页", seg, re.IGNORECASE)
if match:
return [{"action": "wait", "url": "**/app/**"}]
match = re.search(r"等待\s*(\d+)\s*秒", seg, re.IGNORECASE)
if match:
return [{"action": "wait", "ms": int(match.group(1)) * 1000}]
match = re.search(
r"(?:点击|点选|选择)\s*(.+?)(?:菜单|按钮|链接)?$",
seg,
re.IGNORECASE,
)
if match:
target = match.group(1).strip()
target = re.sub(r"(然后|再|并)?\s*(截图|截屏).*$", "", target, flags=re.IGNORECASE).strip()
target = re.sub(r"(然后|再|之后)$", "", target).strip()
target = re.sub(r"(菜单|按钮|链接)$", "", target).strip()
if target:
return [{"action": "click", "target": target}]
match = re.search(
r"(?:访问|打开|进入)\s*(https?://\S+|/\S+|登录页|主页|首页)",
seg,
re.IGNORECASE,
)
if match:
target = match.group(1)
mapping = {"登录页": "/login", "主页": "/app/dashboard", "首页": "/app/dashboard"}
return [{"action": "goto", "target": mapping.get(target, target)}]
return []
def parse_natural_language(text: str) -> BrowserScenario | None:
segments = _split_segments(text)
steps: list[dict[str, Any]] = []
for segment in segments:
steps.extend(_parse_segment(segment))
if not steps:
return None
if not any(step.get("action") == "screenshot" for step in steps):
if re.search(r"截图|截屏", text, re.IGNORECASE):
steps.append({"action": "screenshot"})
if not steps:
return None
return BrowserScenario(
name="natural",
base_url=default_base_url(),
steps=steps,
source="natural-language",
)
def parse_browser_request(text: str) -> BrowserScenario | None:
if not is_browser_intent(text):
return None
raw = _strip_mention(text)
yaml_block = re.search(r"```(?:yaml|yml)\s*\n(.+?)```", raw, re.IGNORECASE | re.DOTALL)
if yaml_block:
data = yaml.safe_load(yaml_block.group(1))
if isinstance(data, dict):
base_url = interpolate(str(data.get("base_url") or default_base_url()))
steps = data.get("steps") or []
return BrowserScenario(
name=data.get("name") or "yaml-inline",
base_url=base_url,
steps=_normalize_steps(steps),
source="yaml-inline",
)
inline = _parse_inline_dsl(text)
if inline:
return inline
match = re.match(r"^(browser|网页|网页操作|操作)\s+([\w\-./]+)\s*$", raw, re.IGNORECASE)
if match:
path = _find_scenario_file(match.group(2))
if not path:
raise RuntimeError(f"未找到场景文件:{match.group(2)}.yaml")
return _load_yaml_scenario(path)
scenario = parse_natural_language(text)
if scenario:
return scenario
default_name = (env_config.env("BROWSER_DEFAULT_SCENARIO") or "").strip()
if default_name:
path = _find_scenario_file(default_name)
if path:
return _load_yaml_scenario(path)
return None

View File

@@ -1,87 +0,0 @@
"""浏览器自动化服务:解析场景 + 启动 dev server + 执行步骤。"""
from __future__ import annotations
import asyncio
import logging
from urllib.parse import urlparse
from browser_executor import run_browser_scenario_sync
from browser_models import BrowserResult, BrowserScenario
from browser_parser import parse_browser_request
from preview_service import (
_package_dev_script,
_preview_port,
_project_cwd,
_startup_timeout,
_wait_for_port,
)
logger = logging.getLogger(__name__)
def _ensure_dev_server(base_url: str) -> bool:
parsed = urlparse(base_url)
host = parsed.hostname or "127.0.0.1"
port = parsed.port or (443 if parsed.scheme == "https" else 80)
if _wait_for_port(host, port, timeout=3):
return False
cwd = _project_cwd()
dev_command = _package_dev_script(cwd)
if not dev_command:
raise RuntimeError(
f"无法访问 {base_url},且未找到可启动的 dev 脚本。"
"请先手动启动前端,或设置 PREVIEW_URL。"
)
import subprocess
logger.info("启动 dev server: %s (cwd=%s)", dev_command, cwd)
proc = subprocess.Popen(
dev_command,
cwd=str(cwd),
shell=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
)
if not _wait_for_port(host, port, timeout=_startup_timeout()):
err = ""
if proc.stderr:
err = proc.stderr.read().decode("utf-8", errors="replace")[-1000:]
proc.kill()
raise RuntimeError(
f"dev server 在 {_startup_timeout()}s 内未就绪 ({base_url})。"
f"{(' 日志: ' + err) if err else ''}"
)
return True
async def run_browser_automation(text: str) -> BrowserResult:
scenario = parse_browser_request(text)
if scenario is None:
raise RuntimeError("无法解析网页操作步骤")
started = await asyncio.to_thread(_ensure_dev_server, scenario.base_url)
result = await asyncio.to_thread(run_browser_scenario_sync, scenario)
result.started_dev_server = started
return result
def format_browser_caption(result: BrowserResult) -> str:
lines = [
"**网页操作完成**",
f"> 场景:`{result.scenario_name or '自定义'}`",
f"> 起始:`{result.base_url}`",
f"> 最终:`{result.final_url}`",
f"> 步骤数:{result.step_count}",
]
if result.started_dev_server:
lines.append("> dev server已自动启动")
if result.step_log:
lines.append("")
lines.append("执行记录:")
for item in result.step_log[-8:]:
lines.append(f"- {item}")
return "\n".join(lines)

View File

@@ -1,106 +0,0 @@
"""通过 Cursor SDK 执行用户任务。"""
from __future__ import annotations
import asyncio
import logging
import re
from typing import Awaitable, Callable
import env_config
from bridge_manager import warm_cursor_bridge
logger = logging.getLogger(__name__)
_cursor_lock = asyncio.Lock()
WECHAT_SYSTEM_PREFIX = """你是企业微信群里的 Skills 助手,正在回复群成员的消息。
要求:
- 用简洁的中文回答(除非用户用其他语言提问)
- 使用企业微信支持的 Markdown 子集(加粗、链接、列表;避免复杂表格)
- 直接给出结论,不要冗长铺垫
- 若任务涉及 skills.sh可说明安装命令 `npx skills add owner/repo/skill-name`
- **不要**在回复里写 `[图片]` 占位符;企微无法通过 Markdown 显示图片
- 若用户要页面截图,请明确告知其发送:`截图` 或 `preview`(由 bot 自动发图)
用户任务:
"""
def _cursor_settings() -> dict[str, str | int]:
timeout_raw = env_config.env("CURSOR_TIMEOUT", "600") or "600"
return {
"api_key": env_config.env("CURSOR_API_KEY"),
"cwd": env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech",
"model": env_config.env("CURSOR_MODEL", "composer-2.5") or "composer-2.5",
"timeout": int(timeout_raw),
}
def strip_mention(text: str) -> str:
return re.sub(r"@\S+\s*", "", text).strip()
def _build_prompt(task: str) -> str:
return WECHAT_SYSTEM_PREFIX + task.strip()
def execute_cursor_task_sync(task: str) -> str:
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
settings = _cursor_settings()
api_key = settings["api_key"]
if not api_key:
raise RuntimeError(
"未配置 CURSOR_API_KEY。请在 bot/.env 中设置,"
"密钥见 https://cursor.com/dashboard/integrations"
)
warm_cursor_bridge()
cwd = str(settings["cwd"])
prompt = _build_prompt(task)
logger.info("Cursor 执行任务 cwd=%s model=%s", cwd, settings["model"])
try:
result = Agent.prompt(
prompt,
AgentOptions(
api_key=api_key,
model=settings["model"],
local=LocalAgentOptions(cwd=cwd),
),
)
except CursorAgentError as exc:
raise RuntimeError(
f"Cursor 启动失败:{exc.message}"
+ ("(可重试)" if exc.is_retryable else "")
) from exc
if result.status == "error":
detail = result.result or "运行失败,无详细错误"
raise RuntimeError(f"Cursor 执行失败:{detail}")
text = (result.result or "").strip()
if not text:
return "Cursor 已完成任务,但没有返回文本内容。"
return text
async def run_cursor_task(
task: str,
on_progress: Callable[[str], Awaitable[None]] | None = None,
) -> str:
timeout = int(_cursor_settings()["timeout"])
if on_progress:
await on_progress("Cursor 正在执行任务,请稍候…")
async with _cursor_lock:
try:
return await asyncio.wait_for(
asyncio.to_thread(execute_cursor_task_sync, task),
timeout=timeout,
)
except asyncio.TimeoutError as exc:
raise RuntimeError(f"Cursor 执行超时(>{timeout}s") from exc

View File

@@ -1,21 +0,0 @@
"""加载 bot/.env供各模块在 import 时统一读取环境变量。"""
from __future__ import annotations
import os
import sys
from pathlib import Path
from dotenv import load_dotenv
_BOT_DIR = Path(__file__).resolve().parent
_REPO_ROOT = _BOT_DIR.parent
if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))
load_dotenv(_BOT_DIR / ".env")
load_dotenv(_BOT_DIR / ".env.local", override=True)
def env(key: str, default: str | None = None) -> str | None:
return os.getenv(key, default)

View File

@@ -1,58 +0,0 @@
"""从文本/Cursor 回复中解析本地截图路径。"""
from __future__ import annotations
import re
from pathlib import Path
import env_config
IMAGE_SUFFIXES = (".png", ".jpg", ".jpeg", ".webp")
def _project_cwd() -> Path:
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
return Path(raw).resolve()
def _resolve_candidate(raw: str, cwd: Path) -> Path | None:
cleaned = raw.strip().strip("`\"'[]()")
if not cleaned or cleaned.startswith("http"):
return None
path = Path(cleaned)
if not path.is_absolute():
path = cwd / path
try:
resolved = path.resolve()
except OSError:
return None
if resolved.is_file() and resolved.suffix.lower() in IMAGE_SUFFIXES:
return resolved
return None
def find_image_paths(text: str) -> list[Path]:
cwd = _project_cwd()
seen: set[Path] = set()
found: list[Path] = []
patterns = [
r"(?:保存(?:至|到)|saved\s+to|screenshot\s*[:])\s*([^\s\n\]]+\.(?:png|jpe?g|webp))",
r"([A-Za-z]:\\[^\s\n\]]+\.(?:png|jpe?g|webp))",
r"([^\s\n\]]+\.(?:png|jpe?g|webp))",
]
for pattern in patterns:
for match in re.finditer(pattern, text, re.IGNORECASE):
path = _resolve_candidate(match.group(1), cwd)
if path and path not in seen:
seen.add(path)
found.append(path)
return found
def strip_fake_image_markdown(text: str) -> str:
text = re.sub(r"^\s*\[图片\]\s*$", "", text, flags=re.MULTILINE)
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip()

View File

@@ -1,156 +0,0 @@
"""企业微信智能机器人 · Skills 助手skills 快查 + 截图预览 + Cursor 执行任务)。"""
from __future__ import annotations
import logging
import sys
import env_config
from bridge_manager import shutdown_cursor_bridge, warm_cursor_bridge
from router import route_message, routing_mode
from skills_service import handle_command, warm_feed_cache
from wecom_media import reply_image, upload_image
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
)
logger = logging.getLogger("skills-bot")
BOT_ID = env_config.env("WECOM_BOT_ID") or env_config.env("WECHAT_BOT_ID")
BOT_SECRET = env_config.env("WECOM_BOT_SECRET") or env_config.env("WECHAT_BOT_SECRET")
def _require_credentials() -> None:
if not BOT_ID or not BOT_SECRET:
print(
"请设置环境变量 WECOM_BOT_ID 和 WECOM_BOT_SECRET\n"
"(企业微信 → 智能机器人 → API 模式 → 长连接)",
file=sys.stderr,
)
sys.exit(1)
def create_client():
from aibot import WSClient, WSClientOptions, generate_req_id
ws_client = WSClient(
WSClientOptions(
bot_id=BOT_ID,
secret=BOT_SECRET,
)
)
@ws_client.on("authenticated")
def on_authenticated():
logger.info("企业微信长连接认证成功,路由模式=%s", routing_mode())
cursor_key = env_config.env("CURSOR_API_KEY")
if cursor_key:
logger.info("CURSOR_API_KEY 已加载(%s…)", cursor_key[:8])
try:
warm_cursor_bridge()
logger.info("Cursor bridge 预启动完成")
except Exception as exc:
logger.warning("Cursor bridge 预启动失败Cursor 任务时会重试): %s", exc)
else:
logger.warning("CURSOR_API_KEY 未配置Cursor 任务将失败")
try:
warm_feed_cache()
logger.info("skills 数据预加载完成")
except Exception as exc:
logger.warning("skills 数据预加载失败: %s", exc)
@ws_client.on("event.enter_chat")
async def on_enter_chat(frame):
help_text = handle_command("help")
extra = (
"\n\n---\n"
"**单页截图**`preview` / `截图` / `预览 [路径或URL]`\n"
"**网页操作**:自然语言多步操作,或 `browser 场景名`\n"
"例:`访问登录页,输入账号密码,点击登录,点击智能体管理,截图`\n"
"场景文件:`bot/scenarios/*.yaml`(可用 `browser xiaobao-agent-manage`"
)
await ws_client.reply_welcome(
frame,
{
"msgtype": "markdown",
"markdown": {"content": help_text + extra},
},
)
@ws_client.on("message.text")
async def on_text(frame):
body = frame.get("body", {})
content = body.get("text", {}).get("content", "")
logger.info("收到消息: %s", content)
stream_id = generate_req_id("stream")
last_progress = ""
async def on_progress(message: str) -> None:
nonlocal last_progress
if message != last_progress:
last_progress = message
await ws_client.reply_stream(frame, stream_id, message, False)
await ws_client.reply_stream(frame, stream_id, "收到,正在处理…", False)
try:
result = await route_message(content, on_progress=on_progress)
reply = result.text
logger.info(
"回复来源: %s, 文本长度=%d, 图片=%s",
result.source,
len(reply),
result.image_path or "-",
)
except Exception as exc:
logger.exception("处理失败")
reply = f"处理失败:{exc}"
result = None
if len(reply) > 3800:
reply = reply[:3800] + "\n\n> …内容已截断"
await ws_client.reply_stream(frame, stream_id, reply, True)
if result and result.image_path:
try:
media_id = await upload_image(ws_client, result.image_path)
await reply_image(ws_client, frame, media_id)
logger.info("图片已发送到企微: %s", result.image_path)
except Exception as exc:
logger.exception("发送图片失败")
await ws_client.reply(
frame,
{
"msgtype": "markdown",
"markdown": {
"content": f"截图文件:`{result.image_path}`\n发图失败:{exc}\n\n请确认 bot 已重启,或发送 `截图` 重试。",
},
},
)
@ws_client.on("error")
def on_error(error):
logger.error("连接错误: %s", error)
@ws_client.on("disconnected")
def on_disconnected(reason):
logger.warning("连接断开: %s", reason)
return ws_client
def main() -> None:
import atexit
atexit.register(shutdown_cursor_bridge)
_require_credentials()
client = create_client()
logger.info("启动 Skills 助手Bot ID=%s", BOT_ID[:8] if BOT_ID else "?")
client.run()
if __name__ == "__main__":
main()

View File

@@ -1,255 +0,0 @@
"""在 CURSOR_CWD 启动/访问前端并截图(单页,不含多步操作)。"""
from __future__ import annotations
import asyncio
import json
import logging
import re
import socket
import subprocess
import time
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from urllib.parse import urlparse
import env_config
logger = logging.getLogger(__name__)
SCREENSHOT_DIR = Path(__file__).resolve().parent / ".cache" / "screenshots"
@dataclass
class PreviewResult:
url: str
screenshot_path: Path
started_dev_server: bool
final_url: str | None = None
@dataclass
class PreviewRequest:
url: str | None
port: int | None
def _project_cwd() -> Path:
raw = env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
return Path(raw).resolve()
def _preview_port() -> int:
raw = env_config.env("PREVIEW_PORT", "5173") or "5173"
return int(raw)
def _startup_timeout() -> int:
raw = env_config.env("PREVIEW_STARTUP_TIMEOUT", "120") or "120"
return int(raw)
def _dev_command() -> str:
return env_config.env("PREVIEW_DEV_COMMAND", "npm run dev") or "npm run dev"
def parse_preview_command(text: str) -> tuple[str | None, int | None] | None:
raw = re.sub(r"@\S+\s*", "", text).strip()
if not raw:
return None
m = re.match(
r"^(preview|截图|预览|截屏)(?:\s+(https?://\S+|/\S*))?(?:\s+(\d{2,5}))?$",
raw,
re.IGNORECASE,
)
if not m:
return None
url_part = m.group(2)
port_part = m.group(3)
port = int(port_part) if port_part else None
if url_part and url_part.startswith("/"):
port = port or _preview_port()
return f"http://127.0.0.1:{port}{url_part}", port
return url_part, port
def resolve_preview_request(text: str) -> PreviewRequest | None:
explicit = parse_preview_command(text)
if explicit is not None:
url_override, port_override = explicit
return PreviewRequest(url=url_override, port=port_override)
if not is_preview_intent(text):
return None
url_override = extract_url_from_text(text)
if not url_override:
env_url = env_config.env("PREVIEW_URL")
url_override = env_url.strip() if env_url else f"http://127.0.0.1:{_preview_port()}/"
return PreviewRequest(url=url_override, port=None)
_PREVIEW_INTENT = re.compile(
r"^(preview|截图|预览|截屏)\b|"
r"(页面预览|运行.*(前端|项目|页面)|"
r"打开.*(前端|页面|项目)|"
r"访问.*(并)?.*(截图|截屏)|"
r"启动.*(前端|项目|dev|服务).*(截图|截屏)?)",
re.IGNORECASE,
)
def is_preview_intent(text: str) -> bool:
raw = re.sub(r"@\S+\s*", "", text).strip()
if parse_preview_command(text) is not None:
return True
return bool(_PREVIEW_INTENT.search(raw))
def extract_url_from_text(text: str) -> str | None:
raw = re.sub(r"@\S+\s*", "", text)
match = re.search(
r"(https?://[^\s\]`\"']+|localhost:\d+[/\w\-./]*)",
raw,
re.IGNORECASE,
)
if not match:
return None
url = match.group(1).rstrip(".,,。")
if url.lower().startswith("localhost"):
url = "http://" + url
return url
def _capture_screenshot_sync(url: str, output: Path) -> str:
from playwright.sync_api import sync_playwright
output.parent.mkdir(parents=True, exist_ok=True)
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page(viewport={"width": 1280, "height": 720})
page.goto(url, wait_until="networkidle", timeout=60_000)
page.wait_for_timeout(1500)
page.screenshot(path=str(output), full_page=False, type="png")
final_url = page.url
browser.close()
return final_url
def _wait_for_port(host: str, port: int, timeout: int) -> bool:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection((host, port), timeout=2):
return True
except OSError:
time.sleep(1)
return False
def _resolve_target_url(url_override: str | None, port_override: int | None) -> tuple[str, str | None]:
if url_override:
parsed = urlparse(url_override)
if parsed.scheme and parsed.netloc:
return url_override, None
raise RuntimeError(f"无效 URL{url_override}")
env_url = env_config.env("PREVIEW_URL")
if env_url:
return env_url.strip(), None
port = port_override or _preview_port()
cwd = _project_cwd()
dev_script = _package_dev_script(cwd)
base = f"http://127.0.0.1:{port}/"
return base, dev_script
def _package_dev_script(cwd: Path) -> str | None:
pkg = cwd / "package.json"
if not pkg.exists():
return None
try:
data = json.loads(pkg.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
scripts = data.get("scripts") or {}
for key in ("dev", "preview", "start"):
if scripts.get(key):
cmd = _dev_command()
if key != "dev" and cmd == "npm run dev":
return f"npm run {key}"
return cmd
return None
def _capture_preview_sync(url: str, dev_command: str | None) -> PreviewResult:
cwd = _project_cwd()
parsed = urlparse(url)
host = parsed.hostname or "127.0.0.1"
port = parsed.port or (443 if parsed.scheme == "https" else 80)
dev_proc: subprocess.Popen | None = None
started = False
if dev_command:
if _wait_for_port(host, port, timeout=3):
logger.info("检测到端口 %s 已监听,跳过启动 dev server", port)
else:
logger.info("启动 dev server: %s (cwd=%s)", dev_command, cwd)
dev_proc = subprocess.Popen(
dev_command,
cwd=str(cwd),
shell=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
)
started = True
if not _wait_for_port(host, port, timeout=_startup_timeout()):
err = ""
if dev_proc.stderr:
err = dev_proc.stderr.read().decode("utf-8", errors="replace")[-1000:]
raise RuntimeError(
f"dev server 在 {_startup_timeout()}s 内未就绪 ({url})。"
f"{(' 日志: ' + err) if err else ''}"
)
else:
if not _wait_for_port(host, port, timeout=5):
raise RuntimeError(
f"无法访问 {url}。请在 CURSOR_CWD 放置前端项目,"
"或先手动启动 dev server或设置 PREVIEW_URL。"
)
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
output = SCREENSHOT_DIR / f"preview-{stamp}.png"
try:
final_url = _capture_screenshot_sync(url, output)
finally:
if dev_proc and dev_proc.poll() is None:
dev_proc.terminate()
try:
dev_proc.wait(timeout=5)
except subprocess.TimeoutExpired:
dev_proc.kill()
return PreviewResult(
url=url,
screenshot_path=output,
started_dev_server=started,
final_url=final_url,
)
async def capture_preview(
url_override: str | None = None,
port_override: int | None = None,
) -> PreviewResult:
url, dev_command = _resolve_target_url(url_override, port_override)
return await asyncio.to_thread(_capture_preview_sync, url, dev_command)

View File

@@ -1,7 +0,0 @@
wecom-aibot-python-sdk>=1.0.2
python-dotenv>=1.0.0
httpx>=0.27.0
certifi>=2024.0.0
cursor-sdk>=0.1.0
playwright>=1.49.0
PyYAML>=6.0.0

View File

@@ -1,109 +0,0 @@
"""消息路由skills 快查 / 网页操作 / 截图预览 / Cursor 通用任务。"""
from __future__ import annotations
import re
import env_config
from browser_parser import is_browser_intent, parse_browser_request
from browser_service import format_browser_caption, run_browser_automation
from cursor_runner import run_cursor_task, strip_mention
from image_extract import find_image_paths, strip_fake_image_markdown
from preview_service import capture_preview, is_preview_intent, resolve_preview_request
from skills_service import handle_command, parse_command
from bot_types import RouteResult
def routing_mode() -> str:
return (env_config.env("ROUTING_MODE", "hybrid") or "hybrid").lower()
def _normalize(text: str) -> str:
return re.sub(r"@\S+\s*", "", text).strip().lower()
def is_skills_fast_command(text: str) -> bool:
raw = _normalize(text)
if not raw:
return True
if raw in {"help", "帮助", "?", "h"}:
return True
cmd = parse_command(text)
if cmd.kind in {"help", "list", "detail"}:
return True
if cmd.kind == "search" and re.match(r"^(search|搜索|find|查)\s+", raw):
return True
return False
async def _run_browser(text: str, on_progress=None) -> RouteResult:
if parse_browser_request(text) is None:
raise RuntimeError("无法解析网页操作步骤")
if on_progress:
await on_progress("正在按步骤执行网页操作…")
result = await run_browser_automation(text)
return RouteResult(
source="browser",
text=format_browser_caption(result),
image_path=str(result.screenshot_path),
)
async def _run_preview(text: str, on_progress=None) -> RouteResult:
preview_req = resolve_preview_request(text)
if preview_req is None:
raise RuntimeError("无法解析截图请求")
if on_progress:
await on_progress(f"正在访问并截图:{preview_req.url or '默认地址'}")
result = await capture_preview(preview_req.url, preview_req.port)
caption = (
f"**页面预览**\n"
f"> URL`{result.final_url or result.url}`\n"
f"> 项目:`{env_config.env('CURSOR_CWD', '')}`\n"
f"> dev server{'已自动启动' if result.started_dev_server else '使用已有服务'}"
)
return RouteResult(
source="preview",
text=caption,
image_path=str(result.screenshot_path),
)
async def route_message(text: str, on_progress=None) -> RouteResult:
task = strip_mention(text)
if not task:
return RouteResult("skills", handle_command("help"))
if is_browser_intent(text):
return await _run_browser(text, on_progress=on_progress)
if resolve_preview_request(text) is not None:
return await _run_preview(text, on_progress=on_progress)
mode = routing_mode()
if mode == "skills":
return RouteResult("skills", handle_command(text))
if mode == "cursor" or not is_skills_fast_command(text):
reply = await run_cursor_task(task, on_progress=on_progress)
reply = strip_fake_image_markdown(reply)
image_path: str | None = None
paths = find_image_paths(reply)
if paths:
image_path = str(paths[0])
elif is_preview_intent(text) or is_browser_intent(text):
if on_progress:
await on_progress("未找到截图文件,改用 Playwright 自动执行…")
if is_browser_intent(text):
return await _run_browser(text, on_progress=on_progress)
return await _run_preview(text, on_progress=on_progress)
return RouteResult("cursor", reply, image_path=image_path)
return RouteResult("skills", handle_command(text))

View File

@@ -1,17 +0,0 @@
name: xiaobao-agent-manage
description: 登录后打开智能体管理并截图
steps:
- goto: /login
- fill:
field: 账号
value: "{{PREVIEW_LOGIN_USER}}"
- fill:
field: 密码
value: "{{PREVIEW_LOGIN_PASSWORD}}"
- click: 登录
- wait:
url: "**/app/**"
timeout: 60000
- click: 智能体管理
- wait: 1500
- screenshot

View File

@@ -1,226 +0,0 @@
"""skills.sh 快查命令解析与企微回复格式化。"""
from __future__ import annotations
import re
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Literal
_REPO_ROOT = Path(__file__).resolve().parent.parent
if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))
from shared.skills_data import Board, board_items, format_installs, load_feed, warm_feed_cache
__all__ = ["Command", "handle_command", "parse_command", "warm_feed_cache"]
@dataclass
class Command:
kind: Literal["help", "list", "search", "detail"]
board: Board = "trending"
limit: int = 10
query: str = ""
def _normalize_text(text: str) -> str:
text = re.sub(r"@\S+\s*", "", text)
return text.strip().lower()
def _parse_limit(raw: str | None, default: int = 10) -> int:
if not raw:
return default
try:
n = int(raw)
except ValueError:
return default
return max(1, min(n, 30))
def _match_list(raw: str, board: Board, aliases: str) -> Command | None:
m = re.match(rf"^({aliases})(?:\s+top)?\s*(\d+)?$", raw)
if m:
return Command(kind="list", board=board, limit=_parse_limit(m.group(2)))
m = re.match(rf"^(查|查询)\s+({aliases})(?:\s+top)?\s*(\d+)?$", raw)
if m:
return Command(kind="list", board=board, limit=_parse_limit(m.group(3)))
return None
def parse_command(text: str) -> Command:
raw = _normalize_text(text)
if not raw or raw in {"help", "帮助", "?", "h"}:
return Command(kind="help")
for board, aliases in (
("trending", "trending|趋势|top"),
("hot", "hot|实时|热门"),
("all", "all|总榜|alltime|all-time"),
):
cmd = _match_list(raw, board, aliases)
if cmd:
return cmd
m = re.match(r"^(search|搜索|find|查)\s+(.+)$", raw)
if m:
return Command(kind="search", query=m.group(2).strip(), limit=5)
m = re.match(r"^(detail|详情|skill|info)\s+(.+)$", raw)
if m:
return Command(kind="detail", query=m.group(2).strip())
if raw.startswith("trending") or raw.startswith("趋势"):
parts = raw.split(maxsplit=1)
return Command(kind="list", board="trending", limit=_parse_limit(parts[1] if len(parts) > 1 else None))
return Command(kind="search", query=raw, limit=5)
def _board_title(board: Board) -> str:
return {
"trending": "Trending近期增长",
"hot": "Hot实时热度",
"all": "All Time总安装榜",
}[board]
def format_list(board: Board, limit: int) -> str:
feed = load_feed()
items = board_items(feed, board)[:limit]
updated = feed.get("updatedAt", "未知")[:10]
lines = [
f"**skills.sh {_board_title(board)} Top {limit}**",
f"> 数据更新:{updated}",
"",
]
for i, item in enumerate(items, 1):
title = item.get("title", "?")
source = item.get("source", "?")
installs = format_installs(item.get("installs", 0))
desc = item.get("description", "")
if len(desc) > 80:
desc = desc[:77] + "..."
link = item.get("link", "")
lines.append(f"{i}. **{title}** · {installs}")
lines.append(f" `{source}`")
if desc:
lines.append(f" {desc}")
if link:
lines.append(f" [查看]({link})")
lines.append("")
return "\n".join(lines).strip()
def format_search(query: str, limit: int) -> str:
feed = load_feed()
q = query.lower()
seen: set[str] = set()
matches: list[dict[str, Any]] = []
for board in ("topTrending", "topHot", "topAllTime"):
for item in feed.get(board, []):
item_id = item.get("id") or item.get("title", "")
if item_id in seen:
continue
haystack = " ".join(
[
item.get("title", ""),
item.get("source", ""),
item.get("description", ""),
]
).lower()
if q in haystack:
seen.add(item_id)
matches.append(item)
if len(matches) >= limit:
break
if len(matches) >= limit:
break
if not matches:
return f"未找到与 **{query}** 相关的 skill。\n\n试试:`trending 10` / `hot 10` / `搜索 react`"
lines = [f"**搜索「{query}」** 共 {len(matches)}", ""]
for i, item in enumerate(matches, 1):
title = item.get("title", "?")
source = item.get("source", "?")
installs = format_installs(item.get("installs", 0))
link = item.get("link", "")
lines.append(f"{i}. **{title}** · {installs} · `{source}`")
if link:
lines.append(f" [查看]({link})")
return "\n".join(lines)
def format_detail(name: str) -> str:
feed = load_feed()
q = name.lower().strip()
best: dict[str, Any] | None = None
for board in ("topTrending", "topHot", "topAllTime"):
for item in feed.get(board, []):
title = (item.get("title") or "").lower()
item_id = (item.get("id") or "").lower()
if title == q or q in title or q in item_id:
if best is None or item.get("installs", 0) > best.get("installs", 0):
best = item
if not best:
return f"未找到 skill**{name}**\n\n试试:`搜索 {name}`"
desc = best.get("description", "无描述")
return "\n".join(
[
f"**{best.get('title', '?')}**",
f"`{best.get('source', '?')}`",
f"安装量:**{format_installs(best.get('installs', 0))}**",
"",
desc,
"",
f"[skills.sh 详情]({best.get('link', 'https://skills.sh')})",
"",
f"安装:`npx skills add {best.get('source', '')}/{best.get('title', '')}`",
]
)
def format_help() -> str:
return "\n".join(
[
"**Skills 助手 · 命令帮助**",
"",
"`trending 10` / `趋势 10` — 近期增长榜",
"`hot 10` / `实时 10` — 实时热度榜",
"`all 10` / `总榜 10` — 历史总安装榜",
"`搜索 react` / `search tdd` — 关键词搜索",
"`详情 find-skills` — 查看单个 skill",
"`preview` / `截图` / `预览` — 单页截图",
"`browser 场景名` — 执行 YAML 场景(见 bot/scenarios/",
"自然语言 — 如:访问登录页,输入账号密码,点击登录,点击智能体管理,截图",
"`preview /about 5173` — 指定路径和端口",
"",
"示例:",
"• trending top10",
"• 查 grill",
"• 详情 remotion-render",
]
)
def handle_command(text: str) -> str:
cmd = parse_command(text)
if cmd.kind == "help":
return format_help()
if cmd.kind == "list":
return format_list(cmd.board, cmd.limit)
if cmd.kind == "search":
return format_search(cmd.query, cmd.limit)
if cmd.kind == "detail":
return format_detail(cmd.query)
return format_help()

View File

@@ -1,96 +0,0 @@
"""企业微信 API 模式:上传图片并回复。"""
from __future__ import annotations
import base64
import hashlib
import logging
from pathlib import Path
from typing import Any
from aibot import generate_req_id
logger = logging.getLogger(__name__)
CHUNK_SIZE = 512 * 1024
MAX_IMAGE_BYTES = 9 * 1024 * 1024
def _ensure_image_size(path: Path) -> bytes:
data = path.read_bytes()
if len(data) > MAX_IMAGE_BYTES:
raise RuntimeError(
f"截图过大({len(data) // 1024}KB请缩小页面或使用 viewport 截图(上限 9MB"
)
return data
def _response_body(frame: dict[str, Any]) -> dict[str, Any]:
if frame.get("errcode", 0) != 0:
raise RuntimeError(
f"企微接口错误 errcode={frame.get('errcode')} errmsg={frame.get('errmsg')}"
)
body = frame.get("body")
return body if isinstance(body, dict) else {}
async def upload_image(ws_client: Any, image_path: str | Path) -> str:
path = Path(image_path)
if not path.exists():
raise RuntimeError(f"截图不存在: {path}")
data = _ensure_image_size(path)
md5 = hashlib.md5(data).hexdigest()
chunks = [data[i : i + CHUNK_SIZE] for i in range(0, len(data), CHUNK_SIZE)]
total_chunks = len(chunks)
manager = ws_client._ws_manager
init_frame = await manager.send_reply(
generate_req_id("upload_init"),
{
"type": "image",
"filename": path.name,
"total_size": len(data),
"total_chunks": total_chunks,
"md5": md5,
},
"aibot_upload_media_init",
)
upload_id = _response_body(init_frame).get("upload_id")
if not upload_id:
raise RuntimeError("上传初始化失败:未返回 upload_id")
for index, chunk in enumerate(chunks):
chunk_frame = await manager.send_reply(
generate_req_id("upload_chunk"),
{
"upload_id": upload_id,
"chunk_index": index,
"base64_data": base64.b64encode(chunk).decode("ascii"),
},
"aibot_upload_media_chunk",
)
_response_body(chunk_frame)
finish_frame = await manager.send_reply(
generate_req_id("upload_finish"),
{"upload_id": upload_id},
"aibot_upload_media_finish",
)
media_id = _response_body(finish_frame).get("media_id")
if not media_id:
raise RuntimeError("上传完成但未返回 media_id")
logger.info("图片已上传 media_id=%s", str(media_id)[:12])
return str(media_id)
async def reply_image(ws_client: Any, frame: dict[str, Any], media_id: str) -> None:
await ws_client.reply(
frame,
{
"msgtype": "image",
"image": {"media_id": media_id},
},
)

View File

@@ -5,6 +5,7 @@ from __future__ import annotations
import sys
from daily.generate import main as generate_main
from daily.scheduler import main as schedule_main
from daily.webhook import main as push_main
@@ -14,7 +15,14 @@ def main() -> int:
return generate_main()
if cmd in {"push", "send", "webhook"}:
return push_main(sys.argv[2:])
print(f"未知命令: {cmd}\n用法: python -m daily [generate|push] [report_path]", file=sys.stderr)
if cmd in {"schedule", "scheduler", "daemon"}:
return schedule_main()
print(
f"未知命令: {cmd}\n"
"用法: python -m daily [generate|push|schedule] [report_path]\n"
" python -m daily schedule [--once] [--dry-run]",
file=sys.stderr,
)
return 1

View File

@@ -59,11 +59,41 @@ def _extract_markdown(text: str) -> str:
def analyze_trends(llm_input: dict[str, Any], *, date_str: str) -> dict[str, Any] | None:
from daily.config import theme_ban_days
from daily.narrative_axis import (
enforce_narrative_axis,
load_recent_axes,
load_recent_theme_summaries,
pick_narrative_axis,
)
skill = _load_skill()
featured_note = ""
if llm_input.get("featured_pick"):
featured_note = (
"\n输入已含 **featured_pick**(编辑指定今日首推);"
"top_picks.skill 必须以 featured_pick 为准;"
"why/opening 不得向读者提及「编辑指定」。\n"
)
used_axes = set(load_recent_axes(date_str))
axis = pick_narrative_axis(used_axes)
llm_input["required_narrative_axis"] = axis
llm_input["narrative_axis"] = axis
theme_ban = load_recent_theme_summaries(date_str, theme_ban_days())
ban_note = ""
if theme_ban:
ban_note = (
"\n近几日已用过的主题/导语(请软避开同类开场,勿原样复用):\n- "
+ "\n- ".join(theme_ban)
+ "\n"
)
system = (
f"{skill}\n\n"
f"{featured_note}"
f"{ban_note}"
"当前执行 **Step 1趋势分析**。\n"
"只输出 trends JSONheadline, opening, themes, top_picks, signals不要 Markdown。"
f"**required_narrative_axis** = `{axis}`;输出 JSON 必须含 `narrative_axis` 且等于该值。\n"
"只输出 trends JSONheadline, opening, themes, top_picks, signals, narrative_axis不要 Markdown。"
)
user = json.dumps(llm_input, ensure_ascii=False, indent=2)
try:
@@ -77,8 +107,9 @@ def analyze_trends(llm_input: dict[str, Any], *, date_str: str) -> dict[str, Any
if not parsed.get("headline") and not parsed.get("opening"):
logger.warning("Agent Step1 JSON 无效")
return None
parsed = enforce_narrative_axis(parsed, axis)
save_json(trends_json_path(date_str), parsed)
logger.info("Agent Step1 完成:%s", parsed.get("headline", "?"))
logger.info("Agent Step1 完成:%s [%s]", parsed.get("headline", "?"), axis)
return parsed
@@ -91,8 +122,17 @@ def write_wecom_report(
updated: str,
) -> str | None:
skill = _load_skill()
featured_note = ""
if llm_input.get("featured_pick"):
featured_note = (
"\n输入 data 已含 **featured_pick**"
"今日首推区块须使用 featured_pick.why_today"
"链接行用 Markdown [标题](URL),勿用反引号裸 URL"
"读者可见文案不得出现「编辑指定」等元信息。\n"
)
system = (
f"{skill}\n\n"
f"{featured_note}"
"当前执行 **Step 2撰写企微早报**。\n"
f"日期={date_str},时间={time_str},数据截至={updated}\n"
"只输出企微 Markdown 正文,不要代码块,不要 JSON。"

163
daily/board_history.py Normal file
View File

@@ -0,0 +1,163 @@
"""企微展示历史:读写 data.wecom_shown_keys与 movement_baseline 严格分离。"""
from __future__ import annotations
import json
import logging
import re
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
from daily.config import OUTPUT_DIR, board_dedup_days
from daily.delta import RECENT_BOARD_KEYS, skill_id
logger = logging.getLogger(__name__)
BOARD_KEYS = RECENT_BOARD_KEYS
_GITHUB_REPO_RE = re.compile(r"github\.com/([\w.-]+/[\w.-]+)", re.I)
_SKILL_SH_RE = re.compile(r"skills\.sh/([\w.-]+/[\w.-]+(?:/[\w.-]+)?)", re.I)
# 与企微正文榜单标题对齐;顺序用于切分相邻 section
_WECOM_SECTION_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
("skills_trending", re.compile(r"Skills\s+Trending", re.I)),
("skills_hot", re.compile(r"Skills\s+Hot", re.I)),
("github_trending", re.compile(r"GitHub\s+Trending", re.I)),
("github_emerging", re.compile(r"GitHub\s+新兴", re.I)),
("github_topic", re.compile(r"Topic\s+", re.I)),
)
def extract_shown_keys(board: str, items: list[dict[str, Any]]) -> list[str]:
"""从最终展示 items 抽取稳定 identity key。
Skills 榜同时写入 skill id 与 source便于周去重按仓屏蔽。
"""
keys: list[str] = []
seen: set[str] = set()
for item in items:
if board.startswith("skills_"):
candidates = [skill_id(item), str(item.get("source") or "").strip()]
else:
candidates = [str(item.get("repo") or "")]
for key in candidates:
if not key or key in seen:
continue
seen.add(key)
keys.append(key)
return keys
def _keys_from_board_items(board: str, data: dict[str, Any]) -> set[str]:
if board == "github_topic":
topic = data.get("github_topic") or {}
items = topic.get("repos") if isinstance(topic, dict) else []
else:
items = data.get(board) or []
if not isinstance(items, list):
return set()
return set(extract_shown_keys(board, items))
def parse_wecom_shown_keys(md: str) -> dict[str, set[str]]:
"""从企微 Markdown 按榜单 section 解析已展示 keys冷启动兼容"""
out: dict[str, set[str]] = {board: set() for board in BOARD_KEYS}
if not (md or "").strip():
return out
hits: list[tuple[int, str]] = []
for board, pattern in _WECOM_SECTION_PATTERNS:
for match in pattern.finditer(md):
hits.append((match.start(), board))
if not hits:
return out
hits.sort(key=lambda x: x[0])
for idx, (start, board) in enumerate(hits):
end = hits[idx + 1][0] if idx + 1 < len(hits) else len(md)
chunk = md[start:end]
if board.startswith("skills_"):
out[board].update(_SKILL_SH_RE.findall(chunk))
else:
out[board].update(_GITHUB_REPO_RE.findall(chunk))
return out
def _load_shown_keys_for_day(path: Path, date_str: str) -> dict[str, set[str]] | None:
"""读一日历史:优先 wecom_shown_keys缺省则回退 wecom.md再回退 data 榜字段。"""
empty = {board: set() for board in BOARD_KEYS}
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
logger.warning("读取 wecom_shown_keys %s 失败:%s", path, exc)
return None
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
return empty
out: dict[str, set[str]] = {board: set() for board in BOARD_KEYS}
shown = data.get("wecom_shown_keys")
if isinstance(shown, dict):
for board in BOARD_KEYS:
keys = shown.get(board) or []
if isinstance(keys, list):
out[board].update(str(k) for k in keys if k)
if any(out.values()):
return out
wecom_path = OUTPUT_DIR / f"{date_str}.wecom.md"
if wecom_path.exists():
try:
md = wecom_path.read_text(encoding="utf-8")
except OSError as exc:
logger.warning("读取 wecom.md 回退 %s 失败:%s", wecom_path, exc)
else:
parsed = parse_wecom_shown_keys(md)
if any(parsed.values()):
return parsed
for board in BOARD_KEYS:
out[board].update(_keys_from_board_items(board, data))
return out
def load_recent_shown_keys(
date_str: str,
*,
lookback_days: int | None = None,
) -> dict[str, set[str]]:
"""近 N 日已展示 keys 并集(不含当日)。缺省或读失败视为空集。"""
empty = {board: set() for board in BOARD_KEYS}
try:
dt = datetime.strptime(date_str, "%Y-%m-%d")
except ValueError:
return empty
days = lookback_days if lookback_days is not None else board_dedup_days()
out: dict[str, set[str]] = {board: set() for board in BOARD_KEYS}
for day_offset in range(1, days + 1):
prev_date = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
path = OUTPUT_DIR / f"{prev_date}.data.json"
if not path.exists():
continue
day_keys = _load_shown_keys_for_day(path, prev_date)
if day_keys is None:
continue
for board in BOARD_KEYS:
out[board].update(day_keys.get(board) or set())
return out
def merge_wecom_shown_into_data(
data: dict[str, Any],
shown: dict[str, list[str]],
) -> dict[str, Any]:
"""写入 wecom_shown_keys不修改 movement_baseline。"""
merged = dict(data)
merged["wecom_shown_keys"] = {
board: list(keys) for board, keys in shown.items()
}
return merged

48
daily/board_select.py Normal file
View File

@@ -0,0 +1,48 @@
"""五榜唯一列表主人:周去重 + 深池补满。"""
from __future__ import annotations
import logging
from typing import Any, Literal
from daily.delta import skill_id
logger = logging.getLogger(__name__)
def board_select(
*,
board: str,
items: list[dict[str, Any]],
recent_keys: set[str],
limit: int,
pool_size: int,
kind: Literal["skill", "github"],
) -> list[dict[str, Any]]:
"""从深池过滤近 N 日已展示 key按原顺序取满 limit不足则短榜。"""
if kind == "skill":
from daily.skills_group import group_skills_by_source
pool = group_skills_by_source(items, limit=pool_size, pool_size=pool_size)
else:
pool = items[: max(pool_size, limit)]
out: list[dict[str, Any]] = []
for item in pool:
if kind == "skill":
key = skill_id(item)
source = str(item.get("source") or "").strip()
if (key and key in recent_keys) or (source and source in recent_keys):
continue
if not key and not source:
continue
else:
key = str(item.get("repo") or "")
if not key or key in recent_keys:
continue
out.append(item)
if len(out) >= limit:
break
if len(out) < limit:
logger.info("board_short:%s:%s", board, len(out))
return out

View File

@@ -12,7 +12,7 @@ import time
from pathlib import Path
from typing import Any, Mapping
import env_config
from daily.config import ROOT, env
logger = logging.getLogger(__name__)
@@ -22,7 +22,7 @@ _bridge_process: subprocess.Popen[bytes] | None = None
def _cursor_cwd() -> str:
return env_config.env("CURSOR_CWD", r"d:\LY\test\tech") or r"d:\LY\test\tech"
return env("DAILY_CURSOR_CWD") or env("CURSOR_CWD") or str(ROOT)
def _parse_discovery_line(line: str) -> Mapping[str, Any] | None:
@@ -124,7 +124,8 @@ def warm_cursor_bridge(force: bool = False) -> None:
)
try:
discovery = _read_discovery_polling(process)
except Exception:
except (RuntimeError, OSError, ValueError) as exc:
logger.warning("Cursor bridge discovery 失败,终止子进程:%s", exc)
process.kill()
process.wait(timeout=5)
raise

View File

@@ -30,6 +30,24 @@ def wecom_skill_desc_limit() -> int:
return env_int("DAILY_WECOM_SKILL_DESC_LIMIT", 56)
def wecom_news_desc_limit() -> int:
"""企微新闻摘要建议字数;在句读/词边界截断,不加省略号。"""
return max(24, env_int("DAILY_WECOM_NEWS_DESC_LIMIT", 72))
def wecom_ai_news_tech_limit() -> int:
"""research 模式下技术类时讯条数(叠加在 DAILY_WECOM_AI_NEWS 之上)。"""
return max(0, env_int("DAILY_WECOM_AI_NEWS_TECH", 5))
def wecom_pad_pool_size(display_limit: int) -> int:
"""Delta 补榜候选池大小(展示条数之上多取,避免去重后凑不满)。"""
explicit = env_int("DAILY_WECOM_PAD_POOL", -1)
if explicit > 0:
return explicit
return max(display_limit * 5, 50)
def full_desc_limit() -> int:
"""完整版早报摘要长度0 表示不截断。"""
return env_int("DAILY_FULL_DESC_LIMIT", 0)
@@ -43,6 +61,7 @@ def wecom_max_bytes() -> int:
"""兼容旧配置名。"""
return wecom_chunk_bytes()
load_dotenv(ROOT / ".env")
load_dotenv(ROOT / ".env.local", override=True)
@@ -71,3 +90,85 @@ def env_int(key: str, default: int) -> int:
return int(raw)
except ValueError:
return default
def env_bool(key: str, default: bool) -> bool:
raw = env(key)
if raw is None:
return default
return raw.strip().lower() in {"1", "true", "yes", "on"}
def wecom_mode() -> str:
raw = (env("DAILY_WECOM_MODE") or "delta").strip().lower()
return raw if raw in {"full", "delta"} else "delta"
def news_dedup_days() -> int:
return max(1, env_int("DAILY_NEWS_DEDUP_DAYS", 7))
def skip_push_when_silent() -> bool:
return env_bool("DAILY_SKIP_PUSH_WHEN_SILENT", True)
def delta_baseline_fallback() -> str:
raw = (env("DAILY_DELTA_BASELINE_FALLBACK") or "full").strip().lower()
return raw if raw in {"full", "empty"} else "full"
def wecom_delta_pad() -> bool:
"""Delta 模式下新入榜优先,不足时用当日 Top 榜补满;补榜排除近 N 天 baseline 已出现条目。"""
return env_bool("DAILY_WECOM_DELTA_PAD", True)
def delta_pad_lookback_days() -> int:
"""补榜时排除近 N 天 baseline 已出现过的条目(默认与异动对比窗口一致)。"""
fallback = env_int("DAILY_DELTA_LOOKBACK_DAYS", 7)
return max(1, env_int("DAILY_DELTA_PAD_LOOKBACK_DAYS", fallback))
def force_push() -> bool:
return env_bool("DAILY_FORCE_PUSH", False)
def schedule_timezone_name() -> str:
return (env("DAILY_SCHEDULE_TZ") or "Asia/Shanghai").strip() or "Asia/Shanghai"
def schedule_generate_at() -> str:
return (env("DAILY_SCHEDULE_GENERATE_AT") or "08:50").strip() or "08:50"
def schedule_push_at() -> str:
return (env("DAILY_SCHEDULE_PUSH_AT") or "09:00").strip() or "09:00"
def board_dedup_days() -> int:
return max(1, env_int("DAILY_BOARD_DEDUP_DAYS", 7))
def board_pool_size() -> int:
fallback = env_int("DAILY_WECOM_SKILL_POOL", 400)
return max(1, env_int("DAILY_BOARD_POOL_SIZE", max(200, fallback)))
def featured_dedup_days() -> int:
return max(1, env_int("DAILY_FEATURED_DEDUP_DAYS", 30))
def theme_ban_days() -> int:
return max(1, env_int("DAILY_THEME_BAN_DAYS", 7))
def narrative_axis_days() -> int:
return max(1, env_int("DAILY_NARRATIVE_AXIS_DAYS", 3))
def news_backfill_enabled() -> bool:
return env_bool("DAILY_NEWS_BACKFILL", False)
def workday_only() -> bool:
"""仅工作日生成/推送;法定节假日与周末跳过(调休补班日照常)。"""
return env_bool("DAILY_WORKDAY_ONLY", True)

View File

@@ -8,13 +8,22 @@ from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Callable
from daily.config import OUTPUT_DIR, env_int
from daily.config import OUTPUT_DIR, delta_baseline_fallback, env_int, wecom_mode
logger = logging.getLogger(__name__)
KeyFn = Callable[[dict[str, Any]], str]
RECENT_BOARD_KEYS = (
"skills_trending",
"skills_hot",
"github_trending",
"github_emerging",
"github_topic",
)
def compare_depth() -> int:
return env_int("DAILY_DELTA_COMPARE_DEPTH", 15)
@@ -39,6 +48,23 @@ def _key_set(items: list[dict[str, Any]], key_fn: KeyFn, *, depth: int) -> set[s
return {key_fn(item) for item in items[:depth] if key_fn(item)}
def load_recent_board_keys(
date_str: str,
*,
lookback_days: int | None = None,
) -> dict[str, set[str]]:
"""近 N 天各榜已展示过的 skill id / repo不含当日供补榜去重
委托 board_history.load_recent_shown_keys只读 wecom_shown_keys
不读 movement_baseline。
"""
from daily.board_history import load_recent_shown_keys
from daily.config import board_dedup_days
days = lookback_days if lookback_days is not None else board_dedup_days()
return load_recent_shown_keys(date_str, lookback_days=days)
def find_previous_data(date_str: str) -> tuple[str, dict[str, Any]] | None:
"""查找最近一份早于 date_str 的 data.json。"""
try:
@@ -311,3 +337,40 @@ def build_movement_context(
"skills_stable": not skills_trending_all and not skills_hot_all,
"github_stable": not github_trending_all and not github_emerging_all and not github_topic_all,
}
def effective_wecom_mode(*, date_str: str, configured_mode: str | None = None) -> str:
mode = configured_mode or wecom_mode()
if mode != "delta":
return "full"
if find_previous_data(date_str) is None and delta_baseline_fallback() == "full":
return "full"
return "delta"
def partition_skill_moves_for_wecom(
trending_moves: list[dict[str, Any]],
hot_moves: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
hot_by_id = {skill_id(m): m for m in hot_moves if skill_id(m)}
trending_out: list[dict[str, Any]] = []
consumed_hot: set[str] = set()
for move in trending_moves:
sid = skill_id(move)
copy = dict(move)
badges = [f"Trending #{move.get('rank', '?')}"]
hot_match = hot_by_id.get(sid)
if hot_match:
badges.append(f"Hot #{hot_match.get('rank', '?')}")
consumed_hot.add(sid)
copy["badge"] = " · ".join(badges)
trending_out.append(copy)
hot_out: list[dict[str, Any]] = []
for move in hot_moves:
sid = skill_id(move)
if sid in consumed_hot:
continue
copy = dict(move)
copy["badge"] = f"Hot #{move.get('rank', '?')}"
hot_out.append(copy)
return trending_out, hot_out

545
daily/featured_pick.py Normal file
View File

@@ -0,0 +1,545 @@
"""今日首推:解析 DAILY_FEATURED_PICK → 定人(月去重)→ LLM 检索 → featured JSON。"""
from __future__ import annotations
import hashlib
import json
import logging
import random
import re
from datetime import datetime, timedelta
from typing import Any
from daily.config import OUTPUT_DIR, ROOT, env, featured_dedup_days
from daily.llm_client import extract_json_object, has_llm_configured, llm_chat
from daily.report_data import featured_json_path, save_json
logger = logging.getLogger(__name__)
_SKILL_DIR = ROOT / "skills" / "daily-featured-pick"
_GITHUB_REPO_RE = re.compile(r"github\.com/([^/\s#?]+/[^/\s#?]+)", re.I)
def featured_identity_key(featured: dict[str, Any] | None) -> str:
"""稳定身份skill→idgithub→repo兜底从 url 解析。"""
if not featured:
return ""
typ = str(featured.get("type") or "").lower()
if typ == "skill" or featured.get("id"):
sid = str(featured.get("id") or "").strip()
if sid:
return sid
repo = str(featured.get("repo") or "").strip()
if repo:
return repo
for field in ("url", "command", "link"):
url = str(featured.get(field) or "")
m = _GITHUB_REPO_RE.search(url)
if m:
return m.group(1)
return ""
def load_recent_featured_keys(date_str: str, days: int | None = None) -> set[str]:
"""近 N 日 data.featured_pick_key 并集(不含当日)。"""
lookback = days if days is not None else featured_dedup_days()
try:
dt = datetime.strptime(date_str, "%Y-%m-%d")
except ValueError:
return set()
out: set[str] = set()
for day_offset in range(1, lookback + 1):
prev = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
path = OUTPUT_DIR / f"{prev}.data.json"
if not path.exists():
continue
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
logger.warning("读取 featured_pick_key %s 失败:%s", path, exc)
continue
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
continue
key = str(data.get("featured_pick_key") or "").strip()
if not key:
featured = data.get("featured_pick")
if isinstance(featured, dict):
key = featured_identity_key(featured)
if key:
out.add(key)
return out
def load_yesterday_featured_key(date_str: str) -> str | None:
try:
dt = datetime.strptime(date_str, "%Y-%m-%d")
except ValueError:
return None
prev = (dt - timedelta(days=1)).strftime("%Y-%m-%d")
path = OUTPUT_DIR / f"{prev}.data.json"
if not path.exists():
return None
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return None
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
return None
key = str(data.get("featured_pick_key") or "").strip()
if key:
return key
featured = data.get("featured_pick")
if isinstance(featured, dict):
return featured_identity_key(featured) or None
return None
def _featured_rng(date_str: str) -> random.Random:
seed = int(hashlib.sha256(f"{date_str}:featured".encode()).hexdigest()[:16], 16)
return random.Random(seed)
def _stub_from_pool_item(item: dict[str, Any]) -> dict[str, Any]:
repo = str(item.get("repo") or "").strip()
if repo:
url = str(item.get("url") or f"https://github.com/{repo}").strip()
return {
"type": "github",
"title": repo.split("/")[-1],
"repo": repo,
"url": url,
"command": url,
"summary": str(item.get("description") or "")[:160],
"why_today": "",
"evidence": [],
"tags": [],
}
sid = str(item.get("id") or "").strip()
source = str(item.get("source") or "").strip()
title = str(item.get("title") or "").strip()
return {
"type": "skill",
"id": sid,
"title": title or sid,
"command": _skill_command(item),
"url": str(item.get("link") or ""),
"summary": str(item.get("description") or "")[:160],
"why_today": "",
"evidence": [],
"tags": [],
"source": source,
}
def featured_resolve(
*,
date_str: str,
candidate: dict[str, Any] | None,
pool_a: list[dict[str, Any]],
pool_b: list[dict[str, Any]],
recent_featured: set[str] | None = None,
yesterday_key: str | None = None,
rng: random.Random | None = None,
) -> tuple[dict[str, Any] | None, str | None]:
"""若与昨日同一身份则改推;返回 (seed_stub, identity_key),不含完整 why。"""
if not candidate:
return None, None
key = featured_identity_key(candidate)
if not yesterday_key or key != yesterday_key:
return candidate, key or None
blocked = set(recent_featured or set()) | {yesterday_key}
picker = rng or _featured_rng(date_str)
def _choices(pool: list[dict[str, Any]]) -> list[tuple[str, dict[str, Any]]]:
out: list[tuple[str, dict[str, Any]]] = []
seen: set[str] = set()
for item in pool:
ik = featured_identity_key(item)
if not ik or ik in blocked or ik in seen:
continue
seen.add(ik)
out.append((ik, item))
return out
options = _choices(pool_a)
if not options:
options = _choices(pool_b)
if not options:
logger.info("featured_fallback_exhausted")
return candidate, key
chosen_key, chosen_item = picker.choice(options)
return _stub_from_pool_item(chosen_item), chosen_key
def _config_from_candidate(candidate: dict[str, Any]) -> dict[str, str]:
typ = str(candidate.get("type") or "").lower()
if typ == "skill" or candidate.get("id"):
query = str(candidate.get("id") or candidate.get("title") or "").strip()
return {"query": query, "url_hint": str(candidate.get("url") or "")}
repo = str(candidate.get("repo") or "").strip()
if repo:
return {
"query": repo,
"url_hint": str(candidate.get("url") or f"https://github.com/{repo}"),
}
query = str(candidate.get("title") or candidate.get("url") or "").strip()
return {"query": query or "featured", "url_hint": str(candidate.get("url") or "")}
def _seed_candidate_from_config(
config: dict[str, str],
llm_input: dict[str, Any],
) -> dict[str, Any]:
matches = match_in_data(llm_input, config["query"])
if matches["skills"]:
return _stub_from_pool_item(matches["skills"][0])
if matches["github"]:
return _stub_from_pool_item(matches["github"][0])
url = config.get("url_hint") or ""
seed: dict[str, Any] = {
"type": "other",
"title": config["query"],
"url": url,
"command": url or config["query"],
}
m = _GITHUB_REPO_RE.search(url)
if m:
seed["type"] = "github"
seed["repo"] = m.group(1)
return seed
def parse_featured_pick() -> dict[str, str] | None:
"""解析 DAILY_FEATURED_PICKquery 或 query|url。"""
raw = (env("DAILY_FEATURED_PICK") or "").strip()
if not raw:
return None
if "|" in raw:
query, url_hint = raw.split("|", 1)
query = query.strip()
url_hint = url_hint.strip()
if not query:
return None
payload: dict[str, str] = {"query": query}
if url_hint:
payload["url_hint"] = url_hint
return payload
return {"query": raw}
def _matches_query(text: str, query: str) -> bool:
return query.lower() in (text or "").lower()
def _skill_matches(item: dict[str, Any], query: str) -> bool:
for key in ("id", "title", "source"):
if _matches_query(str(item.get(key) or ""), query):
return True
for sub in item.get("cluster_skills") or []:
if isinstance(sub, str) and _matches_query(sub, query):
return True
return False
def match_in_data(llm_input: dict[str, Any], query: str) -> dict[str, list[dict[str, Any]]]:
"""在榜单数据中模糊匹配 query。"""
skills: list[dict[str, Any]] = []
seen_skill: set[str] = set()
for board in ("skills_trending", "skills_hot"):
for item in llm_input.get(board) or []:
sid = str(item.get("id") or "")
if sid in seen_skill:
continue
if _skill_matches(item, query):
seen_skill.add(sid)
skills.append({**item, "board": board})
if len(skills) >= 5:
break
if len(skills) >= 5:
break
github: list[dict[str, Any]] = []
seen_repo: set[str] = set()
for board in ("github_trending", "github_emerging"):
for item in llm_input.get(board) or []:
repo = str(item.get("repo") or "")
if not repo or repo in seen_repo:
continue
if _matches_query(repo, query):
seen_repo.add(repo)
github.append({**item, "board": board})
if len(github) >= 5:
break
topic = llm_input.get("github_topic") or {}
for item in topic.get("repos") or []:
repo = str(item.get("repo") or "")
if not repo or repo in seen_repo:
continue
if _matches_query(repo, query):
seen_repo.add(repo)
github.append({**item, "board": "github_topic"})
if len(github) >= 5:
break
return {"skills": skills, "github": github}
def _load_skill() -> str:
path = _SKILL_DIR / "SKILL.md"
if path.exists():
return path.read_text(encoding="utf-8").strip()
return "你是早报编辑。根据输入检索今日首推信息,只输出 JSON。"
def _skill_command(item: dict[str, Any]) -> str:
source = str(item.get("source") or "").strip()
title = str(item.get("title") or "").strip()
if source and title:
return f"npx skills add {source}/{title}"
sid = str(item.get("id") or "").strip()
if sid.count("/") >= 2:
parts = sid.split("/", 2)
return f"npx skills add {parts[0]}/{parts[1]}/{parts[2]}"
if sid.count("/") == 1:
return f"npx skills add {sid}"
return ""
def _evidence_from_skill(item: dict[str, Any]) -> list[str]:
board = item.get("board", "")
board_label = {
"skills_trending": "Skills Trending",
"skills_hot": "Skills Hot",
}.get(str(board), str(board))
installs = item.get("installs_fmt") or item.get("installs")
title = item.get("title") or item.get("id") or "?"
if installs:
return [f"{board_label} 匹配 · {title} · {installs}"]
return [f"{board_label} 匹配 · {title}"]
def _evidence_from_github(item: dict[str, Any]) -> list[str]:
board = item.get("board", "")
board_label = {
"github_trending": "GitHub Trending",
"github_emerging": "GitHub 新兴",
"github_topic": "GitHub Topic",
}.get(str(board), str(board))
repo = item.get("repo") or "?"
stars = item.get("total_stars_fmt") or ""
if stars:
return [f"{board_label} 匹配 · {repo} · ⭐{stars}"]
return [f"{board_label} 匹配 · {repo}"]
def _fallback_featured(
config: dict[str, str],
llm_input: dict[str, Any],
*,
partial: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""LLM 不可用或失败时,用榜单匹配 + 配置回退。"""
partial = partial or {}
matches = match_in_data(llm_input, config["query"])
skill = matches["skills"][0] if matches["skills"] else None
gh = matches["github"][0] if matches["github"] else None
if skill:
featured: dict[str, Any] = {
"title": str(skill.get("title") or config["query"]),
"type": "skill",
"command": _skill_command(skill),
"url": str(skill.get("link") or config.get("url_hint") or ""),
"summary": str(partial.get("summary") or skill.get("description") or "")[:160],
"why_today": str(
partial.get("why_today")
or f"今日 Skills 榜匹配到 **{skill.get('title') or config['query']}**,适合作为首推。"
),
"evidence": list(partial.get("evidence") or _evidence_from_skill(skill)),
"tags": list(partial.get("tags") or []),
}
if skill.get("id"):
featured["id"] = skill["id"]
return featured
if gh:
return {
"title": str(gh.get("repo") or config["query"]).split("/")[-1],
"type": "github",
"command": str(gh.get("url") or config.get("url_hint") or ""),
"url": str(gh.get("url") or config.get("url_hint") or ""),
"summary": str(partial.get("summary") or gh.get("description") or "")[:160],
"why_today": str(
partial.get("why_today")
or f"今日 GitHub 榜匹配到 **{gh.get('repo')}**,适合作为首推。"
),
"evidence": list(partial.get("evidence") or _evidence_from_github(gh)),
"tags": list(partial.get("tags") or []),
"repo": gh.get("repo"),
}
url = config.get("url_hint") or ""
return {
"title": config["query"],
"type": "other",
"command": url or config["query"],
"url": url,
"summary": str(partial.get("summary") or "")[:160],
"why_today": str(
partial.get("why_today")
or f"**{config['query']}** 未出现在今日 Top 榜,仍值得单独关注。"
),
"evidence": list(partial.get("evidence") or ([f"主推 · {config['query']}"])),
"tags": list(partial.get("tags") or []),
}
def _normalize_featured(
raw: dict[str, Any],
config: dict[str, str],
llm_input: dict[str, Any],
) -> dict[str, Any]:
"""补齐 command / url / evidence并与榜单数字对齐。"""
matches = match_in_data(llm_input, config["query"])
skill = matches["skills"][0] if matches["skills"] else None
gh = matches["github"][0] if matches["github"] else None
featured = dict(raw)
featured.setdefault("title", config["query"])
featured.setdefault("type", "other")
if skill and featured.get("type") in {"skill", "other", ""}:
featured.setdefault("id", skill.get("id"))
featured.setdefault("command", _skill_command(skill))
featured.setdefault("url", skill.get("link") or config.get("url_hint") or "")
if not featured.get("evidence"):
featured["evidence"] = _evidence_from_skill(skill)
featured["type"] = "skill"
elif gh and featured.get("type") in {"github", "other", ""}:
featured.setdefault("repo", gh.get("repo"))
featured.setdefault("url", gh.get("url") or config.get("url_hint") or "")
featured.setdefault("command", featured.get("url") or gh.get("url") or "")
if not featured.get("evidence"):
featured["evidence"] = _evidence_from_github(gh)
featured["type"] = "github"
featured.setdefault("command", config.get("url_hint") or config["query"])
featured.setdefault("url", config.get("url_hint") or "")
featured.setdefault("summary", "")
featured.setdefault("why_today", featured.get("summary") or "")
featured.setdefault("evidence", [])
featured.setdefault("tags", [])
return featured
def research_featured_pick(
llm_input: dict[str, Any],
*,
date_str: str,
config: dict[str, str] | None = None,
) -> dict[str, Any] | None:
"""Step 0检索今日首推成功返回 featured dict未配置返回 None。"""
config = config or parse_featured_pick()
if not config:
return None
matches = match_in_data(llm_input, config["query"])
payload = {
"query": config["query"],
"url_hint": config.get("url_hint"),
"cwd": env("DAILY_CURSOR_CWD") or str(ROOT),
"data_matches": matches,
}
if not has_llm_configured():
featured = _fallback_featured(config, llm_input)
save_json(featured_json_path(date_str), featured)
logger.info("Featured pick无 LLM规则回退%s", featured.get("title"))
return featured
skill = _load_skill()
system = (
f"{skill}\n\n"
"当前执行 **Step 0今日首推检索**。\n"
"只输出 featured JSONtitle, type, command, url, summary, why_today, evidence, tags"
"不要 Markdown不要解释。"
)
user = json.dumps(payload, ensure_ascii=False, indent=2)
try:
raw = llm_chat(system, user)
except Exception as exc:
logger.warning("Featured pick LLM 失败,回退规则模式:%s", exc)
featured = _fallback_featured(config, llm_input)
save_json(featured_json_path(date_str), featured)
return featured
if not raw:
featured = _fallback_featured(config, llm_input)
save_json(featured_json_path(date_str), featured)
return featured
parsed = extract_json_object(raw)
if not parsed.get("why_today") and not parsed.get("summary"):
logger.warning("Featured pick JSON 无效,回退规则模式")
featured = _fallback_featured(config, llm_input, partial=parsed)
save_json(featured_json_path(date_str), featured)
return featured
featured = _normalize_featured(parsed, config, llm_input)
save_json(featured_json_path(date_str), featured)
logger.info("Featured pick 完成:%s", featured.get("title"))
return featured
def apply_featured_pick(
llm_input: dict[str, Any],
*,
date_str: str,
pool_a: list[dict[str, Any]] | None = None,
pool_b: list[dict[str, Any]] | None = None,
) -> dict[str, Any] | None:
"""先定人(相对昨日改推 + 月去重),再 research写入 featured_pick / featured_pick_key。"""
config = parse_featured_pick()
if not config:
return None
seed = _seed_candidate_from_config(config, llm_input)
recent = load_recent_featured_keys(date_str)
yesterday = load_yesterday_featured_key(date_str)
resolved, identity_key = featured_resolve(
date_str=date_str,
candidate=seed,
pool_a=pool_a or [],
pool_b=pool_b or [],
recent_featured=recent,
yesterday_key=yesterday,
rng=_featured_rng(date_str),
)
research_config = config
if resolved and identity_key and featured_identity_key(seed) != identity_key:
research_config = _config_from_candidate(resolved)
featured = research_featured_pick(
llm_input, date_str=date_str, config=research_config
)
if featured:
llm_input["featured_pick"] = featured
key = identity_key or featured_identity_key(featured)
if key:
llm_input["featured_pick_key"] = key
return featured
def pick_command_from_featured(featured: dict[str, Any] | None) -> str | None:
cmd = str((featured or {}).get("command") or "").strip()
return cmd or None
def pick_why_from_featured(featured: dict[str, Any] | None) -> str | None:
why = str((featured or {}).get("why_today") or "").strip()
return why or None

View File

@@ -5,8 +5,9 @@ from __future__ import annotations
import re
from typing import Any
from daily.config import wecom_skill_desc_limit
from daily.config import env_bool, wecom_skill_desc_limit
from daily.localize import LocalizeJob, localize_brief_descriptions, needs_chinese
from daily.skills_group import group_skills_by_source
from daily.text_utils import trim_brief
ICONS = {
@@ -42,7 +43,7 @@ def _skill_line(rank: int, item: dict[str, Any], *, badge: str = "") -> list[str
lines = [head]
if sample or desc:
hint = desc or sample
lines.append(f" > {hint}")
lines.append(f" {hint}")
return lines
title = item.get("title", "?")
if link:
@@ -51,29 +52,181 @@ def _skill_line(rank: int, item: dict[str, Any], *, badge: str = "") -> list[str
head = f"{rank}. {badge_prefix}**{title}** · `{source}` · **{installs}**"
lines = [head]
if desc:
lines.append(f" > {desc}")
lines.append(f" {desc}")
return lines
def _ai_news_lines(items: list[dict[str, Any]]) -> list[str]:
def _ai_news_link_label(item: dict[str, Any], *, merged: bool = False) -> str:
title = (item.get("title") or "?").strip() or "?"
source = (item.get("source_name") or "").strip()
if merged and source:
return f"{source} - {title}"
desc = (item.get("desc_short") or "").strip()
if desc:
return desc
return title
def _ai_news_lines(items: list[dict[str, Any]], *, merged: bool = False) -> list[str]:
lines: list[str] = []
for i, item in enumerate(items, 1):
title = item.get("title", "?")
label = _ai_news_link_label(item, merged=merged)
link = item.get("link", "")
source = item.get("source_name", "?")
pub = item.get("published_fmt", "")
desc = item.get("desc_short", "")
pub_suffix = f" · {pub}" if pub else ""
desc = (item.get("desc_short") or "").strip()
pub_suffix = "" if merged else (f" · {pub}" if pub else "")
if link:
head = f"{i}. [**{title}**]({link}) · `{source}`{pub_suffix}"
if merged and desc:
head = f"{i}. [{label}]({link}) — {desc}{pub_suffix}"
elif merged:
head = f"{i}. [{label}]({link}){pub_suffix}"
else:
head = f"{i}. **{title}** · `{source}`{pub_suffix}"
head = f"{i}. [{label}]({link}) · `{source}`{pub_suffix}"
elif merged and desc:
head = f"{i}. {label}{desc}{pub_suffix}"
else:
head = f"{i}. {label} · `{source}`{pub_suffix}"
lines.append(head)
if desc:
lines.append(f" > {desc}")
return lines
_NEWS_BLOCK_END = re.compile(
r"\n\n(📈|🔥|🐙|🌱|🤖|📦|🎯|💡|🌍|🇨🇳|📰|🔧)",
)
def _build_merged_news_block(
items: list[dict[str, Any]],
tech_items: list[dict[str, Any]] | None = None,
) -> str:
combined = list(items or [])
if tech_items:
combined.extend(tech_items)
if not combined:
return ""
parts = [f"📰 **AI 时讯精选 Top {len(combined)}**"]
parts.extend(_ai_news_lines(combined, merged=True))
return "\n".join(parts) + "\n"
def _replace_merged_news_block(
md: str,
items: list[dict[str, Any]],
tech_items: list[dict[str, Any]] | None = None,
) -> str:
if not items and not tech_items:
return md
block = _build_merged_news_block(items, tech_items)
start_pat = re.compile(
r"^📰 \*\*AI 时讯精选[^\n]*\*\*\s*$",
re.MULTILINE,
)
match = start_pat.search(md)
if not match:
anchor = re.search(r"^(💡|🎯).*$", md, re.MULTILINE)
if anchor:
insert_at = anchor.end()
return md[:insert_at] + "\n\n" + block + md[insert_at:].lstrip("\n")
anchor2 = re.search(r"^(🌍|🇨🇳|📈|🔧).*$", md, re.MULTILINE)
if anchor2:
insert_at = anchor2.start()
return md[:insert_at] + block + md[insert_at:].lstrip("\n")
return md.rstrip() + "\n\n" + block
start = match.start()
tail = md[match.end() :]
end_rel = _NEWS_BLOCK_END.search(tail)
end = match.end() + (end_rel.start() if end_rel else len(tail))
return md[:start] + block + md[end:].lstrip("\n")
def _remove_news_blocks(md: str, icons: tuple[str, ...], *, label_must_contain: str = "") -> str:
out = md
for icon in icons:
start_pat = re.compile(rf"^{re.escape(icon)} \*\*[^\n]+\*\*\s*$", re.MULTILINE)
while True:
match = None
for candidate in start_pat.finditer(out):
if label_must_contain and label_must_contain not in candidate.group(0):
continue
match = candidate
break
if not match:
break
start = match.start()
tail = out[match.end() :]
end_rel = _NEWS_BLOCK_END.search(tail)
end = match.end() + (end_rel.start() if end_rel else len(tail))
out = out[:start] + out[end:].lstrip("\n")
return out
def _replace_news_block(
md: str,
icon: str,
items: list[dict[str, Any]],
label: str,
*,
merged: bool = False,
) -> str:
if not items:
return md
if merged:
start_pat = re.compile(
rf"^{re.escape(icon)} \*\*{re.escape(label)}[^\n]*\*\*\s*$",
re.MULTILINE,
)
else:
start_pat = re.compile(rf"^{re.escape(icon)} \*\*[^\n]+\*\*\s*$", re.MULTILINE)
match = start_pat.search(md)
block = (
f"{icon} **{label} Top {len(items)}**\n"
+ "\n".join(_ai_news_lines(items, merged=merged))
+ "\n"
)
if not match:
anchor = re.search(r"^(💡|🎯).*$", md, re.MULTILINE)
if anchor:
insert_at = anchor.end()
return md[:insert_at] + "\n\n" + block + md[insert_at:].lstrip("\n")
# 插在 🌍/🇨🇳 原位置,或 Skills 区块前
anchor2 = re.search(r"^(🌍|🇨🇳|📈).*$", md, re.MULTILINE)
if anchor2:
insert_at = anchor2.start()
return md[:insert_at] + block + md[insert_at:].lstrip("\n")
return md.rstrip() + "\n\n" + block
start = match.start()
tail = md[match.end() :]
end_rel = _NEWS_BLOCK_END.search(tail)
end = match.end() + (end_rel.start() if end_rel else len(tail))
return md[:start] + block + md[end:].lstrip("\n")
def replace_wecom_news_sections(
md: str,
*,
ai_news: list[dict[str, Any]] | None = None,
cn_ai_news: list[dict[str, Any]] | None = None,
tech_ai_news: list[dict[str, Any]] | None = None,
merged: bool = False,
) -> str:
"""用 Python 整理后的新闻列表替换 Agent/模板中的时讯区块。"""
if merged and (ai_news or tech_ai_news):
out = _remove_news_blocks(md, ("🌍", "🇨🇳"))
out = _remove_news_blocks(out, ("📰",), label_must_contain="AI 时讯精选")
out = _remove_news_blocks(out, ("🔧",), label_must_contain="技术类时讯")
return _replace_merged_news_block(out, ai_news or [], tech_ai_news)
out = md
if cn_ai_news:
out = _replace_news_block(out, "🇨🇳", cn_ai_news, "国内 AI 时讯")
if ai_news:
out = _replace_news_block(out, "🌍", ai_news, "国际 AI 时讯")
return out
WECOM_GITHUB_DESC_LIMIT = 40
def _github_repo_lines(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
lines: list[str] = []
for i, repo in enumerate(repos, 1):
@@ -94,12 +247,45 @@ def _github_repo_lines(repos: list[dict[str, Any]], *, show_created: bool = Fals
meta_parts.append(f"创建于 {created}")
meta = f" · {' · '.join(meta_parts)}" if meta_parts else ""
lines.append(f"{i}. [{name}]({url}){meta}")
desc = repo.get("desc_short") or repo.get("description", "")
desc = repo.get("wecom_desc") or repo.get("desc_short") or repo.get("description", "")
if desc:
lines.append(f" > {desc}")
lines.append(f" {desc}")
return lines
def finalize_wecom_github_repos(
items: list[dict[str, Any]],
*,
desc_limit: int | None = None,
) -> list[dict[str, Any]]:
"""为企微 GitHub 条目生成简短中文简介。"""
if desc_limit is None:
desc_limit = WECOM_GITHUB_DESC_LIMIT
limit = desc_limit if desc_limit > 0 else 40
copies: list[tuple[str, dict[str, Any]]] = []
jobs: list[LocalizeJob] = []
for item in items:
copy = dict(item)
desc = (copy.get("description") or copy.get("desc_short") or "").strip()
key = f"github:{copy.get('repo', '?')}"
if needs_chinese(desc) or len(desc) > limit:
jobs.append(LocalizeJob(key, desc, limit))
else:
copy["wecom_desc"] = desc
copies.append((key, copy))
zh_map = localize_brief_descriptions(jobs, archive=True)
out: list[dict[str, Any]] = []
for key, copy in copies:
if key in zh_map:
copy["wecom_desc"] = zh_map[key]
elif "wecom_desc" not in copy:
fallback = (copy.get("description") or copy.get("desc_short") or "").strip()
copy["wecom_desc"] = _brief_fallback_desc(fallback, limit) if fallback else ""
out.append(copy)
return out
def _fallback_skill_desc(item: dict[str, Any]) -> str:
if item.get("cluster"):
count = int(item.get("cluster_count") or 1)
@@ -173,11 +359,13 @@ def _grouped_skill_to_wecom_item(
if not item.get("wecom_desc"):
desc = _brief_fallback_desc(desc, limit)
return {
"id": str(item.get("id") or f"{item.get('source', '?')}/{item.get('title', '')}"),
"title": item.get("title", ""),
"source": item.get("source", "?"),
"installs_fmt": installs_fmt,
"link": item.get("link", ""),
"desc_short": desc,
"badge": item.get("badge", ""),
"cluster": bool(item.get("cluster")),
"cluster_count": item.get("cluster_count"),
"cluster_titles": item.get("cluster_titles"),
@@ -193,19 +381,560 @@ def build_skills_board_section(icon_key: str, board_label: str, items: list[dict
return "\n".join(lines)
def _move_to_wecom_skill_item(move: dict[str, Any]) -> dict[str, Any]:
installs = int(move.get("installs") or 0)
return {
"title": move.get("title", "?"),
"source": move.get("source", "?"),
"installs_fmt": move.get("installs_fmt") or str(installs),
"link": move.get("link", ""),
"description": (move.get("description") or "").strip(),
}
def _move_to_skill_row(move: dict[str, Any]) -> dict[str, Any]:
title = str(move.get("title") or "?")
source = str(move.get("source") or "?")
installs = int(move.get("installs") or 0)
sid = str(move.get("id") or f"{source}/{title}")
return {
"id": sid,
"title": title,
"source": source,
"installs": installs,
"link": move.get("link", ""),
"description": (move.get("description") or "").strip(),
}
def _flatten_skill_board_item(item: dict[str, Any]) -> list[dict[str, Any]]:
if item.get("cluster"):
source = str(item.get("source") or "?")
installs = int(item.get("installs") or 0)
titles = [str(t) for t in (item.get("cluster_skills") or []) if t]
if not titles:
titles = [str(item.get("title") or "?")]
top_title = str(item.get("title") or titles[0])
rows: list[dict[str, Any]] = []
for title in titles:
link = item.get("link", "")
if title != top_title:
link = f"https://www.skills.sh/{source}/{title}"
rows.append(
{
"id": f"{source}/{title}",
"title": title,
"source": source,
"installs": installs,
"link": link,
"description": (item.get("description") or "").strip(),
}
)
return rows
title = str(item.get("title") or "?")
source = str(item.get("source") or "?")
return [
{
"id": str(item.get("id") or f"{source}/{title}"),
"title": title,
"source": source,
"installs": int(item.get("installs") or 0),
"link": item.get("link", ""),
"description": (item.get("description") or "").strip(),
}
]
def _skill_keys_in_board_item(item: dict[str, Any]) -> set[str]:
if item.get("cluster"):
source = str(item.get("source") or "?")
titles = item.get("cluster_skills") or [item.get("title", "")]
return {f"{source}/{t}" for t in titles if t}
return {_skill_group_key(item)}
def _prepare_grouped_wecom_skills(
flat_rows: list[dict[str, Any]],
*,
limit: int,
) -> tuple[list[dict[str, Any]], set[str]]:
if not flat_rows:
return [], set()
grouped = group_skills_by_source(flat_rows, limit=limit, pool_size=max(len(flat_rows), limit))
prepared = finalize_wecom_skill_groups(grouped)
wecom_items = [_grouped_skill_to_wecom_item(x) for x in prepared]
keys: set[str] = set()
for item in grouped:
keys.update(_skill_keys_in_board_item(item))
return wecom_items[:limit], keys
def _normalize_skill_source_groups(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""将条目规范为按 source 合并的榜单项(展示始终为合并态)。"""
flat: list[dict[str, Any]] = []
for item in items:
flat.extend(_flatten_skill_board_item(item))
if not flat:
return []
return group_skills_by_source(flat, limit=len(flat), pool_size=len(flat))
def _skill_primary_id(item: dict[str, Any]) -> str:
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}" or "").strip()
def _source_from_skill_key(key: str) -> str:
from daily.skills_group import source_from_skill_key
return source_from_skill_key(key)
def expand_skill_recent_keys(keys: set[str] | None) -> set[str]:
from daily.skills_group import expand_skill_recent_keys as _expand
return _expand(keys)
def _merge_skill_board_items(
moves: list[dict[str, Any]],
full_items: list[dict[str, Any]],
limit: int,
*,
exclude_keys: set[str] | None = None,
recent_keys: set[str] | None = None,
) -> tuple[list[dict[str, Any]], set[str]]:
"""异动优先,不足时用深池补满;按 source 合并态取条。
周去重按 source含从 skill id 展开);同日避开其它榜时也按 source。
"""
from daily.delta import skill_id as move_skill_id
exclude = exclude_keys or set()
recent = expand_skill_recent_keys(recent_keys)
exclude_sources = {_source_from_skill_key(k) for k in exclude if k}
exclude_sources.update(k for k in exclude if k)
def _blocked(item: dict[str, Any]) -> bool:
primary = _skill_primary_id(item)
source = str(item.get("source") or "").strip()
if primary and primary in recent:
return True
if source and source in recent:
return True
if primary and primary in exclude:
return True
if source and (source in exclude_sources or source in exclude):
return True
return False
groups: list[dict[str, Any]] = []
seen_sources: set[str] = set()
move_rows: list[dict[str, Any]] = []
seen_move_ids: set[str] = set()
for move in moves:
key = move_skill_id(move)
if not key or key in seen_move_ids:
continue
move_source = str(move.get("source") or "").strip()
if key in exclude or (move_source and move_source in exclude_sources):
continue
if key in recent or (move_source and move_source in recent):
continue
seen_move_ids.add(key)
move_rows.append(_move_to_skill_row(move))
for group in _normalize_skill_source_groups(move_rows):
source = str(group.get("source") or "?")
if source in seen_sources or _blocked(group):
continue
seen_sources.add(source)
groups.append(group)
if len(groups) >= limit:
break
if len(groups) < limit:
for group in _normalize_skill_source_groups(full_items):
if len(groups) >= limit:
break
source = str(group.get("source") or "?")
if source in seen_sources or _blocked(group):
continue
seen_sources.add(source)
groups.append(group)
if not groups:
return [], set()
prepared = finalize_wecom_skill_groups(groups[:limit])
wecom_items = [_grouped_skill_to_wecom_item(x) for x in prepared]
# 供同日 Hot 排除:主键 + source
keys: set[str] = set()
for item in groups[:limit]:
primary = _skill_primary_id(item)
if primary:
keys.add(primary)
source = str(item.get("source") or "").strip()
if source:
keys.add(source)
return wecom_items[:limit], keys
def build_skills_delta_sections(
trending_moves: list[dict[str, Any]],
hot_moves: list[dict[str, Any]],
*,
trending_full: list[dict[str, Any]] | None = None,
hot_full: list[dict[str, Any]] | None = None,
trending_limit: int = 5,
hot_limit: int = 5,
pad: bool = False,
recent_trending: set[str] | None = None,
recent_hot: set[str] | None = None,
) -> str:
sections: list[str] = []
trending_keys: set[str] = set()
if pad:
skill_recent = expand_skill_recent_keys(
(recent_trending or set()) | (recent_hot or set())
)
t_items, trending_keys = _merge_skill_board_items(
trending_moves,
trending_full or [],
trending_limit,
recent_keys=skill_recent,
)
if t_items:
lines = [f"{ICONS['trending']} **Skills Trending Top {len(t_items)}**"]
for rank, item in enumerate(t_items, 1):
lines.extend(_skill_line(rank, item))
sections.append("\n".join(lines))
h_items, _ = _merge_skill_board_items(
hot_moves,
hot_full or [],
hot_limit,
exclude_keys=trending_keys,
recent_keys=skill_recent,
)
if h_items:
lines = [f"{ICONS['hot']} **Skills Hot Top {len(h_items)}**"]
for rank, item in enumerate(h_items, 1):
lines.extend(_skill_line(rank, item))
sections.append("\n".join(lines))
return "\n\n".join(sections)
if trending_moves:
flat = [_move_to_skill_row(m) for m in trending_moves]
items, _ = _prepare_grouped_wecom_skills(flat, limit=len(flat))
lines = [f"{ICONS['trending']} **Skills Trending 变化**"]
for rank, item in enumerate(items, 1):
lines.extend(_skill_line(rank, item))
sections.append("\n".join(lines))
if hot_moves:
flat = [_move_to_skill_row(m) for m in hot_moves]
items, _ = _prepare_grouped_wecom_skills(flat, limit=len(flat))
lines = [f"{ICONS['hot']} **Skills Hot 变化**"]
for rank, item in enumerate(items, 1):
lines.extend(_skill_line(rank, item))
sections.append("\n".join(lines))
return "\n\n".join(sections)
def _github_move_to_repo(move: dict[str, Any]) -> dict[str, Any]:
return {
"repo": move.get("repo", "?"),
"url": move.get("url", ""),
"language": move.get("language", ""),
"stars_today_fmt": move.get("stars_today_fmt", ""),
"total_stars_fmt": move.get("total_stars_fmt", ""),
"created_at": move.get("created_at", ""),
"description": move.get("description", ""),
"desc_short": (move.get("description") or "").strip(),
}
def _merge_github_board_items(
moves: list[dict[str, Any]],
full_repos: list[dict[str, Any]],
limit: int,
*,
recent_repos: set[str] | None = None,
) -> list[dict[str, Any]]:
recent = recent_repos or set()
seen: set[str] = set()
merged: list[dict[str, Any]] = []
for move in moves:
repo = _github_move_to_repo(move)
key = str(repo.get("repo") or "")
if not key or key in seen or key in recent:
continue
seen.add(key)
merged.append(repo)
for repo in full_repos:
if len(merged) >= limit:
break
key = str(repo.get("repo") or "")
if not key or key in seen or key in recent:
continue
seen.add(key)
merged.append(repo)
return finalize_wecom_github_repos(merged)[:limit]
def build_github_delta_sections(
movement: dict[str, Any],
*,
topic_name: str,
github_trending: list[dict[str, Any]] | None = None,
github_emerging: list[dict[str, Any]] | None = None,
github_topic: list[dict[str, Any]] | None = None,
trending_limit: int = 5,
emerging_limit: int = 5,
topic_limit: int = 5,
pad: bool = False,
recent_board_keys: dict[str, set[str]] | None = None,
) -> str:
sections: list[str] = []
recent = recent_board_keys or {}
if pad:
github_recent = (
(recent.get("github_trending") or set())
| (recent.get("github_emerging") or set())
| (recent.get("github_topic") or set())
)
mapping = [
("github_trending_moves", "github_trending", github_trending or [], trending_limit, "github", "GitHub Trending", False),
("github_emerging_moves", "github_emerging", github_emerging or [], emerging_limit, "emerging", "GitHub 新兴", True),
("github_topic_moves", "github_topic", github_topic or [], topic_limit, "topic", f"Topic `{topic_name}`", False),
]
for move_key, board_key, full_repos, limit, icon_key, label, show_created in mapping:
repos = _merge_github_board_items(
movement.get(move_key) or [],
full_repos,
limit,
recent_repos=github_recent,
)
github_recent |= {str(r.get("repo") or "") for r in repos if r.get("repo")}
if not repos:
continue
lines = [f"{ICONS[icon_key]} **{label} Top {len(repos)}**"]
lines.extend(_github_repo_lines(repos, show_created=show_created))
sections.append("\n".join(lines))
return "\n\n".join(sections)
mapping = [
("github_trending_moves", "github", "GitHub Trending 变化", False),
("github_emerging_moves", "emerging", "GitHub 新兴 变化", True),
("github_topic_moves", "topic", f"Topic `{topic_name}` 变化", False),
]
for key, icon_key, label, show_created in mapping:
moves = movement.get(key) or []
if not moves:
continue
lines = [f"{ICONS[icon_key]} **{label}**"]
repos = finalize_wecom_github_repos([_github_move_to_repo(m) for m in moves])
lines.extend(_github_repo_lines(repos, show_created=show_created))
sections.append("\n".join(lines))
return "\n\n".join(sections)
_SKILL_SECTIONS = re.compile(
r"📈 \*\*Skills Trending.*?(?=🐙 \*\*GitHub Trending)",
re.DOTALL,
)
_SKILL_TRENDING_BLOCK = re.compile(r"📈 \*\*Skills Trending[^\n]*\n(?:.*?\n)*?(?=\n🔥 \*\*Skills Hot|\n🐙 |\n🌱 |\n🤖 |\Z)", re.DOTALL)
_SKILL_HOT_BLOCK = re.compile(r"🔥 \*\*Skills Hot[^\n]*\n(?:.*?\n)*?(?=\n🐙 |\n🌱 |\n🤖 |\Z)", re.DOTALL)
_GITHUB_SECTIONS = re.compile(r"🐙 \*\*GitHub Trending.*", re.DOTALL)
def replace_wecom_skill_sections(
md: str,
def _strip_board_sections(md: str) -> str:
md = _SKILL_TRENDING_BLOCK.sub("", md)
md = _SKILL_HOT_BLOCK.sub("", md)
if _GITHUB_SECTIONS.search(md):
md = _GITHUB_SECTIONS.sub("", md)
return re.sub(r"\n{3,}", "\n\n", md).rstrip()
def resolve_wecom_board_items(
*,
mode: str,
movement: dict[str, Any],
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
topic_name: str,
github_trending: list[dict[str, Any]] | None = None,
github_emerging: list[dict[str, Any]] | None = None,
github_topic: list[dict[str, Any]] | None = None,
wecom_trending: int = 5,
wecom_hot: int = 5,
wecom_github: int = 5,
wecom_emerging: int = 5,
wecom_topic: int = 5,
pad: bool = False,
date_str: str | None = None,
trending_pad: list[dict[str, Any]] | None = None,
hot_pad: list[dict[str, Any]] | None = None,
github_trending_pad: list[dict[str, Any]] | None = None,
github_emerging_pad: list[dict[str, Any]] | None = None,
github_topic_pad: list[dict[str, Any]] | None = None,
) -> dict[str, list[dict[str, Any]]]:
"""返回最终企微正文各榜 items与 replace_wecom_board_sections 同源),供写回 shown。"""
from daily.delta import load_recent_board_keys, partition_skill_moves_for_wecom
if mode != "delta":
return {
"skills_trending": list(trending),
"skills_hot": list(hot),
"github_trending": list(github_trending or []),
"github_emerging": list(github_emerging or []),
"github_topic": list(github_topic or []),
}
recent_board_keys: dict[str, set[str]] = {}
skill_recent: set[str] = set()
github_recent: set[str] = set()
if pad and date_str:
recent_board_keys = load_recent_board_keys(date_str)
# Trending / Hot 共用周去重:任一类出现过的 source 两边都不再展示
skill_recent = expand_skill_recent_keys(
(recent_board_keys.get("skills_trending") or set())
| (recent_board_keys.get("skills_hot") or set())
)
# GitHub 三榜共用周去重:任一类出现过的 repo 各榜都不再展示
github_recent = (
(recent_board_keys.get("github_trending") or set())
| (recent_board_keys.get("github_emerging") or set())
| (recent_board_keys.get("github_topic") or set())
)
t_moves, h_moves = partition_skill_moves_for_wecom(
movement.get("skills_trending_moves") or [],
movement.get("skills_hot_moves") or [],
)
if pad:
t_items, trending_keys = _merge_skill_board_items(
t_moves,
trending_pad if trending_pad else trending,
wecom_trending,
recent_keys=skill_recent,
)
h_items, _ = _merge_skill_board_items(
h_moves,
hot_pad if hot_pad else hot,
wecom_hot,
exclude_keys=trending_keys,
recent_keys=skill_recent,
)
gt_items = _merge_github_board_items(
movement.get("github_trending_moves") or [],
github_trending_pad if github_trending_pad else (github_trending or []),
wecom_github,
recent_repos=github_recent,
)
github_recent |= {str(r.get("repo") or "") for r in gt_items if r.get("repo")}
ge_items = _merge_github_board_items(
movement.get("github_emerging_moves") or [],
github_emerging_pad if github_emerging_pad else (github_emerging or []),
wecom_emerging,
recent_repos=github_recent,
)
github_recent |= {str(r.get("repo") or "") for r in ge_items if r.get("repo")}
gtopic_items = _merge_github_board_items(
movement.get("github_topic_moves") or [],
github_topic_pad if github_topic_pad else (github_topic or []),
wecom_topic,
recent_repos=github_recent,
)
return {
"skills_trending": t_items,
"skills_hot": h_items,
"github_trending": gt_items,
"github_emerging": ge_items,
"github_topic": gtopic_items,
}
t_flat = [_move_to_skill_row(m) for m in t_moves]
h_flat = [_move_to_skill_row(m) for m in h_moves]
t_items, _ = _prepare_grouped_wecom_skills(t_flat, limit=len(t_flat) or 1) if t_flat else ([], set())
h_items, _ = _prepare_grouped_wecom_skills(h_flat, limit=len(h_flat) or 1) if h_flat else ([], set())
return {
"skills_trending": t_items,
"skills_hot": h_items,
"github_trending": finalize_wecom_github_repos(
[_github_move_to_repo(m) for m in (movement.get("github_trending_moves") or [])]
),
"github_emerging": finalize_wecom_github_repos(
[_github_move_to_repo(m) for m in (movement.get("github_emerging_moves") or [])]
),
"github_topic": finalize_wecom_github_repos(
[_github_move_to_repo(m) for m in (movement.get("github_topic_moves") or [])]
),
}
def replace_wecom_board_sections(
md: str,
*,
mode: str,
movement: dict[str, Any],
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
topic_name: str,
github_trending: list[dict[str, Any]] | None = None,
github_emerging: list[dict[str, Any]] | None = None,
github_topic: list[dict[str, Any]] | None = None,
wecom_trending: int = 5,
wecom_hot: int = 5,
wecom_github: int = 5,
wecom_emerging: int = 5,
wecom_topic: int = 5,
pad: bool = False,
date_str: str | None = None,
trending_pad: list[dict[str, Any]] | None = None,
hot_pad: list[dict[str, Any]] | None = None,
github_trending_pad: list[dict[str, Any]] | None = None,
github_emerging_pad: list[dict[str, Any]] | None = None,
github_topic_pad: list[dict[str, Any]] | None = None,
) -> str:
"""用 Python 合并后的 Skills 榜替换或插入 Agent 早报中的对应区块。"""
from daily.delta import load_recent_board_keys, partition_skill_moves_for_wecom
recent_board_keys: dict[str, set[str]] = {}
if pad and date_str:
recent_board_keys = load_recent_board_keys(date_str)
if mode == "delta":
t_moves, h_moves = partition_skill_moves_for_wecom(
movement.get("skills_trending_moves") or [],
movement.get("skills_hot_moves") or [],
)
skills_sec = build_skills_delta_sections(
t_moves,
h_moves,
trending_full=trending_pad if pad and trending_pad else trending,
hot_full=hot_pad if pad and hot_pad else hot,
trending_limit=wecom_trending,
hot_limit=wecom_hot,
pad=pad,
recent_trending=recent_board_keys.get("skills_trending"),
recent_hot=recent_board_keys.get("skills_hot"),
)
github_sec = build_github_delta_sections(
movement,
topic_name=topic_name,
github_trending=github_trending_pad if pad and github_trending_pad else github_trending,
github_emerging=github_emerging_pad if pad and github_emerging_pad else github_emerging,
github_topic=github_topic_pad if pad and github_topic_pad else github_topic,
trending_limit=wecom_github,
emerging_limit=wecom_emerging,
topic_limit=wecom_topic,
pad=pad,
recent_board_keys=recent_board_keys,
)
board_block = "\n\n".join(x for x in [skills_sec, github_sec] if x)
md = _strip_board_sections(md)
if board_block:
return md + "\n\n" + board_block + "\n"
return md + "\n"
trending_sec = build_skills_board_section("trending", "Skills Trending", trending)
hot_sec = build_skills_board_section("hot", "Skills Hot", hot)
replacement = f"{trending_sec}\n\n{hot_sec}\n\n"
@@ -218,6 +947,76 @@ def replace_wecom_skill_sections(
return md.rstrip() + "\n\n" + replacement
def replace_wecom_skill_sections(
md: str,
*,
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
mode: str = "full",
movement: dict[str, Any] | None = None,
topic_name: str = "llm",
github_trending: list[dict[str, Any]] | None = None,
github_emerging: list[dict[str, Any]] | None = None,
github_topic: list[dict[str, Any]] | None = None,
wecom_trending: int = 5,
wecom_hot: int = 5,
wecom_github: int = 5,
wecom_emerging: int = 5,
wecom_topic: int = 5,
pad: bool = False,
date_str: str | None = None,
trending_pad: list[dict[str, Any]] | None = None,
hot_pad: list[dict[str, Any]] | None = None,
github_trending_pad: list[dict[str, Any]] | None = None,
github_emerging_pad: list[dict[str, Any]] | None = None,
github_topic_pad: list[dict[str, Any]] | None = None,
) -> str:
"""用 Python 合并后的 Skills 榜替换或插入 Agent 早报中的对应区块。"""
return replace_wecom_board_sections(
md,
mode=mode,
movement=movement or {},
trending=trending,
hot=hot,
topic_name=topic_name,
github_trending=github_trending,
github_emerging=github_emerging,
github_topic=github_topic,
wecom_trending=wecom_trending,
wecom_hot=wecom_hot,
wecom_github=wecom_github,
wecom_emerging=wecom_emerging,
wecom_topic=wecom_topic,
pad=pad,
date_str=date_str,
trending_pad=trending_pad,
hot_pad=hot_pad,
github_trending_pad=github_trending_pad,
github_emerging_pad=github_emerging_pad,
github_topic_pad=github_topic_pad,
)
def _format_pick_link(pick_command: str, *, title: str = "", url: str = "") -> str:
cmd = pick_command.strip()
if not cmd:
return ""
label = title.strip()
if cmd.startswith("http://") or cmd.startswith("https://"):
if not label:
m = re.match(r"https?://github\.com/([^/\s#?]+/[^/\s#?]+)", cmd)
label = m.group(1) if m else cmd
return f"[{label}]({cmd})"
m = re.match(r"npx skills add (\S+)", cmd)
if m:
skill_path = m.group(1)
if not label:
label = skill_path.split("/")[-1]
href = url.strip() or f"https://skills.sh/{skill_path}"
return f"[{label}]({href})"
return f"`{cmd}`"
def build_wecom_report(
*,
date_str: str,
@@ -233,7 +1032,13 @@ def build_wecom_report(
topic_repos: list[dict[str, Any]],
ai_news: list[dict[str, Any]] | None = None,
cn_ai_news: list[dict[str, Any]] | None = None,
merged_ai_news: list[dict[str, Any]] | None = None,
merged_tech_ai_news: list[dict[str, Any]] | None = None,
pick_command: str,
pick_why: str = "",
pick_title: str = "",
pick_url: str = "",
include_boards: bool = True,
) -> str:
lines = [
f"{ICONS['header']} **早报 · {date_str}**",
@@ -247,6 +1052,12 @@ def build_wecom_report(
lines.append(f"{ICONS['theme']} {theme_line}")
lines.append("")
if merged_ai_news or merged_tech_ai_news:
block = _build_merged_news_block(merged_ai_news or [], merged_tech_ai_news)
if block:
lines.append(block.rstrip())
lines.append("")
else:
if ai_news:
lines.append(f"{ICONS['ainews']} **国际 AI 时讯 Top {len(ai_news)}**")
lines.extend(_ai_news_lines(ai_news))
@@ -257,6 +1068,7 @@ def build_wecom_report(
lines.extend(_ai_news_lines(cn_ai_news))
lines.append("")
if include_boards:
lines.append(f"{ICONS['trending']} **Skills Trending Top {len(trending)}**")
for rank, item in enumerate(trending, 1):
lines.extend(_skill_line(rank, item, badge=item.get("badge", "")))
@@ -269,20 +1081,23 @@ def build_wecom_report(
if repos:
lines.append(f"{ICONS['github']} **GitHub Trending Top {len(repos)}**")
lines.extend(_github_repo_lines(repos))
lines.extend(_github_repo_lines(finalize_wecom_github_repos(repos)))
lines.append("")
if emerging:
lines.append(f"{ICONS['emerging']} **新兴项目 Top {len(emerging)}**")
lines.extend(_github_repo_lines(emerging, show_created=True))
lines.extend(_github_repo_lines(finalize_wecom_github_repos(emerging), show_created=True))
lines.append("")
if topic_repos:
lines.append(f"{ICONS['topic']} **Topic `{topic_name}` Top {len(topic_repos)}**")
lines.extend(_github_repo_lines(topic_repos))
lines.extend(_github_repo_lines(finalize_wecom_github_repos(topic_repos)))
lines.append("")
lines.append(f"{ICONS['pick']} **今日首推**")
lines.append(f"`{pick_command}`")
lines.append(_format_pick_link(pick_command, title=pick_title, url=pick_url))
# 首推「为什么值得点开」一句理由; DAILY_FEATURED_REASON=0 关闭, 空则省略
if pick_why and env_bool("DAILY_FEATURED_REASON", True):
lines.append(f"> {pick_why}")
return "\n".join(lines)

File diff suppressed because it is too large Load Diff

View File

@@ -43,38 +43,59 @@ def search_github_repos(
logger.warning("GitHub Search 需要 GITHUB_TOKEN: %s", query[:80])
return []
target = max(1, min(int(limit), 1000))
per_page = min(100, target)
repos: list[dict[str, Any]] = []
seen: set[str] = set()
page = 1
try:
with httpx.Client(
timeout=20.0,
verify=certifi.where(),
headers=github_api_headers(),
) as client:
while len(repos) < target:
resp = client.get(
"https://api.github.com/search/repositories",
params={
"q": query,
"sort": sort,
"order": "desc",
"per_page": min(max(limit, 1), 30),
"per_page": per_page,
"page": page,
},
)
if resp.status_code != 200:
logger.warning("GitHub Search 失败 (%s): %s", resp.status_code, query[:80])
return []
logger.warning(
"GitHub Search 失败 (%s page=%s): %s",
resp.status_code,
page,
query[:80],
)
break
items = resp.json().get("items") or []
except Exception as exc:
logger.warning("GitHub Search 异常: %s", exc)
return []
repos: list[dict[str, Any]] = []
if not items:
break
for item in items:
full_name = item.get("full_name") or ""
if not full_name:
if not full_name or full_name in seen:
continue
seen.add(full_name)
repos.append(_repo_from_api_item(item, source="api-search"))
if len(repos) >= limit:
if len(repos) >= target:
break
return repos
if len(items) < per_page:
break
page += 1
# Search API 最多约 1000 条 / 10 页
if page > 10:
break
except Exception as exc:
logger.warning("GitHub Search 异常: %s", exc)
return repos[:target]
return repos[:target]
def _date_days_ago(days: int) -> str:

106
daily/holiday.py Normal file
View File

@@ -0,0 +1,106 @@
"""法定节假日获取、缓存与工作日判定。
数据来自 xiaoai.me 公共接口,一次性抓取全年并缓存到 .cache/
后续判定只读本地缓存;缓存缺失或过期时才重新请求。
"""
from __future__ import annotations
import json
import logging
import ssl
import urllib.request
from datetime import date
from pathlib import Path
from daily.config import CACHE_DIR
logger = logging.getLogger(__name__)
_API_URL = "https://publicapi.xiaoai.me/holiday/year?date={year}"
_TIMEOUT = 15
def _cache_file(year: int) -> Path:
return CACHE_DIR / f"holidays-{year}.json"
def _ssl_context() -> ssl.SSLContext:
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx
def _fetch_year(year: int) -> dict[str, dict]:
"""请求全年节假日,返回 {date_str: {"rest": bool, "name": str}}。
rest=1 表示休息(法定节假日/调休放假rest=0 表示调休补班(需上班)。
"""
url = _API_URL.format(year=year)
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=_TIMEOUT, context=_ssl_context()) as resp:
payload = json.loads(resp.read().decode("utf-8"))
items = payload.get("data") or []
result: dict[str, dict] = {}
for item in items:
day = item.get("date")
if not day:
continue
result[day] = {
"rest": bool(item.get("rest", 0)),
"name": item.get("holiday", ""),
}
if not result:
raise RuntimeError(f"节假日接口返回为空 year={year}")
return result
def load_holidays(year: int, *, refresh: bool = False) -> dict[str, dict]:
"""加载全年节假日缓存;缺失或 refresh 时联网抓取并写入缓存。
联网失败时若已有缓存则回退用缓存,保证离线可用。
"""
cache = _cache_file(year)
if cache.exists() and not refresh:
try:
raw = json.loads(cache.read_text(encoding="utf-8"))
if isinstance(raw, dict) and raw:
return raw
except (OSError, ValueError):
logger.warning("节假日缓存损坏,重新获取 %s", cache)
try:
data = _fetch_year(year)
except Exception as exc:
if cache.exists():
logger.warning("节假日获取失败,回退到本地缓存:%s", exc)
return json.loads(cache.read_text(encoding="utf-8"))
raise
cache.parent.mkdir(parents=True, exist_ok=True)
cache.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
logger.info("已缓存 %d 年节假日 %d 条 -> %s", year, len(data), cache)
return data
def is_workday(day: date, holidays: dict[str, dict] | None = None) -> bool:
"""判断是否为工作日。
规则调休补班日rest=0 且在表中)→ 上班;
法定节假日/调休放假rest=1→ 休息;
周六日 → 休息;其余 → 上班。
"""
holidays = holidays if holidays is not None else load_holidays(day.year)
key = day.isoformat()
entry = holidays.get(key)
if entry is not None:
return not entry["rest"]
return day.weekday() < 5 # 周一~周五
def workday_name(day: date, holidays: dict[str, dict] | None = None) -> str | None:
"""返回该天的节假日名(休息或调休补班),无则 None。用于日志。"""
holidays = holidays if holidays is not None else load_holidays(day.year)
entry = holidays.get(day.isoformat())
return entry["name"] if entry else None

View File

@@ -74,25 +74,26 @@ def _cursor_chat(system: str, user: str) -> str:
api_key = (env("CURSOR_API_KEY") or "").strip()
if not api_key:
return ""
import sys
from cursor_sdk import Agent, AgentOptions, Client, CursorAgentError, LocalAgentOptions
from daily.config import ROOT
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
_bot = str(ROOT / "bot")
if _bot not in sys.path:
sys.path.insert(0, _bot)
try:
from bridge_manager import warm_cursor_bridge
except ImportError:
warm_cursor_bridge = lambda: None # noqa: E731
from daily.bridge_manager import warm_cursor_bridge
cwd = env("DAILY_CURSOR_CWD") or str(ROOT)
# bridge_manager 读 bot env_config 的 CURSOR_CWD早报侧须先对齐工作目录
os.environ["CURSOR_CWD"] = cwd
warm_cursor_bridge()
model = env("CURSOR_MODEL") or "composer-2.5"
prompt = f"{system}\n\n{user}"
# SDK 默认 unary_timeout 只有 60sCURSOR_MODEL=auto 时后端首 token 常超时;
# 自建带大超时的 Client 绕开 _default_client()unary/stream 都放大。
unary_timeout = env_int("DAILY_CURSOR_UNARY_TIMEOUT", 300)
stream_timeout = env_int("DAILY_CURSOR_STREAM_TIMEOUT", 900)
client = Client(
base_url=os.environ["CURSOR_SDK_BRIDGE_URL"],
auth_token=os.environ["CURSOR_SDK_BRIDGE_TOKEN"],
unary_timeout=unary_timeout,
stream_timeout=stream_timeout,
)
try:
result = Agent.prompt(
prompt,
@@ -101,9 +102,12 @@ def _cursor_chat(system: str, user: str) -> str:
model=model,
local=LocalAgentOptions(cwd=cwd),
),
client=client,
)
except CursorAgentError as exc:
raise RuntimeError(f"LLM 调用失败:{exc.message}") from exc
finally:
client.close()
if result.status == "error":
raise RuntimeError(f"LLM 调用失败:{result.result or '未知错误'}")
return (result.result or "").strip()
@@ -118,5 +122,16 @@ def llm_chat(system: str, user: str) -> str:
return ""
def has_cursor_configured() -> bool:
return bool((env("CURSOR_API_KEY") or "").strip())
def cursor_agent_prompt(system: str, user: str) -> str:
"""仅 Cursor SDK Agent可用 WebSearch 等工具),不走 OpenAI 兼容 API。"""
if not has_cursor_configured():
return ""
return _cursor_chat(system, user)
def has_llm_configured() -> bool:
return bool(env("DAILY_LLM_API_KEY") or env("OPENAI_API_KEY") or env("CURSOR_API_KEY"))

149
daily/narrative_axis.py Normal file
View File

@@ -0,0 +1,149 @@
"""叙事轴硬互斥:代码选定轴,注入 Agent Step1 并强制覆写。"""
from __future__ import annotations
import json
import logging
import random
from collections import defaultdict
from datetime import datetime, timedelta
from typing import Any
from daily.config import OUTPUT_DIR, narrative_axis_days
logger = logging.getLogger(__name__)
NARRATIVE_AXES: tuple[str, ...] = (
"政策监管",
"模型发布",
"工具链/Agent",
"芯片算力",
"开源生态",
"应用落地",
"安全/诉讼",
)
def pick_narrative_axis(
used: set[str],
*,
rng: random.Random | None = None,
) -> str:
"""从固定轴枚举中排除已用轴后随机选取;全用尽则回退全表。"""
available = [a for a in NARRATIVE_AXES if a not in used]
pool = available or list(NARRATIVE_AXES)
picker = rng or random.Random()
return picker.choice(pool)
def load_recent_axes(date_str: str, days: int | None = None) -> list[str]:
"""近 N 日 data.narrative_axis不含当日按时间从近到远"""
lookback = days if days is not None else narrative_axis_days()
try:
dt = datetime.strptime(date_str, "%Y-%m-%d")
except ValueError:
return []
axes: list[str] = []
for day_offset in range(1, lookback + 1):
prev = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
path = OUTPUT_DIR / f"{prev}.data.json"
if not path.exists():
continue
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
logger.warning("读取 narrative_axis %s 失败:%s", path, exc)
continue
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
continue
axis = str(data.get("narrative_axis") or "").strip()
if axis:
axes.append(axis)
return axes
def load_recent_theme_summaries(date_str: str, days: int) -> list[str]:
"""近 N 日 theme/opening 摘要,供 Step1 软禁参考。"""
try:
dt = datetime.strptime(date_str, "%Y-%m-%d")
except ValueError:
return []
summaries: list[str] = []
for day_offset in range(1, days + 1):
prev = (dt - timedelta(days=day_offset)).strftime("%Y-%m-%d")
path = OUTPUT_DIR / f"{prev}.data.json"
if not path.exists():
continue
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
continue
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
continue
theme = str(data.get("theme") or data.get("editorial_theme") or "").strip()
opening = ""
trends = data.get("trends") if isinstance(data.get("trends"), dict) else {}
if isinstance(trends, dict):
opening = str(trends.get("opening") or "").strip()
if not theme:
themes = trends.get("themes") or []
if themes and isinstance(themes[0], dict):
theme = str(themes[0].get("title") or "").strip()
bit = " · ".join(x for x in (prev, theme, opening[:40]) if x)
if bit:
summaries.append(bit)
return summaries
def enforce_narrative_axis(trends: dict[str, Any], axis: str) -> dict[str, Any]:
"""强制 trends['narrative_axis'] = axis。"""
out = dict(trends)
out["narrative_axis"] = axis
return out
def theme_clusters(
feed: dict[str, Any],
*,
limit: int = 5,
theme_rules: list[tuple[str, str, list[str]]],
skill_id_fn: Any,
) -> list[tuple[str, list[str]]]:
"""按 THEME_RULES 把 feed 的 topTrending/topHot 聚成 (主题, 示例列表)。
从 generate.py 迁入(原私有 _theme_clusters。theme_rules 与 skill_id_fn
由调用方注入,避免对 generate.py 的反向依赖(防循环 import
"""
buckets: dict[str, list[str]] = defaultdict(list)
seen: set[str] = set()
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
item_id = skill_id_fn(item)
if item_id in seen:
continue
seen.add(item_id)
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, theme, keywords in theme_rules:
if any(k in haystack for k in keywords):
label = f"**{item.get('title')}** (`{item.get('source')}`)"
if label not in buckets[theme]:
buckets[theme].append(label)
break
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
def theme_names(
feed: dict[str, Any],
*,
theme_rules: list[tuple[str, str, list[str]]],
skill_id_fn: Any,
limit: int = 3,
) -> list[str]:
"""仅取主题名(不含 markdown 示例),供「今日看点/theme_line」回退文案。"""
return [theme for theme, _ in theme_clusters(
feed, theme_rules=theme_rules, skill_id_fn=skill_id_fn
)][:limit]

View File

@@ -1,13 +1,26 @@
from daily.news.fetch import (
fetch_ai_news,
fetch_cn_ai_news,
format_cn_news_section,
format_news_section,
)
__all__ = [
"fetch_ai_news",
"fetch_cn_ai_news",
"format_news_section",
"format_cn_news_section",
]

View File

@@ -27,7 +27,7 @@ NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
name="厂商官方",
icon="🏢",
feeds=(
NewsFeed("Anthropic Claude 更新", "https://docs.anthropic.com/en/release-notes/feed"),
NewsFeed("Anthropic Claude 更新", "https://platform.claude.com/docs/en/release-notes/overview"),
NewsFeed("OpenAI", "https://openai.com/news/rss.xml"),
NewsFeed("Google AI", "https://blog.google/technology/ai/rss/"),
NewsFeed("DeepMind", "https://deepmind.google/blog/rss.xml"),

View File

@@ -44,7 +44,6 @@ CN_NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
icon="📰",
feeds=(
NewsFeed("量子位", "https://www.qbitai.com/feed"),
NewsFeed("InfoQ 中文", "https://www.infoq.cn/feed/AI"),
),
),
NewsCategory(
@@ -60,12 +59,4 @@ CN_NEWS_CATEGORIES: tuple[NewsCategory, ...] = (
),
),
),
NewsCategory(
id="dev",
name="开发者社区",
icon="💻",
feeds=(
NewsFeed("掘金", "https://juejin.cn/rss", ai_filter=True),
),
),
)

View File

@@ -8,21 +8,23 @@ import time
import html
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone, timedelta
from datetime import datetime, time as dt_time, timezone, timedelta
from email.utils import parsedate_to_datetime
from typing import Any
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
from zoneinfo import ZoneInfo
import certifi
import httpx
from daily.config import env, env_int, news_summary_limit
from daily.config import env, env_int, news_summary_limit, wecom_news_desc_limit
from daily.news.feeds import NEWS_CATEGORIES, NewsCategory, NewsFeed
from daily.news.feeds_cn import CN_AI_TITLE_KEYWORDS, CN_NEWS_CATEGORIES
from daily.text_utils import trim_brief
logger = logging.getLogger(__name__)
USER_AGENT = "Mozilla/5.0 (compatible; daily-robots/1.0; +https://skills.sh)"
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)"
BROWSER_USER_AGENT = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
@@ -46,15 +48,40 @@ def _cn_enabled() -> bool:
def _hours_window() -> int:
return max(1, env_int("DAILY_AI_NEWS_HOURS", 72))
return max(1, env_int("DAILY_AI_NEWS_HOURS", 24))
def _news_tz_name() -> str:
return (env("DAILY_AI_NEWS_TZ") or env("DAILY_SCHEDULE_TZ") or "Asia/Shanghai").strip()
def _floor_today_enabled() -> bool:
raw = env("DAILY_AI_NEWS_FLOOR_TODAY")
if raw is None:
return True
return raw.strip().lower() not in {"0", "false", "no", "off"}
def _cutoff_datetime(*, floor_today: bool) -> datetime:
"""滚动 N 小时窗口;国际新闻可叠加「不早于今日 0 点(本地时区)」。"""
now = _now_utc()
rolling = now - timedelta(hours=_hours_window())
if not floor_today:
return rolling
tz = ZoneInfo(_news_tz_name())
local = now.astimezone(tz)
start_today = local.replace(hour=0, minute=0, second=0, microsecond=0).astimezone(timezone.utc)
return max(rolling, start_today)
def _per_feed_limit() -> int:
return max(1, env_int("DAILY_AI_NEWS_PER_FEED", 3))
want = max(_wecom_limit(), _wecom_cn_limit())
return max(want // 2, env_int("DAILY_AI_NEWS_PER_FEED", 5))
def _per_category_limit() -> int:
return max(1, env_int("DAILY_AI_NEWS_PER_CATEGORY", 5))
want = max(_wecom_limit(), _wecom_cn_limit())
return max(want, env_int("DAILY_AI_NEWS_PER_CATEGORY", 10))
def _wecom_limit() -> int:
@@ -62,7 +89,7 @@ def _wecom_limit() -> int:
def _wecom_cn_limit() -> int:
return max(1, env_int("DAILY_WECOM_CN_AI_NEWS", 8))
return max(1, env_int("DAILY_WECOM_CN_AI_NEWS", 10))
def _matches_cn_ai_title(title: str) -> bool:
@@ -102,7 +129,6 @@ def _parse_datetime(value: str | None) -> datetime | None:
for fmt in (
"%Y-%m-%dT%H:%M:%SZ",
"%Y-%m-%dT%H:%M:%S%z",
"%Y-%m-%d",
):
try:
dt = datetime.strptime(text[: len(fmt.replace("%z", "+0000"))], fmt.replace("%z", ""))
@@ -111,17 +137,103 @@ def _parse_datetime(value: str | None) -> datetime | None:
return dt.astimezone(timezone.utc)
except ValueError:
continue
if re.match(r"^\d{4}-\d{2}-\d{2}$", text):
try:
tz = ZoneInfo(_news_tz_name())
day = datetime.strptime(text, "%Y-%m-%d").date()
# 仅日期时按本地中午估算,避免 UTC 0 点误判为「前一天」
dt = datetime.combine(day, dt_time(12, 0), tzinfo=tz)
return dt.astimezone(timezone.utc)
except ValueError:
pass
return None
def _clean_text(text: str | None, limit: int = 200) -> str:
if not text:
return ""
plain = STRIP_HTML.sub(" ", html.unescape(text))
plain = WS.sub(" ", plain).strip()
plain = _strip_summary_plain(text)
if limit <= 0 or len(plain) <= limit:
return plain
return plain[: limit - 3] + "..."
return trim_brief(plain, limit)
def _strip_summary_plain(text: str | None) -> str:
if not text:
return ""
plain = STRIP_HTML.sub(" ", html.unescape(text))
return WS.sub(" ", plain).strip()
_JUNK_SUMMARY_RE = re.compile(
r"^(点击查看原文|article url:|comments url:|discussion on hn|read more)",
re.IGNORECASE,
)
def _is_junk_news_summary(text: str) -> bool:
if not text:
return True
if _JUNK_SUMMARY_RE.match(text.strip()):
return True
if text.strip().endswith(">") and "点击" in text:
return True
return False
def brief_news_summary(text: str | None, limit: int | None = None) -> str:
"""企微新闻一句摘要:去 HTML、过滤占位文案、句读处截断。"""
plain = _strip_summary_plain(text)
if _is_junk_news_summary(plain):
return ""
lim = wecom_news_desc_limit() if limit is None else limit
return trim_brief(plain, lim)
def sync_wecom_news_rows(items: list[dict[str, Any]], flat: list[dict[str, Any]]) -> None:
"""中文化后,用 flat 最新 summary 刷新企微 desc_short。"""
by_link = {_normalize_link(str(i.get("link") or "")): i for i in flat if i.get("link")}
for row in items:
link = _normalize_link(str(row.get("link") or ""))
src = by_link.get(link)
if src:
row["desc_short"] = brief_news_summary(src.get("summary"))
def finalize_wecom_news_items(
items: list[dict[str, Any]],
*,
force_chinese: bool = False,
) -> None:
"""企微新闻摘要:确保 desc_short 为中文(国际源 force_chinese=True"""
from daily.localize import LocalizeJob, localize_brief_descriptions, needs_chinese
from daily.news.sanitize import strip_relax_window_prefix
limit = wecom_news_desc_limit()
jobs: list[LocalizeJob] = []
keyed: list[tuple[str, dict[str, Any]]] = []
for idx, item in enumerate(items):
text = strip_relax_window_prefix(
(item.get("desc_short") or item.get("summary_plain") or "").strip()
)
if text:
item["desc_short"] = text
if not text or _is_junk_news_summary(text):
item["desc_short"] = ""
continue
if force_chinese or needs_chinese(text):
key = f"wecom-news:{item.get('link') or idx}"
jobs.append(LocalizeJob(key, text, limit))
keyed.append((key, item))
elif not item.get("desc_short"):
item["desc_short"] = brief_news_summary(text, limit)
if not jobs:
return
zh_map = localize_brief_descriptions(jobs, archive=True)
for key, item in keyed:
if key in zh_map:
item["desc_short"] = strip_relax_window_prefix(zh_map[key])
def _normalize_link(link: str) -> str:
@@ -286,7 +398,7 @@ def _fetch_one(
client: httpx.Client,
category: NewsCategory,
feed: NewsFeed,
) -> tuple[list[dict[str, Any]], bool]:
) -> list[dict[str, Any]]:
last_exc: Exception | None = None
for url in _reddit_fetch_urls(feed.url):
try:
@@ -294,12 +406,12 @@ def _fetch_one(
resp = client.get(url, headers=headers)
resp.raise_for_status()
entries = _parse_feed(resp.text, feed.name, category)
return _filter_ai_entries(entries, ai_filter=feed.ai_filter), True
return _filter_ai_entries(entries, ai_filter=feed.ai_filter)
except Exception as exc:
last_exc = exc
continue
logger.warning("RSS fetch failed [%s] %s: %s", feed.name, feed.url, last_exc)
return [], False
return []
def _dedupe_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
@@ -317,10 +429,119 @@ def _dedupe_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
return result
def _filter_flat_in_window(
flat: list[dict[str, Any]],
*,
floor_today: bool,
) -> list[dict[str, Any]]:
cutoff = _cutoff_datetime(floor_today=floor_today)
items = [i for i in _dedupe_items(flat) if _within_window(i, cutoff)]
items.sort(key=_sort_key, reverse=True)
return items
def _pick_news_items(
flat: list[dict[str, Any]],
limit: int,
preferred: tuple[str, ...],
*,
one_per_source: bool = False,
) -> list[dict[str, Any]]:
picked: list[dict[str, Any]] = []
seen_links: set[str] = set()
seen_sources: set[str] = set()
def _try_take(item: dict[str, Any]) -> bool:
link = _normalize_link(item.get("link", ""))
if not link or link in seen_links:
return False
if one_per_source:
source = item.get("source_name", "?")
if source in seen_sources:
return False
seen_sources.add(source)
seen_links.add(link)
picked.append(item)
return True
for cat in preferred:
for item in flat:
if item.get("category_id") != cat:
continue
if _try_take(item) and len(picked) >= limit:
return picked[:limit]
for item in flat:
if _try_take(item) and len(picked) >= limit:
break
return picked[:limit]
def _fill_picked_to_limit(
picked: list[dict[str, Any]],
pools: list[list[dict[str, Any]]],
limit: int,
) -> list[dict[str, Any]]:
seen_links = {_normalize_link(i.get("link", "")) for i in picked}
for pool in pools:
for item in pool:
if len(picked) >= limit:
return picked[:limit]
link = _normalize_link(item.get("link", ""))
if not link or link in seen_links:
continue
picked.append(item)
seen_links.add(link)
return picked[:limit]
def _to_wecom_news_row(item: dict[str, Any]) -> dict[str, Any]:
plain = _strip_summary_plain(item.get("summary", ""))
return {
"title": item.get("title", "?"),
"link": item.get("link", ""),
"source_name": item.get("source_name", "?"),
"published_fmt": item.get("published_fmt", ""),
"desc_short": brief_news_summary(plain),
"summary_plain": plain,
}
def _apply_pushed_dedup_with_backfill(
items: list[dict[str, Any]],
picked: list[dict[str, Any]],
*,
date_str: str | None,
limit: int,
) -> list[dict[str, Any]]:
if not date_str:
return items[:limit]
from daily.config import news_backfill_enabled
from daily.news.pushed_links import filter_unpushed_items
fresh = filter_unpushed_items(items, date_str=date_str)
if len(fresh) >= limit:
return fresh[:limit]
if not news_backfill_enabled():
if len(fresh) < limit:
logger.info("news_short:%s", len(fresh))
return fresh[:limit]
seen = {_normalize_link(i.get("link", "")) for i in fresh if i.get("link")}
for item in picked:
if len(fresh) >= limit:
break
link = _normalize_link(item.get("link", ""))
if not link or link in seen:
continue
fresh.append(_to_wecom_news_row(item))
seen.add(link)
return fresh[:limit]
def _within_window(item: dict[str, Any], cutoff: datetime) -> bool:
dt = _entry_datetime(item)
if dt is None:
return True
return False
return dt >= cutoff
@@ -331,11 +552,11 @@ def _sort_key(item: dict[str, Any]) -> tuple[int, datetime]:
return (0, dt)
def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
def _fetch_news(categories: tuple[NewsCategory, ...], *, floor_today: bool = False) -> dict[str, Any]:
hours = _hours_window()
per_feed = _per_feed_limit()
per_category = _per_category_limit()
cutoff = _now_utc() - timedelta(hours=hours)
cutoff = _cutoff_datetime(floor_today=floor_today)
headers = {"User-Agent": USER_AGENT, "Accept": "application/rss+xml, application/atom+xml, application/xml, text/xml, */*"}
tasks: list[tuple[NewsCategory, NewsFeed]] = []
@@ -344,12 +565,7 @@ def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
tasks.append((category, feed))
raw_by_category: dict[str, list[dict[str, Any]]] = {c.id: [] for c in categories}
stats: dict[str, Any] = {
"feeds_total": len(tasks),
"feeds_ok": 0,
"items_raw": 0,
"feeds_failed": [],
}
stats = {"feeds_total": len(tasks), "feeds_ok": 0, "items_raw": 0}
with httpx.Client(timeout=15.0, verify=certifi.where(), follow_redirects=True, headers=headers) as client:
fast_tasks = [t for t in tasks if not t[1].slow]
@@ -363,24 +579,19 @@ def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
for future in as_completed(futures):
cat_id, feed_name = futures[future]
try:
entries, ok = future.result()
entries = future.result()
except Exception as exc:
logger.warning("RSS 任务异常 [%s]: %s", feed_name, exc)
stats["feeds_failed"].append(feed_name)
continue
if ok:
if entries:
stats["feeds_ok"] += 1
else:
stats["feeds_failed"].append(feed_name)
stats["items_raw"] += len(entries)
raw_by_category[cat_id].extend(entries[:per_feed])
for cat, feed in slow_tasks:
entries, ok = _fetch_one(client, cat, feed)
if ok:
entries = _fetch_one(client, cat, feed)
if entries:
stats["feeds_ok"] += 1
else:
stats["feeds_failed"].append(feed.name)
stats["items_raw"] += len(entries)
raw_by_category[cat.id].extend(entries[:per_feed])
if _is_reddit_url(feed.url):
@@ -416,6 +627,7 @@ def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
return {
"enabled": True,
"hours": hours,
"floor_today": floor_today,
"categories": categories_out,
"flat": flat,
"stats": stats,
@@ -426,7 +638,7 @@ def fetch_ai_news() -> dict[str, Any]:
"""按类别抓取国际 AI 时讯,返回 {enabled, hours, categories, flat, stats}。"""
if not _enabled():
return {"enabled": False, "categories": [], "flat": [], "stats": {}}
return _fetch_news(NEWS_CATEGORIES)
return _fetch_news(NEWS_CATEGORIES, floor_today=_floor_today_enabled())
def fetch_cn_ai_news() -> dict[str, Any]:
@@ -469,20 +681,15 @@ def _format_news_section(
categories = news.get("categories") or []
hours = news.get("hours", 72)
floor_note = " · 仅今日" if news.get("floor_today") else ""
lines = [
"---",
"",
f"## {section_no}{title}",
"",
f"> 近 **{hours}h** · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用",
f"> 近 **{hours}h**{floor_note} · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用",
"",
]
failed = news.get("stats", {}).get("feeds_failed") or []
if failed:
preview = "".join(failed[:5])
suffix = "" if len(failed) > 5 else ""
lines.append(f"> ⚠️ {len(failed)} 个源抓取失败:{preview}{suffix}")
lines.append("")
if not categories:
lines.append("*暂无可用条目(网络/RSS 源异常或时间窗口内无更新)。*")
@@ -513,93 +720,27 @@ def _format_news_section(
return lines
def _sorted_flat(news: dict[str, Any]) -> list[dict[str, Any]]:
flat = list(news.get("flat") or [])
flat.sort(
key=lambda item: (float(item.get("score") or 0), _sort_key(item)[1]),
reverse=True,
)
return flat
def prepare_wecom_news_items(news: dict[str, Any], *, limit: int | None = None) -> list[dict[str, Any]]:
def prepare_wecom_news_items(news: dict[str, Any], *, date_str: str | None = None) -> list[dict[str, Any]]:
if not news.get("enabled"):
return []
pick_limit = limit or _wecom_limit()
flat = _sorted_flat(news)
limit = _wecom_limit()
floor = bool(news.get("floor_today", _floor_today_enabled()))
flat_strict = _filter_flat_in_window(news.get("flat") or [], floor_today=floor)
flat_relaxed = _filter_flat_in_window(news.get("flat") or [], floor_today=False)
preferred = ("media", "newsletter", "official", "community", "research", "developer")
picked: list[dict[str, Any]] = []
seen: set[str] = set()
for cat in preferred:
for item in flat:
link = _normalize_link(item.get("link", ""))
if item.get("category_id") != cat or link in seen:
continue
picked.append(item)
seen.add(link)
if len(picked) >= pick_limit:
break
if len(picked) >= pick_limit:
break
items: list[dict[str, Any]] = []
for item in picked[:pick_limit]:
items.append(
{
"title": item.get("title", "?"),
"link": item.get("link", ""),
"source_name": item.get("source_name", "?"),
"published_fmt": item.get("published_fmt", ""),
"desc_short": _clean_text(item.get("summary", ""), 36),
"score": item.get("score"),
}
)
return items
picked = _pick_news_items(flat_strict, limit, preferred)
picked = _fill_picked_to_limit(picked, [flat_relaxed, news.get("flat") or []], limit)
items = [_to_wecom_news_row(item) for item in picked[:limit]]
return _apply_pushed_dedup_with_backfill(items, picked, date_str=date_str, limit=limit)
def prepare_wecom_cn_news_items(news: dict[str, Any], *, limit: int | None = None) -> list[dict[str, Any]]:
def prepare_wecom_cn_news_items(news: dict[str, Any], *, date_str: str | None = None) -> list[dict[str, Any]]:
if not news.get("enabled"):
return []
pick_limit = limit or _wecom_cn_limit()
flat = _sorted_flat(news)
preferred = ("media", "tech", "dev")
picked: list[dict[str, Any]] = []
seen_links: set[str] = set()
seen_sources: set[str] = set()
for cat in preferred:
for item in flat:
link = _normalize_link(item.get("link", ""))
source = item.get("source_name", "?")
if item.get("category_id") != cat or not link or link in seen_links or source in seen_sources:
continue
picked.append(item)
seen_links.add(link)
seen_sources.add(source)
if len(picked) >= pick_limit:
break
if len(picked) >= pick_limit:
break
if len(picked) < pick_limit:
for item in flat:
link = _normalize_link(item.get("link", ""))
if not link or link in seen_links:
continue
picked.append(item)
seen_links.add(link)
if len(picked) >= pick_limit:
break
items: list[dict[str, Any]] = []
for item in picked[:pick_limit]:
items.append(
{
"title": item.get("title", "?"),
"link": item.get("link", ""),
"source_name": item.get("source_name", "?"),
"published_fmt": item.get("published_fmt", ""),
"desc_short": _clean_text(item.get("summary", ""), 36),
"score": item.get("score"),
}
)
return items
limit = _wecom_cn_limit()
flat = _filter_flat_in_window(news.get("flat") or [], floor_today=False)
preferred = ("media", "tech")
picked = _pick_news_items(flat, limit, preferred, one_per_source=True)
picked = _fill_picked_to_limit(picked, [news.get("flat") or []], limit)
items = [_to_wecom_news_row(item) for item in picked[:limit]]
return _apply_pushed_dedup_with_backfill(items, picked, date_str=date_str, limit=limit)

View File

@@ -0,0 +1,87 @@
"""已推送企微早报的新闻 link 去重缓存。"""
from __future__ import annotations
import json
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
from daily.config import CACHE_DIR, news_dedup_days
from daily.news.fetch import _normalize_link
def _cache_path() -> Path:
return CACHE_DIR / "pushed-news-links.json"
def _load_raw() -> dict[str, Any]:
path = _cache_path()
if not path.exists():
return {"dates": {}}
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return {"dates": {}}
if not isinstance(data.get("dates"), dict):
return {"dates": {}}
return data
def _save_raw(data: dict[str, Any]) -> None:
path = _cache_path()
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
def _prune(data: dict[str, Any], *, keep_days: int) -> None:
dates: dict[str, list[str]] = data.setdefault("dates", {})
try:
anchor = max(datetime.strptime(d, "%Y-%m-%d") for d in dates)
except ValueError:
return
cutoff = anchor - timedelta(days=keep_days)
for key in list(dates.keys()):
try:
if datetime.strptime(key, "%Y-%m-%d") < cutoff:
dates.pop(key, None)
except ValueError:
dates.pop(key, None)
def load_pushed_link_set() -> set[str]:
data = _load_raw()
out: set[str] = set()
for links in (data.get("dates") or {}).values():
if isinstance(links, list):
out.update(str(x) for x in links if x)
return out
def filter_unpushed_items(
items: list[dict[str, Any]],
*,
date_str: str,
) -> list[dict[str, Any]]:
del date_str # reserved for per-day scoping if needed later
seen = load_pushed_link_set()
out: list[dict[str, Any]] = []
for item in items:
link = _normalize_link(str(item.get("link") or ""))
if not link or link in seen:
continue
out.append(item)
return out
def record_pushed_links(date_str: str, links: list[str]) -> None:
data = _load_raw()
dates: dict[str, list[str]] = data.setdefault("dates", {})
normalized: list[str] = []
for link in links:
clean = _normalize_link(link)
if clean:
normalized.append(clean)
dates[date_str] = sorted(set(normalized))
_prune(data, keep_days=news_dedup_days())
_save_raw(data)

View File

@@ -1,171 +0,0 @@
"""RSS 新闻去重、打分与排序。"""
from __future__ import annotations
import logging
import re
from datetime import datetime, timezone
from typing import Any
from daily.config import env_int
from daily.delta import find_previous_data
from daily.news.fetch import _entry_datetime, _normalize_link, _normalize_title
logger = logging.getLogger(__name__)
_WS = re.compile(r"\s+")
_TOKEN = re.compile(r"[\w]{2,}", re.UNICODE)
_CJK = re.compile(r"[\u4e00-\u9fff]")
CATEGORY_TIER: dict[str, int] = {
"official": 20,
"developer": 16,
"research": 14,
"media": 12,
"newsletter": 10,
"community": 8,
"tech": 12,
"dev": 10,
}
def title_similarity_threshold() -> float:
raw = env_int("DAILY_NEWS_TITLE_SIM", 55)
return max(0, min(raw, 95)) / 100.0
def _title_tokens(title: str) -> set[str]:
normalized = _normalize_title(title)
tokens = set(_TOKEN.findall(normalized))
cjk = "".join(_CJK.findall(title))
for i in range(max(0, len(cjk) - 1)):
tokens.add(cjk[i : i + 2])
if not tokens and normalized:
tokens.add(normalized)
return tokens
def jaccard_similarity(left: set[str], right: set[str]) -> float:
if not left or not right:
return 0.0
inter = len(left & right)
union = len(left | right)
return inter / union if union else 0.0
def fuzzy_dedupe_by_title(
items: list[dict[str, Any]],
*,
threshold: float | None = None,
) -> list[dict[str, Any]]:
"""按标题相似度合并重复报道,保留 score 更高(或更靠前)的条目。"""
if not items:
return []
limit = threshold if threshold is not None else title_similarity_threshold()
kept: list[dict[str, Any]] = []
kept_tokens: list[set[str]] = []
for item in items:
tokens = _title_tokens(str(item.get("title") or ""))
duplicate_idx: int | None = None
for idx, existing_tokens in enumerate(kept_tokens):
if jaccard_similarity(tokens, existing_tokens) >= limit:
duplicate_idx = idx
break
if duplicate_idx is None:
kept.append(item)
kept_tokens.append(tokens)
continue
existing = kept[duplicate_idx]
if float(item.get("score") or 0) > float(existing.get("score") or 0):
kept[duplicate_idx] = item
kept_tokens[duplicate_idx] = tokens
return kept
def _freshness_points(item: dict[str, Any], *, now: datetime | None = None) -> int:
now = now or datetime.now(timezone.utc)
dt = _entry_datetime(item)
if dt is None:
return 4
age_hours = max(0.0, (now - dt).total_seconds() / 3600.0)
if age_hours <= 6:
return 30
if age_hours <= 24:
return 22
if age_hours <= 72:
return 8
return 0
def _source_tier(item: dict[str, Any]) -> int:
category_id = str(item.get("category_id") or "")
return CATEGORY_TIER.get(category_id, 8)
def _novelty_points(item: dict[str, Any], yesterday_links: set[str]) -> int:
link = _normalize_link(str(item.get("link") or ""))
if link and link in yesterday_links:
return -20
return 5
def score_news_item(
item: dict[str, Any],
*,
yesterday_links: set[str] | None = None,
now: datetime | None = None,
) -> int:
links = yesterday_links or set()
total = _source_tier(item) + _freshness_points(item, now=now) + _novelty_points(item, links)
return max(0, total)
def load_yesterday_news_links(date_str: str) -> set[str]:
baseline = find_previous_data(date_str)
if not baseline:
return set()
_, data = baseline
links: set[str] = set()
for key in ("ai_news", "cn_ai_news"):
for item in data.get(key) or []:
if not isinstance(item, dict):
continue
link = _normalize_link(str(item.get("link") or ""))
if link:
links.add(link)
return links
def apply_news_ranking(news: dict[str, Any], *, date_str: str) -> dict[str, Any]:
"""对 categories / flat 打分、模糊去重并写回 score 字段。"""
if not news.get("enabled"):
return news
yesterday_links = load_yesterday_news_links(date_str)
threshold = title_similarity_threshold()
now = datetime.now(timezone.utc)
categories = news.get("categories") or []
for category in categories:
items = list(category.get("items") or [])
for item in items:
item["score"] = score_news_item(item, yesterday_links=yesterday_links, now=now)
items.sort(key=lambda x: float(x.get("score") or 0), reverse=True)
category["items"] = fuzzy_dedupe_by_title(items, threshold=threshold)
flat: list[dict[str, Any]] = []
for category in categories:
flat.extend(category.get("items") or [])
flat.sort(key=lambda x: float(x.get("score") or 0), reverse=True)
news["flat"] = fuzzy_dedupe_by_title(flat, threshold=threshold)
stats = dict(news.get("stats") or {})
stats["ranked_items"] = len(news["flat"])
stats["yesterday_links"] = len(yesterday_links)
news["stats"] = stats
logger.info(
"新闻排序完成:%d 条 flat昨日链接基准 %d",
len(news["flat"]),
len(yesterday_links),
)
return news

386
daily/news/research.py Normal file
View File

@@ -0,0 +1,386 @@
"""Cursor SDK + deep-research 工作流:采集 AI 时讯(方案 A内置 WebSearch"""
from __future__ import annotations
import json
import logging
from datetime import datetime, timezone, timedelta
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
from daily.config import OUTPUT_DIR, ROOT, env, env_int, wecom_ai_news_tech_limit
from daily.llm_client import cursor_agent_prompt, extract_json_object, has_cursor_configured
from daily.news.fetch import brief_news_summary, _normalize_link
from daily.news.pushed_links import filter_unpushed_items
from daily.news.research_quality import post_process_research_news, research_cn_min
logger = logging.getLogger(__name__)
_SKILL_DIR = ROOT / "skills" / "daily-ai-news-research"
_DEEP_RESEARCH_CANDIDATES = (
ROOT / "skills" / "deep-research" / "SKILL.md",
Path.home() / ".agents" / "skills" / "deep-research" / "SKILL.md",
Path.home() / ".cursor" / "skills" / "deep-research" / "SKILL.md",
)
def ai_news_mode() -> str:
return (env("DAILY_AI_NEWS_MODE") or "rss").strip().lower()
def is_research_mode() -> bool:
return ai_news_mode() == "research"
def research_hours() -> int:
return max(1, env_int("DAILY_AI_NEWS_HOURS", 24))
def research_limit() -> int:
return max(1, env_int("DAILY_WECOM_AI_NEWS", 10))
def research_pool_limit(display_limit: int | None = None) -> int:
"""Agent 原始候选条数(展示上限之上多拉,供可信/去重筛)。"""
lim = display_limit if display_limit is not None else research_limit()
explicit = env_int("DAILY_AI_NEWS_RESEARCH_POOL", 0)
if explicit > 0:
return max(lim, explicit)
return max(lim * 2, lim + 8)
def research_tech_pool_limit(display_limit: int | None = None) -> int:
tech = display_limit if display_limit is not None else research_tech_limit()
if tech <= 0:
return 0
explicit = env_int("DAILY_AI_NEWS_RESEARCH_TECH_POOL", 0)
if explicit > 0:
return max(tech, explicit)
return max(tech * 2, tech + 4)
def research_json_path(date_str: str) -> Path:
return OUTPUT_DIR / f"{date_str}.ai-news-research.json"
def _save_research_json(path: Path, data: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
def _load_skill() -> str:
parts: list[str] = []
for path in _DEEP_RESEARCH_CANDIDATES:
if path.exists():
parts.append(path.read_text(encoding="utf-8").strip())
break
local = _SKILL_DIR / "SKILL.md"
if local.exists():
parts.append(local.read_text(encoding="utf-8").strip())
if not parts:
return "你是 AI 时讯调研员,只输出 JSON。"
return "\n\n---\n\n".join(parts)
def _guess_source_name(link: str, explicit: str) -> str:
name = (explicit or "").strip()
if name:
return name
host = urlparse(link).netloc.lower().removeprefix("www.")
mapping = {
"techcrunch.com": "TechCrunch",
"theverge.com": "The Verge",
"openai.com": "OpenAI",
"anthropic.com": "Anthropic",
"arxiv.org": "arXiv",
"qbitai.com": "量子位",
"36kr.com": "36氪",
"leiphone.com": "雷锋网",
}
for key, label in mapping.items():
if host.endswith(key) or key in host:
return label
return host.split(".")[0].capitalize() if host else "?"
def _normalize_research_item(raw: dict[str, Any]) -> dict[str, Any] | None:
title = str(raw.get("title") or "").strip()
link = _normalize_link(str(raw.get("link") or ""))
if not title or not link or not link.startswith("http"):
return None
desc = brief_news_summary(str(raw.get("desc_short") or raw.get("summary") or ""))
item: dict[str, Any] = {
"title": title,
"link": link,
"source_name": _guess_source_name(link, str(raw.get("source_name") or "")),
"published_fmt": str(raw.get("published_fmt") or "").strip(),
"desc_short": desc,
"summary_plain": desc,
}
region = str(raw.get("region") or "").strip().lower()
if region:
item["region"] = region
return item
def research_tech_limit() -> int:
return wecom_ai_news_tech_limit()
def _parse_items_array(
items_raw: Any,
*,
limit: int,
seen: set[str],
) -> list[dict[str, Any]]:
if not isinstance(items_raw, list):
return []
out: list[dict[str, Any]] = []
for row in items_raw:
if not isinstance(row, dict):
continue
item = _normalize_research_item(row)
if not item:
continue
if item["link"] in seen:
continue
seen.add(item["link"])
out.append(item)
if len(out) >= limit:
break
return out
def parse_research_response(
raw: str,
*,
limit: int,
tech_limit: int = 0,
pool_limit: int | None = None,
tech_pool_limit: int | None = None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
parsed = extract_json_object(raw)
item_cap = pool_limit if pool_limit is not None else limit
tech_cap = tech_pool_limit if tech_pool_limit is not None else tech_limit
seen: set[str] = set()
items = _parse_items_array(parsed.get("items"), limit=item_cap, seen=seen)
tech_items = (
_parse_items_array(parsed.get("tech_items"), limit=tech_cap, seen=seen) if tech_cap else []
)
return items, tech_items
def _apply_pushed_dedup(items: list[dict[str, Any]], *, date_str: str, limit: int) -> list[dict[str, Any]]:
from daily.config import news_backfill_enabled
from daily.news.sanitize import strip_relax_window_prefix
for item in items:
if item.get("desc_short"):
item["desc_short"] = strip_relax_window_prefix(str(item.get("desc_short") or ""))
fresh = filter_unpushed_items(items, date_str=date_str)
if len(fresh) >= limit:
return fresh[:limit]
if not news_backfill_enabled():
if len(fresh) < limit:
logger.info("news_short:%s", len(fresh))
return fresh[:limit]
seen = {i.get("link") for i in fresh}
for item in items:
if len(fresh) >= limit:
break
if item.get("link") not in seen:
fresh.append(item)
seen.add(item.get("link"))
return fresh[:limit]
def fetch_ai_news_research(
*,
date_str: str,
hours: int | None = None,
limit: int | None = None,
) -> dict[str, Any]:
"""Cursor Agent 调研 AI 时讯;返回 {enabled, mode, hours, items, flat, stats}。"""
h = hours if hours is not None else research_hours()
lim = limit if limit is not None else research_limit()
tech_lim = research_tech_limit()
if not has_cursor_configured():
logger.warning("DAILY_AI_NEWS_MODE=research 但未配置 CURSOR_API_KEY")
return {
"enabled": False,
"mode": "research",
"items": [],
"tech_items": [],
"flat": [],
"stats": {"error": "no_cursor_key"},
}
skill = _load_skill()
now_cst = datetime.now(timezone(timedelta(hours=8)))
cn_min = research_cn_min(lim)
pool = research_pool_limit(lim)
tech_pool = research_tech_pool_limit(tech_lim)
# 候选池内国内目标略高于展示配额,避免筛完国内不足
cn_pool_target = max(cn_min * 2, cn_min + 2)
# 解析多收一点原始行,输出前/后处理再压成「去重后候选池」
raw_cap = max(pool + 10, (pool * 3) // 2)
tech_raw_cap = max(tech_pool + 4, (tech_pool * 3) // 2) if tech_pool else 0
tech_clause = ""
if tech_pool:
tech_clause = (
f"\n另输出 **去重后** 约 **{tech_pool} 条** tech_items 候选(最终展示约 {tech_lim} 条),聚焦工程技术:"
"模型/框架发布、开源项目、芯片算力、开发者工具、推理与工程实践。"
"不得与 items 重复 link/同事件;输出前自行去重,候选池内每条应为独立事件。"
)
system = (
f"{skill}\n\n"
"当前执行 **早报 AI 时讯调研**。\n"
f"时间窗口:近 **{h}** 小时(截至 {now_cst.strftime('%Y-%m-%d %H:%M')} UTC+8\n"
f"输出 **同事件去重后** 约 **{pool} 条** items 候选(按重要性排序;最终展示约 {lim} 条)。\n"
"候选池条数 = 独立事件数:同一事件多源报道只留一条最权威源,禁止用换源重复充数。\n"
f"去重后的候选中国内可信源尽量不少于 **{cn_pool_target}** 条(展示侧至少 {cn_min} 条)。\n"
f"禁止用低质源凑数;可信独立事件不足才少返回。{tech_clause}\n"
"只采用官方博客/新闻稿、政府监管原文、一线权威媒体、学术官方;"
"禁止二手搬运、标题党、营销号。使用 WebSearch 检索;不要读取本项目文档或 RSS 配置。"
)
user = (
f"/deep-research 获取近 {h} 小时的 AI 人工智能新闻资讯,"
f"国内与国际合并items 去重后约 {pool} 条独立事件(国内可信尽量 ≥{cn_pool_target}"
"输出前完成同事件去重;可信度不足则不写。"
f"只输出 JSONitems 去重后目标约 {pool}"
+ (f"tech_items 去重后目标约 {tech_pool}" if tech_pool else "")
+ ""
)
try:
raw = cursor_agent_prompt(system, user)
except Exception as exc:
logger.warning("AI 时讯 research 失败:%s", exc)
return {
"enabled": False,
"mode": "research",
"items": [],
"tech_items": [],
"flat": [],
"stats": {"error": str(exc)},
}
if not raw:
return {
"enabled": False,
"mode": "research",
"items": [],
"tech_items": [],
"flat": [],
"stats": {"error": "empty_response"},
}
items, tech_items = parse_research_response(
raw,
limit=lim,
tech_limit=tech_lim,
pool_limit=raw_cap,
tech_pool_limit=tech_raw_cap,
)
payload = extract_json_object(raw)
if payload:
_save_research_json(research_json_path(date_str), payload)
if not items and not tech_items:
logger.warning("AI 时讯 research JSON 无效或无条目")
return {
"enabled": False,
"mode": "research",
"items": [],
"tech_items": [],
"flat": [],
"stats": {"error": "invalid_json"},
}
items, tech_items = post_process_research_news(
items,
tech_items,
limit=lim,
tech_limit=tech_lim,
min_cn=cn_min,
)
items = _apply_pushed_dedup(items, date_str=date_str, limit=lim)
if tech_items:
tech_items = _apply_pushed_dedup(tech_items, date_str=date_str, limit=tech_lim)
logger.info(
"AI 时讯 research 完成:%d 条 + %d 技术(候选池 %d/%d",
len(items),
len(tech_items),
pool,
tech_pool,
)
flat = [
{
"title": i["title"],
"link": i["link"],
"summary": i.get("summary_plain") or i.get("desc_short") or "",
"source_name": i["source_name"],
"published_fmt": i.get("published_fmt") or "",
"category_id": "research",
"category_name": "Deep Research",
"category_icon": "🔍",
}
for i in items + tech_items
]
return {
"enabled": True,
"mode": "research",
"hours": h,
"items": items,
"tech_items": tech_items,
"flat": flat,
"stats": {"source": "cursor_research", "items": len(items), "tech_items": len(tech_items)},
}
def format_research_news_section(
research: dict[str, Any],
*,
section_no: int,
wecom_limit: int | None = None,
) -> list[str]:
if not research.get("enabled"):
hint = research.get("stats", {}).get("error", "调研失败或未配置 CURSOR_API_KEY")
return [
"---",
"",
f"## {section_no}、AI 时讯精选Deep Research",
"",
f"*不可用:{hint}*",
"",
]
hours = research.get("hours", 24)
items = (research.get("flat") or [])[: wecom_limit or research_limit()]
lines = [
"---",
"",
f"## {section_no}、AI 时讯精选Deep Research",
"",
f"> 近 **{hours}h** · Cursor Agent WebSearch · {len(items)}",
"",
]
if not items:
lines.append("*暂无可用条目。*")
lines.append("")
return lines
for i, item in enumerate(items, 1):
pub = f" · {item['published_fmt']}" if item.get("published_fmt") else ""
lines.append(
f"{i}. **[{item['title']}]({item['link']})** · `{item['source_name']}`{pub}"
)
summary = item.get("summary") or ""
if summary:
lines.append(f" - {summary}")
lines.append("")
return lines

View File

@@ -0,0 +1,442 @@
"""Research 时讯后处理可信源、同事件去重、tech 主题过滤、国内配额。"""
from __future__ import annotations
import logging
import re
from typing import Any
from urllib.parse import urlparse
from daily.config import env_int
logger = logging.getLogger(__name__)
# 官方域(同事件去重时优先保留)
_OFFICIAL_HOST_SUFFIXES: tuple[str, ...] = (
"openai.com",
"anthropic.com",
"deepmind.google",
"blog.google",
"ai.googleblog.com",
"microsoft.com",
"meta.com",
"engineering.fb.com",
"nvidia.com",
"huggingface.co",
"arxiv.org",
"github.com",
"github.blog",
"modelcontextprotocol.io",
"cursor.com",
"vercel.com",
"langchain.dev",
"cohere.com",
"moonshot.cn",
"moonshot.ai",
"sktelecom.com",
"tether.io",
)
# 权威媒体 / 可信站(国际 + 国内)
_TRUSTED_HOST_SUFFIXES: tuple[str, ...] = _OFFICIAL_HOST_SUFFIXES + (
"techcrunch.com",
"theverge.com",
"wired.com",
"arstechnica.com",
"venturebeat.com",
"technologyreview.com",
"engadget.com",
"cnet.com",
"zdnet.com",
"axios.com",
"reuters.com",
"bloomberg.com",
"bloomberglaw.com",
"bbc.com",
"bbc.co.uk",
"nytimes.com",
"wsj.com",
"ft.com",
"nbcnews.com",
"time.com",
"theguardian.com",
"washingtonpost.com",
"theregister.com",
"nature.com",
"science.org",
"scmp.com",
"caixinglobal.com",
"caixin.com",
"qbitai.com",
"36kr.com",
"leiphone.com",
"jiqizhixin.com",
"ithome.com",
"tmtpost.com",
"huxiu.com",
"solidot.org",
"synched.cn",
"infoq.cn",
"yicai.com",
"news.cn",
"xinhuanet.com",
"people.com.cn",
"cls.cn",
"geekpark.net",
"standard.com",
"business-standard.com",
"siliconvalley.com",
)
_TRUSTED_SOURCE_NAMES: frozenset[str] = frozenset(
{
"techcrunch",
"the verge",
"wired",
"ars technica",
"engadget",
"reuters",
"bloomberg",
"bloomberg law",
"nbc news",
"time",
"the register",
"openai",
"anthropic",
"arxiv",
"hugging face",
"mcp blog",
"github",
"sk telecom",
"tether",
"量子位",
"36氪",
"36kr",
"雷锋网",
"机器之心",
"it之家",
"财新",
"caixin",
"钛媒体",
"虎嗅",
"第一财经",
"新华网",
"新华社",
"财联社",
"极客公园",
"人民日报",
}
)
_CN_HOST_SUFFIXES: tuple[str, ...] = (
"qbitai.com",
"36kr.com",
"leiphone.com",
"jiqizhixin.com",
"ithome.com",
"caixin.com",
"caixinglobal.com",
"tmtpost.com",
"huxiu.com",
"solidot.org",
"synched.cn",
"infoq.cn",
"moonshot.cn",
"yicai.com",
"news.cn",
"xinhuanet.com",
"people.com.cn",
"cls.cn",
"geekpark.net",
"zhihu.com",
"sina.com.cn",
"qq.com",
"163.com",
)
_CN_SOURCE_NAMES: frozenset[str] = frozenset(
{
"量子位",
"36氪",
"36kr",
"雷锋网",
"机器之心",
"it之家",
"财新",
"caixin",
"钛媒体",
"虎嗅",
"月之暗面",
"第一财经",
"新华网",
"新华社",
"财联社",
"极客公园",
"人民日报",
}
)
_ENTITIES: tuple[tuple[str, tuple[str, ...]], ...] = (
("openai", ("openai", "altman", "chatgpt", "奥特曼")),
("anthropic", ("anthropic", "amodei", "claude")),
("kimi", ("kimi", "moonshot", "月之暗面")),
("mcp", ("mcp", "model context protocol", "modelcontextprotocol")),
("nvidia", ("nvidia", "英伟达")),
("amd", ("amd",)),
("hugging_face", ("hugging face", "huggingface")),
("google", ("google", "deepmind", "gemini")),
("meta", ("meta", "llama")),
("microsoft", ("microsoft", "copilot")),
("huawei", ("huawei", "华为", "昇腾", "ascend")),
("moore", ("摩尔线程", "moore threads", "musa")),
)
_EVENT_CLUSTERS: tuple[tuple[str, tuple[str, ...]], ...] = (
("petition", ("petition", "联名", "decelerat", "pace ai", "控制", "减速")),
("hack", ("hack", "入侵", "siege", "breach", "攻击", "逃逸")),
("open_source", ("open-source", "opensource", "open sources", "开源", "open-sources")),
("adapt", ("适配", "adapt", "day-0", "day0", "day 0", "推理部署", "训练适配")),
("release", ("release", "发布", "specification", "规范", "v2.0", "changelog")),
("chip", ("chip", "芯片", "data center", "数据中心", "mi455")),
("regulate", ("framework", "监管", "voluntary", "审核", "ban", "禁止")),
)
_RELEASE_FAMILY = frozenset({"open_source", "adapt", "release"})
# 分发平台,不参与 tech↔items 主题冲突(避免 HF 上架与 HF 被黑误杀)
_PLATFORM_ENTITIES = frozenset({"hugging_face"})
def _host(link: str) -> str:
return urlparse(link).netloc.lower().removeprefix("www.")
def _ends_with_any(host: str, suffixes: tuple[str, ...]) -> bool:
return any(host == s or host.endswith("." + s) for s in suffixes)
def _norm_text(*parts: str) -> str:
text = " ".join(p for p in parts if p).lower()
text = re.sub(r"[\s\-_/|·,。、::()【】\[\]]+", " ", text)
return text.strip()
def _item_text(item: dict[str, Any]) -> str:
return _norm_text(
str(item.get("title") or ""),
str(item.get("desc_short") or item.get("summary_plain") or ""),
)
def _match_labels(text: str, table: tuple[tuple[str, tuple[str, ...]], ...]) -> frozenset[str]:
hit: set[str] = set()
for label, kws in table:
if any(kw in text for kw in kws):
hit.add(label)
return frozenset(hit)
def entities_of(item: dict[str, Any]) -> frozenset[str]:
return _match_labels(_item_text(item), _ENTITIES)
def events_of(item: dict[str, Any]) -> frozenset[str]:
return _match_labels(_item_text(item), _EVENT_CLUSTERS)
def event_key(item: dict[str, Any]) -> tuple[frozenset[str], frozenset[str]] | None:
ents = entities_of(item)
evs = events_of(item)
if not ents or not evs:
return None
return (ents, evs)
def same_event(a: dict[str, Any], b: dict[str, Any]) -> bool:
"""共享至少一实体且共享至少一事件簇 → 同事件(保守合并)。"""
ea, eva = entities_of(a), events_of(a)
eb, evb = entities_of(b), events_of(b)
if not ea or not eb or not eva or not evb:
return False
return bool(ea & eb) and bool(eva & evb)
def is_official_item(item: dict[str, Any]) -> bool:
return _ends_with_any(_host(str(item.get("link") or "")), _OFFICIAL_HOST_SUFFIXES)
def _is_institutional_cn_host(host: str) -> bool:
"""新华社 / 政府站等机构域,默认可信。"""
if host.endswith(".gov.cn") or host.endswith(".gov.cn."):
return True
if host == "news.cn" or host.endswith(".news.cn"):
return True
if host.endswith("xinhuanet.com") or host.endswith("people.com.cn"):
return True
return False
def is_trusted_item(item: dict[str, Any]) -> bool:
host = _host(str(item.get("link") or ""))
if _is_institutional_cn_host(host):
return True
if _ends_with_any(host, _TRUSTED_HOST_SUFFIXES):
return True
name = str(item.get("source_name") or "").strip().lower()
return name in _TRUSTED_SOURCE_NAMES
def is_cn_item(item: dict[str, Any]) -> bool:
region = str(item.get("region") or "").strip().lower()
if region in {"cn", "china", "zh", "zh-cn"}:
return True
if region in {"intl", "international", "global", "en"}:
return False
host = _host(str(item.get("link") or ""))
if host.endswith(".cn") or _ends_with_any(host, _CN_HOST_SUFFIXES):
return True
name = str(item.get("source_name") or "").strip().lower()
return name in {n.lower() for n in _CN_SOURCE_NAMES}
def research_cn_min(limit: int) -> int:
explicit = env_int("DAILY_WECOM_AI_NEWS_CN_MIN", 0)
if explicit > 0:
return min(explicit, max(1, limit))
return max(1, limit * 3 // 10)
def _trust_rank(item: dict[str, Any]) -> int:
if is_official_item(item):
return 0
if is_trusted_item(item):
return 1
return 2
def filter_trusted(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
kept = [i for i in items if is_trusted_item(i)]
dropped = len(items) - len(kept)
if dropped:
logger.info("news_dedup_drop:trusted=%s", dropped)
return kept
def dedupe_same_event(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""同事件只留一条;官方源优先,否则保留更靠前的。"""
kept: list[dict[str, Any]] = []
for item in items:
replaced = False
for idx, prev in enumerate(kept):
if not same_event(item, prev):
continue
if _trust_rank(item) < _trust_rank(prev):
kept[idx] = item
replaced = True
break
if not replaced:
kept.append(item)
dropped = len(items) - len(kept)
if dropped:
logger.info("news_dedup_drop:same_event=%s", dropped)
return kept
def filter_tech_against_items(
tech_items: list[dict[str, Any]],
items: list[dict[str, Any]],
) -> list[dict[str, Any]]:
kept: list[dict[str, Any]] = []
for tech in tech_items:
t_ents = entities_of(tech)
t_evs = events_of(tech)
conflict = False
for item in items:
if same_event(tech, item):
conflict = True
break
shared = (t_ents & entities_of(item)) - _PLATFORM_ENTITIES
if shared and ((t_evs | events_of(item)) & _RELEASE_FAMILY):
conflict = True
break
if not conflict:
kept.append(tech)
dropped = len(tech_items) - len(kept)
if dropped:
logger.info("news_dedup_drop:tech_topic=%s", dropped)
return kept
def pack_with_cn_quota(
items: list[dict[str, Any]],
*,
limit: int,
min_cn: int,
) -> list[dict[str, Any]]:
if limit <= 0:
return []
min_cn = max(0, min(min_cn, limit))
out: list[dict[str, Any]] = []
used: set[str] = set()
cn_got = 0
def _take(item: dict[str, Any]) -> None:
nonlocal cn_got
link = str(item.get("link") or "")
if not link or link in used:
return
out.append(item)
used.add(link)
if is_cn_item(item):
cn_got += 1
for item in items:
if len(out) >= limit:
break
link = str(item.get("link") or "")
if not link or link in used:
continue
slots_left = limit - len(out)
need_cn = max(0, min_cn - cn_got)
if not is_cn_item(item) and slots_left <= need_cn:
continue
_take(item)
if len(out) < limit:
for item in items:
if len(out) >= limit:
break
_take(item)
final_cn = sum(1 for i in out if is_cn_item(i))
if final_cn < min_cn:
logger.info("news_cn_short:%s", final_cn)
return out[:limit]
def build_deduped_candidate_pool(
items: list[dict[str, Any]],
tech_items: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""拉取后立即得到去重候选池:可信过滤 + 同事件去重 + tech 相对 items 主题过滤。"""
items = dedupe_same_event(filter_trusted(items))
tech_items = dedupe_same_event(filter_trusted(tech_items))
tech_items = filter_tech_against_items(tech_items, items)
logger.info("research_pool_deduped:items=%s tech=%s", len(items), len(tech_items))
return items, tech_items
def post_process_research_news(
items: list[dict[str, Any]],
tech_items: list[dict[str, Any]],
*,
limit: int,
tech_limit: int,
min_cn: int | None = None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""去重候选池 → 国内配额打包 → 截到展示上限。"""
cn_min = research_cn_min(limit) if min_cn is None else max(0, min_cn)
items, tech_items = build_deduped_candidate_pool(items, tech_items)
items = pack_with_cn_quota(items, limit=limit, min_cn=cn_min)
tech_items = filter_tech_against_items(tech_items, items)
return items, tech_items[: max(0, tech_limit)]

18
daily/news/sanitize.py Normal file
View File

@@ -0,0 +1,18 @@
"""新闻文案清洗:剥离「放宽窗口」类凑数前缀。"""
from __future__ import annotations
import re
_RELAX_PREFIX = re.compile(
r"^(?:放宽窗口|放宽至[^:]*)\s*[:]\s*",
re.UNICODE,
)
def strip_relax_window_prefix(text: str) -> str:
"""去掉开头的「放宽窗口:」/「放宽至…:」前缀。"""
raw = (text or "").strip()
if not raw:
return ""
return _RELAX_PREFIX.sub("", raw, count=1).strip()

View File

@@ -1 +0,0 @@
"""早报生成流水线子模块。"""

View File

@@ -1,145 +0,0 @@
"""抓取与结构化输入组装。"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime
from typing import Any
from daily.agent_workflow import is_agent_mode
from daily.config import env_int
from daily.delta import compare_depth
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos
from daily.github.trending import fetch_github_trending
from daily.news.fetch import fetch_ai_news, fetch_cn_ai_news
from daily.news.rank import apply_news_ranking
from daily.report_data import build_llm_input
from daily.skills_board import load_boards
from shared.skills_data import load_feed
@dataclass
class ReportLimits:
trending_n: int
hot_n: int
skill_pool: int
wecom_trending: int
wecom_hot: int
github_limit: int
wecom_github: int
emerging_limit: int
wecom_emerging: int
topic_limit: int
wecom_topic: int
@dataclass
class ReportContext:
feed: dict[str, Any]
date_str: str
time_str: str
updated: str
trending: list[dict[str, Any]]
hot: list[dict[str, Any]]
github_trending: list[dict[str, Any]]
github_emerging: list[dict[str, Any]]
github_topic: list[dict[str, Any]]
topic_name: str
ai_news: dict[str, Any]
cn_ai_news: dict[str, Any]
limits: ReportLimits
wecom_limits: dict[str, int]
llm_input: dict[str, Any]
def resolve_limits() -> ReportLimits:
compare_n = compare_depth()
trending_n = env_int("DAILY_TRENDING_LIMIT", 150)
hot_n = max(env_int("DAILY_HOT_LIMIT", 150), compare_n)
skill_pool = max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
wecom_trending = env_int("DAILY_WECOM_TRENDING", 10)
wecom_hot = env_int("DAILY_WECOM_HOT", 10)
github_limit = env_int("DAILY_GITHUB_TRENDING_LIMIT", 10)
wecom_github = env_int("DAILY_WECOM_GITHUB_TRENDING", env_int("DAILY_WECOM_REPOS", 10))
emerging_limit = env_int("DAILY_GITHUB_EMERGING_LIMIT", 10)
wecom_emerging = env_int("DAILY_WECOM_GITHUB_EMERGING", 10)
topic_limit = env_int("DAILY_GITHUB_TOPIC_LIMIT", 10)
wecom_topic = env_int("DAILY_WECOM_GITHUB_TOPIC", 10)
return ReportLimits(
trending_n=trending_n,
hot_n=hot_n,
skill_pool=skill_pool,
wecom_trending=wecom_trending,
wecom_hot=wecom_hot,
github_limit=github_limit,
wecom_github=wecom_github,
emerging_limit=emerging_limit,
wecom_emerging=wecom_emerging,
topic_limit=topic_limit,
wecom_topic=wecom_topic,
)
def collect_report_context(now: datetime) -> ReportContext:
limits = resolve_limits()
compare_n = compare_depth()
github_fetch_n = max(limits.github_limit, compare_n, limits.wecom_github)
emerging_fetch_n = max(limits.emerging_limit, compare_n, limits.wecom_emerging)
topic_fetch_n = max(limits.topic_limit, compare_n, limits.wecom_topic)
feed = load_feed(force=True)
date_str = now.strftime("%Y-%m-%d")
time_str = now.strftime("%H:%M") + " (UTC+8)"
updated = (feed.get("updatedAt") or "")[:10]
trending, hot = load_boards(feed, trending_limit=limits.trending_n, hot_limit=limits.hot_n)
github_trending = fetch_github_trending(github_fetch_n)
seen_repos = {r["repo"] for r in github_trending}
github_emerging = fetch_emerging_repos(emerging_fetch_n, exclude=seen_repos)
seen_repos.update(r["repo"] for r in github_emerging)
topic_name, github_topic = fetch_topic_hot_repos(topic_fetch_n, exclude=seen_repos)
ai_news = apply_news_ranking(fetch_ai_news(), date_str=date_str)
cn_ai_news = apply_news_ranking(fetch_cn_ai_news(), date_str=date_str)
wecom_limits = {
"trending": limits.wecom_trending,
"hot": limits.wecom_hot,
"trending_pool": limits.skill_pool,
"hot_pool": limits.skill_pool,
"github": limits.wecom_github,
"emerging": limits.wecom_emerging,
"topic": limits.wecom_topic,
"ai_news": env_int("DAILY_WECOM_AI_NEWS", 10),
"cn_ai_news": env_int("DAILY_WECOM_CN_AI_NEWS", 8),
}
llm_input = build_llm_input(
date_str=date_str,
updated=updated,
trending=trending,
hot=hot,
github_trending=github_trending,
github_emerging=github_emerging,
github_topic=github_topic,
topic_name=topic_name,
ai_news=ai_news,
cn_ai_news=cn_ai_news,
wecom_limits=wecom_limits,
agent_mode=is_agent_mode(),
)
return ReportContext(
feed=feed,
date_str=date_str,
time_str=time_str,
updated=updated,
trending=trending,
hot=hot,
github_trending=github_trending,
github_emerging=github_emerging,
github_topic=github_topic,
topic_name=topic_name,
ai_news=ai_news,
cn_ai_news=cn_ai_news,
limits=limits,
wecom_limits=wecom_limits,
llm_input=llm_input,
)

View File

@@ -1,180 +0,0 @@
"""归档 / 企微格式化与摘要构建。"""
from __future__ import annotations
import re
import xml.etree.ElementTree as ET
from typing import Any
import certifi
import httpx
from daily.config import full_desc_limit, wecom_skill_desc_limit
from daily.github.auth import github_html_headers
from daily.news.fetch import prepare_wecom_cn_news_items, prepare_wecom_news_items
from shared.skills_data import format_installs
from daily.pipeline.snapshot import skill_id
def short_desc(text: str, limit: int = 72) -> str:
text = re.sub(r"\s+", " ", text or "").strip()
if limit <= 0 or len(text) <= limit:
return text
return text[: limit - 3] + "..."
def archive_desc(text: str) -> str:
return short_desc(text, full_desc_limit())
def wecom_desc(text: str, limit: int = 36) -> str:
return short_desc(text, limit)
def prepare_skill_item(item: dict[str, Any], prev_ids: set[str], rank: int) -> dict[str, Any]:
badge = ""
sid = skill_id(item)
if sid not in prev_ids and prev_ids:
badge = "🆕"
elif rank == 1:
badge = "👑"
installs_fmt = item.get("installs_fmt") or format_installs(item.get("installs", 0))
title = item.get("source", "?") if item.get("cluster") else item.get("title", "?")
desc = item.get("wecom_desc") or item.get("description") or item.get("cluster_titles") or ""
limit = wecom_skill_desc_limit()
desc_short = desc if item.get("wecom_desc") or limit <= 0 else wecom_desc(desc, limit)
return {
"title": title,
"source": item.get("source", "?"),
"installs_fmt": installs_fmt,
"link": item.get("link", ""),
"desc_short": desc_short,
"badge": badge,
"cluster": bool(item.get("cluster")),
"cluster_count": item.get("cluster_count"),
"cluster_titles": item.get("cluster_titles"),
}
def prepare_github_item(item: dict[str, Any]) -> dict[str, Any]:
return {**item, "desc_short": wecom_desc(item.get("description", ""), 40)}
def build_highlights(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
github_trending: list[dict[str, Any]],
github_emerging: list[dict[str, Any]],
ai_news: dict[str, Any] | None = None,
cn_ai_news: dict[str, Any] | None = None,
) -> list[str]:
points: list[str] = []
if ai_news and ai_news.get("enabled"):
top_news = prepare_wecom_news_items(ai_news)
if top_news:
n0 = top_news[0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🌍 AI 时讯 [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
elif ai_news.get("flat"):
n0 = ai_news["flat"][0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(f"🌍 AI 时讯 [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}")
if cn_ai_news and cn_ai_news.get("enabled"):
top_cn = prepare_wecom_cn_news_items(cn_ai_news)
if top_cn:
n0 = top_cn[0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🇨🇳 国内 AI [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
elif cn_ai_news.get("flat"):
n0 = cn_ai_news["flat"][0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🇨🇳 国内 AI [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
if trending:
t0 = trending[0]
points.append(f"📈 Skills 榜首 **{t0.get('title')}**{format_installs(t0.get('installs', 0))}")
if github_trending:
g0 = github_trending[0]
stars = g0.get("stars_today_fmt", "")
total = g0.get("total_stars_fmt", "")
star_hint = f"+{stars} today · " if stars else (f"{total} · " if total else "")
points.append(f"🐙 GitHub Trending [{g0['repo']}]({g0['url']}){star_hint}{g0.get('language', '')}")
if github_emerging:
e0 = github_emerging[0]
points.append(f"🌱 新兴 [{e0['repo']}]({e0['url']})(⭐ {e0.get('total_stars_fmt', '?')}")
elif hot:
h0 = hot[0]
points.append(f"🔥 Skills Hot 榜首 **{h0.get('title')}**1H {format_installs(h0.get('installs', 0))}")
while len(points) < 3 and len(trending) > len(points):
item = trending[len(points)]
points.append(f"✨ **{item.get('title')}** · `{item.get('source')}`")
return points[:3]
def format_github_repo_section(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
lines: list[str] = []
for i, repo in enumerate(repos, 1):
lang = repo.get("language") or ""
stars_today = repo.get("stars_today_fmt") or ""
total = repo.get("total_stars_fmt") or ""
created = repo.get("created_at") or ""
meta_parts = [lang]
if stars_today:
meta_parts.append(f"+{stars_today} today")
if total:
meta_parts.append(f"总 ⭐ {total}")
if show_created and created:
meta_parts.append(f"创建于 {created}")
lines.append(f"{i}. **[{repo['repo']}]({repo['url']})** · {' · '.join(meta_parts)}")
desc = archive_desc(repo.get("description", ""))
if desc:
lines.append(f" - {desc}")
lines.append("")
return lines
def format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) -> list[str]:
lines: list[str] = []
for i, item in enumerate(items, 1):
sid = item.get("id") or f"{item.get('source', '?')}/{item.get('title', '?')}"
link = item.get("link", "")
installs = format_installs(item.get("installs", 0))
meta = f"1H {installs}" if hot else f"总安装 {installs}"
if link:
lines.append(f"{i}. **[{sid}]({link})** · {meta}")
else:
lines.append(f"{i}. **{sid}** · {meta}")
desc = archive_desc(item.get("description", ""))
if desc:
lines.append(f" - {desc}")
lines.append("")
return lines
def fetch_latest_release_title(repo: str) -> str | None:
atom_url = f"https://github.com/{repo}/releases.atom"
try:
with httpx.Client(
timeout=12.0,
verify=certifi.where(),
follow_redirects=True,
headers=github_html_headers(),
) as client:
resp = client.get(atom_url)
if resp.status_code != 200:
return None
root = ET.fromstring(resp.text)
ns = {"a": "http://www.w3.org/2005/Atom"}
entry = root.find("a:entry", ns)
if entry is None:
return None
title = entry.find("a:title", ns)
return title.text.strip() if title is not None and title.text else None
except Exception:
return None

View File

@@ -1,116 +0,0 @@
"""归档内容中文化。"""
from __future__ import annotations
from typing import Any
from daily.config import full_desc_limit, news_summary_limit
from daily.localize import LocalizeJob, localize_descriptions, needs_chinese
from daily.pipeline.snapshot import skill_id
def localize_descriptions_in_place(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
github_trending: list[dict[str, Any]],
github_emerging: list[dict[str, Any]],
github_topic: list[dict[str, Any]],
ai_news: dict[str, Any],
cn_ai_news: dict[str, Any] | None = None,
) -> None:
full_limit = full_desc_limit()
news_limit = news_summary_limit()
jobs: list[LocalizeJob] = []
seen_skill: set[str] = set()
for item in trending + hot:
sid = skill_id(item)
if sid in seen_skill:
continue
seen_skill.add(sid)
desc = (item.get("description") or "").strip()
if desc:
jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
seen_repo: set[str] = set()
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
repo = item.get("repo", "")
if not repo or repo in seen_repo:
continue
seen_repo.add(repo)
desc = (item.get("description") or "").strip()
if desc:
jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
if ai_news.get("enabled"):
seen_news: set[str] = set()
for item in ai_news.get("flat") or []:
link = item.get("link", "")
if not link or link in seen_news:
continue
seen_news.add(link)
summary = (item.get("summary") or "").strip()
if summary:
jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
zh_map = localize_descriptions(jobs, archive=True)
if not zh_map and not jobs:
return
def _apply_zh(mapping: dict[str, str]) -> None:
for item in trending + hot:
key = f"skill:{skill_id(item)}"
if key in mapping:
item["description"] = mapping[key]
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
key = f"github:{item.get('repo', '')}"
if key in mapping:
item["description"] = mapping[key]
if ai_news.get("enabled"):
for cat in ai_news.get("categories") or []:
for item in cat.get("items") or []:
key = f"news:{item.get('link', '')}"
if key in mapping:
item["summary"] = mapping[key]
for item in ai_news.get("flat") or []:
key = f"news:{item.get('link', '')}"
if key in mapping:
item["summary"] = mapping[key]
_apply_zh(zh_map)
retry_jobs: list[LocalizeJob] = []
seen_skill.clear()
for item in trending + hot:
sid = skill_id(item)
if sid in seen_skill:
continue
seen_skill.add(sid)
desc = (item.get("description") or "").strip()
if needs_chinese(desc):
retry_jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
seen_repo.clear()
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
repo = item.get("repo", "")
if not repo or repo in seen_repo:
continue
seen_repo.add(repo)
desc = (item.get("description") or "").strip()
if needs_chinese(desc):
retry_jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
if ai_news.get("enabled"):
seen_news.clear()
for item in ai_news.get("flat") or []:
link = item.get("link", "")
if not link or link in seen_news:
continue
seen_news.add(link)
summary = (item.get("summary") or "").strip()
if needs_chinese(summary):
retry_jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
if retry_jobs:
_apply_zh(localize_descriptions(retry_jobs, archive=True))

View File

@@ -1,36 +0,0 @@
"""Skills 快照读写。"""
from __future__ import annotations
import json
from typing import Any
from daily.config import CACHE_DIR, SNAPSHOT_FILE
def skill_id(item: dict[str, Any]) -> str:
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
def load_snapshot() -> set[str]:
if not SNAPSHOT_FILE.exists():
return set()
try:
data = json.loads(SNAPSHOT_FILE.read_text(encoding="utf-8"))
return set(str(x) for x in (data.get("skill_ids") or []))
except (OSError, json.JSONDecodeError):
return set()
def save_snapshot(feed: dict[str, Any], date_str: str) -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
ids: list[str] = []
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
sid = skill_id(item)
if sid not in ids:
ids.append(sid)
SNAPSHOT_FILE.write_text(
json.dumps({"date": date_str, "skill_ids": ids}, ensure_ascii=False, indent=2),
encoding="utf-8",
)

View File

@@ -1,53 +0,0 @@
"""主题检测与聚类。"""
from __future__ import annotations
from collections import defaultdict
from typing import Any
THEME_RULES: list[tuple[str, str, list[str]]] = [
("🎬", "AI 多媒体 / 视频", ["runcomfy", "remotion", "video", "seedance", "inpaint", "lipsync"]),
("🔧", "工程协作 / Skill 元能力", ["grill", "tdd", "architecture", "find-skills", "to-issues"]),
("📱", "飞书 / Lark", ["lark", "feishu"]),
("📣", "内容营销", ["viral", "tiktok", "instagram", "reels"]),
("🎨", "设计 / 前端", ["frontend", "design", "ui-ux", "tailwind"]),
]
def detect_theme_line(feed: dict[str, Any]) -> str:
scores: dict[str, int] = defaultdict(int)
for board in ("topTrending", "topHot"):
for rank, item in enumerate(feed.get(board, [])[:10], 1):
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, label, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
scores[label] += max(1, 11 - rank)
break
if not scores:
return "**今日主题**Agent Skills 生态持续活跃"
return f"**今日主题**{max(scores.items(), key=lambda x: x[1])[0]}"
def theme_clusters(feed: dict[str, Any], limit: int = 5) -> list[tuple[str, list[str]]]:
from daily.pipeline.snapshot import skill_id
buckets: dict[str, list[str]] = defaultdict(list)
seen: set[str] = set()
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
item_id = skill_id(item)
if item_id in seen:
continue
seen.add(item_id)
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, theme, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
label = f"**{item.get('title')}** (`{item.get('source')}`)"
if label not in buckets[theme]:
buckets[theme].append(label)
break
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]

61
daily/push_gate.py Normal file
View File

@@ -0,0 +1,61 @@
"""企微早报推送闸门。"""
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from typing import Any
from daily.config import force_push, skip_push_when_silent
logger = logging.getLogger(__name__)
@dataclass
class PushGateResult:
should_push: bool
reasons: list[str] = field(default_factory=list)
silent: bool = False
def _has_board_moves(movement: dict[str, Any]) -> bool:
keys = (
"skills_trending_moves",
"skills_hot_moves",
"github_trending_moves",
"github_emerging_moves",
"github_topic_moves",
)
return any(movement.get(k) for k in keys)
def evaluate_push_gate(
*,
movement: dict[str, Any],
ai_news_items: list[dict[str, Any]],
cn_ai_news_items: list[dict[str, Any]],
featured_pick: dict[str, Any] | None,
) -> PushGateResult:
if force_push():
logger.info("push_gate: force_push=on, 强制推送")
return PushGateResult(should_push=True, reasons=["force_push"], silent=False)
reasons: list[str] = []
if _has_board_moves(movement):
reasons.append("board_moves")
if ai_news_items:
reasons.append("ai_news")
if cn_ai_news_items:
reasons.append("cn_ai_news")
if featured_pick:
reasons.append("featured_pick")
should = bool(reasons)
silent = not should and skip_push_when_silent()
if should:
logger.info("push_gate: 推送 (原因: %s)", ",".join(reasons))
elif silent:
logger.info("push_gate: 静默日, 跳过推送 (无任何更新信号)")
else:
logger.info("push_gate: 无更新但 skip_push_when_silent=off, 仍推送")
return PushGateResult(should_push=should, reasons=reasons, silent=silent)

View File

@@ -6,10 +6,11 @@ import json
from pathlib import Path
from typing import Any
from daily.config import OUTPUT_DIR, env_int
from daily.delta import build_movement_baseline, build_movement_context, compare_depth
from daily.config import OUTPUT_DIR, env_int, wecom_mode
from daily.delta import build_movement_baseline, build_movement_context, compare_depth, effective_wecom_mode
from daily.news.fetch import prepare_wecom_cn_news_items, prepare_wecom_news_items
from daily.skills_group import group_skills_by_source
from daily.text_utils import trim_brief
def skill_id(item: dict[str, Any]) -> str:
@@ -59,51 +60,41 @@ def _slim_news_items(
limit: int,
*,
prepare=prepare_wecom_news_items,
date_str: str | None = None,
) -> list[dict[str, Any]]:
items: list[dict[str, Any]] = []
for item in prepare(ai_news, limit=limit):
payload = {
for item in prepare(ai_news, date_str=date_str):
items.append(
{
"link": item.get("link", ""),
"title": item.get("title", ""),
"source_name": item.get("source_name", ""),
"published_fmt": item.get("published_fmt", ""),
"summary": item.get("desc_short") or "",
"summary": trim_brief(
item.get("summary_plain") or item.get("desc_short") or "",
120,
),
}
if item.get("score") is not None:
payload["score"] = item.get("score")
items.append(payload)
)
if len(items) >= limit:
break
if items:
return items
flat = sorted(
ai_news.get("flat") or [],
key=lambda row: float(row.get("score") or 0),
reverse=True,
)
for item in flat[:limit]:
payload = {
for item in (ai_news.get("flat") or [])[:limit]:
items.append(
{
"link": item.get("link", ""),
"title": item.get("title", ""),
"source_name": item.get("source_name", ""),
"published_fmt": item.get("published_fmt", ""),
"summary": item.get("summary", ""),
}
if item.get("score") is not None:
payload["score"] = item.get("score")
items.append(payload)
)
return items
def _news_pool_limit(wecom_limit: int, *, env_key: str, default_pool: int, agent_mode: bool) -> int:
if not agent_mode:
return wecom_limit
pool = env_int(env_key, default_pool)
return max(wecom_limit, pool)
def _wecom_skill_pool() -> int:
return max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
return max(10, env_int("DAILY_WECOM_SKILL_POOL", 400))
def build_llm_input(
@@ -119,21 +110,46 @@ def build_llm_input(
ai_news: dict[str, Any],
cn_ai_news: dict[str, Any],
wecom_limits: dict[str, int],
agent_mode: bool = False,
research_items: list[dict[str, Any]] | None = None,
research_tech_items: list[dict[str, Any]] | None = None,
boards_for_wecom: dict[str, list[dict[str, Any]]] | None = None,
) -> dict[str, Any]:
"""供 Cursor 编辑的精简 JSON不含完整 markdown"""
news_limit = _news_pool_limit(
wecom_limits.get("ai_news", 10),
env_key="DAILY_AGENT_NEWS_POOL",
default_pool=40,
agent_mode=agent_mode,
)
cn_news_limit = _news_pool_limit(
wecom_limits.get("cn_ai_news", 8),
env_key="DAILY_AGENT_CN_NEWS_POOL",
default_pool=30,
agent_mode=agent_mode,
news_limit = wecom_limits.get("ai_news", 10)
cn_news_limit = wecom_limits.get("cn_ai_news", 8)
if research_items is not None:
slim_research = [
{
"link": item.get("link", ""),
"title": item.get("title", ""),
"source_name": item.get("source_name", ""),
"published_fmt": item.get("published_fmt", ""),
"summary": trim_brief(item.get("desc_short") or "", 120),
}
for item in research_items[:news_limit]
]
ai_news_payload = slim_research
tech_news_payload = [
{
"link": item.get("link", ""),
"title": item.get("title", ""),
"source_name": item.get("source_name", ""),
"published_fmt": item.get("published_fmt", ""),
"summary": trim_brief(item.get("desc_short") or "", 120),
}
for item in (research_tech_items or [])
]
cn_news_payload: list[dict[str, Any]] = []
ai_news_mode = "research"
else:
ai_news_payload = _slim_news_items(ai_news, news_limit, date_str=date_str) if ai_news.get("enabled") else []
cn_news_payload = (
_slim_news_items(cn_ai_news, cn_news_limit, prepare=prepare_wecom_cn_news_items, date_str=date_str)
if cn_ai_news.get("enabled")
else []
)
ai_news_mode = "rss"
tech_news_payload: list[dict[str, Any]] = []
depth = compare_depth()
trend_cmp = trending[:depth]
hot_cmp = hot[:depth]
@@ -141,6 +157,13 @@ def build_llm_input(
emerging_cmp = github_emerging[:depth]
topic_cmp = github_topic[:depth]
if boards_for_wecom:
trending_slice = boards_for_wecom.get("skills_trending") or []
hot_slice = boards_for_wecom.get("skills_hot") or []
github_slice = boards_for_wecom.get("github_trending") or []
emerging_slice = boards_for_wecom.get("github_emerging") or []
topic_slice = boards_for_wecom.get("github_topic") or []
else:
trending_slice = group_skills_by_source(
trending,
limit=wecom_limits.get("trending", 10),
@@ -172,10 +195,13 @@ def build_llm_input(
github_topic=[_slim_github(x) for x in topic_cmp],
depth=depth,
)
eff_mode = effective_wecom_mode(date_str=date_str)
return {
"date": date_str,
"data_updated": updated,
"wecom_mode": wecom_mode(),
"effective_wecom_mode": eff_mode,
"skills_trending": [_slim_skill(x) for x in trending_slice],
"skills_hot": [_slim_skill(x) for x in hot_slice],
"github_trending": [_slim_github(x) for x in github_slice],
@@ -184,12 +210,10 @@ def build_llm_input(
"topic": topic_name,
"repos": [_slim_github(x) for x in topic_slice],
},
"ai_news": _slim_news_items(ai_news, news_limit) if ai_news.get("enabled") else [],
"cn_ai_news": _slim_news_items(
cn_ai_news, cn_news_limit, prepare=prepare_wecom_cn_news_items
)
if cn_ai_news.get("enabled")
else [],
"ai_news": ai_news_payload,
"tech_ai_news": tech_news_payload,
"cn_ai_news": cn_news_payload,
"ai_news_mode": ai_news_mode,
"movement": movement,
"movement_baseline": movement_baseline,
}
@@ -211,6 +235,10 @@ def editorial_json_path(date_str: str) -> Path:
return OUTPUT_DIR / f"{date_str}.editorial.json"
def featured_json_path(date_str: str) -> Path:
return OUTPUT_DIR / f"{date_str}.featured.json"
def save_json(path: Path, data: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")

307
daily/scheduler.py Normal file
View File

@@ -0,0 +1,307 @@
"""常驻调度:按配置时刻生成早报并推送企微。"""
from __future__ import annotations
import json
import logging
import subprocess
import sys
import time
from dataclasses import dataclass
from datetime import date, datetime, time as dt_time, timedelta
from pathlib import Path
from zoneinfo import ZoneInfo
from daily.config import (
CACHE_DIR,
LOG_DIR,
OUTPUT_DIR,
ROOT,
schedule_generate_at,
schedule_push_at,
schedule_timezone_name,
workday_only,
)
from daily.holiday import is_workday, load_holidays, workday_name
logger = logging.getLogger(__name__)
_STATE_FILE = CACHE_DIR / "scheduler-state.json"
_POLL_SECONDS = 15
@dataclass(frozen=True)
class ClockTime:
hour: int
minute: int
@dataclass
class SchedulerState:
last_generate_date: str | None = None
last_push_date: str | None = None
@classmethod
def load(cls) -> SchedulerState:
if not _STATE_FILE.exists():
return cls()
try:
raw = json.loads(_STATE_FILE.read_text(encoding="utf-8"))
except (OSError, ValueError):
return cls()
if not isinstance(raw, dict):
return cls()
return cls(
last_generate_date=raw.get("last_generate_date"),
last_push_date=raw.get("last_push_date"),
)
def save(self) -> None:
_STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
payload = {
"last_generate_date": self.last_generate_date,
"last_push_date": self.last_push_date,
}
_STATE_FILE.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
def parse_hhmm(value: str) -> ClockTime:
raw = (value or "").strip()
parts = raw.split(":", 1)
if len(parts) != 2:
raise ValueError(f"无效时间格式: {value!r},应为 HH:MM")
hour = int(parts[0])
minute = int(parts[1])
if not (0 <= hour <= 23 and 0 <= minute <= 59):
raise ValueError(f"无效时间: {value!r}")
return ClockTime(hour=hour, minute=minute)
def load_timezone() -> ZoneInfo:
name = schedule_timezone_name()
try:
return ZoneInfo(name)
except Exception as exc:
raise RuntimeError(f"无效时区 DAILY_SCHEDULE_TZ={name!r}") from exc
def _localize(day: date, clock: ClockTime, tz: ZoneInfo) -> datetime:
return datetime.combine(day, dt_time(clock.hour, clock.minute), tz)
def next_occurrence_after(clock: ClockTime, tz: ZoneInfo, after: datetime) -> datetime:
local = after.astimezone(tz)
candidate = local.replace(hour=clock.hour, minute=clock.minute, second=0, microsecond=0)
if candidate <= local:
candidate += timedelta(days=1)
return candidate
def _today_slot(day: date, clock: ClockTime, tz: ZoneInfo) -> datetime:
return _localize(day, clock, tz)
def _run_daily_subcommand(subcmd: str, *extra: str) -> int:
cmd = [sys.executable, "-m", "daily", subcmd, *extra]
logger.info("执行: %s", " ".join(cmd))
proc = subprocess.run(cmd, cwd=str(ROOT), check=False)
return int(proc.returncode)
def run_generate() -> int:
return _run_daily_subcommand("generate")
def run_push_for_date(date_str: str) -> int:
report = OUTPUT_DIR / f"{date_str}.wecom.md"
if not report.exists():
logger.error("推送失败:报告不存在 %s", report)
return 1
return _run_daily_subcommand("push", str(report))
def _setup_logging() -> Path:
LOG_DIR.mkdir(parents=True, exist_ok=True)
log_path = LOG_DIR / "scheduler.log"
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
handlers=[
logging.FileHandler(log_path, encoding="utf-8"),
logging.StreamHandler(sys.stdout),
],
)
return log_path
def _sleep_until(target: datetime, tz: ZoneInfo) -> None:
while True:
now = datetime.now(tz)
seconds = (target - now).total_seconds()
if seconds <= 0:
return
time.sleep(min(seconds, _POLL_SECONDS))
def plan_next_action(
*,
now: datetime,
tz: ZoneInfo,
state: SchedulerState,
generate_at: ClockTime,
push_at: ClockTime,
) -> tuple[datetime, str] | None:
"""返回下一次应执行的动作;若今日已全部完成则返回明日 generate。"""
today = now.astimezone(tz).date()
today_str = today.isoformat()
gen_done = state.last_generate_date == today_str
push_done = state.last_push_date == today_str
gen_slot = _today_slot(today, generate_at, tz)
push_slot = _today_slot(today, push_at, tz)
# 推送窗口内generate 未做则立即补跑(须先于 push
if not gen_done and gen_slot <= now <= push_slot:
return now, "generate"
# 已过推送时刻:仅当 generate 已完成时补跑 push
if not push_done and gen_done and now >= push_slot:
return now, "push"
candidates: list[tuple[datetime, str]] = []
if not gen_done and gen_slot > now:
candidates.append((gen_slot, "generate"))
if not push_done and push_slot > now:
candidates.append((push_slot, "push"))
if candidates:
return min(candidates, key=lambda item: item[0])
tomorrow_gen = next_occurrence_after(generate_at, tz, now)
return tomorrow_gen, "generate"
def run_scheduled_action(action: str, *, today_str: str) -> int:
if action == "generate":
return run_generate()
if action == "push":
return run_push_for_date(today_str)
raise ValueError(f"未知动作: {action}")
def tick_once(
*,
now: datetime | None = None,
tz: ZoneInfo | None = None,
state: SchedulerState | None = None,
generate_at: ClockTime | None = None,
push_at: ClockTime | None = None,
dry_run: bool = False,
) -> SchedulerState:
tz = tz or load_timezone()
now = now or datetime.now(tz)
state = state or SchedulerState.load()
generate_at = generate_at or parse_hhmm(schedule_generate_at())
push_at = push_at or parse_hhmm(schedule_push_at())
today_str = now.astimezone(tz).date().isoformat()
# 仅工作日运行:非工作日(法定节假日/周末,调休补班日除外)跳过当日 generate/push。
# 将两者标记为已完成,避免 plan_next_action 在当天反复补跑。
if workday_only() and state.last_generate_date != today_str:
today = now.astimezone(tz).date()
try:
holidays = load_holidays(today.year)
if not is_workday(today, holidays):
name = workday_name(today, holidays) or "周末"
logger.info("今日 %s 非工作日(%s),跳过生成与推送", today_str, name)
state.last_generate_date = today_str
state.last_push_date = today_str
if not dry_run:
state.save()
if dry_run:
logger.info("[dry-run] 非工作日,将跳过")
return state
except Exception as exc:
# 节假日数据不可用时按常规工作日处理,避免因接口故障漏跑
logger.warning("工作日判定失败(%s),按正常工作日处理", exc)
planned = plan_next_action(
now=now,
tz=tz,
state=state,
generate_at=generate_at,
push_at=push_at,
)
if not planned:
return state
run_at, action = planned
if run_at > now:
if not dry_run:
logger.info("下次 %s @ %s (%s)", action, run_at.isoformat(), tz.key)
_sleep_until(run_at, tz)
elif not dry_run:
slot = _today_slot(now.astimezone(tz).date(), generate_at if action == "generate" else push_at, tz)
logger.info(
"补跑 %s(计划 %02d:%02d,当前 %s",
action,
slot.hour,
slot.minute,
now.astimezone(tz).strftime("%H:%M"),
)
if dry_run:
logger.info("[dry-run] 将执行 %s @ %s", action, run_at.isoformat())
return state
logger.info("开始 %s%s", action, today_str)
code = run_scheduled_action(action, today_str=today_str)
if code != 0:
logger.error("%s 失败exit=%s", action, code)
else:
if action == "generate":
state.last_generate_date = today_str
elif action == "push":
state.last_push_date = today_str
state.save()
logger.info("%s 完成", action)
return state
def main() -> int:
dry_run = "--dry-run" in sys.argv[1:]
once = "--once" in sys.argv[1:]
log_path = _setup_logging()
tz = load_timezone()
generate_at = parse_hhmm(schedule_generate_at())
push_at = parse_hhmm(schedule_push_at())
logger.info(
"调度器启动 tz=%s generate=%02d:%02d push=%02d:%02d log=%s",
tz.key,
generate_at.hour,
generate_at.minute,
push_at.hour,
push_at.minute,
log_path,
)
state = SchedulerState.load()
try:
while True:
state = tick_once(
tz=tz,
state=state,
generate_at=generate_at,
push_at=push_at,
dry_run=dry_run,
)
if once or dry_run:
break
except KeyboardInterrupt:
logger.info("调度器已停止KeyboardInterrupt")
return 0
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -2,20 +2,97 @@
from __future__ import annotations
import json
import logging
import re
import time
from typing import Any, Literal
import certifi
import httpx
from daily.config import env
from daily.config import CACHE_DIR, env
logger = logging.getLogger(__name__)
Board = Literal["trending", "hot"]
SKILLS_SITE = "https://www.skills.sh"
USER_AGENT = "Mozilla/5.0 (compatible; daily-robots/1.0; +https://skills.sh)"
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)"
FEED_URLS = [
"https://cdn.jsdelivr.net/gh/NeverSight/skills.sh_feed@main/data/feed.json",
"https://raw.githubusercontent.com/NeverSight/skills.sh_feed/main/data/feed.json",
]
FEED_CACHE_TTL = 600
FEED_CACHE_FILE = CACHE_DIR / "feed.json"
_feed_cache: dict[str, Any] = {"data": None, "fetched_at": 0.0}
def format_installs(n: int | float) -> str:
if n >= 1_000_000:
return f"{n / 1_000_000:.1f}M"
if n >= 1_000:
return f"{n / 1_000:.1f}K"
return str(int(n))
def _fetch_feed_json(url: str) -> dict[str, Any]:
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
with httpx.Client(
timeout=httpx.Timeout(20.0, connect=10.0),
verify=certifi.where(),
follow_redirects=True,
) as client:
resp = client.get(url, headers=headers)
resp.raise_for_status()
return resp.json()
def _load_feed_disk_cache() -> dict[str, Any] | None:
if not FEED_CACHE_FILE.exists():
return None
try:
return json.loads(FEED_CACHE_FILE.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
logger.warning("读取 feed 本地缓存失败: %s", exc)
return None
def _save_feed_disk_cache(data: dict[str, Any]) -> None:
FEED_CACHE_FILE.parent.mkdir(parents=True, exist_ok=True)
FEED_CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
def load_feed(force: bool = False) -> dict[str, Any]:
now = time.time()
if not force and _feed_cache["data"] and now - _feed_cache["fetched_at"] < FEED_CACHE_TTL:
return _feed_cache["data"]
errors: list[str] = []
for url in FEED_URLS:
for attempt in range(3):
try:
data = _fetch_feed_json(url)
_feed_cache["data"] = data
_feed_cache["fetched_at"] = now
_save_feed_disk_cache(data)
logger.info("skills feed 已更新: %s", url)
return data
except Exception as exc:
msg = f"{url} (#{attempt + 1}): {exc}"
errors.append(msg)
logger.debug("拉取失败 %s", msg)
time.sleep(0.5 * (attempt + 1))
stale = _load_feed_disk_cache()
if stale:
logger.warning("网络不可用,回退到 feed 本地缓存")
_feed_cache["data"] = stale
_feed_cache["fetched_at"] = now
return stale
raise RuntimeError(f"无法获取 skills 数据。最近错误: {errors[-1] if errors else 'unknown'}")
_SKILL_RE = re.compile(
r'\{"source":"(?P<source>[^"]+)","skillId":"(?P<skill_id>[^"]+)",'

View File

@@ -9,6 +9,28 @@ def skill_id(item: dict[str, Any]) -> str:
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
def source_from_skill_key(key: str) -> str:
"""从 skill idsource/title…还原 source去掉最后一段 title。"""
parts = [p for p in str(key or "").split("/") if p]
if len(parts) >= 2:
return "/".join(parts[:-1])
return str(key or "").strip()
def expand_skill_recent_keys(keys: set[str] | None) -> set[str]:
"""周去重 blocklist保留原始 key并展开为 source。"""
out: set[str] = set()
for key in keys or set():
k = str(key or "").strip()
if not k:
continue
out.add(k)
src = source_from_skill_key(k)
if src:
out.add(src)
return out
def format_installs(n: int | float) -> str:
if n >= 1_000_000:
return f"{n / 1_000_000:.1f}M"

View File

@@ -15,7 +15,7 @@ def clip_text(text: str, limit: int) -> str:
def trim_brief(text: str, limit: int) -> str:
"""企微简要:控制在 limit 内,优先在句读截断,不加省略号。"""
"""企微简要:控制在 limit 内,优先在句读/词边界截断,不加省略号。"""
text = _WS.sub(" ", (text or "").strip())
if not text or limit <= 0 or len(text) <= limit:
return text
@@ -27,4 +27,12 @@ def trim_brief(text: str, limit: int) -> str:
pos = text.find(sep)
if pos != -1 and pos + 1 <= limit:
return text[: pos + 1]
return text[:limit].rstrip(",、;: ")
for sep in (". ", "! ", "? ", "; "):
pos = text.rfind(sep, 0, limit + 1)
if pos != -1 and pos + 1 >= min(limit // 2, 20):
return text[: pos + 1].rstrip()
if len(text) > limit:
space = text.rfind(" ", 0, limit + 1)
if space >= min(limit // 2, 20):
return text[:space].rstrip(",、;: ,.;")
return text[:limit].rstrip(",、;: ,.;")

View File

@@ -2,6 +2,8 @@
from __future__ import annotations
import json
import re
import sys
import time
from pathlib import Path
@@ -13,6 +15,7 @@ from daily.config import OUTPUT_DIR, ROOT, env_int, wecom_chunk_bytes
from daily.wecom_split import split_wecom_messages
_PUSH_GAP_MS = 300
_DATE_RE = re.compile(r"(\d{4}-\d{2}-\d{2})\.wecom\.md$")
def _load_webhook_key() -> str:
@@ -43,6 +46,31 @@ def _resolve_report_path(arg: str | None) -> Path:
raise RuntimeError("未找到 .wecom.md 报告,请先运行 python -m daily")
def _push_gate_for_report(path: Path) -> dict | None:
match = _DATE_RE.search(path.name)
if not match:
return None
data_path = path.parent / f"{match.group(1)}.data.json"
if not data_path.exists():
data_path = OUTPUT_DIR / f"{match.group(1)}.data.json"
if not data_path.exists():
return None
try:
payload = json.loads(data_path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return None
meta = payload.get("meta") or {}
gate = meta.get("push_gate")
return gate if isinstance(gate, dict) else None
def should_skip_push(report_path: Path) -> bool:
gate = _push_gate_for_report(report_path)
if not gate:
return False
return bool(gate.get("silent")) and not gate.get("should_push")
def _post_markdown(client: httpx.Client, url: str, content: str) -> None:
payload = {"msgtype": "markdown", "markdown": {"content": content}}
resp = client.post(url, json=payload)
@@ -54,6 +82,9 @@ def _post_markdown(client: httpx.Client, url: str, content: str) -> None:
def send_report(report_path: Path | None = None) -> None:
path = _resolve_report_path(str(report_path) if report_path else None)
if should_skip_push(path):
print(f"[silent] no push gate matched for {path.name}")
return
if not path.exists():
raise RuntimeError(f"报告文件不存在: {path}")

View File

@@ -1,14 +0,0 @@
services:
daily:
build: .
env_file: .env
environment:
TZ: Asia/Shanghai
volumes:
- ./output:/app/output
- ./.cache:/app/.cache
- ./logs:/app/logs
# Examples:
# docker compose run --rm daily
# docker compose run --rm daily --skip-push
# docker compose run --rm daily --force

View File

@@ -0,0 +1,273 @@
# Design: 企微早报 Delta 模式
Generated: 2026-07-09
Repo: daily-robots
Status: DRAFT
Mode: Builder
## Problem Statement
企微早报每天推送五榜 Top 10 + 18 条新闻,内容与前几日高度重复(`find-skills``openclaw`、飞书集群等长期霸榜)。读者真实需求是「今天有什么新变化」,而非「再读一遍黄页」。
根因:
1. `daily-agent/SKILL.md` 要求即使较昨日无新增,仍须完整列出 Top 榜。
2. `movement` 仅用于 opening / signals列表区块仍全量渲染。
3. Trending 与 Hot 独立展示,同一 skill 描述写两遍。
4. 新闻 `DAILY_AI_NEWS_HOURS=72`,无已推送 link 去重,旧闻可连续出现。
## What Makes This Cool
把早报从「日报复印机」变成「变化通知」:只有新入榜、新新闻、编辑推荐时才占版面;榜全稳且无新新闻时静默不推。读者打开企微即知「今天值得扫一眼的是什么」。
## Explicit Non-Goals已否决方案
以下方案**不在本设计范围内**
| 方案 | 状态 |
|------|------|
| 静态页 / 外链档案库 | ❌ 不做 |
| 今日一装(每天一个 `npx skills add` | ❌ 不做 |
| 按星期轮换版面 | ❌ 不做 |
| 榜首锚点(稳定日仍展示 #1 | ❌ 不做 |
## Premises
1. 重复感主要来自**列表区块全量复印**,而非 opening 里引用榜首数字。
2. `output/*.data.json``daily/delta.py` 已具备新入榜对比能力,应上升为**列表渲染主数据源**。
3. 企微消息仍在应用内读完,不依赖外部页面。
4. 叙事层opening、信号、首推、新闻保持充实缩短的是**榜单列表**,不是整报。
## Recommended Approach: Delta 模式
### 环境变量
```env
# full = 现有行为(全量 Top 榜列表)
# delta = 本设计(默认推荐)
DAILY_WECOM_MODE=delta
# 无对比基准时(首日或缺历史 data.json是否自动 full 一次
DAILY_DELTA_BASELINE_FALLBACK=full # full | empty
# 推送闸门全不满足时是否跳过 webhook仍写 output 文件)
DAILY_SKIP_PUSH_WHEN_SILENT=1
# 强制推送(忽略静默)
# DAILY_FORCE_PUSH=1
# 新闻:缩短窗口 + 去重天数
DAILY_AI_NEWS_HOURS=24
DAILY_NEWS_DEDUP_DAYS=7
```
### 推送闸门Push Gate
满足**任一**条件则生成并推送企微早报:
| 条件 | 数据源 |
|------|--------|
| 任榜单有新入条目 | `movement.*_moves` 非空 |
| 去重后仍有新新闻 | 国际或国内 AI 时讯 |
| 存在 `featured_pick` | Step 0 编辑推荐 |
| `DAILY_FORCE_PUSH=1` | 环境变量 |
**静默日**:以上皆不满足 → 不调用 webhook`DAILY_SKIP_PUSH_WHEN_SILENT=1` 时)。
仍执行 `daily generate`,写入 `output/{date}.md``output/{date}.wecom.md``output/{date}.data.json` 留档。
**注意**:仅新闻有新、榜单全稳时**仍推送**,但 Skills/GitHub 列表区块整块省略(不是全天静默)。
### 列表渲染Delta 列表)
`DAILY_WECOM_MODE=delta` 时:
#### Skills
- **仅展示** `movement.skills_trending_moves` / `movement.skills_hot_moves` 中的新入榜条目。
- **跨榜去重**:按 `skill_id``id``source/title`)合并;同一 skill 只出现一次,标注来源榜(如 `Trending #4 · Hot #2`)。
- **无新入**:该榜区块**整块不出现**(不写多行「较昨日无新增」)。
#### GitHub
- 仅展示 `movement.github_trending_moves``github_emerging_moves``github_topic_moves`
- 无新入则区块省略。
#### 不包含
- 全量 Top N 列表
- 榜首锚点
- `(新入 … #n` 括号标注(与现 `agent_workflow._strip_new_entry_notes` 一致,列表标题用 `[新入 #n]` 前缀即可)
### 固定骨架(不因 Delta 缩短)
Agent 模式(`DAILY_REPORT_MODE=agent`)下,以下区块**保持**
- opening23 句,首句含具体证据)
- headline / 今日主题
- 今日信号35 条)
- 今日首推
- 国际 AI / 国内 AI 精选(条数仍由 `DAILY_WECOM_AI_NEWS` 等控制)
榜单变短;叙事与新闻不主动砍到 0。
### Full 模式逃生口
`DAILY_WECOM_MODE=full` 时行为与**当前生产一致**`format_wecom.build_wecom_report` / `replace_wecom_skill_sections` 全量 Top N。用于手动切回或对比测试。
### 首日 / 无历史基准
`find_previous_data(date)` 返回 `None` 时:
| `DAILY_DELTA_BASELINE_FALLBACK` | 行为 |
|----------------------------------|------|
| `full`(推荐) | 当日按 full 模式渲染列表一次;次日起 delta |
| `empty` | 当日列表区块为空opening 须说明「首日报,暂无对比基准」 |
实现时在 `generate_report``build_llm_input` 传入 `baseline_date` 供 Agent 引用。
## News Dedup
### P0本阶段
- 维护 `cache/pushed-news-links.json`(或写入 `output/` 旁 cache最近 `DAILY_NEWS_DEDUP_DAYS` 天已出现在企微早报中的 `link` 集合。
- `prepare_wecom_news_items` / `prepare_wecom_cn_news_items` 输出前过滤已见 link。
- `DAILY_AI_NEWS_HOURS` 默认改为 `24``.env.example` 同步)。
### P1可选后续
- 标题归一化去重(同一事件多源报道)
-`source_name` 每日上限 N 条
## Agent Skill 变更
文件:`skills/daily-agent/SKILL.md`
### 删除 / 修改
- 删除规则:「即使某榜较昨日无新增,仍须完整列出 Top 榜条目」。
- 删除:「禁止改用 movement 作为列表来源」(在 delta 模式下反转)。
### 新增
`DAILY_WECOM_MODE=delta`(或 llm_input 含 `wecom_mode: delta`
1. Agent **不写** Skills Trending / Hot / GitHub 列表(仍由 Python 插入,与现流程一致)。
2. opening / signals **可引用**榜首与 movement 摘要;禁止在 signals 重复列表已展示的同一事实。
3. 榜全稳时signals 聚焦新闻与首推,不必编造榜单变化。
`wecom_mode: full` 时保持现有 SKILL 规则。
## Python 模块变更
| 模块 | 变更 |
|------|------|
| `daily/config.py` | `wecom_mode()`, `news_dedup_days()`, `skip_push_when_silent()`, `delta_baseline_fallback()` |
| `daily/delta.py` | 可选:`merge_skill_moves_for_wecom(trending_moves, hot_moves)` 跨榜去重 |
| `daily/format_wecom.py` | `build_skills_delta_section()`, `build_github_delta_section()``replace_wecom_skill_sections` 支持 delta |
| `daily/news/fetch.py` | `filter_pushed_news()` + cache 读写 |
| `daily/generate.py` | 推送闸门baseline fallback静默 skip push |
| `daily/report_data.py` | `llm_input` 增加 `wecom_mode`, `push_gate` 摘要 |
| `daily/agent_workflow.py` | 无逻辑变更;依赖 Python 插入 delta 列表 |
| `.env.example` | 新 env 文档 |
## 企微消息示例
### 有变化日
```markdown
📰 **早报 · 2026-07-10**
[opening今天最大变化含数字/条目名]
🎯 **{headline}**
💡 **今日信号**
> ...
📦 **今日首推**
[...]
🌍 **国际 AI · 精选 N**
...
📈 **Skills Trending 变化**
1. [新入 #4] [**xxx**](...) · ...
描述一行
🔥 **Skills Hot 变化**
1. [新入 #2] [**yyy**](...) · ...
🐙 **GitHub Trending 变化**
1. [新入 #4] [owner/repo](...) · ...
```
### 仅新闻有新(榜稳)
- 无 📈/🔥/🐙 区块
- opening 可一句:「榜单较昨日 Top15 无新入;以下为今日 AI 时讯。」
### 静默日
- 不推送企微
- `output/` 仍落盘;日志:`[silent] no push gate matched for 2026-07-10`
## Approaches Considered
### Approach A: 配置瘦身(缩 Top N、24h 新闻)
- Effort: S | Risk: Low
- 只减篇幅,榜头仍天天重复;未解决根因。
### Approach B: Delta 列表 + 推送闸门 + 新闻去重(本设计)
- Effort: M | Risk: Med
- 复用 `delta.py`;改 format + skill + push 逻辑。
### Approach C: 仅改 Agent 文案
- Effort: S | Risk: High
- Python 仍插入全量列表,规则冲突,不可持续。
**Recommendation: B** — 数据层与展示层一致,静默日与跨榜去重可测。
## Success Criteria
1. 连续 3 天对比 `output/*.data.json`:企微列表区块**重复 skill_id 占比**显著下降。
2. 榜全稳且无新新闻日:`DAILY_SKIP_PUSH_WHEN_SILENT=1` 时不发 webhook。
3. Trending/Hot 同一 skill 在列表中**最多出现 1 次**。
4. `DAILY_WECOM_MODE=full` 与现网行为一致(回归用)。
5. 首日 `DAILY_DELTA_BASELINE_FALLBACK=full` 不产生空列表投诉。
## Open Questions
1. 静默日是否需要在企微发一行「今日无更新」?当前设计:**不发**。
2. 新闻去重 cache 是否纳入 git建议**否**,放 `cache/`(已在 `.gitignore`)。
3. Classic 模式(非 agent是否同步 delta建议**是**,同一 `format_wecom` 路径。
## Implementation Tasks
| ID | Priority | Task | Files |
|----|----------|------|-------|
| T1 | P1 | 新增 config helpers + `.env.example` | `daily/config.py`, `.env.example` |
| T2 | P1 | 新闻 link 去重 cache | `daily/news/fetch.py`, `daily/config.py` |
| T3 | P1 | Delta 列表渲染 + 跨榜去重 | `daily/format_wecom.py`, `daily/delta.py` |
| T4 | P1 | 推送闸门 + 静默 skip push | `daily/generate.py`, `daily/webhook.py` |
| T5 | P1 | baseline fallback full 一次 | `daily/generate.py` |
| T6 | P1 | 更新 `daily-agent/SKILL.md` | `skills/daily-agent/SKILL.md` |
| T7 | P2 | `llm_input``wecom_mode` / push 摘要 | `daily/report_data.py` |
| T8 | P2 | 单元测试:跨榜去重、推送闸门、新闻去重 | `tests/test_wecom_delta.py` |
## Test Plan
- [ ]`2026-07-09.data.json` 时生成 `2026-07-10`:列表仅含新入项
- [ ] 人造「全稳 + 无新新闻」:不 push
- [ ] 人造「全稳 + 有新新闻」push无 Skills/GitHub 块
- [ ] `DAILY_WECOM_MODE=full` 输出与改前 `2026-07-09.wecom.md` 结构一致
- [ ] 无 baseline + `DAILY_DELTA_BASELINE_FALLBACK=full`:首日全量列表
- [ ] 同一 skill 在 Trending/Hot moves 均出现:列表只 1 条
## What I Noticed
- 重复感是**产品形态**问题,不是 Agent 文笔问题。
- 明确否决静态页、今日一装、轮换、锚点后,方案边界清晰,实现可分期。
- 推送闸门必须**把新闻算进去**,否则静默日会被新闻绕过。

View File

@@ -1,35 +0,0 @@
<!-- 示例Agent 模式企微早报节选,日期 2026-07-03 -->
📰 **早报 · 2026-07-03**
> ⏱ 09:30 (UTC+8) · 数据截至 2026-07-02
> remotion-render 以 21549 次安装稳坐 Skills Trending 榜首halt-catch-fire 五件套把「TSX 直出 MP4」推到台前。GitHub 侧 openclaw 仍以 381.5K star 居 Trending 第一;新闻头条则是 OpenAI 拟向特朗普政府让渡 AI 繁荣 5% 分成——多媒体 skill 装机、本地 Agent 开源栈与 AI 治理议题同日撞屏。
🎯 **代码出片霸榜Agent基建共振**
💡 **今日信号**
> 🔥 Skills Hotxixu-me/skills 12 件套新占 Top10 前十,含 xdrop、openclaw-secure-linux-cloud 等开发者工具簇
> 📦 Skills较昨日 Trending Top15 新入 7 条,飞书 lark-approval、lark-wiki 等办公 skill 批量涌入
> 🐙 GitHubaffaan-m/ECC 225.2K star 新入 Trending #2面向 Claude Code/Codex/Cursor 的 Agent harness 优化成显学
> 🌱 GitHublangflow、dify 新入 Trending Top10可视化 Agent 工作流平台与 DeepSpec 推测解码基建各据一角
> 📰 时讯Anthropic 搁置已久的 Fable 5 获准回归,模型内容政策与 Zuckerberg「Agent 进展慢于预期」形成对照
📦 **今日首推**
`npx skills add halt-catch-fire/skills/remotion-render`
> Trending #1、21549 安装5 skill 集群覆盖 remotion-render 到 ai-video-generation把 React/Remotion 组件程序化渲染 MP4 做成默认可装工作流。
🌍 **国际 AI · 精选 10**
1. [OpenAI拟向特朗普政府让渡AI繁荣5%分成](https://www.theverge.com/ai-artificial-intelligence/960588/openai-government-5-percent-stake-trump) — The Verge 头条:头部模型公司探讨向政府让渡 AI 收益分成,商业化与治理边界被推上舆论前台
2. [Anthropic搁置已久的Fable 5获准回归](https://www.theverge.com/ai-artificial-intelligence/958964/anthropic-claude-fable-5-is-back) — 经数周协商后,此前被搁置的 Claude Fable 5 内容政策模型重新获准上线
3. [Zuckerberg称AI Agent进展慢于预期](https://techcrunch.com/2026/07/02/mark-zuckerberg-tells-staff-that-ai-agents-havent-progressed-as-quickly-as-hed-hoped/) — Meta 内部会议透露Agent 落地节奏未达此前预期
🇨🇳 **国内 AI · 精选 8**
1. [让Agent越用越强AReaL 2.0开源打造面向自演进智能体的RL基础设施](https://www.qbitai.com/2026/07/442134.html) — 量子位:面向自演进 Agent 的 RL 训练基建开源
2. [Agent Loop 深度调研:把决定权交给模型的一次换代,为什么发生在现在](https://juejin.cn/post/7657771483503018036) — 掘金长文梳理 Agent Loop 架构演进
📈 **Skills Trending Top 9**
1. [**halt-catch-fire/skills** · 5 skills · **21.5K21.5K**](https://www.skills.sh/halt-catch-fire/skills/remotion-render)
> 用 inference.sh 将 Remotion 组件代码直接渲染为 MP4可配分辨率帧率时长适合程序化视
2. [**find-skills**](https://www.skills.sh/vercel-labs/skills/find-skills) · `vercel-labs/skills` · **18.0K**
> 用户询问如何做某事或寻找能力扩展时,帮助发现并安装合适的 Agent Skill。
<!-- 完整版还包含 Hot 榜、GitHub Trending/新兴/Topic 等分区,见本地 output/ -->

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,604 @@
# 企微早报多样性与去重 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 实现企微早报硬去重多样性:五榜周去重(深池补满)、首推与昨日相同则改推(月去重)、叙事轴代码互斥、取消新闻「放宽窗口」凑数;且 `movement_baseline``wecom_shown_keys` 严格分离。
**Architecture:**`daily generate` 管线加代码选择器:`board_select` 为 full/delta 唯一列表主人;周历史只读写 `data.wecom_shown_keys`post-render`movement_baseline` 仍为 raw Top compare`featured_resolve` 先定人再 research`pick_narrative_axis` 代码选轴注入 Agent Step1新闻关 backfill + 剥「放宽」前缀。
**Tech Stack:** Python 3.13+、现有 `unittest`/`pytest``daily/delta.py` / `format_wecom.py` / `featured_pick.py` / `news/fetch.py``output/*.data.json`
**Spec:** `docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md`Status: APPROVED
## Global Constraints
- `movement_baseline` **禁止**被展示历史覆写;周去重只读 `wecom_shown_keys`
- full/delta **唯一列表主人** = `board_select`delta 的 pad 共用同一套 shown 历史)
- `DAILY_BOARD_DEDUP_DAYS` 默认 `7`;与 pad lookback 对齐且数据源同一
- `DAILY_FEATURED_DEDUP_DAYS` 默认 `30`
- `DAILY_NARRATIVE_AXIS_DAYS` 默认 `3`;轴枚举固定 7 个(见 Task 5
- `DAILY_NEWS_BACKFILL` 默认 `0`(禁止旧闻凑数)
- research 补新闻年龄上限 = `DAILY_AI_NEWS_HOURS`(不得变相 48h 放宽)
- 不做:语义「同一类」、同日 Trending↔Hot 互斥、`FEATURED_FORCE`、关键短语硬匹配
- Commit 信息正文用中文(若本任务含 Commit 步)
---
## File Structure
| 文件 | 职责 |
|------|------|
| `daily/config.py` | 新 env`board_dedup_days``board_pool_size``featured_dedup_days``theme_ban_days``narrative_axis_days``news_backfill_enabled` |
| `daily/board_history.py` | **新建**`load_recent_shown_keys` / `extract_shown_keys` / `attach_wecom_shown_keys`(读写 `data.wecom_shown_keys` |
| `daily/board_select.py` | **新建**`board_select(...)` 周过滤+深池 |
| `daily/delta.py` | `load_recent_board_keys` 改为委托 `load_recent_shown_keys`(保留函数名兼容);**不**改 `build_movement_baseline` |
| `daily/format_wecom.py` | pad 使用 shown keys可选返回最终展示 items 供写回 |
| `daily/featured_pick.py` | `featured_identity_key``featured_resolve`、先定人再 research |
| `daily/narrative_axis.py` | **新建**`NARRATIVE_AXES``pick_narrative_axis``load_recent_axes` |
| `daily/news/fetch.py` | `_apply_pushed_dedup_with_backfill` 尊重 `news_backfill_enabled()`;默认不塞回 |
| `daily/news/research.py` + `skills/daily-ai-news-research/SKILL.md` | 禁放宽文案;补入不超时窗 |
| `daily/text_utils.py``daily/news/sanitize.py` | `strip_news_relax_prefix(desc)` |
| `daily/agent_workflow.py` | Step1 注入 axis + 近 7 日 theme 软禁;强制覆写冲突轴 |
| `daily/generate.py` | 串联select → featured → editorial → render → persist shown/key/axis |
| `daily/report_data.py` | data.json 可携 `wecom_shown_keys` / `featured_pick_key` / `narrative_axis`(写回可由 generate 合并) |
| `.env.example` | 文档化新变量 |
| `skills/daily-agent/SKILL.md` | `narrative_axis` 必填且等于输入指定轴 |
| `tests/test_board_select.py` | **新建** |
| `tests/test_board_history.py` | **新建** |
| `tests/test_featured_resolve.py` | **新建** |
| `tests/test_narrative_axis.py` | **新建** |
| `tests/test_news_relax.py` | **新建** |
| `tests/test_wecom_delta.py` | 回归pad 不读 movement 当展示史 |
---
### Task 1: Config + shown-keys 历史层
**Files:**
- Modify: `daily/config.py`(文件末尾追加)
- Create: `daily/board_history.py`
- Modify: `daily/delta.py``load_recent_board_keys` 改委托)
- Test: `tests/test_board_history.py`
- Modify: `.env.example`
**Interfaces:**
- Consumes: `OUTPUT_DIR`、现有 `delta.skill_id` / repo key 约定
- Produces:
- `board_dedup_days() -> int`(默认 7
- `board_pool_size() -> int`(默认 `max(50, env WECOM_SKILL_POOL)`
- `featured_dedup_days() -> int`(默认 30
- `theme_ban_days() -> int`(默认 7
- `narrative_axis_days() -> int`(默认 3
- `news_backfill_enabled() -> bool`(默认 Falseenv `DAILY_NEWS_BACKFILL`
- `extract_shown_keys(board: str, items: list[dict]) -> list[str]`
- `load_recent_shown_keys(date_str: str, *, lookback_days: int | None = None) -> dict[str, set[str]]`
- `merge_wecom_shown_into_data(data: dict, shown: dict[str, list[str]]) -> dict`
- [ ] **Step 1: Write the failing test**
```python
# tests/test_board_history.py
from __future__ import annotations
import json
import os
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from daily.board_history import extract_shown_keys, load_recent_shown_keys, merge_wecom_shown_into_data
from daily.config import board_dedup_days, news_backfill_enabled
class ConfigDiversityTests(unittest.TestCase):
def test_board_dedup_days_default(self):
with patch.dict(os.environ, {}, clear=True):
self.assertEqual(board_dedup_days(), 7)
def test_news_backfill_default_off(self):
with patch.dict(os.environ, {}, clear=True):
self.assertFalse(news_backfill_enabled())
class ShownKeysTests(unittest.TestCase):
def test_extract_github_repo_keys(self):
items = [{"repo": "a/b"}, {"repo": "c/d"}]
self.assertEqual(extract_shown_keys("github_trending", items), ["a/b", "c/d"])
def test_load_recent_reads_wecom_shown_not_baseline(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp)
# 前日shown 只有 x/ybaseline raw 含 a/b —— 周去重只能看到 x/y
payload = {
"data": {
"date": "2026-07-13",
"movement_baseline": {
"github_trending": [{"repo": "a/b"}, {"repo": "x/y"}],
},
"wecom_shown_keys": {"github_trending": ["x/y"]},
}
}
(out / "2026-07-13.data.json").write_text(
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
)
with patch("daily.board_history.OUTPUT_DIR", out):
keys = load_recent_shown_keys("2026-07-14", lookback_days=7)
self.assertEqual(keys["github_trending"], {"x/y"})
self.assertNotIn("a/b", keys["github_trending"])
def test_merge_shown_does_not_touch_baseline(self):
data = {
"movement_baseline": {"github_trending": [{"repo": "raw/one"}]},
}
merged = merge_wecom_shown_into_data(
data, {"github_trending": ["shown/one"]}
)
self.assertEqual(
merged["movement_baseline"]["github_trending"][0]["repo"], "raw/one"
)
self.assertEqual(merged["wecom_shown_keys"]["github_trending"], ["shown/one"])
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_board_history.py -v`
Expected: FAIL模块/函数不存在)
- [ ] **Step 3: Implement config + board_history + delta 委托**
`daily/config.py` 追加:
```python
def board_dedup_days() -> int:
return max(1, env_int("DAILY_BOARD_DEDUP_DAYS", 7))
def board_pool_size() -> int:
fallback = env_int("DAILY_WECOM_SKILL_POOL", 50)
return max(1, env_int("DAILY_BOARD_POOL_SIZE", max(50, fallback)))
def featured_dedup_days() -> int:
return max(1, env_int("DAILY_FEATURED_DEDUP_DAYS", 30))
def theme_ban_days() -> int:
return max(1, env_int("DAILY_THEME_BAN_DAYS", 7))
def narrative_axis_days() -> int:
return max(1, env_int("DAILY_NARRATIVE_AXIS_DAYS", 3))
def news_backfill_enabled() -> bool:
return env_bool("DAILY_NEWS_BACKFILL", False)
```
新建 `daily/board_history.py`:实现 `BOARD_KEYS``delta.RECENT_BOARD_KEYS` 同五榜Skills 用 `delta.skill_id`GitHub 用 `repo``load_recent_shown_keys` **只**读各日 `data.wecom_shown_keys`,缺省空集,读写失败打 log 后当空集。
修改 `daily/delta.py``load_recent_board_keys`:改为
```python
def load_recent_board_keys(date_str: str, *, lookback_days: int | None = None) -> dict[str, set[str]]:
from daily.board_history import load_recent_shown_keys
from daily.config import board_dedup_days
days = lookback_days if lookback_days is not None else board_dedup_days()
return load_recent_shown_keys(date_str, lookback_days=days)
```
删除(或不再走)原「从 movement_baseline 抽 keys」逻辑避免 pad 继续把 raw Top 当展示史。
`.env.example` 追加注释块:
```env
# 多样性 / 去重(见 docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md
# DAILY_BOARD_DEDUP_DAYS=7
# DAILY_BOARD_POOL_SIZE=50
# DAILY_FEATURED_DEDUP_DAYS=30
# DAILY_THEME_BAN_DAYS=7
# DAILY_NARRATIVE_AXIS_DAYS=3
DAILY_NEWS_BACKFILL=0
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_board_history.py tests/test_wecom_delta.py -v`
Expected: `test_board_history` PASS既有 delta 测试若依赖「baseline 即 recent」行为按 Task 1 语义改断言为 shown_keys本 Task 内修回归,勿留红)。
- [ ] **Step 5: Commit**
```bash
git add daily/config.py daily/board_history.py daily/delta.py .env.example tests/test_board_history.py tests/test_wecom_delta.py
git commit -m "feat: 拆分 wecom_shown_keys 与 movement_baseline 历史层"
```
---
### Task 2: `board_select` 周去重 + 深池
**Files:**
- Create: `daily/board_select.py`
- Test: `tests/test_board_select.py`
**Interfaces:**
- Consumes: `extract_shown_keys` / `skill_id``group_skills_by_source`Skills 板)
- Produces:
- `board_select(*, board: str, items: list[dict], recent_keys: set[str], limit: int, pool_size: int, kind: Literal["skill","github"]) -> list[dict]`
- 日志短榜:`board_short:{board}:{n}``logging.getLogger(__name__).info`
- [ ] **Step 1: Write the failing test**
```python
# tests/test_board_select.py
from __future__ import annotations
import unittest
from daily.board_select import board_select
def _gh(repo: str) -> dict:
return {"repo": repo, "description": repo}
class BoardSelectTests(unittest.TestCase):
def test_filters_recent_and_keeps_order(self):
pool = [_gh(f"o/r{i}") for i in range(20)]
recent = {"o/r0", "o/r1", "o/r2"}
out = board_select(
board="github_trending",
items=pool,
recent_keys=recent,
limit=5,
pool_size=20,
kind="github",
)
keys = [x["repo"] for x in out]
self.assertEqual(keys, ["o/r3", "o/r4", "o/r5", "o/r6", "o/r7"])
def test_deep_pool_fills_after_filter(self):
pool = [_gh(f"o/r{i}") for i in range(8)]
recent = {f"o/r{i}" for i in range(6)} # 前 6 全封
out = board_select(
board="github_emerging",
items=pool,
recent_keys=recent,
limit=5,
pool_size=8,
kind="github",
)
self.assertEqual([x["repo"] for x in out], ["o/r6", "o/r7"]) # 短榜
def test_skill_uses_skill_id(self):
items = [
{"id": "a/b/s1", "source": "a/b", "title": "s1", "installs": 10},
{"id": "c/d/s2", "source": "c/d", "title": "s2", "installs": 9},
]
out = board_select(
board="skills_trending",
items=items,
recent_keys={"a/b/s1"},
limit=10,
pool_size=50,
kind="skill",
)
self.assertEqual([x["id"] for x in out], ["c/d/s2"])
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_board_select.py -v`
Expected: FAIL
- [ ] **Step 3: Implement `board_select`**
```python
# daily/board_select.py — 核心逻辑示意
def board_select(*, board, items, recent_keys, limit, pool_size, kind):
if kind == "skill":
from daily.skills_group import group_skills_by_source
pool = group_skills_by_source(items, limit=pool_size, pool_size=pool_size)
def key_fn(x): return skill_id(x)
else:
pool = items[: max(pool_size, limit)]
def key_fn(x): return str(x.get("repo") or "")
out = []
for item in pool:
k = key_fn(item)
if not k or k in recent_keys:
continue
out.append(item)
if len(out) >= limit:
break
if len(out) < limit:
logger.info("board_short:%s:%s", board, len(out))
return out
```
Skills输入可为未 group 的 raw函数内 group。GitHub输入为 repo 列表。
- [ ] **Step 4: Run tests — expect PASS**
Run: `pytest tests/test_board_select.py -v`
- [ ] **Step 5: Commit**
```bash
git add daily/board_select.py tests/test_board_select.py
git commit -m "feat: 实现 board_select 周去重与深池补满"
```
---
### Task 3: 接入 generate / format_wecom唯一列表主人 + post-render 写回)
**Files:**
- Modify: `daily/generate.py`(选榜、传入 pad、渲染后 merge shown
- Modify: `daily/format_wecom.py`delta pad 已通过改写后的 `load_recent_board_keys` 读 shown确保传入的 `*_pad` 池已是 `board_select` 深池结果)
- Modify: `daily/report_data.py`可选llm_input 切片改为 board_select 后列表,避免 Agent 看见未去重 Top
- Test: `tests/test_board_history.py` 增补「shown ≠ baseline 推导」集成断言;`tests/test_wecom_delta.py` pad 用例
**Interfaces:**
- Consumes: Task12
- Produces: 每次成功 generate 后 `output/{date}.data.json``data.wecom_shown_keys`
- [ ] **Step 1: Write / extend failing integration test**
```python
def test_persist_shown_keys_differs_from_baseline_keys(self):
# 构造raw trending 前 3 名本周已 shownboard_select 选出 3..
# movement_baseline 仍含 0..compare_depth
# 断言 data["wecom_shown_keys"]["github_trending"] 与 baseline repos 集合不等
...
```
(可用临时 `OUTPUT_DIR` + 调用抽取出的 `persist` 辅助,或测 `merge_wecom_shown_into_data` + `board_select` 组合。)
- [ ] **Step 2: Run — expect FAILgenerate 尚未写 shown**
- [ ] **Step 3: Wire generate**
`generate_report` 中,在组装 wecom 榜之前:
1. `recent = load_recent_shown_keys(date_str)`
2. 对五榜分别 `board_select(...)` 得到 `selected_*`limit=wecom_*pool=`board_pool_size()`
3. full渲染用 `selected_*`
4. delta`trending_pad`/`github_*_pad` = 同规则更大 pool 的 select 结果(或 raw 深池再 select`replace_wecom_board_sections(..., pad=True)` 内部 recent 已是 shown
5. 渲染后根据**最终写入正文的 items**full=selecteddelta=函数返回或并行计算最终列表)调用 `extract_shown_keys``merge_wecom_shown_into_data`,写回 data.json在现有 `save_json` 路径合并字段)
注意:`movement_baseline` 仍用 **raw** compare 切片构建(`report_data.build_llm_input` 现逻辑保留)。
`build_llm_input` 当前把未过滤 Top 塞进 Agent改为传入 `selected_*`(或另字段 `boards_for_wecom`),避免 opening 引用已周封杀的榜首。
辅助:在 `format_wecom` 增加 `resolve_wecom_board_items(...)` 返回最终 items dict供写回与 featured 池 A 共用,避免正文与 history 分叉。
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_board_select.py tests/test_board_history.py tests/test_wecom_delta.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add daily/generate.py daily/format_wecom.py daily/report_data.py tests/
git commit -m "feat: generate 以 board_select 为唯一列表主人并写回 shown keys"
```
---
### Task 4: `featured_resolve`(先定人再 research
**Files:**
- Modify: `daily/featured_pick.py`
- Modify: `daily/generate.py`(调用顺序)
- Test: `tests/test_featured_resolve.py`
**Interfaces:**
- Consumes: 最终展示 items池 A、raw 深池(池 B、近 30 日 `featured_pick_key`
- Produces:
- `featured_identity_key(featured: dict) -> str``type==skill` → idgithub → repo兜底 url path
- `load_recent_featured_keys(date_str, days) -> set[str]`
- `featured_resolve(*, date_str, candidate: dict | None, pool_a: list[dict], pool_b: list[dict], rng: random.Random | None) -> tuple[dict | None, str | None]`
返回 `(resolved_seed_or_featured_stub, identity_key)`**不含**完整 whywhy 由后续 research 写)
-`apply_featured_pick`:先 resolve 身份(若与昨日冲突则换候选写入 query`research_featured_pick`
- [ ] **Step 1: Failing tests**
```python
# tests/test_featured_resolve.py
def test_same_as_yesterday_picks_from_pool_a(self):
yesterday_key = "headroomlabs-ai/headroom"
pool_a = [
{"repo": "headroomlabs-ai/headroom", "board": "github_topic"},
{"repo": "ollama/ollama", "board": "github_trending"},
]
rng = random.Random(0)
resolved, key = featured_resolve(
date_str="2026-07-14",
candidate={"type": "github", "url": "https://github.com/headroomlabs-ai/headroom", "title": "headroom"},
pool_a=pool_a,
pool_b=[],
recent_featured={yesterday_key},
yesterday_key=yesterday_key,
rng=rng,
)
self.assertNotEqual(key, yesterday_key)
self.assertEqual(key, "ollama/ollama")
def test_pool_a_before_pool_b(self):
...
def test_exhausted_keeps_original(self):
...
```
- [ ] **Step 2: Run — FAIL**
- [ ] **Step 3: Implement**
`featured_resolve`:若无 candidate 或与 `yesterday_key` 不同 → 原样返回。
冲突时:过滤 `recent_featured | {yesterday_key}`,先从 pool_a 建可选项(每项抽 identity`rng.choice`;空则 pool_b仍空 log `featured_fallback_exhausted` 并保留原 candidate。
`apply_featured_pick` / generate 流程:
1. 解析 env 得到初始 query/candidate
2. `featured_resolve`(此时池 A 已是 board_select 结果)
3. 若换人:用新 repo/skill 构造 config`research_featured_pick`
4. 写入 `llm_input["featured_pick"]` 与之后 data.`featured_pick_key`
随机默认:`random.Random(int(hashlib.sha256(f"{date_str}:featured".encode()).hexdigest()[:16], 16))`
- [ ] **Step 4: pytest PASS**
- [ ] **Step 5: Commit**
```bash
git add daily/featured_pick.py daily/generate.py tests/test_featured_resolve.py
git commit -m "feat: 首推与昨日冲突时改推并保证一月不重复"
```
---
### Task 5: `narrative_axis` 硬互斥 + Step1 软禁 theme
**Files:**
- Create: `daily/narrative_axis.py`
- Modify: `daily/agent_workflow.py``analyze_trends`
- Modify: `skills/daily-agent/SKILL.md`
- Modify: `daily/generate.py`(落盘 `narrative_axis`;注入 llm_input
- Test: `tests/test_narrative_axis.py`
**Interfaces:**
- Produces:
- `NARRATIVE_AXES: tuple[str, ...] = ("政策监管", "模型发布", "工具链/Agent", "芯片算力", "开源生态", "应用落地", "安全/诉讼")`
- `pick_narrative_axis(used: set[str], *, rng: random.Random | None = None) -> str`
- `load_recent_axes(date_str, days) -> list[str]`(近 N 日 data.`narrative_axis`
- `enforce_narrative_axis(trends: dict, axis: str) -> dict`(强制 trends["narrative_axis"]=axis
- [ ] **Step 1: Failing tests**
```python
def test_pick_excludes_used(self):
used = {"政策监管", "模型发布", "工具链/Agent"}
for _ in range(20):
axis = pick_narrative_axis(used, rng=random.Random(1))
self.assertNotIn(axis, used)
def test_enforce_overwrites_llm(self):
trends = {"narrative_axis": "开源生态", "opening": "..."}
out = enforce_narrative_axis(trends, "芯片算力")
self.assertEqual(out["narrative_axis"], "芯片算力")
```
- [ ] **Step 2: FAIL → Step 3 implement**
`analyze_trends`:计算 `axis = pick_narrative_axis(set(load_recent_axes(...)))`;把 `required_narrative_axis` 与近 `theme_ban_days` 的 theme/opening 摘要列表注入 system prompt要求 JSON 含 `narrative_axis` 且必须等于 required。解析后 `enforce_narrative_axis`
generate 将 axis 写入 data.json。
SKILL.md Step1 schema 增加 `narrative_axis` 字段说明。
- [ ] **Step 45: pytest + commit**
```bash
git commit -m "feat: 代码选定叙事轴并注入 Agent 开场约束"
```
---
### Task 6: 取消新闻「放宽窗口」
**Files:**
- Modify: `daily/news/fetch.py``_apply_pushed_dedup_with_backfill`
- Modify: `daily/news/research.py`(确认只 filter_unpushed不足不拉超窗
- Create: `daily/news/sanitize.py`(或放入 `text_utils`)— `strip_relax_window_prefix(text: str) -> str`
- Modify: `skills/daily-ai-news-research/SKILL.md`(删除「放宽至 48h 并注明」;改为不足则少返回、禁止标注)
- Modify: `daily/generate.py` / news finalize对 desc_short 剥前缀)
- Test: `tests/test_news_relax.py`;扩展 `tests/test_news_fetch_window.py`
**Interfaces:**
- `news_backfill_enabled() == False` 时:`_apply_pushed_dedup_with_backfill` 等价于只返回 `filter_unpushed_items(...)[:limit]`**不**再从 `picked` 塞回
- `strip_relax_window_prefix`:去掉开头的 `放宽窗口[:]?` / `放宽至[^:]*[:]`
- [ ] **Step 1: Failing tests**
```python
def test_backfill_disabled_does_not_reinsert_pushed(self):
# fresh 不足 limitpicked 含已推BACKFILL=0 → 结果不含已推 link
...
def test_strip_relax_prefix(self):
self.assertEqual(
strip_relax_window_prefix("放宽窗口:苹果起诉 OpenAI"),
"苹果起诉 OpenAI",
)
```
- [ ] **Step 24: 实现并跑 `pytest tests/test_news_relax.py tests/test_news_fetch_window.py -v`**
Research 路径SKUILL 改完后,代码侧对 items 统一 `strip`;不足时 log `news_short:{n}`,接受短列表。
- [ ] **Step 5: Commit**
```bash
git commit -m "fix: 关闭新闻放宽凑数并剥离放宽窗口文案"
```
---
### Task 7: 端到端回归与文档对齐
**Files:**
- Modify: 如有遗漏的 `.env.example` / SKILL
- Test: 全量相关测试
- [ ] **Step 1: 跑全套**
Run:
```bash
pytest tests/test_board_history.py tests/test_board_select.py tests/test_featured_resolve.py tests/test_narrative_axis.py tests/test_news_relax.py tests/test_news_fetch_window.py tests/test_wecom_delta.py tests/test_featured_pick.py -v
```
Expected: 全部 PASS
- [ ] **Step 2: Spec 对照清单(人工)**
| Spec 要求 | 任务 |
|-----------|------|
| wecom_shown_keys ≠ movement_baseline | T1, T3 |
| board_select 唯一主人 + delta pad 共用 shown | T2, T3 |
| 首推月去重 A→B、先定人再 why | T4 |
| narrative_axis 硬保证 | T5 |
| 禁放宽 backfill + 剥前缀 + hours 窗 | T6 |
| 成功标准可测 | 各测覆盖 |
- [ ] **Step 3: Commit若有收尾文档**
```bash
git add -u
git commit -m "test: 多样性去重全链路回归通过"
```
---
## Spec Coverage Self-Review
| Spec 节 | 计划任务 |
|---------|----------|
| 1.1 双基准分离 | T1 |
| 1.2 唯一列表主人 | T2T3 |
| 1.3 真相表 | T1, T3T5 落盘字段 |
| 2.1 board_select | T2 |
| 2.2 featured_resolve | T4 |
| 3.1 narrative_axis + 软 theme | T5 |
| 3.2 取消放宽 | T6 |
| 4.x 配置/降级/测试 | T1T7 |
无 TBDCommit 信息均为中文描述体。
## Execution Handoff
Plan complete and saved to `docs/superpowers/plans/2026-07-14-wecom-diversity-dedup.md`.
**两种执行方式:**
1. **Subagent-Driven推荐** — 每任务新开子代理,任务间审查
2. **Inline Execution** — 本会话按 `executing-plans` 连续做完,设检查点
要哪个?

View File

@@ -0,0 +1,271 @@
# Design: 企微早报多样性与去重
Generated: 2026-07-14
Repo: daily-robots
Status: APPROVED
Mode: Builder
Related: `docs/design-wecom-delta-mode.md`Delta 列表模式)
Revision: office-hours A —— 拆分展示历史、单一列表主人、历史真相表2026-07-14
## Problem Statement
近两日企微早报(如 2026-07-13 / 07-14骨架相同
- **今日首推**连续两天同为 `headroom`
- **开场主题**同属「上下文压缩 + 视频/Skills」腔调
- **AI 时讯**出现「放宽窗口」旧闻凑数标注
- **Skills / GitHub 各榜**(尤其新兴榜)周内大量重复展示
读者需要「今天新信息」,而不是换日期的复印机。
## Decisions已确认
| 决策点 | 选择 |
|--------|------|
| 实现路径 | **A管线选择器**代码硬保证去重LLM 只写开场/理由/摘要;中文化仍走现有 `localize` |
| Skills「同一类」 | 暂不管;沿用现有 `group_skills_by_source`**已知残留**:同 source 换 skill id 仍可能周内再出现) |
| 周去重后不足 Top N | **深池补满**,仍保证周内未出现;池空则短榜,不破周约束 |
| 首推改推候选 | **先展示榜(池 A再 raw 深池(池 B**;一月内首推不重复 |
| 开场主题 | **近 7 天禁主题/句式(软)+ 叙事轴与近 3 天不同(硬,代码选轴)** |
| 取消「放宽窗口」 | **禁止旧闻/已推凑数**;不够则深度检索补新闻;禁止任何「放宽」文案标注;仍不够则短列表 |
| 展示历史 vs 异动基准 | **必须拆开**`movement_baseline``wecom_shown_keys` |
| full/delta 列表主人 | **唯一主人** = `board_select`(含 full 与 delta 的 movespad禁止二次独立选榜 |
## Explicit Non-Goals
| 项 | 状态 |
|----|------|
| 语义级 Skill「同一类」分类 | ❌ 本期不做 |
| 同日 Skills Trending ↔ Hot 互斥 | ❌ 本期不做 |
| 独立 editorial 微服务 | ❌ 不做 |
| 改写 `localize` 为脚本机翻 | ❌ 保持 LLM + 缓存 |
| 编辑指定首推豁免改推(`FEATURED_FORCE` | ❌ 本期不做 |
| 「开场关键短语」硬匹配算法 | ❌ 本期不做(仅 prompt 软约束;不进硬成功标准) |
## Recommended Approach: 管线选择器(路径 A
在现有 `daily generate` 内增加选条/裁决层,不新起进程:
```
采集 raw 榜 + 新闻
→ build movement_baselineraw Top compare_depth —— 仅供次日「新入榜」,禁止写展示历史)
→ board_select读 wecom_shown_keys 周历史;周去重 + 深池;输出当日最终展示列表)
· full直接取 board_select 结果前 N
· delta在 board_select 候选池内做 movespadpad 也只从该池/同规则深池取,不再另起一套历史)
→ featured_resolve与昨日首推相同则改推池 A = 本 run 最终展示 keys
→ research why先定人再写 why_today
→ news_select禁放宽凑数深检索补满剥「放宽」前缀
→ editorial代码选 narrative_axisprompt 附近 7 天 theme 软禁)
→ 渲染 wecom
→ 写回 wecom_shown_keys = 最终进入企微正文的榜条目 keyspost-render
→ 其余 history首推月、axis写入 data.json 约定字段
```
**LLM 负责**`opening` / `theme_line`、首推 `why_today`、新闻与榜单项中文摘要(`localize`)。
**代码负责**:谁上榜、首推换谁、周/月去重、`narrative_axis` 选取、是否允许旧闻。
### Approaches Considered
| | A 管线选择器(采用) | B 偏 LLM 约束 | C 独立 editorial 服务 |
|--|--|--|--|
| 优点 | 可测;与 pushed-links 模式一致 | 改 prompt 快 | 边界清晰 |
| 缺点 | 需动 generate / featured / news / format | 易漏、难测 | 过重 |
---
## Section 1 — 总览、列表主人、历史真相表
### 1.1 两种「基准」禁止混用
| 字段 | 含义 | 写入时机 | 读者 |
|------|------|----------|------|
| `movement_baseline` | **Raw** 各榜 Top `compare_depth`(现网语义不变) | `build_llm_input` / 采集后尽早 | `build_movement_context`(新入榜) |
| `wecom_shown_keys` | **读者实际见到**的各榜 key 集合(及可选 rank | **wecom 渲染完成之后** | `board_select` 周去重delta pad测试 |
**禁止**:把 `wecom_shown_keys` 写入或覆写 `movement_baseline`
**禁止**:让 `load_recent_board_keys` 继续读 `movement_baseline` 充当「已展示」——应改为读近 N 日 `wecom_shown_keys`(可保留函数名,换数据源;或新建 `load_recent_shown_keys`)。
### 1.2 唯一列表主人
`board_select`(模块可挂在 `daily/board_select.py` 或扩 `delta.py`)是各榜**最终展示行**的唯一生产者:
| 模式 | 行为 |
|------|------|
| `full` | `board_select(raw, shown_history) →` 至多 N 条,直接渲染 |
| `delta` | 先算相对 `movement_baseline` 的 moves展示 = `moves`(已在候选内)∪ `pad`**pad 候选必须来自同一周去重池**(与 full 同一套 `board_select` 规则),不得再读 raw baseline 当「已展示」 |
交互影响(非「完全正交」):周去重会减少可展示重复项 → delta 日可能更短、silent/gate 行为可能变化。`DAILY_WECOM_MODE` 枚举语义不变,但列表密度会变。
### 1.3 历史真相表(单一来源)
全部落在 `output/{date}.data.json`(新闻 pushed-links 例外,沿用现网 cache
| 字段路径 | 窗口 | Key 规则 | 写者 | 读者 |
|----------|------|----------|------|------|
| `data.movement_baseline` | 次日对比用 | raw 条目切片 | `build_movement_baseline` | movement |
| `data.wecom_shown_keys.{board}` | 滚动 7 天(读近 7 日文件) | Skills与现网 `_skill_keys_in_board_item` / `skill_id` 一致GitHub`owner/repo` | post-render persist | `board_select` / pad |
| `data.featured_pick_key` | 滚动 30 天 | skill id 或 `owner/repo` | `featured_resolve` 成功后 | 月去重 |
| `data.narrative_axis` | 滚动 3 天 | 枚举字符串 | 代码 `pick_narrative_axis` | Step 1 约束 / 校验 |
| `data.theme_line` / trends opening | 近 7 日供 prompt | 原文 | editorial 落盘 | Step 1 软禁(不硬匹配) |
| `CACHE_DIR/pushed-news-links.json` | `DAILY_NEWS_DEDUP_DAYS` | 规范化 URL | 推送成功后 | news filter |
不另建平行 CACHE「board-history.json」避免双源漂移。冷启动缺文件 = 空集合。
### 1.4 数据流挂点
| 逻辑 | 挂点 |
|------|------|
| `movement_baseline` | 现网raw 榜入库时(不变) |
| `board_select` | 渲染前;输出写入供 Agent/`llm_input` 与 wecom 共用的最终列表字段 |
| delta pad | **调用同一周去重历史**`wecom_shown_keys`),不再独立解释 `movement_baseline` 为展示史 |
| `featured_resolve` | **先于** why 检索;池 A = 本 run `board_select`delta 则为本 run 最终展示列表) |
| 新闻 | 所有 prepare 路径关 backfillresearch SKILL 改文案规则;后处理剥「放宽」 |
| `narrative_axis` | 代码先选轴再注入 Step 1LLM 不得另选冲突轴 |
| `wecom_shown_keys` 写回 | `replace_wecom_*` / `build_wecom_report` 之后,与最终正文列表一致 |
---
## Section 2 — 各榜选条 + 今日首推改推
### 2.1 `board_select`(五榜共用)
适用:`skills_trending` / `skills_hot` / `github_trending` / `github_emerging` / `github_topic`
```
输入:当日 raw 池pool ≥ DAILY_BOARD_POOL_SIZE
历史:近 DAILY_BOARD_DEDUP_DAYS 的 wecom_shown_keys[board]
输出:至多 N 条N = 现有 wecom Top 配置)
1. 现有整理Skillssource 合并GitHubrepo key
2. 滤掉近 7 天该榜 wecom_shown_keys
3. 按原排名取前 N
4. 不足 → 继续扫深池,仍排除周历史,直到满 N 或池空
5. 池空仍不足 → 短榜;日志 board_short:{board}:{n};不回填周内已展示条目
```
分榜独立历史Trending 出过的 skillHot 仍可出。
Post-render将**实际写入企微的** keys 写入当日 `wecom_shown_keys`测试断言history ⊆ / == 渲染列表,**≠** `movement_baseline`)。
### 2.2 `featured_resolve`
**触发**:本 run 拟用首推身份与**前一天** `featured_pick_key`(或等价 data 字段)相同。
身份函数skill → `skill_id`github → `owner/repo`
`DAILY_FEATURED_PICK` 与自动首推;本期不豁免。无昨日文件 → 不改推。
**顺序(硬)**:定候选 → 再 `research`/`why_today`(禁止先写旧条目 why 再改人却不重写)。
**候选**
1. **池 A**:本 run **最终会展示**的 Skills + GitHub 榜条目(与 `wecom_shown_keys` 同源结构)
2. **池 B**raw 深池中尚未进入本 run 展示者
**过滤**:近 30 天 `featured_pick_key`;排除冲突项自身。
**抽取**`hash(date_str + "featured")` 可复现;测试可注入 RNG。先 A 后 B仍空 → 保留原首推 + `featured_fallback_exhausted`
**落盘**`data.featured_pick_key`
---
## Section 3 — 开场主题 + AI 时讯
### 3.1 开场主题
| 机制 | 强度 | 规则 |
|------|------|------|
| `narrative_axis` | **硬** | 代码 `pick_narrative_axis(used_last_N)` 从剩余枚举选取;注入 promptLLM 输出须等于该轴;冲突则重试 1 次,再失败则**强制覆写为代码所选轴**再落盘(保证成功标准可测) |
| theme/opening 软禁 | **软** | prompt 附近 7 天 `theme_line`/opening 摘要;禁止复述;**无** n-gram 硬匹配;**不**列入硬成功标准 |
**叙事轴枚举**
`政策监管` · `模型发布` · `工具链/Agent` · `芯片算力` · `开源生态` · `应用落地` · `安全/诉讼`
`opening` 首句证据须来自当日数据;首推改推后须跟新首推或当日主轴新闻。
### 3.2 AI 时讯:取消「放宽窗口」
目标条数 = 现网配置之和(如 `DAILY_WECOM_AI_NEWS` + tech/CN 等文档不写死「15」。
1. **所有 prepare 路径**关闭「不够塞回已推/旧条」(`DAILY_NEWS_BACKFILL=0` 默认);`pushed-news-links` 过滤保留。
2. 不够 → 深度检索补新闻https link、未 pushed、可核实**补入年龄上限** = `DAILY_AI_NEWS_HOURS`(与主窗一致),禁止借 research 变相放宽到任意旧闻。
3. 改 research SKILL删除「放宽至 48h 并注明」;后处理剥 `放宽窗口`/`放宽至` 前缀或丢弃。
4. 仍不足 → 短列表 + `news_short:{n}`
中文化:`daily/localize.py`(不变)。
---
## Section 4 — 配置、错误处理、测试
### 4.1 环境变量
| 变量 | 默认 | 含义 |
|------|------|------|
| `DAILY_BOARD_DEDUP_DAYS` | `7` | 读 `wecom_shown_keys` 的滚动天数 |
| `DAILY_BOARD_POOL_SIZE` | ≥50 / 与现有 skill pool 对齐 | 深池扫描深度 |
| `DAILY_FEATURED_DEDUP_DAYS` | `30` | 今日首推月去重 |
| `DAILY_THEME_BAN_DAYS` | `7` | 软禁:注入 prompt 的 theme 天数 |
| `DAILY_NARRATIVE_AXIS_DAYS` | `3` | 叙事轴互斥窗 |
| `DAILY_NEWS_BACKFILL` | `0` | `0`=禁止旧闻凑数 |
| `DAILY_NEWS_DEDUP_DAYS` | 已有 `7` | pushed-links |
`DAILY_DELTA_PAD_LOOKBACK_DAYS` 应与 `DAILY_BOARD_DEDUP_DAYS` 对齐,且 **pad 与 board_select 共用 `wecom_shown_keys`**(窗口对齐不够,数据源必须同一)。
### 4.2 错误与降级
| 情况 | 行为 |
|------|------|
| 无 `wecom_shown_keys` 历史 | 空集合,正常满榜 |
| 周去重后深池不足 | 短榜 + `board_short` |
| 首推冲突且 A/B 空 | 保留原首推 + `featured_fallback_exhausted` |
| LLM 轴与代码轴冲突 | 覆写为代码轴 |
| 深检索仍不足时讯 | 短列表;禁止 backfill |
| history 读写失败 | 当次按空历史 + error 日志 |
### 4.3 测试pytest
1. `board_select`:假 `wecom_shown_keys` + 深池 → 无周交集;深池补满;不足短榜
2. **回归钉死**:写回后 `wecom_shown_keys` ≠ 用 `movement_baseline` 推导的集合(构造 raw Top 与展示 Top 故意不同)
3. deltapad 不引入近 7 日 `wecom_shown_keys` 内 key
4. `featured_resolve`:先定人再 whyA 优先 B月未见可注入 RNG
5. news`BACKFILL=0`;剥「放宽*」research 补入不超 hours 窗
6. `pick_narrative_axis`:近 3 天互斥;落盘轴 == 代码轴
7. 既有 delta / pushed_links / wecom 回归不挂
### 4.4 成功标准(硬)
- 连续两天:**首推 key 不同**(除非 `featured_fallback_exhausted`
- 同一榜近 7 日 `wecom_shown_keys`**无重复 key**(池足够时)
- 时讯:无「放宽*」标注;无 backfill 已推 link
- 近 3 天 `narrative_axis`**两两不同**(代码保证)
软标准不闸门opening 读感不像连续复印。
---
## Implementation Sketch非计划明细
1. `data.json` 增加 `wecom_shown_keys`;改 `load_recent_*` 数据源
2. `board_select` + 让 delta pad 共用
3. post-render persist shown keys
4. `featured_resolve` 时序修正
5. news backfill off + SKILL + 剥前缀
6. `pick_narrative_axis` + prompt 注入
7. 测试如上
正式任务拆解 → `writing-plans`
## Office-hours Review Notes
- 对抗审阅质量约 4/10 → 本修订处理三大硬伤(存储拆分、列表主人、真相表)。
- 未纳入本期(原选项 B首推质量加权、关键短语硬匹配。
- 已知残留source 级「同类」周内可再现。
## Spec Self-Review
- [x] `movement_baseline``wecom_shown_keys` 职责分离写死
- [x] 单一列表主人 + full/delta 交互说明
- [x] 历史真相表无「与/或」双源
- [x] 轴硬 / 短语软;成功标准不含无法验证的短语匹配
- [x] 周不足=深池、首推=A→B、新闻禁放宽 与访谈一致

View File

@@ -0,0 +1,141 @@
# Design: Research 模式 AI 时讯质量(去重 · 国内配额 · 可信源)
Generated: 2026-07-29
Repo: daily-robots
Status: APPROVED
Related: `skills/daily-ai-news-research/SKILL.md`, `daily/news/research.py`, `docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md`
## Problem Statement
当前 `DAILY_AI_NEWS_MODE=research` 下:
1. **内容重复高**:同事件多源报道并列入榜(如 OpenAI 联名 / Altman 减速 / HF 入侵同簇Kimi K3 开源与多条适配);`items``tech_items` 只按 link 去重skill 还允许主题重叠。
2. **几乎无国内资讯**prompt 写「不区分国内国外合并精选」模型偏国际源research 分支关闭国内 RSS企微也不再出国内独立区块。
3. **可信度不稳**:低质搬运 / 营销号可进榜(如证券之星类),与「以官方/权威为准」不符。
样例:`output/2026-07-29.ai-news-research.json`
## Decisions已确认
| 决策点 | 选择 |
|--------|------|
| 国内资讯路径 | **A继续 research 合并展示**;不恢复国内独立区块;不引入 RSS 回填 |
| 国内配额 | 主列表至少约 30%10 条 → **≥3 条国内**`tech_items` **不强制**国内 |
| 去重范围 | **C**:同事件多源只留 1 条 + `tech_items` 相对 `items` 主题不得复述 |
| 可信度 | 国内/国际均以 **官方 + 权威媒体** 为准;不可信则不写,宁缺毋滥 |
| 实现路径 | **Prompt/skill 约束 + `research.py` 解析后硬校验**(推荐方案 2 |
| 条数不足 | **少返回**;禁止为凑满 limit / 国内配额灌低质或同事件条目 |
## Explicit Non-Goals
| 项 | 状态 |
|----|------|
| 恢复「国内 AI 时讯」独立企微区块 | ❌ 本期不做 |
| research 不足时用国内 RSS 补齐 | ❌ 本期不做(偏离路径 A |
| 切回纯 RSS | ❌ 本期不做 |
| LLM 语义嵌入聚类 | ❌ 本期不做(规则 + 实体/关键词即可) |
| 改 `generate.py` research/rss 分支结构 | ❌ 不动 |
## Architecture
```
Cursor Agent (WebSearch)
skills/daily-ai-news-research/SKILL.md + research prompt
│ JSON: items / tech_items
parse_research_response (link 去重)
post_process_research_items
1. trusted-source filter
2. same-event dedupe (items)
3. tech vs items topic filter
4. CN quota packing (min_cn)
_apply_pushed_dedup → generate / wecom
```
仍走现有 `fetch_ai_news_research``generate.py` research 分支;企微保持一块合并时讯。
## Components
### 1. Skill / prompt`skills/daily-ai-news-research/SKILL.md` + `fetch_ai_news_research` 文案)
更新要点:
- 中英文检索都要做;主列表国内 ≥ `min_cn`(默认 30%),但**只收可信国内源**。
- 删除「主题可重叠」「必须恰好 N 条硬凑」;改为「不足少返回」。
- 同事件只保留一条最权威源;`tech_items` 不得复述 `items` 已覆盖的事件/产品发布。
- 来源优先级:官方 blog/changelog/新闻稿、政府/监管原文、一线权威媒体、学术官方;禁止二手搬运、标题党、不明自媒体、证券营销号(无权威交叉验证则不写)。
- 用户 prompt 从「不区分国内国外,合并精选」改为「合并展示,国内配额硬性,可信度优先」。
### 2. 后处理(`daily/news/research.py`
#### 2.1 国内判定(任一即国内)
1. 可选字段 `region: "cn"|"intl"`(有则优先)。
2. `source_name` / host 命中国内白名单量子位、36氪、雷锋网、IT之家、财新、机器之心、钛媒体、虎嗅等
3. 常见中文域(`.cn` 及已知国内 host
#### 2.2 可信源过滤
- 维护 `TRUSTED_HOSTS` / `TRUSTED_SOURCE_NAMES`(国际 + 国内权威;含官方域如 `openai.com``anthropic.com``blog.google` 等)。
- 不在白名单 → **剔除**(保守:宁可短列表)。
- 官方域优先于二手报道;同事件去重时官方源胜出。
#### 2.3 同事件去重items
- 在 link 去重之后。
- 归一化 title+desc抽取主实体公司/产品)+ 事件动词簇(开源、入侵、联名、适配…)。
- 高置信重叠 → 只留排序更靠前且更可信的一条;保守策略避免误杀明显不同事件。
#### 2.4 tech vs items
- tech 与任一 item 同事件,或同主实体且同属「发布/适配/SDK」簇 → 丢弃 tech。
-items 已有 Kimi K3 开源 → tech 中 HF / 昇腾 / 摩尔线程适配全部剔除。
#### 2.5 国内配额
- `min_cn = env DAILY_WECOM_AI_NEWS_CN_MIN`;若未设或 ≤0`max(1, lim * 3 // 10)`10 → 3
- 去重后尽量保证最终 `items` 中国内 ≥ `min_cn`:国内可信条优先占位,再用国际可信条补满。
- 可信国内不足 → 短列表 + 日志 `news_cn_short:N`**不用低质源灌满**。
### 3. 配置
| 变量 | 默认 | 含义 |
|------|------|------|
| `DAILY_WECOM_AI_NEWS_CN_MIN` | `0`= 按 30% 推算) | 主列表国内最少条数;正整数覆盖 |
`.env.example` 注明;`.env` 可不改(沿用推算)。
### 4. 测试(`tests/test_ai_news_research.py`
- 国内判定(白名单 / `.cn` / `region`)。
- 可信过滤:证券之星类 host 剔除;官方/权威保留。
- 同事件Altman 减速与员工联名若同属「减速/治理请愿」簇 → 只留 1 条;与 HF 入侵若实体+事件动词不同则可分簇保留(保守,避免误杀)。
- techitems 已有 Kimi K3 → 多条 Kimi 适配 tech 全剔除。
- 配额:混合列表打包后国内 ≥ `min_cn`(在可信国内足够时)。
验收可对 `2026-07-29.ai-news-research.json` 跑后处理做人工对照。
## Failure Behavior
| 情况 | 行为 |
|------|------|
| 无 `CURSOR_API_KEY` / Agent 失败 | 与现网一致:`enabled: False` |
| 去重/可信过滤后变短 | 接受短列表;日志 `news_dedup_drop` / `news_cn_short` |
| 可信国内为 0 | 不编造、不 RSS 回填;国际可信条照常(若有) |
## Files Touched
- `skills/daily-ai-news-research/SKILL.md`
- `daily/news/research.py`
- `tests/test_ai_news_research.py`
- `.env.example`(可选配置说明)
## Out of Scope Reminder
不改 RSS 抓取路径;不改 research/rss 模式切换结构;不恢复国内独立企微区块。

View File

@@ -1,3 +0,0 @@
[pytest]
testpaths = tests
pythonpath = .

View File

@@ -1,4 +0,0 @@
python-dotenv>=1.0.0
httpx>=0.27.0
certifi>=2024.0.0
pytest>=8.0

View File

@@ -3,21 +3,16 @@
# .\run-daily.ps1
# .\run-daily.ps1 -SkipPush
# .\run-daily.ps1 -SkipGenerate
# .\run-daily.ps1 -Force # bypass duplicate-run lock
param(
[switch]$SkipPush,
[switch]$SkipGenerate,
[switch]$Force
[switch]$SkipGenerate
)
$ErrorActionPreference = "Stop"
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path
Set-Location $Root
$LockFile = Join-Path $Root ".cache\run-daily.lock"
$LockMaxMinutes = 30
function Import-DotEnvFile {
param([string]$Path)
if (-not (Test-Path $Path)) { return }
@@ -34,49 +29,22 @@ function Import-DotEnvFile {
}
}
function Test-RunDailyLock {
if (-not (Test-Path $LockFile)) { return $false }
$age = (Get-Date) - (Get-Item $LockFile).LastWriteTime
return $age.TotalMinutes -lt $LockMaxMinutes
}
Import-DotEnvFile (Join-Path $Root ".env")
Import-DotEnvFile (Join-Path $Root ".env.local")
function Set-RunDailyLock {
$dir = Split-Path $LockFile -Parent
if (-not (Test-Path $dir)) {
New-Item -ItemType Directory -Path $dir -Force | Out-Null
}
Set-Content -Path $LockFile -Value (Get-Date -Format "o") -Encoding UTF8
}
$python = "python"
$date = Get-Date -Format "yyyy-MM-dd"
$reportWecom = Join-Path $Root "output\$date.wecom.md"
function Clear-RunDailyLock {
if (Test-Path $LockFile) {
Remove-Item $LockFile -Force -ErrorAction SilentlyContinue
}
}
if (-not $Force -and (Test-RunDailyLock)) {
Write-Host "Skip: run-daily already ran within ${LockMaxMinutes} minutes (lock: $LockFile). Use -Force to override."
exit 0
}
Set-RunDailyLock
try {
Import-DotEnvFile (Join-Path $Root ".env")
Import-DotEnvFile (Join-Path $Root ".env.local")
$python = "python"
$date = Get-Date -Format "yyyy-MM-dd"
$reportWecom = Join-Path $Root "output\$date.wecom.md"
if (-not $SkipGenerate) {
if (-not $SkipGenerate) {
Write-Host "Generating daily report: $date"
& $python -m daily generate
if ($LASTEXITCODE -ne 0) {
throw "daily generate failed with exit code $LASTEXITCODE"
}
}
}
if (-not $SkipPush) {
if (-not $SkipPush) {
if (-not (Test-Path $reportWecom)) {
throw "Report not found: $reportWecom"
}
@@ -85,10 +53,6 @@ try {
if ($LASTEXITCODE -ne 0) {
throw "daily push failed with exit code $LASTEXITCODE"
}
}
}
Write-Host "Done: $date"
}
finally {
Clear-RunDailyLock
}
Write-Host "Done: $date"

View File

@@ -1,106 +0,0 @@
#!/usr/bin/env bash
# Daily report: generate + push to WeCom webhook
# Usage:
# ./run-daily.sh
# ./run-daily.sh --skip-push
# ./run-daily.sh --skip-generate
# ./run-daily.sh --force
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$ROOT"
LOCK_FILE="$ROOT/.cache/run-daily.lock"
LOCK_MAX_MINUTES=30
SKIP_PUSH=0
SKIP_GENERATE=0
FORCE=0
while [[ $# -gt 0 ]]; do
case "$1" in
--skip-push) SKIP_PUSH=1 ;;
--skip-generate) SKIP_GENERATE=1 ;;
--force) FORCE=1 ;;
-h|--help)
echo "Usage: $0 [--skip-push] [--skip-generate] [--force]"
exit 0
;;
*)
echo "Unknown option: $1" >&2
exit 1
;;
esac
shift
done
load_dotenv() {
local file="$1"
[[ -f "$file" ]] || return 0
while IFS= read -r line || [[ -n "$line" ]]; do
[[ "$line" =~ ^[[:space:]]*# ]] && continue
[[ "$line" != *"="* ]] && continue
local name="${line%%=*}"
local value="${line#*=}"
name="${name#"${name%%[![:space:]]*}"}"
name="${name%"${name##*[![:space:]]}"}"
value="${value#"${value%%[![:space:]]*}"}"
value="${value%"${value##*[![:space:]]}"}"
value="${value%\"}"
value="${value#\"}"
value="${value%\'}"
value="${value#\'}"
if [[ -n "$name" && -n "$value" ]]; then
export "$name=$value"
fi
done < "$file"
}
lock_active() {
[[ -f "$LOCK_FILE" ]] || return 1
local now lock_ts age_minutes
now="$(date +%s)"
lock_ts="$(date -r "$LOCK_FILE" +%s 2>/dev/null || stat -c %Y "$LOCK_FILE" 2>/dev/null || echo 0)"
age_minutes=$(( (now - lock_ts) / 60 ))
[[ "$age_minutes" -lt "$LOCK_MAX_MINUTES" ]]
}
set_lock() {
mkdir -p "$(dirname "$LOCK_FILE")"
date -Iseconds > "$LOCK_FILE"
}
clear_lock() {
rm -f "$LOCK_FILE"
}
if [[ "$FORCE" -eq 0 ]] && lock_active; then
echo "Skip: run-daily already ran within ${LOCK_MAX_MINUTES} minutes (lock: $LOCK_FILE). Use --force to override."
exit 0
fi
set_lock
trap clear_lock EXIT
load_dotenv "$ROOT/.env"
load_dotenv "$ROOT/.env.local"
PYTHON="${PYTHON:-python}"
DATE="$(TZ=Asia/Shanghai date +%Y-%m-%d)"
REPORT_WECOM="$ROOT/output/${DATE}.wecom.md"
if [[ "$SKIP_GENERATE" -eq 0 ]]; then
echo "Generating daily report: $DATE"
"$PYTHON" -m daily generate
fi
if [[ "$SKIP_PUSH" -eq 0 ]]; then
if [[ ! -f "$REPORT_WECOM" ]]; then
echo "Report not found: $REPORT_WECOM" >&2
exit 1
fi
echo "Pushing to WeCom webhook..."
"$PYTHON" -m daily push "$REPORT_WECOM"
fi
echo "Done: $DATE"

Some files were not shown because too many files have changed in this diff Show More