Compare commits

...

13 Commits

Author SHA1 Message Date
85614f5921 chore(skills): 接入 caveman 系列 agent skills
同步 skills-lock,便于压缩沟通、提交与评审等快捷模式。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-31 15:48:00 +08:00
e2f609a027 feat(news): research 时讯加可信过滤、去重与国内配额
候选池按独立事件拉取,白名单过滤低质源,同事件与 tech 主题去重后按展示上限打包。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-31 15:47:50 +08:00
0384e2c7a9 docs: research AI 时讯质量设计(去重、国内配额、可信源)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-29 16:19:55 +08:00
f166d1e504 fix: LLM 客户端自建大超时 Client,绕开 SDK 默认 60s unary 超时
CURSOR_MODEL=auto 时后端首 token 常超过 SDK 默认 unary_timeout,
自建带 DAILY_CURSOR_UNARY_TIMEOUT(300s)/DAILY_CURSOR_STREAM_TIMEOUT(900s)
的 Client 并显式传入 Agent.prompt,finally 中关闭释放资源。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 17:58:09 +08:00
36085f107d feat: 新增节假日缓存,仅工作日生成与推送
- daily/holiday.py: 抓取 xiaoai.me 全年节假日并缓存到 .cache/holidays-<year>.json,
  is_workday() 判定(法定节假日/周末休息,调休补班日上班),联网失败回退本地缓存
- scheduler: tick_once 前置工作日闸门,非工作日标记完成并跳过,判定失败降级为正常工作日避免漏跑
- config/.env.example: 新增 DAILY_WORKDAY_ONLY 开关(默认开)
- tests: test_holiday 8 例 + scheduler 非工作日闸门 2 例

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 17:51:09 +08:00
d66f2c716c docs: env 示例补充 DAILY_WECOM_TOP_LINE / DAILY_FEATURED_REASON
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 17:08:35 +08:00
d9aef4b340 chore: 删 tmp 调试文件,裸 except 改命名异常+日志
- 删除被跟踪的 tmp_check_desc.py / tmp_desc_snip.txt
- bridge_manager discovery、generate release 抓取的裸
  except Exception 改命名异常 + logger.warning(T8)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 17:01:20 +08:00
d7992e7a7a feat: push_gate 判定分支加结构化日志
force/推送/静默日/无更新仍推 四分支各记一条 logger.info,
无人值守时可定位「早报没推是静默日还是别的原因」(T7)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:55:47 +08:00
3321a307a0 feat: 首推理由行加 DAILY_FEATURED_REASON 开关
pick_why 本已渲染,现加开关默认开,空则省略(T6)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:53:19 +08:00
33bfed0e79 feat: theme_line 取数升级,无评分命中时回退 theme_names
新增 _top_line: 优先评分最高主题,无命中时回退 _theme_clusters 的
主题名(非 markdown 示例),DAILY_WECOM_TOP_LINE=0 退回原逻辑。
复用现有 theme_line 行,不新增头部行(T5)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:44:15 +08:00
ea8de9fe61 test: 新增 generate_report 黄金/确定性回归
冻结 _now_cst、mock 网络/文件/LLM(featured_pick 与 push_gate
有独立测试,此处冻结),验证同一输入下完整版与企微版产物逐字节
一致——拆分 _collect/_select/_render 不得改变行为(T1+T4)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 16:35:42 +08:00
cd92644be4 refactor: 拆 generate_report 为 _collect/_select/_render 三段
所有选择逻辑(9 处 board_select + featured_pick + _localize + pad)
全归 _select,_render 纯拼装写盘,薄壳 generate_report 仅编排。
结构化 bundle 传参;副作用(record_pushed_links/_save_snapshot/
save_json)保持原时序在 _render 末尾。行为不变,89 测试绿(T2)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 14:02:32 +08:00
92e0ed9a27 refactor: _theme_clusters 迁入 narrative_axis 并以注入消除反向依赖
theme_clusters/theme_names 经参数注入 THEME_RULES 与 skill_id_fn,
避免 generate.py 的循环 import(T3,为拆分 generate_report 铺路)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 13:49:37 +08:00
46 changed files with 3635 additions and 164 deletions

View File

@@ -0,0 +1,61 @@
# cavecrew
Decision guide. When to delegate to caveman subagents instead of doing the work inline.
## What it does
Tells the main thread when to spawn a caveman-style subagent versus the vanilla equivalent. The win: subagent tool-results inject back into main context verbatim, and caveman output is roughly 1/3 the size of vanilla prose. Across 20 delegations in one session, that is the difference between context exhaustion and finishing the task.
Three subagents:
| Subagent | Job | Use when |
|----------|-----|----------|
| `cavecrew-investigator` | Locate code (read-only) | "Where is X defined / what calls Y / list uses of Z" |
| `cavecrew-builder` | Surgical edit, 1-2 files | Scope is obvious, ≤2 files. Refuses 3+ file scope. |
| `cavecrew-reviewer` | Diff/file review | One-line findings with severity emoji |
Use vanilla `Explore` or `Code Reviewer` when you want prose, architecture commentary, or rationale. Use main thread directly for one-line answers and 3+ file refactors.
This skill is a decision guide, not a slash command. It activates when the conversation mentions delegation.
## How to invoke
Triggers on phrases like "delegate to subagent", "use cavecrew", "spawn investigator", "save context", "compressed agent output".
## Example chaining
Locate → fix → verify (most common):
1. `cavecrew-investigator` returns site list (`path:line — symbol — note`)
2. Main thread picks 1-2 sites, hands paths to `cavecrew-builder`
3. `cavecrew-reviewer` audits the resulting diff
Parallel scout: spawn 2-3 `cavecrew-investigator` calls in one message with different angles (defs, callers, tests). Aggregate in main.
## Model overrides
By default, `cavecrew-reviewer` and `cavecrew-investigator` pin `model: haiku` in their frontmatter; `cavecrew-builder` has no `model:` line (uses the API session default). Set env vars in your shell before launching Claude Code to override per-agent:
| Env var | Agent |
|---|---|
| `CAVECREW_REVIEWER_MODEL` | `cavecrew-reviewer` |
| `CAVECREW_BUILDER_MODEL` | `cavecrew-builder` |
| `CAVECREW_INVESTIGATOR_MODEL` | `cavecrew-investigator` |
Example — run reviewer on sonnet, keep others on default:
```sh
export CAVECREW_REVIEWER_MODEL=sonnet
```
Use the same model name strings you'd use in any Claude Code agent frontmatter (e.g. `haiku`, `sonnet`, `opus`).
Overrides patch only the `model:` line in the installed agent's frontmatter; the prompt body is untouched and keeps receiving upstream updates. Plugin installs only — standalone hook installs have no local agent files to patch. Unset or blank = no change. The patch persists in the installed file until the plugin is updated or reinstalled.
## See also
- [`SKILL.md`](./SKILL.md) — full decision matrix and output contracts
- [`agents/cavecrew-investigator.md`](../../agents/cavecrew-investigator.md)
- [`agents/cavecrew-builder.md`](../../agents/cavecrew-builder.md)
- [`agents/cavecrew-reviewer.md`](../../agents/cavecrew-reviewer.md)
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,82 @@
---
name: cavecrew
description: >
Decision guide for delegating to caveman-style subagents. Tells the main
thread WHEN to spawn `cavecrew-investigator` (locate code), `cavecrew-builder`
(1-2 file edit), or `cavecrew-reviewer` (diff review) instead of doing the
work inline or using vanilla `Explore`. Subagent output is caveman-compressed
so the tool-result injected back into main context is ~60% smaller — main
context lasts longer across long sessions.
Trigger: "delegate to subagent", "use cavecrew", "spawn investigator/builder/reviewer",
"save context", "compressed agent output".
---
Cavecrew = three subagent presets that emit caveman output. Same job as Anthropic defaults (`Explore`, edit-style agents, reviewer); difference is the tool-result they return is compressed, so main context shrinks per delegation.
## When to use cavecrew vs alternatives
| Task | Use |
|---|---|
| "Where is X defined / what calls Y / list uses of Z" | `cavecrew-investigator` |
| Same but you also want suggestions/architecture commentary | `Explore` (vanilla) |
| Surgical edit, ≤2 files, scope obvious | `cavecrew-builder` |
| New feature / 3+ files / cross-cutting refactor | Main thread or `feature-dev:code-architect` |
| Review diff, branch, or file for bugs | `cavecrew-reviewer` |
| Deep code review with rationale + alternatives | `Code Reviewer` (vanilla) |
| One-line answer you already know | Main thread, no subagent |
Rule of thumb: **if you'd want the subagent's output in 1/3 the tokens, pick cavecrew. If you'd want prose, pick vanilla.**
## Why this exists (the real win)
Subagent tool results get injected into main context verbatim. A vanilla `Explore` that returns 2k tokens of prose costs 2k tokens of main-context budget every time. The same finding from `cavecrew-investigator` returns ~700 tokens. Across 20 delegations in one session that's the difference between context exhaustion and finishing the task.
## Output contracts
What main thread can rely on per agent:
**`cavecrew-investigator`**
```
<Header>:
- path:line — `symbol` — short note
totals: <counts>.
```
Or `No match.` Always file-path-first, line-number-attached, backticked symbols. Safe to grep with `path:\d+`.
**`cavecrew-builder`**
```
<path:line-range> — <change ≤10 words>.
verified: <re-read OK | mismatch @ path:line>.
```
Or one of: `too-big.` / `needs-confirm.` / `ambiguous.` / `regressed.` (terminal first token).
**`cavecrew-reviewer`**
```
path:line: <emoji> <severity>: <problem>. <fix>.
totals: N🔴 N🟡 N🔵 N❓
```
Or `No issues.` Findings sorted file → line ascending.
## Chaining patterns
**Locate → fix → verify** (most common):
1. `cavecrew-investigator` returns site list.
2. Main thread picks 1-2 sites, hands paths to `cavecrew-builder`.
3. `cavecrew-reviewer` audits the diff.
**Parallel scout** (when investigation is broad):
Spawn 2-3 `cavecrew-investigator` calls in one message (different angles: defs vs callers vs tests). Aggregate in main thread.
**Single-shot edit** (when site is already known):
Skip investigator. Hand exact path:line to `cavecrew-builder` directly.
## What NOT to do
- Don't use `cavecrew-builder` when you don't already know the file. Spawn investigator first or main thread will eat tokens passing context.
- Don't chain `cavecrew-investigator → cavecrew-builder` for a 5-file refactor. Builder will return `too-big.` and you'll have wasted a turn.
- Don't ask `cavecrew-reviewer` for "general feedback" — it returns findings only, no architecture opinions. Use `Code Reviewer` for that.
- Don't expect prose. Cavecrew output is structured, sometimes terse to the point of cryptic. If a human will read it directly, paraphrase.
## Auto-clarity (inherited)
Subagents drop caveman → normal English for security warnings, irreversible-action confirmations, and any output where fragment ambiguity could be misread. Resume caveman after.

View File

@@ -0,0 +1,44 @@
# caveman-commit
Terse Conventional Commits. Why over what.
## What it does
Generates commit messages in Conventional Commits format. Subject ≤50 chars, hard cap 72. Imperative mood. Body only when the *why* is non-obvious or there are breaking changes. No AI attribution, no "this commit does X", no emoji unless the project uses them. Body always required for breaking changes, security fixes, data migrations, and reverts — future debuggers need the context.
Outputs only the message. Does not stage, commit, or amend.
## How to invoke
```
/caveman-commit
```
Also triggers on phrases like "write a commit", "commit message", "generate commit".
## Example output
Diff: new endpoint for user profile.
```
feat(api): add GET /users/:id/profile
Mobile client needs profile data without the full user payload
to reduce LTE bandwidth on cold-launch screens.
Closes #128
```
Diff: breaking API rename.
```
feat(api)!: rename /v1/orders to /v1/checkout
BREAKING CHANGE: clients on /v1/orders must migrate to /v1/checkout
before 2026-06-01. Old route returns 410 after that date.
```
## See also
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,65 @@
---
name: caveman-commit
description: >
Ultra-compressed commit message generator. Cuts noise from commit messages while preserving
intent and reasoning. Conventional Commits format. Subject ≤50 chars, body only when "why"
isn't obvious. Use when user says "write a commit", "commit message", "generate commit",
"/commit", or invokes /caveman-commit. Auto-triggers when staging changes.
---
Write commit messages terse and exact. Conventional Commits format. No fluff. Why over what.
## Rules
**Subject line:**
- `<type>(<scope>): <imperative summary>``<scope>` optional
- Types: `feat`, `fix`, `refactor`, `perf`, `docs`, `test`, `chore`, `build`, `ci`, `style`, `revert`
- Imperative mood: "add", "fix", "remove" — not "added", "adds", "adding"
- ≤50 chars when possible, hard cap 72
- No trailing period
- Match project convention for capitalization after the colon
**Body (only if needed):**
- Skip entirely when subject is self-explanatory
- Add body only for: non-obvious *why*, breaking changes, migration notes, linked issues
- Wrap at 72 chars
- Bullets `-` not `*`
- Reference issues/PRs at end: `Closes #42`, `Refs #17`
**What NEVER goes in:**
- "This commit does X", "I", "we", "now", "currently" — the diff says what
- "As requested by..." — use Co-authored-by trailer
- "Generated with Claude Code" or any AI attribution — unless the user's own rule requires an `Assisted-by`/AI-attribution trailer, then add it as a trailer
- Emoji (unless project convention requires)
- Restating the file name when scope already says it
## Examples
Diff: new endpoint for user profile with body explaining the why
- ❌ "feat: add a new endpoint to get user profile information from the database"
-
```
feat(api): add GET /users/:id/profile
Mobile client needs profile data without the full user payload
to reduce LTE bandwidth on cold-launch screens.
Closes #128
```
Diff: breaking API change
- ✅
```
feat(api)!: rename /v1/orders to /v1/checkout
BREAKING CHANGE: clients on /v1/orders must migrate to /v1/checkout
before 2026-06-01. Old route returns 410 after that date.
```
## Auto-Clarity
Always include body for: breaking changes, security fixes, data migrations, anything reverting a prior commit. Never compress these into subject-only — future debuggers need the context.
## Boundaries
Only generates the commit message. Does not run `git commit`, does not stage files, does not amend. Output the message as a code block ready to paste. "stop caveman-commit" or "normal mode": revert to verbose commit style.

View File

@@ -0,0 +1,163 @@
<p align="center">
<img src="https://em-content.zobj.net/source/apple/391/rock_1faa8.png" width="80" />
</p>
<h1 align="center">caveman-compress</h1>
<p align="center">
<strong>shrink memory file. save token every session.</strong>
</p>
---
A Claude Code skill that compresses your project memory files (`CLAUDE.md`, todos, preferences) into caveman format — so every session loads fewer tokens automatically.
Claude read `CLAUDE.md` on every session start. If file big, cost big. Caveman make file small. Cost go down forever.
## What It Do
```
/caveman-compress CLAUDE.md
```
```
CLAUDE.md ← compressed (Claude reads this — fewer tokens every session)
CLAUDE.original.md ← human-readable backup (you edit this)
```
Original never lost. You can read and edit `.original.md`. Run skill again to re-compress after edits.
## Benchmarks
Real results on real project files:
| File | Original | Compressed | Saved |
|------|----------:|----------:|------:|
| `claude-md-preferences.md` | 706 | 285 | **59.6%** |
| `project-notes.md` | 1145 | 535 | **53.3%** |
| `claude-md-project.md` | 1122 | 636 | **43.3%** |
| `todo-list.md` | 627 | 388 | **38.1%** |
| `mixed-with-code.md` | 888 | 560 | **36.9%** |
| **Average** | **898** | **481** | **46%** |
All validations passed ✅ — headings, code blocks, URLs, file paths preserved exactly.
## Before / After
<table>
<tr>
<td width="50%">
### 📄 Original (706 tokens)
> "I strongly prefer TypeScript with strict mode enabled for all new code. Please don't use `any` type unless there's genuinely no way around it, and if you do, leave a comment explaining the reasoning. I find that taking the time to properly type things catches a lot of bugs before they ever make it to runtime."
</td>
<td width="50%">
### <img src="../../docs/assets/dancing-rock.svg" width="20" height="20" alt="rock"/> Caveman (285 tokens)
> "Prefer TypeScript strict mode always. No `any` unless unavoidable — comment why if used. Proper types catch bugs early."
</td>
</tr>
</table>
**Same instructions. 60% fewer tokens. Every. Single. Session.**
## Security
`caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O patterns detected by static analysis. This is a false positive — see [SECURITY.md](./SECURITY.md) for a full explanation of what the skill does and does not do.
## Install
Compress is built in with the `caveman` plugin. Install `caveman` once, then use `/caveman-compress`.
If you need local files, the compress skill lives at:
```bash
caveman-compress/
```
**Requires:** Python 3.10+
## Usage
```
/caveman-compress <filepath>
```
Examples:
```
/caveman-compress CLAUDE.md
/caveman-compress docs/preferences.md
/caveman-compress todos.md
```
### What files work
| Type | Compress? |
|------|-----------|
| `.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex` | ✅ Yes |
| Extensionless natural language | ✅ Yes |
| `.py`, `.js`, `.ts`, `.json`, `.yaml` | ❌ Skip (code/config) |
| `*.original.md` | ❌ Skip (backup files) |
## How It Work
```
/caveman-compress CLAUDE.md
detect file type (no tokens)
Claude compresses (tokens — one call)
validate output (no tokens)
checks: headings, code blocks, URLs, file paths, bullets
if errors: Claude fixes cherry-picked issues only (tokens — targeted fix)
does NOT recompress — only patches broken parts
retry up to 2 times
write compressed → CLAUDE.md
write original → CLAUDE.original.md
```
Only two things use tokens: initial compression + targeted fix if validation fails. Everything else is local Python.
## What Is Preserved
Caveman compress natural language. It never touch:
- Code blocks (` ``` ` fenced or indented)
- Inline code (`` `backtick content` ``)
- URLs and links
- File paths (`/src/components/...`)
- Commands (`npm install`, `git commit`)
- Technical terms, library names, API names
- Headings (exact text preserved)
- Tables (structure preserved, cell text compressed)
- Dates, version numbers, numeric values
## Why This Matter
`CLAUDE.md` loads on **every session start**. A 1000-token project memory file costs tokens every single time you open a project. Over 100 sessions that's 100,000 tokens of overhead — just for context you already wrote.
Caveman cut that by ~46% on average. Same instructions. Same accuracy. Less waste.
```
┌────────────────────────────────────────────┐
│ TOKEN SAVINGS PER FILE █████ 46% │
│ SESSIONS THAT BENEFIT ██████████ 100% │
│ INFORMATION PRESERVED ██████████ 100% │
│ SETUP TIME █ 1x │
└────────────────────────────────────────────┘
```
## Part of Caveman
This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit — making Claude use fewer tokens without losing accuracy.
- **caveman** — make Claude *speak* like caveman (cuts response tokens ~65%)
- **caveman-compress** — make Claude *read* less (cuts context tokens ~46%)

View File

@@ -0,0 +1,31 @@
# Security
## Snyk High Risk Rating
`caveman-compress` receives a Snyk High Risk rating due to static analysis heuristics. This document explains what the skill does and does not do.
### What triggers the rating
1. **subprocess usage**: The skill calls the `claude` CLI via `subprocess.run()` as a fallback when `ANTHROPIC_API_KEY` is not set. The subprocess call uses a fixed argument list — no shell interpolation occurs. User file content is passed via stdin, not as a shell argument.
2. **File read/write**: The skill reads the file the user explicitly points it at, compresses it, and writes the result back to the same path. A `.original.md` backup is saved alongside it. No files outside the user-specified path are read or written.
### What the skill does NOT do
- Does not execute user file content as code
- Does not make network requests except to Anthropic's API (via SDK or CLI)
- Does not access files outside the path the user provides
- Does not use shell=True or string interpolation in subprocess calls
- Does not collect or transmit any data beyond the file being compressed
### Auth behavior
If `ANTHROPIC_API_KEY` is set, the skill uses the Anthropic Python SDK directly (no subprocess). If not set, it falls back to the `claude` CLI, which uses the user's existing Claude desktop authentication.
### File size limit
Files larger than 500KB are rejected before any API call is made.
### Reporting a vulnerability
If you believe you've found a genuine security issue, please open a GitHub issue with the label `security`.

View File

@@ -0,0 +1,111 @@
---
name: caveman-compress
description: >
Compress natural language memory files (CLAUDE.md, todos, preferences) into caveman format
to save input tokens. Preserves all technical substance, code, URLs, and structure.
Compressed version overwrites the original file. Human-readable backup saved as FILE.original.md.
Trigger: /caveman-compress FILEPATH or "compress memory file"
---
# Caveman Compress
## Purpose
Compress natural language files (CLAUDE.md, todos, preferences) into caveman-speak to reduce input tokens. Compressed version overwrites original. Human-readable backup saved as `<filename>.original.md`.
## Trigger
`/caveman-compress <filepath>` or when user asks to compress a memory file.
## Process
1. The compression scripts live in `scripts/` (adjacent to this SKILL.md). If the path is not immediately available, search for `scripts/__main__.py` next to this SKILL.md.
2. From the directory containing this SKILL.md, run:
python3 -m scripts <absolute_filepath>
3. The CLI will:
- detect file type (no tokens)
- call Claude to compress
- validate output (no tokens)
- if errors: cherry-pick fix with Claude (targeted fixes only, no recompression)
- retry up to 2 times
- if still failing after 2 retries: report error to user, leave original file untouched
4. Return result to user
## Compression Rules
### Remove
- Articles: a, an, the
- Filler: just, really, basically, actually, simply, essentially, generally
- Pleasantries: "sure", "certainly", "of course", "happy to", "I'd recommend"
- Hedging: "it might be worth", "you could consider", "it would be good to"
- Redundant phrasing: "in order to" → "to", "make sure to" → "ensure", "the reason is because" → "because"
- Connective fluff: "however", "furthermore", "additionally", "in addition"
### Preserve EXACTLY (never modify)
- Code blocks (fenced ``` and indented)
- Inline code (`backtick content`)
- URLs and links (full URLs, markdown links)
- File paths (`/src/components/...`, `./config.yaml`)
- Commands (`npm install`, `git commit`, `docker build`)
- Technical terms (library names, API names, protocols, algorithms)
- Proper nouns (project names, people, companies)
- Dates, version numbers, numeric values
- Environment variables (`$HOME`, `NODE_ENV`)
### Preserve Structure
- All markdown headings (keep exact heading text, compress body below)
- Bullet point hierarchy (keep nesting level)
- Numbered lists (keep numbering)
- Tables (compress cell text, keep structure)
- Frontmatter/YAML headers in markdown files
### Compress
- Use short synonyms: "big" not "extensive", "fix" not "implement a solution for", "use" not "utilize"
- Fragments OK: "Run tests before commit" not "You should always run tests before committing"
- Drop "you should", "make sure to", "remember to" — just state the action
- Merge redundant bullets that say the same thing differently
- Keep one example where multiple examples show the same pattern
CRITICAL RULE:
Anything inside ``` ... ``` must be copied EXACTLY.
Do not:
- remove comments
- remove spacing
- reorder lines
- shorten commands
- simplify anything
Inline code (`...`) must be preserved EXACTLY.
Do not modify anything inside backticks.
If file contains code blocks:
- Treat code blocks as read-only regions
- Only compress text outside them
- Do not merge sections around code
## Pattern
Original:
> You should always make sure to run the test suite before pushing any changes to the main branch. This is important because it helps catch bugs early and prevents broken builds from being deployed to production.
Compressed:
> Run tests before push to main. Catch bugs early, prevent broken prod deploys.
Original:
> The application uses a microservices architecture with the following components. The API gateway handles all incoming requests and routes them to the appropriate service. The authentication service is responsible for managing user sessions and JWT tokens.
Compressed:
> Microservices architecture. API gateway route all requests to services. Auth service manage user sessions + JWT tokens.
## Boundaries
- ONLY compress natural language files (.md, .txt, .typ, .typst, .tex, extensionless)
- NEVER modify: .py, .js, .ts, .json, .yaml, .yml, .toml, .env, .lock, .css, .html, .xml, .sql, .sh
- If file has mixed content (prose + code), compress ONLY the prose sections
- If unsure whether something is code or prose, leave it unchanged
- Original file is backed up as FILE.original.md before overwriting
- Never compress FILE.original.md (skip it)

View File

@@ -0,0 +1,9 @@
"""Caveman compress scripts.
This package provides tools to compress natural language markdown files
into caveman format to save input tokens.
"""
__all__ = ["cli", "compress", "detect", "validate"]
__version__ = "1.0.0"

View File

@@ -0,0 +1,3 @@
from .cli import main
main()

View File

@@ -0,0 +1,80 @@
#!/usr/bin/env python3
from pathlib import Path
import sys
# Support both direct execution and module import
try:
from .validate import validate
except ImportError:
sys.path.insert(0, str(Path(__file__).parent))
from validate import validate
try:
import tiktoken
_enc = tiktoken.get_encoding("o200k_base")
except ImportError:
_enc = None
def count_tokens(text):
if _enc is None:
return len(text.split()) # fallback: word count
return len(_enc.encode(text))
def benchmark_pair(orig_path: Path, comp_path: Path):
orig_text = orig_path.read_text()
comp_text = comp_path.read_text()
orig_tokens = count_tokens(orig_text)
comp_tokens = count_tokens(comp_text)
saved = 100 * (orig_tokens - comp_tokens) / orig_tokens if orig_tokens > 0 else 0.0
result = validate(orig_path, comp_path)
return (comp_path.name, orig_tokens, comp_tokens, saved, result.is_valid)
def print_table(rows):
print("\n| File | Original | Compressed | Saved % | Valid |")
print("|------|----------|------------|---------|-------|")
for r in rows:
print(f"| {r[0]} | {r[1]} | {r[2]} | {r[3]:.1f}% | {'' if r[4] else ''} |")
def main():
# Direct file pair: python3 benchmark.py original.md compressed.md
if len(sys.argv) == 3:
orig = Path(sys.argv[1]).resolve()
comp = Path(sys.argv[2]).resolve()
if not orig.exists():
print(f"❌ Not found: {orig}")
sys.exit(1)
if not comp.exists():
print(f"❌ Not found: {comp}")
sys.exit(1)
print_table([benchmark_pair(orig, comp)])
return
# Glob mode: repo_root/tests/caveman-compress/
# __file__ lives at <repo_root>/skills/caveman-compress/scripts/benchmark.py
# Walk up four dirs: scripts → caveman-compress → skills → repo_root.
tests_dir = Path(__file__).resolve().parents[3] / "tests" / "caveman-compress"
if not tests_dir.exists():
print(f"❌ Tests dir not found: {tests_dir}")
sys.exit(1)
rows = []
for orig in sorted(tests_dir.glob("*.original.md")):
comp = orig.with_name(orig.stem.removesuffix(".original") + ".md")
if comp.exists():
rows.append(benchmark_pair(orig, comp))
if not rows:
print("No compressed file pairs found.")
return
print_table(rows)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,85 @@
#!/usr/bin/env python3
"""
Caveman Compress CLI
Usage:
caveman <filepath>
"""
import sys
# Force UTF-8 on stdout/stderr before any code can print. Windows consoles
# default to cp1252 and crash on the ❌ glyphs in error/validation branches,
# masking the real error and leaving the user with a half-compressed file.
for _stream in (sys.stdout, sys.stderr):
reconfigure = getattr(_stream, "reconfigure", None)
if callable(reconfigure):
try:
reconfigure(encoding="utf-8", errors="replace")
except Exception:
pass
from pathlib import Path
from .compress import backup_dir_for, compress_file
from .detect import detect_file_type, should_compress
def print_usage():
print("Usage: caveman <filepath>")
def main():
if len(sys.argv) != 2:
print_usage()
sys.exit(1)
filepath = Path(sys.argv[1])
# Check file exists
if not filepath.exists():
print(f"❌ File not found: {filepath}")
sys.exit(1)
if not filepath.is_file():
print(f"❌ Not a file: {filepath}")
sys.exit(1)
filepath = filepath.resolve()
# Detect file type
file_type = detect_file_type(filepath)
print(f"Detected: {file_type}")
# Check if compressible
if not should_compress(filepath):
print("Skipping: file is not natural language (code/config)")
sys.exit(0)
print("Starting caveman compression...\n")
try:
success = compress_file(filepath)
if success:
print("\nCompression completed successfully")
backup_path = backup_dir_for(filepath) / (filepath.stem + ".original.md")
print(f"Compressed: {filepath}")
print(f"Original: {backup_path}")
sys.exit(0)
else:
print("\n❌ Compression failed after retries")
sys.exit(2)
except KeyboardInterrupt:
print("\nInterrupted by user")
sys.exit(130)
except Exception as e:
print(f"\n❌ Error: {e}")
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,342 @@
#!/usr/bin/env python3
"""
Caveman Memory Compression Orchestrator
Usage:
python scripts/compress.py <filepath>
"""
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path
from typing import List
OUTER_FENCE_REGEX = re.compile(
r"\A\s*(`{3,}|~{3,})[^\n]*\n(.*)\n\1\s*\Z", re.DOTALL
)
# YAML frontmatter: starts at file start with --- on its own line, ends with --- on its own line.
# Captures the entire block (including delimiters and trailing newline) and the body after.
FRONTMATTER_REGEX = re.compile(
r"\A(---\r?\n.*?\r?\n---\r?\n)(.*)", re.DOTALL
)
def split_frontmatter(text: str):
"""Split YAML frontmatter from body. Returns (frontmatter, body).
Memory files (and many other markdown docs) start with a YAML frontmatter
block delimited by `---` lines. The compression LLM has a habit of stripping
or rewriting these despite preserve-structure rules in the prompt — so we
surgically remove the frontmatter before compression and prepend it back
verbatim to the output. Files without frontmatter pass through unchanged.
"""
m = FRONTMATTER_REGEX.match(text)
if m:
return m.group(1), m.group(2)
return "", text
# Filenames and paths that almost certainly hold secrets or PII. Compressing
# them ships raw bytes to the Anthropic API — a third-party data boundary that
# developers on sensitive codebases cannot cross. detect.py already skips .env
# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would
# slip through the natural-language filter. This is a hard refuse before read.
SENSITIVE_BASENAME_REGEX = re.compile(
r"(?ix)^("
r"\.env(\..+)?"
r"|\.netrc"
r"|credentials(\..+)?"
r"|secrets?(\..+)?"
r"|passwords?(\..+)?"
r"|id_(rsa|dsa|ecdsa|ed25519)(\.pub)?"
r"|authorized_keys"
r"|known_hosts"
r"|.*\.(pem|key|p12|pfx|crt|cer|jks|keystore|asc|gpg)"
r")$"
)
SENSITIVE_PATH_COMPONENTS = frozenset({".ssh", ".aws", ".gnupg", ".kube", ".docker"})
SENSITIVE_NAME_TOKENS = (
"secret", "credential", "password", "passwd",
"apikey", "accesskey", "token", "privatekey",
)
def backup_dir_for(filepath: Path) -> Path:
"""Resolve the out-of-tree backup directory for a given source file.
Backups must live OUTSIDE the source directory so skill auto-loaders
(Claude Code rules/, opencode instructions/, etc.) stop re-ingesting the
`.original.md` copies as live files. Base dir is platform-aware:
- Windows: %LOCALAPPDATA%\\caveman-compress\\backups
- else: $XDG_DATA_HOME/caveman-compress/backups if set,
else ~/.local/share/caveman-compress/backups
The source file's parent-dir name is mirrored under the base to reduce
cross-project collisions (e.g. two `task.md` files in different repos).
"""
if os.name == "nt" or sys.platform == "win32":
local_appdata = os.environ.get("LOCALAPPDATA")
base = Path(local_appdata) if local_appdata else Path.home() / "AppData" / "Local"
base = base / "caveman-compress" / "backups"
else:
xdg = os.environ.get("XDG_DATA_HOME")
base = Path(xdg) if xdg else Path.home() / ".local" / "share"
base = base / "caveman-compress" / "backups"
return base / filepath.parent.name
def is_sensitive_path(filepath: Path) -> bool:
"""Heuristic denylist for files that must never be shipped to a third-party API."""
name = filepath.name
if SENSITIVE_BASENAME_REGEX.match(name):
return True
lowered_parts = {p.lower() for p in filepath.parts}
if lowered_parts & SENSITIVE_PATH_COMPONENTS:
return True
# Normalize separators so "api-key" and "api_key" both match "apikey".
lower = re.sub(r"[_\-\s.]", "", name.lower())
return any(tok in lower for tok in SENSITIVE_NAME_TOKENS)
def strip_llm_wrapper(text: str) -> str:
"""Strip outer ```markdown ... ``` fence when it wraps the entire output."""
m = OUTER_FENCE_REGEX.match(text)
if m:
return m.group(2)
return text
from .detect import should_compress
from .validate import validate
MAX_RETRIES = 2
# ---------- Claude Calls ----------
def call_claude(prompt: str) -> str:
"""Send a prompt to Claude.
Prefers the Anthropic SDK when ANTHROPIC_API_KEY is set; otherwise falls
back to the ``claude --print`` CLI (which handles desktop auth).
On Windows the CLI subprocess decoding defaults to the system codepage
(cp1251 / cp1252) and crashes on UTF-8 output — see issue #152. Pinning
``encoding="utf-8"`` with ``errors="replace"`` matches the CLI's actual
native I/O and prevents the UnicodeDecodeError before validation can
report. Windows users with non-ASCII content can also set
``ANTHROPIC_API_KEY`` to route through the SDK and skip the subprocess.
"""
api_key = os.environ.get("ANTHROPIC_API_KEY")
if api_key:
try:
import anthropic
client = anthropic.Anthropic(api_key=api_key)
msg = client.messages.create(
model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),
max_tokens=8192,
messages=[{"role": "user", "content": prompt}],
)
return strip_llm_wrapper(msg.content[0].text.strip())
except ImportError:
pass # anthropic not installed, fall back to CLI
# Fallback: use claude CLI (handles desktop auth).
# Resolve binary via shutil.which so Windows .cmd/.bat shims (e.g.
# %APPDATA%\npm\claude.CMD) work without shell=True. On POSIX,
# shutil.which returns the same absolute path as the implicit lookup,
# so this is a no-op there. Falls back to bare "claude" if not found
# on PATH so subprocess raises a clear FileNotFoundError.
claude_bin = shutil.which("claude") or "claude"
try:
result = subprocess.run(
[claude_bin, "--print"],
input=prompt,
text=True,
capture_output=True,
check=True,
encoding="utf-8",
errors="replace",
)
return strip_llm_wrapper(result.stdout.strip())
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Claude call failed:\n{e.stderr}")
def build_compress_prompt(original: str) -> str:
return f"""
Compress this markdown into caveman format.
STRICT RULES:
- Do NOT modify anything inside ``` code blocks
- Do NOT modify anything inside inline backticks
- Preserve ALL URLs exactly
- Preserve ALL headings exactly
- Preserve file paths and commands
- Return ONLY the compressed markdown body — do NOT wrap the entire output in a ```markdown fence or any other fence. Inner code blocks from the original stay as-is; do not add a new outer fence around the whole file.
Only compress natural language.
TEXT:
{original}
"""
def build_fix_prompt(original: str, compressed: str, errors: List[str]) -> str:
errors_str = "\n".join(f"- {e}" for e in errors)
return f"""You are fixing a caveman-compressed markdown file. Specific validation errors were found.
CRITICAL RULES:
- DO NOT recompress or rephrase the file
- ONLY fix the listed errors — leave everything else exactly as-is
- The ORIGINAL is provided as reference only (to restore missing content)
- Preserve caveman style in all untouched sections
ERRORS TO FIX:
{errors_str}
HOW TO FIX:
- Missing URL: find it in ORIGINAL, restore it exactly where it belongs in COMPRESSED
- Code block mismatch: find the exact code block in ORIGINAL, restore it in COMPRESSED
- Heading mismatch: restore the exact heading text from ORIGINAL into COMPRESSED
- Do not touch any section not mentioned in the errors
ORIGINAL (reference only):
{original}
COMPRESSED (fix this):
{compressed}
Return ONLY the fixed compressed file. No explanation.
"""
# ---------- Core Logic ----------
def compress_file(filepath: Path) -> bool:
# Resolve and validate path
filepath = filepath.resolve()
MAX_FILE_SIZE = 500_000 # 500KB
if not filepath.exists():
raise FileNotFoundError(f"File not found: {filepath}")
if filepath.stat().st_size > MAX_FILE_SIZE:
raise ValueError(f"File too large to compress safely (max 500KB): {filepath}")
# Refuse files that look like they contain secrets or PII. Compressing ships
# the raw bytes to the Anthropic API — a third-party boundary — so we fail
# loudly rather than silently exfiltrate credentials or keys. Override is
# intentional: the user must rename the file if the heuristic is wrong.
if is_sensitive_path(filepath):
raise ValueError(
f"Refusing to compress {filepath}: filename looks sensitive "
"(credentials, keys, secrets, or known private paths). "
"Compression sends file contents to the Anthropic API. "
"Rename the file if this is a false positive."
)
print(f"Processing: {filepath}")
if not should_compress(filepath):
print("Skipping (not natural language)")
return False
original_text = filepath.read_text(errors="ignore")
# Store backup outside the source directory so skill auto-loaders don't
# re-ingest the `.original.md` copy as a live file. Mirror the source's
# parent-dir name + stem under a platform-aware base to reduce collisions.
backup_dir = backup_dir_for(filepath)
backup_dir.mkdir(parents=True, exist_ok=True)
backup_path = backup_dir / (filepath.stem + ".original.md")
if not original_text.strip():
print("❌ Refusing to compress: file is empty or whitespace-only.")
return False
# Check if backup already exists to prevent accidental overwriting
if backup_path.exists():
print(f"⚠️ Backup file already exists: {backup_path}")
print("The original backup may contain important content.")
print("Aborting to prevent data loss. Please remove or rename the backup file if you want to proceed.")
return False
# Split YAML frontmatter off before compression. Claude tends to strip or
# rewrite frontmatter despite preserve-structure rules; we keep it verbatim
# by removing it from the input and re-prepending it to the output.
frontmatter, body = split_frontmatter(original_text)
if frontmatter:
print(f"Detected YAML frontmatter ({len(frontmatter)} chars) — preserving verbatim")
if not body.strip():
print("❌ Refusing to compress: body is empty after frontmatter removal.")
return False
# Step 1: Compress (body only, frontmatter excluded)
print("Compressing with Claude...")
compressed_body = call_claude(build_compress_prompt(body))
if compressed_body is None or not compressed_body.strip():
print("❌ Compression aborted: Claude returned an empty response.")
print(" Original file is untouched (no backup created).")
return False
# Compare the BODY (not the whole file) — frontmatter is preserved verbatim
# and would never change, so identity must be judged on the compressible part.
if compressed_body.strip() == body.strip():
print("❌ Compression aborted: output is identical to input.")
print(" Likely causes: Claude refused, returned the prompt verbatim, or the file is")
print(" already in caveman form. Original file is untouched (no backup created).")
return False
# Reassemble: frontmatter (verbatim) + compressed body
compressed = frontmatter + compressed_body
# Save original as backup, then verify the backup readback before
# touching the input file. If the filesystem dropped bytes (encoding,
# antivirus, disk full), unlink the bad backup and abort instead of
# leaving the user with a corrupt backup + compressed primary.
backup_path.write_text(original_text)
backup_readback = backup_path.read_text(errors="ignore")
if backup_readback != original_text:
print(f"❌ Backup write verification failed: {backup_path}")
print(" In-memory original differs from on-disk backup. Aborting before touching the input file.")
try:
backup_path.unlink()
except OSError:
pass
return False
filepath.write_text(compressed)
# Step 2: Validate + Retry
for attempt in range(MAX_RETRIES):
print(f"\nValidation attempt {attempt + 1}")
result = validate(backup_path, filepath)
if result.is_valid:
print("Validation passed")
break
print("❌ Validation failed:")
for err in result.errors:
print(f" - {err}")
if attempt == MAX_RETRIES - 1:
# Restore original on failure
filepath.write_text(original_text)
backup_path.unlink(missing_ok=True)
print("❌ Failed after retries — original restored")
return False
print("Fixing with Claude...")
compressed = call_claude(
build_fix_prompt(original_text, compressed, result.errors)
)
filepath.write_text(compressed)
return True

View File

@@ -0,0 +1,139 @@
#!/usr/bin/env python3
"""Detect whether a file is natural language (compressible) or code/config (skip)."""
import json
import re
from pathlib import Path
# Extensions that are natural language and compressible
COMPRESSIBLE_EXTENSIONS = {".md", ".txt", ".markdown", ".rst", ".typ", ".typst", ".tex"}
# Extensions that are code/config and should be skipped
SKIP_EXTENSIONS = {
".py", ".js", ".ts", ".tsx", ".jsx", ".json", ".yaml", ".yml",
".toml", ".env", ".lock", ".css", ".scss", ".html", ".xml",
".sql", ".sh", ".bash", ".zsh", ".go", ".rs", ".java", ".c",
".cpp", ".h", ".hpp", ".rb", ".php", ".swift", ".kt", ".lua",
".dockerfile", ".makefile", ".csv", ".ini", ".cfg",
}
# Well-known build/config files that carry no (or a misleading) extension —
# `Dockerfile` has no suffix so `.dockerfile` above never matches it, and
# `CMakeLists.txt` would ride the compressible `.txt` rule. Checked by
# basename before any extension rule.
KNOWN_CODE_FILENAMES = {
"dockerfile", "makefile", "gnumakefile", "jenkinsfile", "vagrantfile",
"rakefile", "gemfile", "justfile", "procfile", "brewfile",
"cmakelists.txt",
}
# Patterns that indicate a line is code
CODE_PATTERNS = [
re.compile(r"^\s*(import |from .+ import |require\(|const |let |var )"),
re.compile(r"^\s*(def |class |function |async function |export )"),
re.compile(r"^\s*(if\s*\(|for\s*\(|while\s*\(|switch\s*\(|try\s*\{)"),
re.compile(r"^\s*[\}\]\);]+\s*$"), # closing braces/brackets
re.compile(r"^\s*@\w+"), # decorators/annotations
re.compile(r'^\s*"[^"]+"\s*:\s*'), # JSON-like key-value
re.compile(r"^\s*\w+\s*=\s*[{\[\(\"']"), # assignment with literal
]
def _is_code_line(line: str) -> bool:
"""Check if a line looks like code."""
return any(p.match(line) for p in CODE_PATTERNS)
def _is_json_content(text: str) -> bool:
"""Check if content is valid JSON."""
try:
json.loads(text)
return True
except (json.JSONDecodeError, ValueError):
return False
def _is_yaml_content(lines: list[str]) -> bool:
"""Heuristic: check if content looks like YAML."""
yaml_indicators = 0
for line in lines[:30]:
stripped = line.strip()
if stripped.startswith("---"):
yaml_indicators += 1
elif re.match(r"^\w[\w\s]*:\s", stripped):
yaml_indicators += 1
elif stripped.startswith("- ") and ":" in stripped:
yaml_indicators += 1
# If most non-empty lines look like YAML
non_empty = sum(1 for l in lines[:30] if l.strip())
return non_empty > 0 and yaml_indicators / non_empty > 0.6
def detect_file_type(filepath: Path) -> str:
"""Classify a file as 'natural_language', 'code', 'config', or 'unknown'.
Returns:
One of: 'natural_language', 'code', 'config', 'unknown'
"""
ext = filepath.suffix.lower()
# Known code filenames win over any extension rule
if filepath.name.lower() in KNOWN_CODE_FILENAMES:
return "code"
# Extension-based classification
if ext in COMPRESSIBLE_EXTENSIONS:
return "natural_language"
if ext in SKIP_EXTENSIONS:
return "code" if ext not in {".json", ".yaml", ".yml", ".toml", ".ini", ".cfg", ".env"} else "config"
# Extensionless files (like CLAUDE.md, TODO) — check content
if not ext:
try:
text = filepath.read_text(errors="ignore")
except (OSError, PermissionError):
return "unknown"
lines = text.splitlines()[:50]
# Shebang means executable script, never prose
if text.startswith("#!"):
return "code"
if _is_json_content(text[:10000]):
return "config"
if _is_yaml_content(lines):
return "config"
code_lines = sum(1 for l in lines if l.strip() and _is_code_line(l))
non_empty = sum(1 for l in lines if l.strip())
if non_empty > 0 and code_lines / non_empty > 0.4:
return "code"
return "natural_language"
return "unknown"
def should_compress(filepath: Path) -> bool:
"""Return True if the file is natural language and should be compressed."""
if not filepath.is_file():
return False
# Skip backup files
if filepath.name.endswith(".original.md"):
return False
return detect_file_type(filepath) == "natural_language"
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("Usage: python detect.py <file1> [file2] ...")
sys.exit(1)
for path_str in sys.argv[1:]:
p = Path(path_str).resolve()
file_type = detect_file_type(p)
compress = should_compress(p)
print(f" {p.name:30s} type={file_type:20s} compress={compress}")

View File

@@ -0,0 +1,213 @@
#!/usr/bin/env python3
import re
from collections import Counter
from pathlib import Path
URL_REGEX = re.compile(r"https?://[^\s)]+")
FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
# crude but effective path detection
# Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
class ValidationResult:
def __init__(self):
self.is_valid = True
self.errors = []
self.warnings = []
def add_error(self, msg):
self.is_valid = False
self.errors.append(msg)
def add_warning(self, msg):
self.warnings.append(msg)
def read_file(path: Path) -> str:
return path.read_text(errors="ignore")
# ---------- Extractors ----------
def extract_headings(text):
return [(level, title.strip()) for level, title in HEADING_REGEX.findall(text)]
def extract_code_blocks(text):
"""Line-based fenced code block extractor.
Handles ``` and ~~~ fences with variable length (CommonMark: closing
fence must use same char and be at least as long as opening). Supports
nested fences (e.g. an outer 4-backtick block wrapping inner 3-backtick
content).
"""
blocks = []
lines = text.split("\n")
i = 0
n = len(lines)
while i < n:
m = FENCE_OPEN_REGEX.match(lines[i])
if not m:
i += 1
continue
fence_char = m.group(2)[0]
fence_len = len(m.group(2))
open_line = lines[i]
block_lines = [open_line]
i += 1
closed = False
while i < n:
close_m = FENCE_OPEN_REGEX.match(lines[i])
if (
close_m
and close_m.group(2)[0] == fence_char
and len(close_m.group(2)) >= fence_len
and close_m.group(3).strip() == ""
):
block_lines.append(lines[i])
closed = True
i += 1
break
block_lines.append(lines[i])
i += 1
if closed:
blocks.append("\n".join(block_lines))
# Unclosed fences are silently skipped — they indicate malformed markdown
# and including them would cause false-positive validation failures.
return blocks
def extract_urls(text):
return set(URL_REGEX.findall(text))
def extract_paths(text):
return set(PATH_REGEX.findall(text))
def count_bullets(text):
return len(BULLET_REGEX.findall(text))
def extract_inline_codes(text):
text_without_fences = re.sub(r"^```[\s\S]*?^```", "", text, flags=re.MULTILINE)
text_without_fences = re.sub(r"^~~~[\s\S]*?^~~~", "", text_without_fences, flags=re.MULTILINE)
return re.findall(r"`([^`]+)`", text_without_fences)
# ---------- Validators ----------
def validate_headings(orig, comp, result):
h1 = extract_headings(orig)
h2 = extract_headings(comp)
if len(h1) != len(h2):
result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
if h1 != h2:
result.add_warning("Heading text/order changed")
def validate_code_blocks(orig, comp, result):
c1 = extract_code_blocks(orig)
c2 = extract_code_blocks(comp)
if c1 != c2:
result.add_error("Code blocks not preserved exactly")
def validate_urls(orig, comp, result):
u1 = extract_urls(orig)
u2 = extract_urls(comp)
if u1 != u2:
result.add_error(f"URL mismatch: lost={u1 - u2}, added={u2 - u1}")
def validate_paths(orig, comp, result):
p1 = extract_paths(orig)
p2 = extract_paths(comp)
if p1 != p2:
result.add_warning(f"Path mismatch: lost={p1 - p2}, added={p2 - p1}")
def validate_bullets(orig, comp, result):
b1 = count_bullets(orig)
b2 = count_bullets(comp)
if b1 == 0:
return
diff = abs(b1 - b2) / b1
if diff > 0.15:
result.add_warning(f"Bullet count changed too much: {b1} -> {b2}")
def validate_inline_codes(orig, comp, result):
c1 = Counter(extract_inline_codes(orig))
c2 = Counter(extract_inline_codes(comp))
if c1 != c2:
lost = set(c1.keys()) - set(c2.keys())
added = set(c2.keys()) - set(c1.keys())
for code, count in c1.items():
if code in c2 and c2[code] < count:
lost.add(f"{code} (lost {count - c2[code]} of {count} occurrences)")
if lost:
result.add_error(f"Inline code lost: {lost}")
if added:
result.add_warning(f"Inline code added: {added}")
# ---------- Main ----------
def validate(original_path: Path, compressed_path: Path) -> ValidationResult:
result = ValidationResult()
orig = read_file(original_path)
comp = read_file(compressed_path)
validate_headings(orig, comp, result)
validate_code_blocks(orig, comp, result)
validate_urls(orig, comp, result)
validate_paths(orig, comp, result)
validate_bullets(orig, comp, result)
validate_inline_codes(orig, comp, result)
return result
# ---------- CLI ----------
if __name__ == "__main__":
import sys
if len(sys.argv) != 3:
print("Usage: python validate.py <original> <compressed>")
sys.exit(1)
orig = Path(sys.argv[1]).resolve()
comp = Path(sys.argv[2]).resolve()
res = validate(orig, comp)
print(f"\nValid: {res.is_valid}")
if res.errors:
print("\nErrors:")
for e in res.errors:
print(f" - {e}")
if res.warnings:
print("\nWarnings:")
for w in res.warnings:
print(f" - {w}")

View File

@@ -0,0 +1,38 @@
# caveman-help
Quick-reference card. One shot, no mode change.
## What it does
Prints a cheat sheet of all caveman modes, sibling skills, deactivation triggers, and how to set the default mode via env var or config file. One-shot display — does not flip the active mode, write flag files, or persist anything. Use when you forget the slash commands.
## How to invoke
```
/caveman-help
```
Also triggers on "caveman help", "what caveman commands", "how do I use caveman".
## Example output
```
Modes:
/caveman full (default)
/caveman lite lighter
/caveman ultra extreme
/caveman wenyan classical Chinese
Skills:
/caveman-commit terse Conventional Commits
/caveman-review one-line PR comments
/caveman-stats session token savings
Deactivate:
"stop caveman" or "normal mode"
```
## See also
- [`SKILL.md`](./SKILL.md) — full reference card
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,63 @@
---
name: caveman-help
description: >
Quick-reference card for all caveman modes, skills, and commands.
One-shot display, not a persistent mode. Trigger: /caveman-help,
"caveman help", "what caveman commands", "how do I use caveman".
---
# Caveman Help
Display this reference card when invoked. One-shot — do NOT change mode, write flag files, or persist anything. Output in caveman style.
## Modes
| Mode | Trigger | What change |
|------|---------|-------------|
| **Lite** | `/caveman lite` | Drop filler. Keep sentence structure. |
| **Full** | `/caveman` | Drop articles, filler, pleasantries, hedging. Fragments OK. Default. |
| **Ultra** | `/caveman ultra` | Extreme compression. Bare fragments. Tables over prose. |
| **Wenyan-Lite** | `/caveman wenyan-lite` | Classical Chinese style, light compression. |
| **Wenyan-Full** | `/caveman wenyan` | Full 文言文. Maximum classical terseness. |
| **Wenyan-Ultra** | `/caveman wenyan-ultra` | Extreme. Ancient scholar on a budget. |
Mode stick until changed or session end.
## Skills
| Skill | Trigger | What it do |
|-------|---------|-----------|
| **caveman-commit** | `/caveman-commit` | Terse commit messages. Conventional Commits. ≤50 char subject. |
| **caveman-review** | `/caveman-review` | One-line PR comments: `L42: bug: user null. Add guard.` |
| **caveman-compress** | `/caveman-compress <file>` | Compress .md files to caveman prose. Saves ~46% input tokens. |
| **caveman-help** | `/caveman-help` | This card. |
## Deactivate
Say "stop caveman" or "normal mode". Resume anytime with `/caveman`.
## Language
Keep user's language by default. User write Portuguese → reply Portuguese caveman. Compress the style, not the language. Technical terms, code, commands, commit types, and exact error strings stay verbatim unless user ask for translation.
## Configure Default Mode
Default mode = `full`. Change it:
**Environment variable** (highest priority):
```bash
export CAVEMAN_DEFAULT_MODE=ultra
```
**Config file** (`~/.config/caveman/config.json`):
```json
{ "defaultMode": "lite" }
```
Set `"off"` to disable auto-activation on session start. User can still activate manually with `/caveman`.
Resolution: env var > config file > `full`.
## More
Full docs: https://github.com/JuliusBrussee/caveman

View File

@@ -0,0 +1,33 @@
# caveman-review
One-line PR comments. Location, problem, fix. No throat-clearing.
## What it does
Generates code review comments in `L<line>: <severity> <problem>. <fix>.` format. One line per finding. Severity emoji: 🔴 bug, 🟡 risk, 🔵 nit, ❓ question. Drops "I noticed that...", hedging, and restating what the diff already shows. Keeps exact line numbers, backticked symbols, and concrete fixes.
Auto-clarity: drops terse mode for CVE-class security findings, architectural disagreements, and onboarding contexts where the author needs the *why*. Resumes terse for the rest.
Output only — does not approve, request changes, or run linters.
## How to invoke
```
/caveman-review
```
Also triggers on "review this PR", "code review", "review the diff".
## Example output
```
L42: 🔴 bug: user can be null after .find(). Add guard before .email.
L88-140: 🔵 nit: 50-line fn does 4 things. Extract validate/normalize/persist.
L23: 🟡 risk: no retry on 429. Wrap in withBackoff(3).
L107: ❓ q: why drop the cache here? Reads on next request will miss.
```
## See also
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,55 @@
---
name: caveman-review
description: >
Ultra-compressed code review comments. Cuts noise from PR feedback while preserving
the actionable signal. Each comment is one line: location, problem, fix. Use when user
says "review this PR", "code review", "review the diff", "/review", or invokes
/caveman-review. Auto-triggers when reviewing pull requests.
---
Write code review comments terse and actionable. One line per finding. Location, problem, fix. No throat-clearing.
## Rules
**Format:** `L<line>: <problem>. <fix>.` — or `<file>:L<line>: ...` when reviewing multi-file diffs.
**Severity prefix (optional, when mixed):**
- `🔴 bug:` — broken behavior, will cause incident
- `🟡 risk:` — works but fragile (race, missing null check, swallowed error)
- `🔵 nit:` — style, naming, micro-optim. Author can ignore
- `❓ q:` — genuine question, not a suggestion
**Drop:**
- "I noticed that...", "It seems like...", "You might want to consider..."
- "This is just a suggestion but..." — use `nit:` instead
- "Great work!", "Looks good overall but..." — say it once at the top, not per comment
- Restating what the line does — the reviewer can read the diff
- Hedging ("perhaps", "maybe", "I think") — if unsure use `q:`
**Keep:**
- Exact line numbers
- Exact symbol/function/variable names in backticks
- Concrete fix, not "consider refactoring this"
- The *why* if the fix isn't obvious from the problem statement
## Examples
❌ "I noticed that on line 42 you're not checking if the user object is null before accessing the email property. This could potentially cause a crash if the user is not found in the database. You might want to add a null check here."
`L42: 🔴 bug: user can be null after .find(). Add guard before .email.`
❌ "It looks like this function is doing a lot of things and might benefit from being broken up into smaller functions for readability."
`L88-140: 🔵 nit: 50-line fn does 4 things. Extract validate/normalize/persist.`
❌ "Have you considered what happens if the API returns a 429? I think we should probably handle that case."
`L23: 🟡 risk: no retry on 429. Wrap in withBackoff(3).`
## Auto-Clarity
Drop terse mode for: security findings (CVE-class bugs need full explanation + reference), architectural disagreements (need rationale, not just a one-liner), and onboarding contexts where the author is new and needs the "why". In those cases write a normal paragraph, then resume terse for the rest.
## Boundaries
Reviews only — does not write the code fix, does not approve/request-changes, does not run linters. Output the comment(s) ready to paste into the PR. "stop caveman-review" or "normal mode": revert to verbose review style.

View File

@@ -0,0 +1,30 @@
# caveman-stats
Real session token receipts. No AI estimation.
## What it does
Reads the current Claude Code session log directly and reports actual input/output token usage plus estimated savings versus a non-caveman baseline. Numbers come from the JSONL session log on disk — the model itself does not compute or estimate them. Output is injected by the `caveman-mode-tracker` hook, which intercepts `/caveman-stats` and returns the formatted stats as a blocked-decision reason.
Each run also writes a lifetime-savings suffix file used by the statusline badge (`⛏ 12.4k`).
## How to invoke
```
/caveman-stats
```
## Example output
```
Session: 47 turns
Input: 12,304 tokens
Output: 3,891 tokens (caveman)
Baseline: 11,247 tokens (estimated without caveman)
Saved: 7,356 tokens (~65%)
```
## See also
- [`SKILL.md`](./SKILL.md) — hook contract and mechanics
- [Caveman README](../../README.md) — repo overview

View File

@@ -0,0 +1,10 @@
---
name: caveman-stats
description: >
Show real token usage and estimated savings for the current session.
Reads directly from the Claude Code session log — no AI estimation.
Triggers on /caveman-stats. Output is injected by the mode-tracker hook;
the model itself does not compute the numbers.
---
This skill is delivered by `hooks/caveman-stats.js` (read by `hooks/caveman-mode-tracker.js` on `/caveman-stats`). The model does not need to do anything when this skill fires — the hook returns `decision: "block"` with the formatted stats as the reason. The user sees the numbers immediately.

View File

@@ -0,0 +1,48 @@
# caveman
Talk like smart caveman. Same brain, fewer tokens.
## What it does
Compress every model response to caveman-style prose. Drops articles, filler, pleasantries, and hedging. Keeps every technical detail, code block, error string, and symbol exact. Cuts 65% of output tokens (measured) with full accuracy preserved. Mode persists for the whole session until changed or stopped.
Six intensity levels:
| Level | What change |
|-------|-------------|
| `lite` | Drop filler/hedging. Sentences stay full. Professional but tight. |
| `full` | Default. Drop articles, fragments OK, short synonyms. |
| `ultra` | Bare fragments. Abbreviations (DB, auth, fn). Arrows for causality. |
| `wenyan-lite` | Classical Chinese register, light compression. |
| `wenyan-full` | Maximum 文言文. 80-90% character reduction. |
| `wenyan-ultra` | Extreme classical compression. |
Auto-clarity rule: caveman drops to normal prose for security warnings, irreversible-action confirmations, multi-step sequences where fragment ambiguity risks misread, and when user repeats a question. Resumes after the clear part.
## How to invoke
```
/caveman # full mode (default)
/caveman lite # lighter compression
/caveman ultra # extreme compression
/caveman wenyan # classical Chinese
stop caveman # back to normal prose
```
## Example output
Question: "Why does my React component re-render?"
Normal prose:
> Your component re-renders because you create a new object reference each render. Wrapping it in `useMemo` will fix the issue.
Caveman (full):
> New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`.
Caveman (ultra):
> Inline obj prop → new ref → re-render. `useMemo`.
## See also
- [`SKILL.md`](./SKILL.md) — full LLM-facing instructions
- [Caveman README](../../README.md) — repo overview, install, benchmarks

View File

@@ -0,0 +1,78 @@
---
name: caveman
description: >
Ultra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman
while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra,
wenyan-lite, wenyan-full, wenyan-ultra.
Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens",
"be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested.
---
Respond terse like smart caveman. All technical substance stay. Only fluff die.
## Persistence
ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode".
Default: **full**. Switch: `/caveman lite|full|ultra`.
## Rules
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). No tool-call narration, no decorative tables/emoji, no dumping long raw error logs unless asked — quote shortest decisive line. Standard well-known tech acronyms OK (DB/API/HTTP); never invent new abbreviations (cfg/impl/req/res/fn) — tokenizer split them same as full word: zero token saved, reader still decode. Full word cheaper AND clearer. No causal arrows (→) either — own token, save nothing. Technical terms exact. Code blocks unchanged. Errors quoted exact.
Preserve user's dominant language. User write Portuguese → reply Portuguese caveman. User write Spanish → reply Spanish caveman. Compress the style, not the language. No forced English openings or status phrases. ALWAYS keep technical terms, code, API names, CLI commands, commit-type keywords (feat/fix/...), and exact error strings verbatim — unless user explicitly ask for translation.
No self-reference. Never name or announce the style. No "caveman mode on", "me caveman think", no third-person caveman tags. Output caveman-only — never normal answer plus "Caveman:" recap. Exception: user explicitly ask what the mode is.
Pattern: `[thing] [action] [reason]. [next step].`
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
## Intensity
| Level | What change |
|-------|------------|
| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |
| **full** | Drop articles, fragments OK, short synonyms. Classic caveman. No tool-call narration, no decorative tables/emoji, no long raw error-log dumps unless asked. Standard acronyms OK; no invented abbreviations |
| **ultra** | Strip conjunctions when cause-then-effect stay unambiguous. One word when one word enough. State each fact once. NO prose abbreviations (cfg/impl/req/res/fn/auth), NO arrows (X → Y) — measured zero token saving under tokenizer, cost decode clarity. Code symbols, function names, API names, error strings: never touch |
| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |
| **wenyan-full** | Maximum classical terseness. Fully 文言文. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (之/乃/為/其) |
| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |
Example — "Why React component re-render?"
- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`."
- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`."
- ultra: "Inline obj prop, new ref, re-render. `useMemo`."
- wenyan-lite: "組件頻重繪,以每繪新生對象參照故。以 useMemo 包之。"
- wenyan-full: "每繪新生對象參照,故重繪;以 useMemo 包之則免。"
- wenyan-ultra: "新參照則重繪。useMemo 包之。"
Example — "Explain database connection pooling."
- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead."
- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead."
- ultra: "Pool reuse open DB connections. No per-request handshake."
- wenyan-full: "池蓄已開之連,不逐請而新開,省握手之費。"
- wenyan-ultra: "池蓄連,免逐請新開,省握手。"
## Auto-Clarity
Drop caveman when:
- Security warnings
- Irreversible action confirmations
- Multi-step sequences where fragment order or omitted conjunctions risk misread
- Compression itself creates technical ambiguity (e.g., `"migrate table drop column backup first"` — order unclear without articles/conjunctions)
- User asks to clarify or repeats question
Resume caveman after clear part done.
Example — destructive op:
> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.
> ```sql
> DROP TABLE users;
> ```
> Caveman resume. Verify backup exist first.
## Boundaries
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.

View File

@@ -8,6 +8,10 @@ DAILY_GITHUB_TRENDING_LIMIT=10
DAILY_GITHUB_EMERGING_LIMIT=10
DAILY_GITHUB_TOPIC_LIMIT=10
# 企微版可选行(默认均开)
# DAILY_WECOM_TOP_LINE=1 # 顶部「今日主题」行,无评分命中时回退主题名
# DAILY_FEATURED_REASON=1 # 首推下「为什么值得点开」理由行
# GitHub Trending
GITHUB_TRENDING_SINCE=daily
# GITHUB_TRENDING_LANGUAGE=python
@@ -34,6 +38,11 @@ DAILY_WECOM_GITHUB_TOPIC=5
DAILY_WECOM_AI_NEWS=10
# research 模式额外技术类时讯条数(叠加在 AI 时讯精选之上)
DAILY_WECOM_AI_NEWS_TECH=5
# research 主列表国内最少条数0=按约 30% 推算10→3
# DAILY_WECOM_AI_NEWS_CN_MIN=3
# research 去重后候选池独立事件数默认展示×20=自动)
# DAILY_AI_NEWS_RESEARCH_POOL=20
# DAILY_AI_NEWS_RESEARCH_TECH_POOL=10
DAILY_WECOM_CN_AI_NEWS=10
# 企微新闻摘要字数(句读/词边界截断,不加省略号)
# DAILY_WECOM_NEWS_DESC_LIMIT=72
@@ -65,6 +74,9 @@ DAILY_NEWS_DEDUP_DAYS=7
DAILY_SCHEDULE_TZ=Asia/Shanghai
DAILY_SCHEDULE_GENERATE_AT=08:50
DAILY_SCHEDULE_PUSH_AT=09:00
# 仅工作日生成/推送(法定节假日、周末跳过;调休补班日照常)
# 节假日数据取自 xiaoai.me缓存于 .cache/holidays-<year>.json每年首次自动获取一次
DAILY_WORKDAY_ONLY=1
# 编辑指定今日首推(可选):关键词,或 关键词|URL
# Python Step 0 检索 → featured.jsonAgent / classic 企微「今日首推」优先使用

View File

@@ -124,7 +124,8 @@ def warm_cursor_bridge(force: bool = False) -> None:
)
try:
discovery = _read_discovery_polling(process)
except Exception:
except (RuntimeError, OSError, ValueError) as exc:
logger.warning("Cursor bridge discovery 失败,终止子进程:%s", exc)
process.kill()
process.wait(timeout=5)
raise

View File

@@ -167,3 +167,8 @@ def narrative_axis_days() -> int:
def news_backfill_enabled() -> bool:
return env_bool("DAILY_NEWS_BACKFILL", False)
def workday_only() -> bool:
"""仅工作日生成/推送;法定节假日与周末跳过(调休补班日照常)。"""
return env_bool("DAILY_WORKDAY_ONLY", True)

View File

@@ -5,7 +5,7 @@ from __future__ import annotations
import re
from typing import Any
from daily.config import wecom_skill_desc_limit
from daily.config import env_bool, wecom_skill_desc_limit
from daily.localize import LocalizeJob, localize_brief_descriptions, needs_chinese
from daily.skills_group import group_skills_by_source
from daily.text_utils import trim_brief
@@ -1096,7 +1096,8 @@ def build_wecom_report(
lines.append(f"{ICONS['pick']} **今日首推**")
lines.append(_format_pick_link(pick_command, title=pick_title, url=pick_url))
if pick_why:
# 首推「为什么值得点开」一句理由; DAILY_FEATURED_REASON=0 关闭, 空则省略
if pick_why and env_bool("DAILY_FEATURED_REASON", True):
lines.append(f"> {pick_why}")
return "\n".join(lines)

View File

@@ -24,6 +24,7 @@ from daily.config import (
SNAPSHOT_FILE,
board_pool_size,
env,
env_bool,
env_int,
full_desc_limit,
news_summary_limit,
@@ -65,6 +66,7 @@ from daily.github.auth import github_html_headers
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos
from daily.github.trending import fetch_github_trending, trending_data_source_note
from daily.localize import LocalizeJob, localize_descriptions, needs_chinese
from daily.narrative_axis import theme_clusters, theme_names
from daily.news.fetch import (
fetch_ai_news,
fetch_cn_ai_news,
@@ -344,6 +346,33 @@ def _detect_theme_line(feed: dict[str, Any]) -> str:
return f"**今日主题**{max(scores.items(), key=lambda x: x[1])[0]}"
def _top_line(feed: dict[str, Any]) -> str:
"""今日看点行: 优先评分最高的主题, 回退 _theme_clusters 的主题名(非 markdown 示例)。
DAILY_WECOM_TOP_LINE=0 关闭时退回 _detect_theme_line 原逻辑。
与原 _detect_theme_line 的差别仅在「无评分命中」时的兜底文案:
用 theme_names 的真实主题取代硬编码「Agent Skills 生态持续活跃」。
"""
if not env_bool("DAILY_WECOM_TOP_LINE", True):
return _detect_theme_line(feed)
scores: dict[str, int] = defaultdict(int)
for board in ("topTrending", "topHot"):
for rank, item in enumerate(feed.get(board, [])[:10], 1):
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, label, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
scores[label] += max(1, 11 - rank)
break
if scores:
return f"**今日主题**{max(scores.items(), key=lambda x: x[1])[0]}"
names = theme_names(feed, theme_rules=THEME_RULES, skill_id_fn=_skill_id)
if names:
return f"**今日主题**{' · '.join(names)}"
return "**今日主题**Agent Skills 生态持续活跃"
def _build_highlights(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
@@ -444,31 +473,11 @@ def _fetch_latest_release_title(repo: str) -> str | None:
return None
title = entry.find("a:title", ns)
return title.text.strip() if title is not None and title.text else None
except Exception:
except (httpx.HTTPError, ET.ParseError, OSError) as exc:
logger.warning("读取 %s release 失败:%s", atom_url, exc)
return None
def _theme_clusters(feed: dict[str, Any], limit: int = 5) -> list[tuple[str, list[str]]]:
buckets: dict[str, list[str]] = defaultdict(list)
seen: set[str] = set()
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
item_id = _skill_id(item)
if item_id in seen:
continue
seen.add(item_id)
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, theme, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
label = f"**{item.get('title')}** (`{item.get('source')}`)"
if label not in buckets[theme]:
buckets[theme].append(label)
break
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
def _format_github_repo_section(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
lines: list[str] = []
for i, repo in enumerate(repos, 1):
@@ -509,7 +518,11 @@ def _format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) ->
return lines
def generate_report() -> tuple[str, str, Path, Path]:
def _collect(date_str: str) -> dict[str, Any]:
"""抓取 skills/github/news 数据并归一化,计算 wecom 限额与周去重 recent keys。
无副作用(不写快照/不记已推)。返回供 _select/_render 消费的 bundle。
"""
# Hot/Trending 前排同 source 极密,需更深抓取才能凑够展示用的唯一 source
trending_n = env_int("DAILY_TRENDING_LIMIT", 400)
hot_n = max(env_int("DAILY_HOT_LIMIT", 400), compare_depth())
@@ -590,11 +603,79 @@ def generate_report() -> tuple[str, str, Path, Path]:
| recent_shown["github_emerging"]
| recent_shown["github_topic"]
)
return {
"trending_n": trending_n,
"hot_n": hot_n,
"github_limit": github_limit,
"emerging_limit": emerging_limit,
"topic_limit": topic_limit,
"wecom_trending": wecom_trending,
"wecom_hot": wecom_hot,
"wecom_github": wecom_github,
"wecom_emerging": wecom_emerging,
"wecom_topic": wecom_topic,
"wecom_limits": wecom_limits,
"pool": pool,
"pad_pool": pad_pool,
"feed": feed,
"prev_ids": prev_ids,
"now": now,
"date_str": date_str,
"time_str": time_str,
"updated": updated,
"trending": trending,
"hot": hot,
"github_trending": github_trending,
"github_emerging": github_emerging,
"github_topic": github_topic,
"topic_name": topic_name,
"news_merged": news_merged,
"ai_news": ai_news,
"cn_ai_news": cn_ai_news,
"ai_news_research": ai_news_research,
"wecom_news": wecom_news,
"wecom_tech_news": wecom_tech_news,
"recent_shown": recent_shown,
"skill_recent": skill_recent,
"github_recent": github_recent,
}
def _select(c: dict[str, Any]) -> dict[str, Any]:
"""全部「选择 + 数据突变」:9 处 board_select + featured_pick + LLM 步骤 + pad。
时序约束内聚本段:featured pick 依赖 deep-pool;_localize 在 featured 后、
pad 前执行;写回 shown keys/llm_input 也在此。_render 不再做任何选择。
副作用: apply_featured_pick 写回 shown keys(保持原时序)。
"""
date_str = c["date_str"]
feed = c["feed"]
prev_ids = c["prev_ids"]
updated = c["updated"]
time_str = c["time_str"]
trending = c["trending"]
hot = c["hot"]
github_trending = c["github_trending"]
github_emerging = c["github_emerging"]
github_topic = c["github_topic"]
topic_name = c["topic_name"]
news_merged = c["news_merged"]
ai_news = c["ai_news"]
cn_ai_news = c["cn_ai_news"]
wecom_news = c["wecom_news"]
wecom_tech_news = c["wecom_tech_news"]
wecom_limits = c["wecom_limits"]
pool = c["pool"]
pad_pool = c["pad_pool"]
recent_shown = c["recent_shown"]
skill_recent = c["skill_recent"]
github_recent = c["github_recent"]
selected_trending = board_select(
board="skills_trending",
items=trending,
recent_keys=skill_recent,
limit=wecom_trending,
limit=c["wecom_trending"],
pool_size=pool,
kind="skill",
)
@@ -602,7 +683,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
board="skills_hot",
items=hot,
recent_keys=skill_recent,
limit=wecom_hot,
limit=c["wecom_hot"],
pool_size=pool,
kind="skill",
)
@@ -610,7 +691,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
board="github_trending",
items=github_trending,
recent_keys=github_recent,
limit=wecom_github,
limit=c["wecom_github"],
pool_size=pool,
kind="github",
)
@@ -619,7 +700,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
board="github_emerging",
items=github_emerging,
recent_keys=github_recent,
limit=wecom_emerging,
limit=c["wecom_emerging"],
pool_size=pool,
kind="github",
)
@@ -628,7 +709,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
board="github_topic",
items=github_topic,
recent_keys=github_recent,
limit=wecom_topic,
limit=c["wecom_topic"],
pool_size=pool,
kind="github",
)
@@ -759,7 +840,179 @@ def generate_report() -> tuple[str, str, Path, Path]:
github_topic=github_topic,
)
themes = _theme_clusters(feed)
themes = theme_clusters(feed, theme_rules=THEME_RULES, skill_id_fn=_skill_id)
pick_src = trending[0].get("source", "") if trending else ""
pick_name = trending[0].get("title", "") if trending else ""
pick_command = pick_command_from_featured(featured) or (
f"npx skills add {pick_src}/{pick_name}"
if pick_src and pick_name
else "npx skills add vercel-labs/skills/find-skills"
)
pick_why = pick_why_from_featured(featured) or ""
pick_title = str((featured or {}).get("title") or pick_name or "").strip()
pick_url = str((featured or {}).get("url") or "").strip()
gt = selected_trending
gh = selected_hot
gt_pad = board_select(
board="skills_trending",
items=trending,
recent_keys=skill_recent,
limit=pad_pool,
pool_size=pool,
kind="skill",
)
gh_pad = board_select(
board="skills_hot",
items=hot,
recent_keys=skill_recent,
limit=pad_pool,
pool_size=pool,
kind="skill",
)
wecom_github_items = [_prepare_github_item(item) for item in selected_github]
wecom_emerging_items = [_prepare_github_item(item) for item in selected_emerging]
wecom_topic_items = [_prepare_github_item(item) for item in selected_topic]
github_pad_recent = (
recent_shown["github_trending"]
| recent_shown["github_emerging"]
| recent_shown["github_topic"]
)
wecom_github_pad = [
_prepare_github_item(item)
for item in board_select(
board="github_trending",
items=github_trending,
recent_keys=github_pad_recent,
limit=pad_pool,
pool_size=pool,
kind="github",
)
]
wecom_emerging_pad = [
_prepare_github_item(item)
for item in board_select(
board="github_emerging",
items=github_emerging,
recent_keys=github_pad_recent,
limit=pad_pool,
pool_size=pool,
kind="github",
)
]
wecom_topic_pad = [
_prepare_github_item(item)
for item in board_select(
board="github_topic",
items=github_topic,
recent_keys=github_pad_recent,
limit=pad_pool,
pool_size=pool,
kind="github",
)
]
delta_pad = eff_mode == "delta" and wecom_delta_pad()
board_kwargs = {
"mode": eff_mode,
"movement": movement,
"trending": gt,
"hot": gh,
"topic_name": topic_name,
"github_trending": wecom_github_items,
"github_emerging": wecom_emerging_items,
"github_topic": wecom_topic_items,
"wecom_trending": c["wecom_trending"],
"wecom_hot": c["wecom_hot"],
"wecom_github": c["wecom_github"],
"wecom_emerging": c["wecom_emerging"],
"wecom_topic": c["wecom_topic"],
"pad": delta_pad,
"date_str": date_str,
"trending_pad": gt_pad,
"hot_pad": gh_pad,
"github_trending_pad": wecom_github_pad,
"github_emerging_pad": wecom_emerging_pad,
"github_topic_pad": wecom_topic_pad,
}
return {
"selected_trending": selected_trending,
"selected_hot": selected_hot,
"boards_for_wecom": boards_for_wecom,
"llm_input": llm_input,
"featured": featured,
"movement": movement,
"eff_mode": eff_mode,
"push_gate": push_gate,
"wecom_ai": wecom_ai,
"wecom_cn": wecom_cn,
"agent_wecom": agent_wecom,
"editorial_theme": editorial_theme,
"editorial_highlights": editorial_highlights,
"themes": themes,
"pick_command": pick_command,
"pick_why": pick_why,
"pick_title": pick_title,
"pick_url": pick_url,
"board_kwargs": board_kwargs,
"prev_ids": prev_ids,
"gt": gt,
"gh": gh,
"wecom_github_items": wecom_github_items,
"wecom_emerging_items": wecom_emerging_items,
"wecom_topic_items": wecom_topic_items,
}
def _render(c: dict[str, Any], s: dict[str, Any]) -> tuple[str, str, Path, Path]:
"""纯拼装: 拼完整版 markdown + 企微 wecom_md + 写盘。
零选择逻辑。副作用: record_pushed_links + _save_snapshot + save_json + 写文件,
全部保持原时序(在拼装完成后执行)。
"""
feed = c["feed"]
now = c["now"]
date_str = c["date_str"]
time_str = c["time_str"]
updated = c["updated"]
trending = c["trending"]
hot = c["hot"]
github_trending = c["github_trending"]
github_emerging = c["github_emerging"]
github_topic = c["github_topic"]
topic_name = c["topic_name"]
news_merged = c["news_merged"]
ai_news = c["ai_news"]
cn_ai_news = c["cn_ai_news"]
ai_news_research = c["ai_news_research"]
wecom_news = c["wecom_news"]
wecom_tech_news = c["wecom_tech_news"]
trending_n = c["trending_n"]
hot_n = c["hot_n"]
github_limit = c["github_limit"]
emerging_limit = c["emerging_limit"]
topic_limit = c["topic_limit"]
prev_ids = s["prev_ids"]
llm_input = s["llm_input"]
featured = s["featured"]
eff_mode = s["eff_mode"]
push_gate = s["push_gate"]
wecom_ai = s["wecom_ai"]
wecom_cn = s["wecom_cn"]
agent_wecom = s["agent_wecom"]
editorial_theme = s["editorial_theme"]
editorial_highlights = s["editorial_highlights"]
themes = s["themes"]
pick_command = s["pick_command"]
pick_why = s["pick_why"]
pick_title = s["pick_title"]
pick_url = s["pick_url"]
board_kwargs = s["board_kwargs"]
gt = s["gt"]
gh = s["gh"]
wecom_github_items = s["wecom_github_items"]
wecom_emerging_items = s["wecom_emerging_items"]
wecom_topic_items = s["wecom_topic_items"]
lines = [
f"# 早报 · {date_str}",
@@ -834,17 +1087,6 @@ def generate_report() -> tuple[str, str, Path, Path]:
lines.append(f"- {ex}")
lines.append("")
pick_src = trending[0].get("source", "") if trending else ""
pick_name = trending[0].get("title", "") if trending else ""
pick_command = pick_command_from_featured(featured) or (
f"npx skills add {pick_src}/{pick_name}"
if pick_src and pick_name
else "npx skills add vercel-labs/skills/find-skills"
)
pick_why = pick_why_from_featured(featured) or ""
pick_title = str((featured or {}).get("title") or pick_name or "").strip()
pick_url = str((featured or {}).get("url") or "").strip()
lines.extend(["---", "", "## 安装示例", "", "```bash"])
for item in trending[:4]:
src, name = item.get("source", ""), item.get("title", "")
@@ -853,88 +1095,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
lines.extend(["```", "", f"*企微短版见 `output/{date_str}.wecom.md`*"])
markdown = "\n".join(lines)
gt = selected_trending
gh = selected_hot
gt_pad = board_select(
board="skills_trending",
items=trending,
recent_keys=skill_recent,
limit=pad_pool,
pool_size=pool,
kind="skill",
)
gh_pad = board_select(
board="skills_hot",
items=hot,
recent_keys=skill_recent,
limit=pad_pool,
pool_size=pool,
kind="skill",
)
wecom_github_items = [_prepare_github_item(item) for item in selected_github]
wecom_emerging_items = [_prepare_github_item(item) for item in selected_emerging]
wecom_topic_items = [_prepare_github_item(item) for item in selected_topic]
github_pad_recent = (
recent_shown["github_trending"]
| recent_shown["github_emerging"]
| recent_shown["github_topic"]
)
wecom_github_pad = [
_prepare_github_item(item)
for item in board_select(
board="github_trending",
items=github_trending,
recent_keys=github_pad_recent,
limit=pad_pool,
pool_size=pool,
kind="github",
)
]
wecom_emerging_pad = [
_prepare_github_item(item)
for item in board_select(
board="github_emerging",
items=github_emerging,
recent_keys=github_pad_recent,
limit=pad_pool,
pool_size=pool,
kind="github",
)
]
wecom_topic_pad = [
_prepare_github_item(item)
for item in board_select(
board="github_topic",
items=github_topic,
recent_keys=github_pad_recent,
limit=pad_pool,
pool_size=pool,
kind="github",
)
]
delta_pad = eff_mode == "delta" and wecom_delta_pad()
board_kwargs = {
"mode": eff_mode,
"movement": movement,
"trending": gt,
"hot": gh,
"topic_name": topic_name,
"github_trending": wecom_github_items,
"github_emerging": wecom_emerging_items,
"github_topic": wecom_topic_items,
"wecom_trending": wecom_trending,
"wecom_hot": wecom_hot,
"wecom_github": wecom_github,
"wecom_emerging": wecom_emerging,
"wecom_topic": wecom_topic,
"pad": delta_pad,
"date_str": date_str,
"trending_pad": gt_pad,
"hot_pad": gh_pad,
"github_trending_pad": wecom_github_pad,
"github_emerging_pad": wecom_emerging_pad,
"github_topic_pad": wecom_topic_pad,
}
if agent_wecom:
wecom_md = replace_wecom_skill_sections(agent_wecom, **board_kwargs)
else:
@@ -944,7 +1105,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
updated=updated,
highlights=editorial_highlights
or _build_highlights(trending, hot, github_trending, github_emerging, ai_news, cn_ai_news),
theme_line=editorial_theme or _detect_theme_line(feed),
theme_line=editorial_theme or _top_line(feed),
ai_news=wecom_ai if not news_merged else None,
cn_ai_news=wecom_cn if not news_merged else None,
merged_ai_news=wecom_news if news_merged else None,
@@ -1020,6 +1181,13 @@ def generate_report() -> tuple[str, str, Path, Path]:
return markdown, wecom_md, out_md, out_wecom
def generate_report() -> tuple[str, str, Path, Path]:
"""编排三段: 抓取(_collect) → 选择(_select) → 拼装(_render)。"""
collected = _collect(_now_cst().strftime("%Y-%m-%d"))
selected = _select(collected)
return _render(collected, selected)
def main() -> int:
LOG_DIR.mkdir(parents=True, exist_ok=True)
log_file = LOG_DIR / f"{_now_cst():%Y-%m-%d}.log"

106
daily/holiday.py Normal file
View File

@@ -0,0 +1,106 @@
"""法定节假日获取、缓存与工作日判定。
数据来自 xiaoai.me 公共接口,一次性抓取全年并缓存到 .cache/
后续判定只读本地缓存;缓存缺失或过期时才重新请求。
"""
from __future__ import annotations
import json
import logging
import ssl
import urllib.request
from datetime import date
from pathlib import Path
from daily.config import CACHE_DIR
logger = logging.getLogger(__name__)
_API_URL = "https://publicapi.xiaoai.me/holiday/year?date={year}"
_TIMEOUT = 15
def _cache_file(year: int) -> Path:
return CACHE_DIR / f"holidays-{year}.json"
def _ssl_context() -> ssl.SSLContext:
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return ctx
def _fetch_year(year: int) -> dict[str, dict]:
"""请求全年节假日,返回 {date_str: {"rest": bool, "name": str}}。
rest=1 表示休息(法定节假日/调休放假rest=0 表示调休补班(需上班)。
"""
url = _API_URL.format(year=year)
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=_TIMEOUT, context=_ssl_context()) as resp:
payload = json.loads(resp.read().decode("utf-8"))
items = payload.get("data") or []
result: dict[str, dict] = {}
for item in items:
day = item.get("date")
if not day:
continue
result[day] = {
"rest": bool(item.get("rest", 0)),
"name": item.get("holiday", ""),
}
if not result:
raise RuntimeError(f"节假日接口返回为空 year={year}")
return result
def load_holidays(year: int, *, refresh: bool = False) -> dict[str, dict]:
"""加载全年节假日缓存;缺失或 refresh 时联网抓取并写入缓存。
联网失败时若已有缓存则回退用缓存,保证离线可用。
"""
cache = _cache_file(year)
if cache.exists() and not refresh:
try:
raw = json.loads(cache.read_text(encoding="utf-8"))
if isinstance(raw, dict) and raw:
return raw
except (OSError, ValueError):
logger.warning("节假日缓存损坏,重新获取 %s", cache)
try:
data = _fetch_year(year)
except Exception as exc:
if cache.exists():
logger.warning("节假日获取失败,回退到本地缓存:%s", exc)
return json.loads(cache.read_text(encoding="utf-8"))
raise
cache.parent.mkdir(parents=True, exist_ok=True)
cache.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
logger.info("已缓存 %d 年节假日 %d 条 -> %s", year, len(data), cache)
return data
def is_workday(day: date, holidays: dict[str, dict] | None = None) -> bool:
"""判断是否为工作日。
规则调休补班日rest=0 且在表中)→ 上班;
法定节假日/调休放假rest=1→ 休息;
周六日 → 休息;其余 → 上班。
"""
holidays = holidays if holidays is not None else load_holidays(day.year)
key = day.isoformat()
entry = holidays.get(key)
if entry is not None:
return not entry["rest"]
return day.weekday() < 5 # 周一~周五
def workday_name(day: date, holidays: dict[str, dict] | None = None) -> str | None:
"""返回该天的节假日名(休息或调休补班),无则 None。用于日志。"""
holidays = holidays if holidays is not None else load_holidays(day.year)
entry = holidays.get(day.isoformat())
return entry["name"] if entry else None

View File

@@ -74,7 +74,7 @@ def _cursor_chat(system: str, user: str) -> str:
api_key = (env("CURSOR_API_KEY") or "").strip()
if not api_key:
return ""
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
from cursor_sdk import Agent, AgentOptions, Client, CursorAgentError, LocalAgentOptions
from daily.bridge_manager import warm_cursor_bridge
@@ -83,6 +83,17 @@ def _cursor_chat(system: str, user: str) -> str:
warm_cursor_bridge()
model = env("CURSOR_MODEL") or "composer-2.5"
prompt = f"{system}\n\n{user}"
# SDK 默认 unary_timeout 只有 60sCURSOR_MODEL=auto 时后端首 token 常超时;
# 自建带大超时的 Client 绕开 _default_client()unary/stream 都放大。
unary_timeout = env_int("DAILY_CURSOR_UNARY_TIMEOUT", 300)
stream_timeout = env_int("DAILY_CURSOR_STREAM_TIMEOUT", 900)
client = Client(
base_url=os.environ["CURSOR_SDK_BRIDGE_URL"],
auth_token=os.environ["CURSOR_SDK_BRIDGE_TOKEN"],
unary_timeout=unary_timeout,
stream_timeout=stream_timeout,
)
try:
result = Agent.prompt(
prompt,
@@ -91,9 +102,12 @@ def _cursor_chat(system: str, user: str) -> str:
model=model,
local=LocalAgentOptions(cwd=cwd),
),
client=client,
)
except CursorAgentError as exc:
raise RuntimeError(f"LLM 调用失败:{exc.message}") from exc
finally:
client.close()
if result.status == "error":
raise RuntimeError(f"LLM 调用失败:{result.result or '未知错误'}")
return (result.result or "").strip()

View File

@@ -5,6 +5,7 @@ from __future__ import annotations
import json
import logging
import random
from collections import defaultdict
from datetime import datetime, timedelta
from typing import Any
@@ -101,3 +102,48 @@ def enforce_narrative_axis(trends: dict[str, Any], axis: str) -> dict[str, Any]:
out = dict(trends)
out["narrative_axis"] = axis
return out
def theme_clusters(
feed: dict[str, Any],
*,
limit: int = 5,
theme_rules: list[tuple[str, str, list[str]]],
skill_id_fn: Any,
) -> list[tuple[str, list[str]]]:
"""按 THEME_RULES 把 feed 的 topTrending/topHot 聚成 (主题, 示例列表)。
从 generate.py 迁入(原私有 _theme_clusters。theme_rules 与 skill_id_fn
由调用方注入,避免对 generate.py 的反向依赖(防循环 import
"""
buckets: dict[str, list[str]] = defaultdict(list)
seen: set[str] = set()
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
item_id = skill_id_fn(item)
if item_id in seen:
continue
seen.add(item_id)
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, theme, keywords in theme_rules:
if any(k in haystack for k in keywords):
label = f"**{item.get('title')}** (`{item.get('source')}`)"
if label not in buckets[theme]:
buckets[theme].append(label)
break
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
def theme_names(
feed: dict[str, Any],
*,
theme_rules: list[tuple[str, str, list[str]]],
skill_id_fn: Any,
limit: int = 3,
) -> list[str]:
"""仅取主题名(不含 markdown 示例),供「今日看点/theme_line」回退文案。"""
return [theme for theme, _ in theme_clusters(
feed, theme_rules=theme_rules, skill_id_fn=skill_id_fn
)][:limit]

View File

@@ -13,6 +13,7 @@ from daily.config import OUTPUT_DIR, ROOT, env, env_int, wecom_ai_news_tech_limi
from daily.llm_client import cursor_agent_prompt, extract_json_object, has_cursor_configured
from daily.news.fetch import brief_news_summary, _normalize_link
from daily.news.pushed_links import filter_unpushed_items
from daily.news.research_quality import post_process_research_news, research_cn_min
logger = logging.getLogger(__name__)
@@ -40,6 +41,25 @@ def research_limit() -> int:
return max(1, env_int("DAILY_WECOM_AI_NEWS", 10))
def research_pool_limit(display_limit: int | None = None) -> int:
"""Agent 原始候选条数(展示上限之上多拉,供可信/去重筛)。"""
lim = display_limit if display_limit is not None else research_limit()
explicit = env_int("DAILY_AI_NEWS_RESEARCH_POOL", 0)
if explicit > 0:
return max(lim, explicit)
return max(lim * 2, lim + 8)
def research_tech_pool_limit(display_limit: int | None = None) -> int:
tech = display_limit if display_limit is not None else research_tech_limit()
if tech <= 0:
return 0
explicit = env_int("DAILY_AI_NEWS_RESEARCH_TECH_POOL", 0)
if explicit > 0:
return max(tech, explicit)
return max(tech * 2, tech + 4)
def research_json_path(date_str: str) -> Path:
return OUTPUT_DIR / f"{date_str}.ai-news-research.json"
@@ -90,7 +110,7 @@ def _normalize_research_item(raw: dict[str, Any]) -> dict[str, Any] | None:
if not title or not link or not link.startswith("http"):
return None
desc = brief_news_summary(str(raw.get("desc_short") or raw.get("summary") or ""))
return {
item: dict[str, Any] = {
"title": title,
"link": link,
"source_name": _guess_source_name(link, str(raw.get("source_name") or "")),
@@ -98,6 +118,10 @@ def _normalize_research_item(raw: dict[str, Any]) -> dict[str, Any] | None:
"desc_short": desc,
"summary_plain": desc,
}
region = str(raw.get("region") or "").strip().lower()
if region:
item["region"] = region
return item
def research_tech_limit() -> int:
@@ -133,11 +157,17 @@ def parse_research_response(
*,
limit: int,
tech_limit: int = 0,
pool_limit: int | None = None,
tech_pool_limit: int | None = None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
parsed = extract_json_object(raw)
item_cap = pool_limit if pool_limit is not None else limit
tech_cap = tech_pool_limit if tech_pool_limit is not None else tech_limit
seen: set[str] = set()
items = _parse_items_array(parsed.get("items"), limit=limit, seen=seen)
tech_items = _parse_items_array(parsed.get("tech_items"), limit=tech_limit, seen=seen) if tech_limit else []
items = _parse_items_array(parsed.get("items"), limit=item_cap, seen=seen)
tech_items = (
_parse_items_array(parsed.get("tech_items"), limit=tech_cap, seen=seen) if tech_cap else []
)
return items, tech_items
@@ -189,25 +219,38 @@ def fetch_ai_news_research(
skill = _load_skill()
now_cst = datetime.now(timezone(timedelta(hours=8)))
cn_min = research_cn_min(lim)
pool = research_pool_limit(lim)
tech_pool = research_tech_pool_limit(tech_lim)
# 候选池内国内目标略高于展示配额,避免筛完国内不足
cn_pool_target = max(cn_min * 2, cn_min + 2)
# 解析多收一点原始行,输出前/后处理再压成「去重后候选池」
raw_cap = max(pool + 10, (pool * 3) // 2)
tech_raw_cap = max(tech_pool + 4, (tech_pool * 3) // 2) if tech_pool else 0
tech_clause = ""
if tech_lim:
if tech_pool:
tech_clause = (
f"\n另输出 **tech_items 恰好 {tech_lim}**,聚焦工程技术:"
f"\n另输出 **去重后** 约 **{tech_pool} 条** tech_items 候选(最终展示约 {tech_lim},聚焦工程技术:"
"模型/框架发布、开源项目、芯片算力、开发者工具、推理与工程实践。"
"与 items 不得重复 link。"
"不得与 items 重复 link/同事件;输出前自行去重,候选池内每条应为独立事件"
)
system = (
f"{skill}\n\n"
"当前执行 **早报 AI 时讯调研**。\n"
f"时间窗口:近 **{h}** 小时(截至 {now_cst.strftime('%Y-%m-%d %H:%M')} UTC+8\n"
f"输出 **恰好 {lim} 条** items按重要性排序{tech_clause}\n"
"使用 WebSearch 检索;不要读取本项目文档或 RSS 配置。"
f"输出 **同事件去重后** 约 **{pool} 条** items 候选(按重要性排序;最终展示约 {lim} 条)。\n"
"候选池条数 = 独立事件数:同一事件多源报道只留一条最权威源,禁止用换源重复充数。\n"
f"去重后的候选中国内可信源尽量不少于 **{cn_pool_target}** 条(展示侧至少 {cn_min} 条)。\n"
f"禁止用低质源凑数;可信独立事件不足才少返回。{tech_clause}\n"
"只采用官方博客/新闻稿、政府监管原文、一线权威媒体、学术官方;"
"禁止二手搬运、标题党、营销号。使用 WebSearch 检索;不要读取本项目文档或 RSS 配置。"
)
user = (
f"/deep-research 获取近 {h} 小时的 AI 人工智能新闻资讯,"
"不区分国内国外,合并精选。"
f"输出 JSONitems 长度={lim}"
+ (f"tech_items 长度={tech_lim}" if tech_lim else "")
f"国内与国际合并items 去重后约 {pool} 条独立事件(国内可信尽量 ≥{cn_pool_target}"
"输出前完成同事件去重;可信度不足则不写。"
f"只输出 JSONitems 去重后目标约 {pool}"
+ (f"tech_items 去重后目标约 {tech_pool}" if tech_pool else "")
+ ""
)
@@ -234,7 +277,13 @@ def fetch_ai_news_research(
"stats": {"error": "empty_response"},
}
items, tech_items = parse_research_response(raw, limit=lim, tech_limit=tech_lim)
items, tech_items = parse_research_response(
raw,
limit=lim,
tech_limit=tech_lim,
pool_limit=raw_cap,
tech_pool_limit=tech_raw_cap,
)
payload = extract_json_object(raw)
if payload:
_save_research_json(research_json_path(date_str), payload)
@@ -250,10 +299,23 @@ def fetch_ai_news_research(
"stats": {"error": "invalid_json"},
}
items, tech_items = post_process_research_news(
items,
tech_items,
limit=lim,
tech_limit=tech_lim,
min_cn=cn_min,
)
items = _apply_pushed_dedup(items, date_str=date_str, limit=lim)
if tech_items:
tech_items = _apply_pushed_dedup(tech_items, date_str=date_str, limit=tech_lim)
logger.info("AI 时讯 research 完成:%d 条 + %d 技术", len(items), len(tech_items))
logger.info(
"AI 时讯 research 完成:%d 条 + %d 技术(候选池 %d/%d",
len(items),
len(tech_items),
pool,
tech_pool,
)
flat = [
{

View File

@@ -0,0 +1,442 @@
"""Research 时讯后处理可信源、同事件去重、tech 主题过滤、国内配额。"""
from __future__ import annotations
import logging
import re
from typing import Any
from urllib.parse import urlparse
from daily.config import env_int
logger = logging.getLogger(__name__)
# 官方域(同事件去重时优先保留)
_OFFICIAL_HOST_SUFFIXES: tuple[str, ...] = (
"openai.com",
"anthropic.com",
"deepmind.google",
"blog.google",
"ai.googleblog.com",
"microsoft.com",
"meta.com",
"engineering.fb.com",
"nvidia.com",
"huggingface.co",
"arxiv.org",
"github.com",
"github.blog",
"modelcontextprotocol.io",
"cursor.com",
"vercel.com",
"langchain.dev",
"cohere.com",
"moonshot.cn",
"moonshot.ai",
"sktelecom.com",
"tether.io",
)
# 权威媒体 / 可信站(国际 + 国内)
_TRUSTED_HOST_SUFFIXES: tuple[str, ...] = _OFFICIAL_HOST_SUFFIXES + (
"techcrunch.com",
"theverge.com",
"wired.com",
"arstechnica.com",
"venturebeat.com",
"technologyreview.com",
"engadget.com",
"cnet.com",
"zdnet.com",
"axios.com",
"reuters.com",
"bloomberg.com",
"bloomberglaw.com",
"bbc.com",
"bbc.co.uk",
"nytimes.com",
"wsj.com",
"ft.com",
"nbcnews.com",
"time.com",
"theguardian.com",
"washingtonpost.com",
"theregister.com",
"nature.com",
"science.org",
"scmp.com",
"caixinglobal.com",
"caixin.com",
"qbitai.com",
"36kr.com",
"leiphone.com",
"jiqizhixin.com",
"ithome.com",
"tmtpost.com",
"huxiu.com",
"solidot.org",
"synched.cn",
"infoq.cn",
"yicai.com",
"news.cn",
"xinhuanet.com",
"people.com.cn",
"cls.cn",
"geekpark.net",
"standard.com",
"business-standard.com",
"siliconvalley.com",
)
_TRUSTED_SOURCE_NAMES: frozenset[str] = frozenset(
{
"techcrunch",
"the verge",
"wired",
"ars technica",
"engadget",
"reuters",
"bloomberg",
"bloomberg law",
"nbc news",
"time",
"the register",
"openai",
"anthropic",
"arxiv",
"hugging face",
"mcp blog",
"github",
"sk telecom",
"tether",
"量子位",
"36氪",
"36kr",
"雷锋网",
"机器之心",
"it之家",
"财新",
"caixin",
"钛媒体",
"虎嗅",
"第一财经",
"新华网",
"新华社",
"财联社",
"极客公园",
"人民日报",
}
)
_CN_HOST_SUFFIXES: tuple[str, ...] = (
"qbitai.com",
"36kr.com",
"leiphone.com",
"jiqizhixin.com",
"ithome.com",
"caixin.com",
"caixinglobal.com",
"tmtpost.com",
"huxiu.com",
"solidot.org",
"synched.cn",
"infoq.cn",
"moonshot.cn",
"yicai.com",
"news.cn",
"xinhuanet.com",
"people.com.cn",
"cls.cn",
"geekpark.net",
"zhihu.com",
"sina.com.cn",
"qq.com",
"163.com",
)
_CN_SOURCE_NAMES: frozenset[str] = frozenset(
{
"量子位",
"36氪",
"36kr",
"雷锋网",
"机器之心",
"it之家",
"财新",
"caixin",
"钛媒体",
"虎嗅",
"月之暗面",
"第一财经",
"新华网",
"新华社",
"财联社",
"极客公园",
"人民日报",
}
)
_ENTITIES: tuple[tuple[str, tuple[str, ...]], ...] = (
("openai", ("openai", "altman", "chatgpt", "奥特曼")),
("anthropic", ("anthropic", "amodei", "claude")),
("kimi", ("kimi", "moonshot", "月之暗面")),
("mcp", ("mcp", "model context protocol", "modelcontextprotocol")),
("nvidia", ("nvidia", "英伟达")),
("amd", ("amd",)),
("hugging_face", ("hugging face", "huggingface")),
("google", ("google", "deepmind", "gemini")),
("meta", ("meta", "llama")),
("microsoft", ("microsoft", "copilot")),
("huawei", ("huawei", "华为", "昇腾", "ascend")),
("moore", ("摩尔线程", "moore threads", "musa")),
)
_EVENT_CLUSTERS: tuple[tuple[str, tuple[str, ...]], ...] = (
("petition", ("petition", "联名", "decelerat", "pace ai", "控制", "减速")),
("hack", ("hack", "入侵", "siege", "breach", "攻击", "逃逸")),
("open_source", ("open-source", "opensource", "open sources", "开源", "open-sources")),
("adapt", ("适配", "adapt", "day-0", "day0", "day 0", "推理部署", "训练适配")),
("release", ("release", "发布", "specification", "规范", "v2.0", "changelog")),
("chip", ("chip", "芯片", "data center", "数据中心", "mi455")),
("regulate", ("framework", "监管", "voluntary", "审核", "ban", "禁止")),
)
_RELEASE_FAMILY = frozenset({"open_source", "adapt", "release"})
# 分发平台,不参与 tech↔items 主题冲突(避免 HF 上架与 HF 被黑误杀)
_PLATFORM_ENTITIES = frozenset({"hugging_face"})
def _host(link: str) -> str:
return urlparse(link).netloc.lower().removeprefix("www.")
def _ends_with_any(host: str, suffixes: tuple[str, ...]) -> bool:
return any(host == s or host.endswith("." + s) for s in suffixes)
def _norm_text(*parts: str) -> str:
text = " ".join(p for p in parts if p).lower()
text = re.sub(r"[\s\-_/|·,。、::()【】\[\]]+", " ", text)
return text.strip()
def _item_text(item: dict[str, Any]) -> str:
return _norm_text(
str(item.get("title") or ""),
str(item.get("desc_short") or item.get("summary_plain") or ""),
)
def _match_labels(text: str, table: tuple[tuple[str, tuple[str, ...]], ...]) -> frozenset[str]:
hit: set[str] = set()
for label, kws in table:
if any(kw in text for kw in kws):
hit.add(label)
return frozenset(hit)
def entities_of(item: dict[str, Any]) -> frozenset[str]:
return _match_labels(_item_text(item), _ENTITIES)
def events_of(item: dict[str, Any]) -> frozenset[str]:
return _match_labels(_item_text(item), _EVENT_CLUSTERS)
def event_key(item: dict[str, Any]) -> tuple[frozenset[str], frozenset[str]] | None:
ents = entities_of(item)
evs = events_of(item)
if not ents or not evs:
return None
return (ents, evs)
def same_event(a: dict[str, Any], b: dict[str, Any]) -> bool:
"""共享至少一实体且共享至少一事件簇 → 同事件(保守合并)。"""
ea, eva = entities_of(a), events_of(a)
eb, evb = entities_of(b), events_of(b)
if not ea or not eb or not eva or not evb:
return False
return bool(ea & eb) and bool(eva & evb)
def is_official_item(item: dict[str, Any]) -> bool:
return _ends_with_any(_host(str(item.get("link") or "")), _OFFICIAL_HOST_SUFFIXES)
def _is_institutional_cn_host(host: str) -> bool:
"""新华社 / 政府站等机构域,默认可信。"""
if host.endswith(".gov.cn") or host.endswith(".gov.cn."):
return True
if host == "news.cn" or host.endswith(".news.cn"):
return True
if host.endswith("xinhuanet.com") or host.endswith("people.com.cn"):
return True
return False
def is_trusted_item(item: dict[str, Any]) -> bool:
host = _host(str(item.get("link") or ""))
if _is_institutional_cn_host(host):
return True
if _ends_with_any(host, _TRUSTED_HOST_SUFFIXES):
return True
name = str(item.get("source_name") or "").strip().lower()
return name in _TRUSTED_SOURCE_NAMES
def is_cn_item(item: dict[str, Any]) -> bool:
region = str(item.get("region") or "").strip().lower()
if region in {"cn", "china", "zh", "zh-cn"}:
return True
if region in {"intl", "international", "global", "en"}:
return False
host = _host(str(item.get("link") or ""))
if host.endswith(".cn") or _ends_with_any(host, _CN_HOST_SUFFIXES):
return True
name = str(item.get("source_name") or "").strip().lower()
return name in {n.lower() for n in _CN_SOURCE_NAMES}
def research_cn_min(limit: int) -> int:
explicit = env_int("DAILY_WECOM_AI_NEWS_CN_MIN", 0)
if explicit > 0:
return min(explicit, max(1, limit))
return max(1, limit * 3 // 10)
def _trust_rank(item: dict[str, Any]) -> int:
if is_official_item(item):
return 0
if is_trusted_item(item):
return 1
return 2
def filter_trusted(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
kept = [i for i in items if is_trusted_item(i)]
dropped = len(items) - len(kept)
if dropped:
logger.info("news_dedup_drop:trusted=%s", dropped)
return kept
def dedupe_same_event(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""同事件只留一条;官方源优先,否则保留更靠前的。"""
kept: list[dict[str, Any]] = []
for item in items:
replaced = False
for idx, prev in enumerate(kept):
if not same_event(item, prev):
continue
if _trust_rank(item) < _trust_rank(prev):
kept[idx] = item
replaced = True
break
if not replaced:
kept.append(item)
dropped = len(items) - len(kept)
if dropped:
logger.info("news_dedup_drop:same_event=%s", dropped)
return kept
def filter_tech_against_items(
tech_items: list[dict[str, Any]],
items: list[dict[str, Any]],
) -> list[dict[str, Any]]:
kept: list[dict[str, Any]] = []
for tech in tech_items:
t_ents = entities_of(tech)
t_evs = events_of(tech)
conflict = False
for item in items:
if same_event(tech, item):
conflict = True
break
shared = (t_ents & entities_of(item)) - _PLATFORM_ENTITIES
if shared and ((t_evs | events_of(item)) & _RELEASE_FAMILY):
conflict = True
break
if not conflict:
kept.append(tech)
dropped = len(tech_items) - len(kept)
if dropped:
logger.info("news_dedup_drop:tech_topic=%s", dropped)
return kept
def pack_with_cn_quota(
items: list[dict[str, Any]],
*,
limit: int,
min_cn: int,
) -> list[dict[str, Any]]:
if limit <= 0:
return []
min_cn = max(0, min(min_cn, limit))
out: list[dict[str, Any]] = []
used: set[str] = set()
cn_got = 0
def _take(item: dict[str, Any]) -> None:
nonlocal cn_got
link = str(item.get("link") or "")
if not link or link in used:
return
out.append(item)
used.add(link)
if is_cn_item(item):
cn_got += 1
for item in items:
if len(out) >= limit:
break
link = str(item.get("link") or "")
if not link or link in used:
continue
slots_left = limit - len(out)
need_cn = max(0, min_cn - cn_got)
if not is_cn_item(item) and slots_left <= need_cn:
continue
_take(item)
if len(out) < limit:
for item in items:
if len(out) >= limit:
break
_take(item)
final_cn = sum(1 for i in out if is_cn_item(i))
if final_cn < min_cn:
logger.info("news_cn_short:%s", final_cn)
return out[:limit]
def build_deduped_candidate_pool(
items: list[dict[str, Any]],
tech_items: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""拉取后立即得到去重候选池:可信过滤 + 同事件去重 + tech 相对 items 主题过滤。"""
items = dedupe_same_event(filter_trusted(items))
tech_items = dedupe_same_event(filter_trusted(tech_items))
tech_items = filter_tech_against_items(tech_items, items)
logger.info("research_pool_deduped:items=%s tech=%s", len(items), len(tech_items))
return items, tech_items
def post_process_research_news(
items: list[dict[str, Any]],
tech_items: list[dict[str, Any]],
*,
limit: int,
tech_limit: int,
min_cn: int | None = None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""去重候选池 → 国内配额打包 → 截到展示上限。"""
cn_min = research_cn_min(limit) if min_cn is None else max(0, min_cn)
items, tech_items = build_deduped_candidate_pool(items, tech_items)
items = pack_with_cn_quota(items, limit=limit, min_cn=cn_min)
tech_items = filter_tech_against_items(tech_items, items)
return items, tech_items[: max(0, tech_limit)]

View File

@@ -2,11 +2,14 @@
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from typing import Any
from daily.config import force_push, skip_push_when_silent
logger = logging.getLogger(__name__)
@dataclass
class PushGateResult:
@@ -34,6 +37,7 @@ def evaluate_push_gate(
featured_pick: dict[str, Any] | None,
) -> PushGateResult:
if force_push():
logger.info("push_gate: force_push=on, 强制推送")
return PushGateResult(should_push=True, reasons=["force_push"], silent=False)
reasons: list[str] = []
@@ -48,4 +52,10 @@ def evaluate_push_gate(
should = bool(reasons)
silent = not should and skip_push_when_silent()
if should:
logger.info("push_gate: 推送 (原因: %s)", ",".join(reasons))
elif silent:
logger.info("push_gate: 静默日, 跳过推送 (无任何更新信号)")
else:
logger.info("push_gate: 无更新但 skip_push_when_silent=off, 仍推送")
return PushGateResult(should_push=should, reasons=reasons, silent=silent)

View File

@@ -20,7 +20,9 @@ from daily.config import (
schedule_generate_at,
schedule_push_at,
schedule_timezone_name,
workday_only,
)
from daily.holiday import is_workday, load_holidays, workday_name
logger = logging.getLogger(__name__)
@@ -200,6 +202,27 @@ def tick_once(
push_at = push_at or parse_hhmm(schedule_push_at())
today_str = now.astimezone(tz).date().isoformat()
# 仅工作日运行:非工作日(法定节假日/周末,调休补班日除外)跳过当日 generate/push。
# 将两者标记为已完成,避免 plan_next_action 在当天反复补跑。
if workday_only() and state.last_generate_date != today_str:
today = now.astimezone(tz).date()
try:
holidays = load_holidays(today.year)
if not is_workday(today, holidays):
name = workday_name(today, holidays) or "周末"
logger.info("今日 %s 非工作日(%s),跳过生成与推送", today_str, name)
state.last_generate_date = today_str
state.last_push_date = today_str
if not dry_run:
state.save()
if dry_run:
logger.info("[dry-run] 非工作日,将跳过")
return state
except Exception as exc:
# 节假日数据不可用时按常规工作日处理,避免因接口故障漏跑
logger.warning("工作日判定失败(%s),按正常工作日处理", exc)
planned = plan_next_action(
now=now,
tz=tz,

View File

@@ -0,0 +1,141 @@
# Design: Research 模式 AI 时讯质量(去重 · 国内配额 · 可信源)
Generated: 2026-07-29
Repo: daily-robots
Status: APPROVED
Related: `skills/daily-ai-news-research/SKILL.md`, `daily/news/research.py`, `docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md`
## Problem Statement
当前 `DAILY_AI_NEWS_MODE=research` 下:
1. **内容重复高**:同事件多源报道并列入榜(如 OpenAI 联名 / Altman 减速 / HF 入侵同簇Kimi K3 开源与多条适配);`items``tech_items` 只按 link 去重skill 还允许主题重叠。
2. **几乎无国内资讯**prompt 写「不区分国内国外合并精选」模型偏国际源research 分支关闭国内 RSS企微也不再出国内独立区块。
3. **可信度不稳**:低质搬运 / 营销号可进榜(如证券之星类),与「以官方/权威为准」不符。
样例:`output/2026-07-29.ai-news-research.json`
## Decisions已确认
| 决策点 | 选择 |
|--------|------|
| 国内资讯路径 | **A继续 research 合并展示**;不恢复国内独立区块;不引入 RSS 回填 |
| 国内配额 | 主列表至少约 30%10 条 → **≥3 条国内**`tech_items` **不强制**国内 |
| 去重范围 | **C**:同事件多源只留 1 条 + `tech_items` 相对 `items` 主题不得复述 |
| 可信度 | 国内/国际均以 **官方 + 权威媒体** 为准;不可信则不写,宁缺毋滥 |
| 实现路径 | **Prompt/skill 约束 + `research.py` 解析后硬校验**(推荐方案 2 |
| 条数不足 | **少返回**;禁止为凑满 limit / 国内配额灌低质或同事件条目 |
## Explicit Non-Goals
| 项 | 状态 |
|----|------|
| 恢复「国内 AI 时讯」独立企微区块 | ❌ 本期不做 |
| research 不足时用国内 RSS 补齐 | ❌ 本期不做(偏离路径 A |
| 切回纯 RSS | ❌ 本期不做 |
| LLM 语义嵌入聚类 | ❌ 本期不做(规则 + 实体/关键词即可) |
| 改 `generate.py` research/rss 分支结构 | ❌ 不动 |
## Architecture
```
Cursor Agent (WebSearch)
skills/daily-ai-news-research/SKILL.md + research prompt
│ JSON: items / tech_items
parse_research_response (link 去重)
post_process_research_items
1. trusted-source filter
2. same-event dedupe (items)
3. tech vs items topic filter
4. CN quota packing (min_cn)
_apply_pushed_dedup → generate / wecom
```
仍走现有 `fetch_ai_news_research``generate.py` research 分支;企微保持一块合并时讯。
## Components
### 1. Skill / prompt`skills/daily-ai-news-research/SKILL.md` + `fetch_ai_news_research` 文案)
更新要点:
- 中英文检索都要做;主列表国内 ≥ `min_cn`(默认 30%),但**只收可信国内源**。
- 删除「主题可重叠」「必须恰好 N 条硬凑」;改为「不足少返回」。
- 同事件只保留一条最权威源;`tech_items` 不得复述 `items` 已覆盖的事件/产品发布。
- 来源优先级:官方 blog/changelog/新闻稿、政府/监管原文、一线权威媒体、学术官方;禁止二手搬运、标题党、不明自媒体、证券营销号(无权威交叉验证则不写)。
- 用户 prompt 从「不区分国内国外,合并精选」改为「合并展示,国内配额硬性,可信度优先」。
### 2. 后处理(`daily/news/research.py`
#### 2.1 国内判定(任一即国内)
1. 可选字段 `region: "cn"|"intl"`(有则优先)。
2. `source_name` / host 命中国内白名单量子位、36氪、雷锋网、IT之家、财新、机器之心、钛媒体、虎嗅等
3. 常见中文域(`.cn` 及已知国内 host
#### 2.2 可信源过滤
- 维护 `TRUSTED_HOSTS` / `TRUSTED_SOURCE_NAMES`(国际 + 国内权威;含官方域如 `openai.com``anthropic.com``blog.google` 等)。
- 不在白名单 → **剔除**(保守:宁可短列表)。
- 官方域优先于二手报道;同事件去重时官方源胜出。
#### 2.3 同事件去重items
- 在 link 去重之后。
- 归一化 title+desc抽取主实体公司/产品)+ 事件动词簇(开源、入侵、联名、适配…)。
- 高置信重叠 → 只留排序更靠前且更可信的一条;保守策略避免误杀明显不同事件。
#### 2.4 tech vs items
- tech 与任一 item 同事件,或同主实体且同属「发布/适配/SDK」簇 → 丢弃 tech。
-items 已有 Kimi K3 开源 → tech 中 HF / 昇腾 / 摩尔线程适配全部剔除。
#### 2.5 国内配额
- `min_cn = env DAILY_WECOM_AI_NEWS_CN_MIN`;若未设或 ≤0`max(1, lim * 3 // 10)`10 → 3
- 去重后尽量保证最终 `items` 中国内 ≥ `min_cn`:国内可信条优先占位,再用国际可信条补满。
- 可信国内不足 → 短列表 + 日志 `news_cn_short:N`**不用低质源灌满**。
### 3. 配置
| 变量 | 默认 | 含义 |
|------|------|------|
| `DAILY_WECOM_AI_NEWS_CN_MIN` | `0`= 按 30% 推算) | 主列表国内最少条数;正整数覆盖 |
`.env.example` 注明;`.env` 可不改(沿用推算)。
### 4. 测试(`tests/test_ai_news_research.py`
- 国内判定(白名单 / `.cn` / `region`)。
- 可信过滤:证券之星类 host 剔除;官方/权威保留。
- 同事件Altman 减速与员工联名若同属「减速/治理请愿」簇 → 只留 1 条;与 HF 入侵若实体+事件动词不同则可分簇保留(保守,避免误杀)。
- techitems 已有 Kimi K3 → 多条 Kimi 适配 tech 全剔除。
- 配额:混合列表打包后国内 ≥ `min_cn`(在可信国内足够时)。
验收可对 `2026-07-29.ai-news-research.json` 跑后处理做人工对照。
## Failure Behavior
| 情况 | 行为 |
|------|------|
| 无 `CURSOR_API_KEY` / Agent 失败 | 与现网一致:`enabled: False` |
| 去重/可信过滤后变短 | 接受短列表;日志 `news_dedup_drop` / `news_cn_short` |
| 可信国内为 0 | 不编造、不 RSS 回填;国际可信条照常(若有) |
## Files Touched
- `skills/daily-ai-news-research/SKILL.md`
- `daily/news/research.py`
- `tests/test_ai_news_research.py`
- `.env.example`(可选配置说明)
## Out of Scope Reminder
不改 RSS 抓取路径;不改 research/rss 模式切换结构;不恢复国内独立企微区块。

View File

@@ -7,6 +7,48 @@
"skillPath": "skills/brand-voice/SKILL.md",
"computedHash": "f07173b4a886e800df150f9824535c0bfceed289f00a27ad4a3b1df5226dfa4d"
},
"cavecrew": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/cavecrew/SKILL.md",
"computedHash": "9633c1391fa246091ce68ea522c0e424b2bc93aeb69fc44221a30b53e8a2c23d"
},
"caveman": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/caveman/SKILL.md",
"computedHash": "723fb2a8bec1156c0f0b5bf020cc739ed09702b7726ec6377480038871339f6e"
},
"caveman-commit": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/caveman-commit/SKILL.md",
"computedHash": "f028652defd5fdeddcce2994083cb1a7b201ee827bba8e2495546ee159fca3de"
},
"caveman-compress": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/caveman-compress/SKILL.md",
"computedHash": "1055abaf7cb2f8c0ca78b64101b84dfc910d9819733ed9f0277b661441797aeb"
},
"caveman-help": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/caveman-help/SKILL.md",
"computedHash": "4dba39eea07a050108d47940b39600bc8f45489201ecff0ccf03627180fd8e50"
},
"caveman-review": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/caveman-review/SKILL.md",
"computedHash": "b9091dbc51de0f3710ea818fd4d638539f8c1784f8fda931eb159c44861e702e"
},
"caveman-stats": {
"source": "JuliusBrussee/caveman",
"sourceType": "github",
"skillPath": "skills/caveman-stats/SKILL.md",
"computedHash": "331f720e2fa97b68cacdae44384878071e8cac6013479edea68f4c8eca308852"
},
"python-patterns": {
"source": "affaan-m/everything-claude-code",
"sourceType": "github",

View File

@@ -1,15 +1,16 @@
# AI 时讯 Deep Research早报专用
你是 **AI 时讯调研员**。使用 **WebSearch** 与网页抓取工具,收集近 N 小时全球 AI 新闻(不区分国内/国外),输出供企微早报使用的结构化 JSON。
你是 **AI 时讯调研员**。使用 **WebSearch** 与网页抓取工具,收集近 N 小时 AI 新闻(国内 + 国际合并展示),输出供企微早报使用的结构化 JSON。
## 工作流
1. 将任务拆成 35 个子问题(模型发布、监管政策、大厂动态、芯片算力、研究突破等)
1. 将任务拆成 35 个子问题(模型发布、监管政策、大厂动态、芯片算力、研究突破等)**中英文检索都要做**
2. 每个子问题用 WebSearch 检索 23 组关键词(中英文混合)
3. 交叉验证:优先权威媒体 / 官方博客 / 学术来源
4. 精选最多 **10 条**最重要、可核实的新闻(`items`);窗口内不足则少返回,勿凑数
5.精选最多 **5 条**工程技术向新闻(`tech_items`):模型/框架发布、开源、芯片算力、开发者工具、推理与工程实践;不得与 `items` 重复 link;不足则少返回
6. **只输出 JSON**,不要 Markdown 报告,不要代码块
3. 交叉验证:只采用 **官方博客 / 新闻稿、政府或监管原文、一线权威媒体、学术官方**
4. 按请求输出 **已去重候选池**(通常多于最终展示,如展示 10 → 去重后约 20**条数 = 独立事件数**。输出前必须完成同事件/同 link 去重;多源只留最权威一条。国内可信独立事件也要明显多于展示配额;禁止换源重复充数或用低质源灌满
5.输出已去重的 `tech_items` 候选;不得与 `items` 重复 link/同事件;技术区不强制国内
6. **候选池内同事件只允许一条**(官方 > 一线媒体)
7. **只输出 JSON**,不要 Markdown 报告,不要代码块
## 质量规则
@@ -19,6 +20,8 @@
4. `desc_short` 用中文一句话摘要≤72 字)
5. `title` 保留原文标题;中文源可用中文标题
6. `source_name` 为媒体/站点简称(如 TechCrunch、量子位、OpenAI Blog
7. 可选 `region`: `"cn"``"intl"`(国内源标 `cn`
8. **禁止**二手搬运、标题党、不明自媒体、证券营销号;无权威源交叉验证则 **不写**
## 输出格式(严格 JSON
@@ -29,6 +32,7 @@
"title": "Apple sues OpenAI over trade secret theft",
"link": "https://techcrunch.com/...",
"source_name": "TechCrunch",
"region": "intl",
"desc_short": "苹果起诉 OpenAI 涉嫌窃取硬件商业机密",
"published_fmt": "07-11 05:00"
}
@@ -36,8 +40,9 @@
"tech_items": [
{
"title": "Meta Iris AI chip enters production",
"link": "https://example.com/...",
"link": "https://techcrunch.com/...",
"source_name": "TechCrunch",
"region": "intl",
"desc_short": "Meta 自研 Iris 芯片 9 月量产",
"published_fmt": ""
}
@@ -46,8 +51,8 @@
}
```
- `items` 数组长度 **必须等于** 请求的 limit默认 10
- `tech_items` 数组长度 **必须等于** 请求的 tech limit默认 5聚焦工程技术可与 `items` 主题重叠但 link 不得重复
- `items` / `tech_items`:条数以请求的**去重后候选目标**为准(独立事件数;可略少,不可灌重复或低质源
- `tech_items` 聚焦工程技术可与 `items` 领域相近,但 **事件与产品不得重复**
- `published_fmt` 格式 `MM-DD HH:MM`UTC+8无法确定则留空字符串
- 不要输出 `items` 以外的长文;`methodology` 可选,一行即可
@@ -56,3 +61,4 @@
- 不要输出 ```json 代码块包裹(直接输出 JSON 对象)
- 不要输出 Executive Summary / Key Takeaways 等报告章节
- 不要使用本项目 RSS 或本地文档作为来源
- 不要为凑国内配额或条数而写入低可信来源

View File

@@ -5,7 +5,24 @@ from __future__ import annotations
import unittest
from daily.format_wecom import _ai_news_lines, replace_wecom_news_sections
from daily.news.research import parse_research_response
from daily.news.research import (
parse_research_response,
research_pool_limit,
research_tech_pool_limit,
)
class TestAiNewsResearchPool(unittest.TestCase):
def test_pool_defaults_above_display(self):
self.assertEqual(research_pool_limit(10), 20)
self.assertEqual(research_tech_pool_limit(5), 10)
def test_pool_env_override(self):
import os
from unittest import mock
with mock.patch.dict(os.environ, {"DAILY_AI_NEWS_RESEARCH_POOL": "24"}, clear=False):
self.assertEqual(research_pool_limit(10), 24)
class TestAiNewsResearchParse(unittest.TestCase):

View File

@@ -0,0 +1,293 @@
"""Research 时讯质量可信源、同事件去重、tech 主题过滤、国内配额。"""
from __future__ import annotations
import json
import unittest
from pathlib import Path
from unittest import mock
from daily.news.research_quality import (
build_deduped_candidate_pool,
dedupe_same_event,
filter_tech_against_items,
is_cn_item,
is_trusted_item,
pack_with_cn_quota,
post_process_research_news,
research_cn_min,
)
def _item(
title: str,
link: str,
*,
source_name: str = "",
desc_short: str = "",
region: str | None = None,
) -> dict:
row = {
"title": title,
"link": link,
"source_name": source_name,
"desc_short": desc_short,
"summary_plain": desc_short,
"published_fmt": "",
}
if region is not None:
row["region"] = region
return row
class TestTrustedAndCn(unittest.TestCase):
def test_trusted_official_and_authority(self):
self.assertTrue(
is_trusted_item(_item("x", "https://openai.com/blog/x", source_name="OpenAI"))
)
self.assertTrue(
is_trusted_item(_item("x", "https://techcrunch.com/a", source_name="TechCrunch"))
)
self.assertTrue(
is_trusted_item(_item("x", "https://www.qbitai.com/a", source_name="量子位"))
)
def test_rejects_low_quality(self):
self.assertFalse(
is_trusted_item(
_item("x", "https://wap.stockstar.com/detail/IG1", source_name="证券之星")
)
)
self.assertFalse(
is_trusted_item(_item("x", "https://random-blog.xyz/a", source_name="Unknown"))
)
self.assertFalse(
is_trusted_item(
_item("x", "https://www.techtimes.com/articles/1.htm", source_name="TechTimes")
)
)
def test_trusted_cn_majors_and_engadget(self):
self.assertTrue(
is_trusted_item(_item("x", "https://www.yicai.com/news/1.html", source_name="第一财经"))
)
self.assertTrue(
is_trusted_item(
_item("x", "https://www.news.cn/world/20260729/a/c.html", source_name="新华网")
)
)
self.assertTrue(
is_trusted_item(
_item("x", "https://www.engadget.com/2225849/google/", source_name="Engadget")
)
)
def test_cn_by_whitelist_region_and_cn_tld(self):
self.assertTrue(is_cn_item(_item("x", "https://www.qbitai.com/a", source_name="量子位")))
self.assertTrue(
is_cn_item(_item("x", "https://techcrunch.com/a", source_name="TechCrunch", region="cn"))
)
self.assertTrue(is_cn_item(_item("x", "https://news.example.cn/a", source_name="X")))
self.assertFalse(is_cn_item(_item("x", "https://techcrunch.com/a", source_name="TechCrunch")))
class TestSameEventAndTech(unittest.TestCase):
def test_petition_cluster_keeps_one(self):
items = [
_item(
"OpenAI, Anthropic scientists ask U.S. for tools to pace AI development",
"https://www.nbcnews.com/tech/a",
source_name="NBC News",
desc_short="超千名前沿实验室员工联名,吁美政府支持控制 AI 研发节奏",
),
_item(
"Sam Altman is ready to decelerate",
"https://techcrunch.com/2026/07/28/sam-altman-is-ready-to-decelerate/",
source_name="TechCrunch",
desc_short="奥特曼称或需控制 AI 发展速度,并支持员工联名请愿",
),
]
out = dedupe_same_event(items)
self.assertEqual(len(out), 1)
def test_distinct_clusters_kept(self):
items = [
_item(
"Sam Altman is ready to decelerate",
"https://techcrunch.com/a",
source_name="TechCrunch",
desc_short="奥特曼称或需控制 AI 发展速度",
),
_item(
"OpenAIs agent siege forced rebuild at Hugging Face",
"https://www.theregister.com/ai/a",
source_name="The Register",
desc_short="Hugging Face 因 OpenAI 智能体入侵重建基础设施",
),
]
out = dedupe_same_event(items)
self.assertEqual(len(out), 2)
def test_tech_drops_kimi_adapt_when_items_have_kimi_open_source(self):
items = [
_item(
"Moonshot Open-Sources Kimi K3",
"https://www.caixinglobal.com/a",
source_name="Caixin",
desc_short="月之暗面开放 Kimi K3 权重与技术报告",
)
]
tech = [
_item(
"moonshotai/Kimi-K3 · Hugging Face",
"https://huggingface.co/moonshotai/Kimi-K3",
source_name="Hugging Face",
desc_short="Kimi K3 开源权重上线",
),
_item(
"华为官宣昇腾 Day0 支持 Kimi K3",
"https://www.ithome.com/0/982/615.htm",
source_name="IT之家",
desc_short="昇腾宣布适配 Kimi K3 训练与推理",
),
_item(
"MCP Specification 2026-07-28",
"https://blog.modelcontextprotocol.io/posts/2026-07-28/",
source_name="MCP Blog",
desc_short="MCP 正式发布新规范",
),
]
kept = filter_tech_against_items(tech, items)
self.assertEqual(len(kept), 1)
self.assertIn("MCP", kept[0]["title"])
class TestDedupedCandidatePool(unittest.TestCase):
def test_pool_is_unique_events_after_fetch(self):
items = [
_item(
"OpenAI, Anthropic scientists ask U.S. for tools to pace AI development",
"https://www.nbcnews.com/tech/a",
source_name="NBC News",
desc_short="超千名前沿实验室员工联名,吁美政府支持控制 AI 研发节奏",
),
_item(
"Sam Altman is ready to decelerate",
"https://techcrunch.com/2026/07/28/sam-altman-is-ready-to-decelerate/",
source_name="TechCrunch",
desc_short="奥特曼称或需控制 AI 发展速度,并支持员工联名请愿",
),
_item(
"junk",
"https://wap.stockstar.com/detail/1",
source_name="证券之星",
desc_short="营销稿",
),
]
pool, tech = build_deduped_candidate_pool(items, [])
self.assertEqual(tech, [])
self.assertEqual(len(pool), 1)
self.assertTrue(all(is_trusted_item(i) for i in pool))
class TestCnQuota(unittest.TestCase):
def test_cn_min_default_30_percent(self):
with mock.patch.dict("os.environ", {"DAILY_WECOM_AI_NEWS_CN_MIN": "0"}, clear=False):
self.assertEqual(research_cn_min(10), 3)
with mock.patch.dict("os.environ", {"DAILY_WECOM_AI_NEWS_CN_MIN": "4"}, clear=False):
self.assertEqual(research_cn_min(10), 4)
def test_pack_reserves_cn_slots(self):
items = [
_item("I1", "https://techcrunch.com/1", source_name="TechCrunch", desc_short="国际1"),
_item("I2", "https://techcrunch.com/2", source_name="TechCrunch", desc_short="国际2"),
_item("I3", "https://techcrunch.com/3", source_name="TechCrunch", desc_short="国际3"),
_item("C1", "https://www.qbitai.com/1", source_name="量子位", desc_short="国内1"),
_item("C2", "https://www.36kr.com/1", source_name="36氪", desc_short="国内2"),
_item("C3", "https://www.jiqizhixin.com/1", source_name="机器之心", desc_short="国内3"),
]
out = pack_with_cn_quota(items, limit=5, min_cn=3)
self.assertEqual(len(out), 5)
self.assertGreaterEqual(sum(1 for i in out if is_cn_item(i)), 3)
class TestPostProcessIntegration(unittest.TestCase):
def test_sample_day_filters_stockstar_and_dedupes(self):
sample = Path(__file__).resolve().parents[1] / "output" / "2026-07-29.ai-news-research.json"
if not sample.exists():
self.skipTest("sample research json missing")
raw = json.loads(sample.read_text(encoding="utf-8"))
items = [
_item(
r["title"],
r["link"],
source_name=r.get("source_name", ""),
desc_short=r.get("desc_short", ""),
)
for r in raw["items"]
]
tech = [
_item(
r["title"],
r["link"],
source_name=r.get("source_name", ""),
desc_short=r.get("desc_short", ""),
)
for r in raw["tech_items"]
]
out_items, out_tech = post_process_research_news(
items, tech, limit=10, tech_limit=5, min_cn=3
)
links = {i["link"] for i in out_items + out_tech}
self.assertTrue(all("stockstar" not in link for link in links))
# 联名/减速同簇只留一条
petitionish = [
i
for i in out_items
if "decelerat" in i["title"].lower()
or "联名" in (i.get("desc_short") or "")
or "pace AI" in i["title"]
]
self.assertLessEqual(len(petitionish), 1)
# Kimi 适配不应再堆在 tech
kimi_tech = [t for t in out_tech if "kimi" in (t["title"] + t.get("desc_short", "")).lower()]
self.assertEqual(kimi_tech, [])
def test_sample_2026_07_30_keeps_cn_majors(self):
sample = Path(__file__).resolve().parents[1] / "output" / "2026-07-30.ai-news-research.json"
if not sample.exists():
self.skipTest("sample research json missing")
raw = json.loads(sample.read_text(encoding="utf-8"))
items = [
_item(
r["title"],
r["link"],
source_name=r.get("source_name", ""),
desc_short=r.get("desc_short", ""),
region=r.get("region"),
)
for r in raw["items"]
]
tech = [
_item(
r["title"],
r["link"],
source_name=r.get("source_name", ""),
desc_short=r.get("desc_short", ""),
region=r.get("region"),
)
for r in raw["tech_items"]
]
out_items, out_tech = post_process_research_news(
items, tech, limit=10, tech_limit=5, min_cn=3
)
# 不应再被白名单误杀成只剩 1 条
self.assertGreaterEqual(len(out_items) + len(out_tech), 4)
self.assertGreaterEqual(sum(1 for i in out_items if is_cn_item(i)), 2)
hosts = " ".join(i["link"] for i in out_items + out_tech)
self.assertNotIn("techtimes.com", hosts)
self.assertTrue("yicai.com" in hosts or "news.cn" in hosts)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,55 @@
# tests/test_featured_reason.py
"""T6: 首推理由行(pick_why + DAILY_FEATURED_REASON 开关)测试。"""
from __future__ import annotations
import unittest
from unittest import mock
import daily.format_wecom as fw
def _build(pick_why: str) -> str:
return fw.build_wecom_report(
date_str="2026-07-18",
time_str="08:50 (UTC+8)",
updated="2026-07-18",
highlights=[],
theme_line="**今日主题**:测试",
ai_news=None,
cn_ai_news=None,
merged_ai_news=None,
merged_tech_ai_news=None,
trending=[],
hot=[],
repos=[],
emerging=[],
topic_name="t",
topic_repos=[],
pick_command="npx skills add src/alpha",
pick_why=pick_why,
pick_title="alpha",
pick_url="https://x/a",
include_boards=False,
)
class FeaturedReasonTests(unittest.TestCase):
def test_reason_shown_when_present_and_enabled(self):
with mock.patch.object(fw, "env_bool", return_value=True):
md = _build("昨日 star 增速第一")
self.assertIn("> 昨日 star 增速第一", md)
def test_reason_hidden_when_switch_off(self):
with mock.patch.object(fw, "env_bool", return_value=False):
md = _build("昨日 star 增速第一")
self.assertNotIn("> 昨日 star 增速第一", md)
def test_reason_omitted_when_empty(self):
with mock.patch.object(fw, "env_bool", return_value=True):
md = _build("")
self.assertIn("今日首推", md)
self.assertNotIn("> \n", md)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,103 @@
# tests/test_generate_golden.py
"""黄金文件/确定性回归: 冻结时间、mock 网络与 LLM, 验证 generate_report
在同一输入下产出逐字节一致(拆分 _collect/_select/_render 不得改变行为)。"""
from __future__ import annotations
import unittest
from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest import mock
import daily.generate as g
FIXED_NOW = datetime(2026, 7, 18, 8, 50, tzinfo=timezone(timedelta(hours=8)))
def _feed() -> dict:
return {
"updatedAt": "2026-07-18T08:00:00",
"topTrending": [
{"id": "a", "source": "src", "title": "alpha", "description": "desc a",
"installs": 100, "link": "https://x/a"},
{"id": "b", "source": "src", "title": "beta", "description": "desc b",
"installs": 90, "link": "https://x/b"},
],
"topHot": [
{"id": "c", "source": "src", "title": "gamma", "description": "desc c",
"installs": 80, "link": "https://x/c"},
],
}
def _patches(tmp: Path):
"""集中 mock 所有外部边界: 网络/文件/时间/LLM。返回 patcher 列表。"""
return [
mock.patch.object(g, "_now_cst", return_value=FIXED_NOW),
mock.patch.object(g, "load_feed", return_value=_feed()),
mock.patch.object(g, "_load_snapshot", return_value=set()),
mock.patch.object(g, "_save_snapshot", lambda *a, **k: None),
mock.patch.object(g, "load_boards", return_value=(
_feed()["topTrending"], _feed()["topHot"])),
mock.patch.object(g, "fetch_github_trending", return_value=[]),
mock.patch.object(g, "fetch_emerging_repos", return_value=[]),
mock.patch.object(g, "fetch_topic_hot_repos", return_value=("topic", [])),
mock.patch.object(g, "fetch_ai_news", return_value={
"enabled": False, "categories": [], "flat": [], "stats": {}}),
mock.patch.object(g, "fetch_cn_ai_news", return_value={
"enabled": False, "categories": [], "flat": [], "stats": {}}),
mock.patch.object(g, "is_research_mode", return_value=False),
mock.patch.object(g, "is_agent_mode", return_value=False),
mock.patch.object(g, "run_editorial", return_value=None),
mock.patch.object(g, "cursor_editor_enabled", return_value=False),
mock.patch.object(g, "record_pushed_links", lambda *a, **k: None),
mock.patch.object(g, "save_json", lambda *a, **k: None),
mock.patch.object(g, "load_recent_shown_keys", return_value={
"skills_trending": set(), "skills_hot": set(),
"github_trending": set(), "github_emerging": set(),
"github_topic": set()}),
mock.patch.object(g, "OUTPUT_DIR", tmp),
mock.patch.object(g, "_localize_descriptions_in_place", lambda *a, **k: None),
# featured_pick / push_gate 有独立测试;此处冻结以免触网(LLM)与读盘。
mock.patch.object(g, "apply_featured_pick", return_value={
"title": "alpha", "url": "https://x/a",
"command": "npx skills add src/alpha", "why": "昨日 star 增速第一"}),
mock.patch.object(g, "evaluate_push_gate", return_value=mock.Mock(
should_push=False, silent=True, reasons=["golden-mock"])),
]
class GenerateGoldenTests(unittest.TestCase):
def _run(self, tmp: Path) -> tuple[str, str]:
patches = _patches(tmp)
for p in patches:
p.start()
try:
markdown, wecom_md, _md, _we = g.generate_report()
return markdown, wecom_md
finally:
for p in patches:
try:
p.stop()
except Exception:
pass
def test_deterministic_same_input_same_output(self):
import tempfile
with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
md1, we1 = self._run(Path(d1))
md2, we2 = self._run(Path(d2))
self.assertEqual(md1, md2, "完整版 markdown 在相同输入下必须逐字节一致")
self.assertEqual(we1, we2, "企微 wecom_md 在相同输入下必须逐字节一致")
def test_output_contains_core_sections(self):
import tempfile
with tempfile.TemporaryDirectory() as d:
markdown, wecom_md = self._run(Path(d))
self.assertIn("# 早报 · 2026-07-18", markdown)
self.assertIn("主题聚类", markdown)
self.assertIsInstance(wecom_md, str)
self.assertTrue(len(wecom_md) > 0)
if __name__ == "__main__":
unittest.main()

57
tests/test_holiday.py Normal file
View File

@@ -0,0 +1,57 @@
"""Tests for daily.holiday."""
from __future__ import annotations
import unittest
from datetime import date
from daily.holiday import is_workday, workday_name
# 模拟 2026 节假日表1/1 元旦(休)1/4 元旦调休(补班)
HOLIDAYS = {
"2026-01-01": {"rest": True, "name": "元旦节"},
"2026-01-02": {"rest": True, "name": "元旦节"},
"2026-01-03": {"rest": True, "name": "元旦节"},
"2026-01-04": {"rest": False, "name": "元旦节调休"},
"2026-02-16": {"rest": True, "name": "春节"},
"2026-02-14": {"rest": False, "name": "春节调休"},
}
class IsWorkdayTests(unittest.TestCase):
def test_legal_holiday_is_rest(self):
# 2026-01-01 周四,元旦 → 休息
self.assertFalse(is_workday(date(2026, 1, 1), HOLIDAYS))
def test_tiaoxiu_makeup_day_is_workday(self):
# 2026-01-04 周日,调休补班 → 上班
self.assertEqual(date(2026, 1, 4).weekday(), 6)
self.assertTrue(is_workday(date(2026, 1, 4), HOLIDAYS))
def test_spring_festival_holiday(self):
self.assertFalse(is_workday(date(2026, 2, 16), HOLIDAYS))
def test_spring_festival_makeup_saturday(self):
# 2026-02-14 周六,调休 → 上班
self.assertEqual(date(2026, 2, 14).weekday(), 5)
self.assertTrue(is_workday(date(2026, 2, 14), HOLIDAYS))
def test_normal_weekday(self):
# 2026-07-23 周四,非节假日 → 上班
self.assertTrue(is_workday(date(2026, 7, 23), HOLIDAYS))
def test_normal_weekend(self):
# 2026-07-25 周六,非节假日 → 休息
self.assertFalse(is_workday(date(2026, 7, 25), HOLIDAYS))
class WorkdayNameTests(unittest.TestCase):
def test_holiday_name(self):
self.assertEqual(workday_name(date(2026, 1, 1), HOLIDAYS), "元旦节")
def test_normal_day_has_no_name(self):
self.assertIsNone(workday_name(date(2026, 7, 23), HOLIDAYS))
if __name__ == "__main__":
unittest.main()

View File

@@ -4,6 +4,7 @@ from __future__ import annotations
import unittest
from datetime import datetime
from unittest import mock
from zoneinfo import ZoneInfo
from daily.scheduler import (
@@ -12,6 +13,7 @@ from daily.scheduler import (
next_occurrence_after,
parse_hhmm,
plan_next_action,
tick_once,
)
@@ -116,3 +118,39 @@ class NextOccurrenceTests(unittest.TestCase):
nxt = next_occurrence_after(ClockTime(8, 50), tz, now)
self.assertEqual(nxt.date().isoformat(), "2026-07-10")
self.assertEqual((nxt.hour, nxt.minute), (8, 50))
class WorkdayGateTests(unittest.TestCase):
"""非工作日 tick_once 直接标记完成并跳过,不触发 generate。"""
def setUp(self) -> None:
self.tz = ZoneInfo("Asia/Shanghai")
self.gen = ClockTime(8, 50)
self.push = ClockTime(9, 0)
def _tick(self, day, workday):
now = datetime(day.year, day.month, day.day, 8, 30, tzinfo=self.tz)
holidays = {} if workday else {day.isoformat(): {"rest": True, "name": "测试假"}}
with mock.patch("daily.scheduler.load_holidays", return_value=holidays), \
mock.patch("daily.scheduler.run_scheduled_action") as run:
state = tick_once(
now=now, tz=self.tz, state=SchedulerState(),
generate_at=self.gen, push_at=self.push, dry_run=True,
)
return state, run
def test_holiday_skips_and_marks_done(self):
# 2026-07-25 是周六;强制为非工作日
from datetime import date
day = date(2026, 7, 25)
state, run = self._tick(day, workday=False)
self.assertFalse(run.called)
self.assertEqual(state.last_generate_date, "2026-07-25")
self.assertEqual(state.last_push_date, "2026-07-25")
def test_workday_proceeds(self):
from datetime import date
day = date(2026, 7, 23) # 周四,工作日
state, run = self._tick(day, workday=True)
# 工作日不提前标记完成dry-run 不执行动作,故 last_generate_date 仍为 None
self.assertIsNone(state.last_generate_date)

60
tests/test_top_line.py Normal file
View File

@@ -0,0 +1,60 @@
# tests/test_top_line.py
"""T5: _top_line 看点行(theme_line 取数升级)测试。"""
from __future__ import annotations
import unittest
from unittest import mock
import daily.generate as g
FEED_HIT = {
"topTrending": [
{"title": "remotion-video", "source": "src", "description": "video tool"},
],
"topHot": [],
}
FEED_MISS = {
"topTrending": [
{"title": "zzz-nomatch", "source": "src", "description": "nothing"},
],
"topHot": [],
}
class TopLineTests(unittest.TestCase):
def test_scores_hit_returns_theme(self):
# 命中 THEME_RULES(video) -> 评分最高主题
with mock.patch.object(g, "env_bool", return_value=True):
line = g._top_line(FEED_HIT)
self.assertIn("今日主题", line)
self.assertIn("AI 多媒体", line)
def test_fallback_to_theme_names_when_no_score(self):
# 无评分命中但 theme_clusters 能聚类 -> 用主题名
feed = {
"topTrending": [
{"id": "x", "title": "runcomfy-x", "source": "s", "description": "video x"},
],
"topHot": [],
}
with mock.patch.object(g, "env_bool", return_value=True):
# 让评分落空(前 10 无命中)但 clusters(前 20)命中
line = g._top_line(feed)
self.assertIn("今日主题", line)
self.assertIn("AI 多媒体", line)
def test_switch_off_uses_legacy_detect(self):
# DAILY_WECOM_TOP_LINE=0 -> 退回 _detect_theme_line
with mock.patch.object(g, "env_bool", return_value=False):
line = g._top_line(FEED_MISS)
self.assertEqual(line, g._detect_theme_line(FEED_MISS))
def test_empty_feed_hardcoded_fallback(self):
# 完全无数据 -> 硬编码兜底, 不抛异常
with mock.patch.object(g, "env_bool", return_value=True):
line = g._top_line({"topTrending": [], "topHot": []})
self.assertIn("今日主题", line)
if __name__ == "__main__":
unittest.main()

View File

@@ -1,14 +0,0 @@
import re
import certifi
import httpx
url = "https://skills.sh/vercel-labs/skills/find-skills"
r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20, verify=certifi.where())
chunks = re.findall(r'self\.__next_f\.push\(\[1,"(.*?)"\]\)', r.text, re.DOTALL)
blob = "\n".join(chunks).encode("utf-8").decode("unicode_escape", errors="ignore")
for needle in ("description", "Helps users", "SKILL.md", "summary"):
print(needle, blob.count(needle))
idx = blob.find("Helps users")
if idx >= 0:
print(blob[idx : idx + 300])

Binary file not shown.