Compare commits

...

7 Commits

Author SHA1 Message Date
ce04d5a342 refactor: Phase 3 拆分 generate 流水线并补全测试
Some checks failed
test / pytest (push) Failing after 3s
daily / report (push) Failing after 1s
提取 collect/formatters/themes 等 pipeline 模块,新增 wecom 分条、RSS、delta、Agent 工作流测试。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:59:12 +08:00
391f887d73 feat: Phase 2 新闻打分去重与 Agent 输入池裁剪
标题相似度合并、信源/时效/昨日重复加权排序,Agent 模式扩大 LLM 新闻候选池并记录抓取失败源。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:53:30 +08:00
728bd1b9d1 feat: Phase 1 添加 GHA 定时、run-daily.sh 与 Docker 部署
让早报可在无本地服务器环境下定时生成并推送企微,补齐 Linux/CI 入口。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:51:11 +08:00
8075ab78e8 test: Phase 3 添加 smoke test、CI 与 run-daily 去重锁
离线 mock 测试 generate 产出,GitHub Actions 跑 pytest,30 分钟内重复调度自动 skip。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:45:58 +08:00
ebc25c9e9b docs: Phase 2 重写 onboarding 文档与示例产出
主 README 聚焦早报 Quick Start,Bot 文档迁至 bot/README.md 并标 experimental。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:45:48 +08:00
f533e31adb refactor: Phase 1 抽出 shared/skills_data 解耦 daily 与 bot
daily 不再通过 sys.path 导入 bot/skills_service,skills feed 数据层上移到 shared/。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:45:39 +08:00
88d08c0da9 chore: Phase 0 统一命名与仓库卫生
统一 daily-robots 命名、修正示例路径与 User-Agent,并清理 tmp 与 IDE 追踪文件。

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 14:45:19 +08:00
48 changed files with 2175 additions and 1046 deletions

18
.dockerignore Normal file
View File

@@ -0,0 +1,18 @@
.git
.github
.env
.env.local
output
.cache
logs
bot/.env
bot/.venv
bot/.cache
__pycache__
*.pyc
.idea
tmp_*
tests
.pytest_cache
*.md
!skills/**/*.md

View File

@@ -1,6 +1,9 @@
# 企微群机器人 webhook早报推送与 bot API 模式凭证不同) # 企微群机器人 webhook早报推送与 bot API 模式凭证不同)
WECOM_WEBHOOK_KEY=your-webhook-key WECOM_WEBHOOK_KEY=your-webhook-key
# GitHub Actions将上述 key 与下方可选项写入 repo Secrets / Variables
# 详见 README「部署GitHub Actions / Docker
# 早报内容 # 早报内容
DAILY_TRENDING_LIMIT=150 DAILY_TRENDING_LIMIT=150
DAILY_HOT_LIMIT=150 DAILY_HOT_LIMIT=150
@@ -61,7 +64,7 @@ DAILY_CN_AI_NEWS=1
# DAILY_REPORT_MODE=agent # DAILY_REPORT_MODE=agent
# CURSOR_API_KEY=cursor_... # CURSOR_API_KEY=cursor_...
# CURSOR_MODEL=composer-2.5 # CURSOR_MODEL=composer-2.5
# DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily # DAILY_CURSOR_CWD=.
# 新增榜对比(较昨日 Top15供 Agent 导语/signals列表展示 Top N # 新增榜对比(较昨日 Top15供 Agent 导语/signals列表展示 Top N
# DAILY_DELTA_COMPARE_DEPTH=15 # DAILY_DELTA_COMPARE_DEPTH=15
@@ -73,6 +76,13 @@ DAILY_AI_NEWS_HOURS=72
DAILY_AI_NEWS_PER_FEED=3 DAILY_AI_NEWS_PER_FEED=3
DAILY_AI_NEWS_PER_CATEGORY=5 DAILY_AI_NEWS_PER_CATEGORY=5
# 新闻排序 / 去重Phase 2
# 标题 Jaccard 相似度阈值0-95默认 55 表示 0.55
# DAILY_NEWS_TITLE_SIM=55
# Agent 模式 LLM 输入池classic 仍用 DAILY_WECOM_* 条数)
# DAILY_AGENT_NEWS_POOL=40
# DAILY_AGENT_CN_NEWS_POOL=30
# Reddit RSS403/429 时在 Reddit 偏好设置 → RSS feeds 复制 user / feed 参数) # Reddit RSS403/429 时在 Reddit 偏好设置 → RSS feeds 复制 user / feed 参数)
# REDDIT_RSS_USER=your_username # REDDIT_RSS_USER=your_username
# REDDIT_RSS_FEED=your_feed_token # REDDIT_RSS_FEED=your_feed_token

64
.github/workflows/daily.yml vendored Normal file
View File

@@ -0,0 +1,64 @@
name: daily
on:
schedule:
# 08:30 CST (UTC+8) = 00:30 UTC
- cron: "30 0 * * *"
workflow_dispatch:
inputs:
skip_push:
description: Skip WeCom push (generate only)
type: boolean
default: false
concurrency:
group: daily-report
cancel-in-progress: false
jobs:
report:
runs-on: ubuntu-latest
timeout-minutes: 30
env:
TZ: Asia/Shanghai
WECOM_WEBHOOK_KEY: ${{ secrets.WECOM_WEBHOOK_KEY }}
GITHUB_TOKEN: ${{ secrets.GH_PAT || github.token }}
CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }}
DAILY_LLM_API_KEY: ${{ secrets.DAILY_LLM_API_KEY }}
DAILY_LLM_API_BASE: ${{ vars.DAILY_LLM_API_BASE }}
DAILY_LLM_MODEL: ${{ vars.DAILY_LLM_MODEL }}
DAILY_REPORT_MODE: ${{ vars.DAILY_REPORT_MODE }}
CURSOR_MODEL: ${{ vars.CURSOR_MODEL }}
DAILY_AI_NEWS: ${{ vars.DAILY_AI_NEWS }}
DAILY_CN_AI_NEWS: ${{ vars.DAILY_CN_AI_NEWS }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: pip install -r requirements.txt
- name: Generate and push daily report
run: |
ARGS=()
if [ "${{ inputs.skip_push }}" = "true" ]; then
ARGS+=(--skip-push)
fi
chmod +x ./run-daily.sh
./run-daily.sh --force "${ARGS[@]}"
- name: Upload report artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: daily-report-${{ github.run_id }}
path: |
output/*.md
output/*.json
if-no-files-found: ignore
retention-days: 14

20
.github/workflows/test.yml vendored Normal file
View File

@@ -0,0 +1,20 @@
name: test
on:
push:
branches: [main]
pull_request:
jobs:
pytest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: pip install -r requirements-dev.txt
- name: Run tests
run: pytest -q

6
.gitignore vendored
View File

@@ -5,4 +5,8 @@ logs/
bot/.env bot/.env
bot/.venv/ bot/.venv/
bot/.cache/ bot/.cache/
output output
__pycache__/
*.pyc
.idea/
tmp_*

3
.idea/.gitignore generated vendored
View File

@@ -1,3 +0,0 @@
# Default ignored files
/shelf/
/workspace.xml

View File

@@ -1,9 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$" />
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>

2
.idea/misc.xml generated
View File

@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<project version="4"> <project version="4">
<component name="ProjectRootManager" version="2" languageLevel="JDK_21" default="true" project-jdk-name="21" project-jdk-type="JavaSDK"> <component name="ProjectRootManager" version="2" default="true">
<output url="file://$PROJECT_DIR$/out" /> <output url="file://$PROJECT_DIR$/out" />
</component> </component>
</project> </project>

8
.idea/modules.xml generated
View File

@@ -1,8 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/daily-robots.iml" filepath="$PROJECT_DIR$/.idea/daily-robots.iml" />
</modules>
</component>
</project>

2
.idea/vcs.xml generated
View File

@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<project version="4"> <project version="4">
<component name="VcsDirectoryMappings"> <component name="VcsDirectoryMappings">
<mapping directory="" vcs="Git" /> <mapping directory="$PROJECT_DIR$" vcs="Git" />
</component> </component>
</project> </project>

27
Dockerfile Normal file
View File

@@ -0,0 +1,27 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
TZ=Asia/Shanghai
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends tzdata \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY daily/ daily/
COPY shared/ shared/
COPY skills/ skills/
COPY bot/ bot/
COPY run-daily.sh .
RUN chmod +x run-daily.sh \
&& mkdir -p output .cache logs
VOLUME ["/app/output", "/app/.cache", "/app/logs"]
ENTRYPOINT ["./run-daily.sh"]

485
README.md
View File

@@ -1,338 +1,169 @@
# skills-hot-daily # daily-robots
Skills / GitHub 早报推送 + 企微对话机器人(同一仓库、两套企微接入)。 > **Daily Briefing** · **WeCom Push** · **Agent Mode**
给 AI 工程小团队推送**每日 curated 情报**skills.sh 榜单、GitHub AI 趋势、国际/国内 RSS生成叙事化早报并推到企业微信群。
与 TLDR AI / Ben's Bites 不同,这里专注 **Agent Skills 生态** + **可 fork 的自托管流水线**(不是又一个 email newsletter
**产出示例**[docs/sample-output/wecom-agent-sample.md](docs/sample-output/wecom-agent-sample.md)Agent 模式企微版节选)
---
## Quick Start
1. **Clone** 本仓库
2. **安装依赖**`pip install -r requirements.txt`
3. **配置环境**`copy .env.example .env`
4. **填写 webhook**:企微群 → 群机器人 → 添加 → 把 `key=` 写入 `.env`
```env
WECOM_WEBHOOK_KEY=your-webhook-key
```
5. **生成并推送**`.\run-daily.ps1`
生成文件在 `output/`
| 文件 | 说明 |
|------|------|
| `YYYY-MM-DD.wecom.md` | 企微推送版(主产物) |
| `YYYY-MM-DD.md` | 完整归档版 |
| `YYYY-MM-DD.data.json` | 结构化数据(供 LLM / 调试) |
常用变体:
```powershell
.\run-daily.ps1 -SkipPush # 只生成
python -m daily push output\2026-07-03.wecom.md # 只推送
```
Linux / macOS / CI 等价脚本:
```bash
chmod +x run-daily.sh
./run-daily.sh # 生成 + 推送
./run-daily.sh --skip-push # 只生成
./run-daily.sh --force # 忽略 30 分钟内重复运行锁
```
**前置条件**Python 3.10+、企业微信群机器人 webhook。无需 Bot API 凭证。
---
## 部署GitHub Actions / Docker
### GitHub Actions零服务器定时推送
仓库已包含 [`.github/workflows/daily.yml`](.github/workflows/daily.yml),默认每天 **08:30北京时间** 生成并推送。
1. Fork 或启用本仓库的 Actions
2. **Settings → Secrets and variables → Actions** 添加 Secrets
| Secret | 必需 | 说明 |
|--------|------|------|
| `WECOM_WEBHOOK_KEY` | 是 | 企微群机器人 webhook key |
| `GH_PAT` | 否 | 个人 GitHub PAT提高 API 限额;不设则用 Actions 内置 token |
| `CURSOR_API_KEY` | 否 | Agent 模式Cursor SDK |
| `DAILY_LLM_API_KEY` | 否 | Agent / 编辑层OpenAI 兼容 API |
3. 可选 **Variables**(不设则用代码默认值):`DAILY_REPORT_MODE`、`DAILY_LLM_MODEL`、`DAILY_AI_NEWS` 等
4. **Actions → daily → Run workflow** 可手动触发;勾选 *Skip WeCom push* 则只生成不上报
运行产物可在 workflow 的 **Artifacts** 中下载(`.md` / `.json`,保留 14 天)。
### Docker
```bash
cp .env.example .env # 填入 WECOM_WEBHOOK_KEY 等
docker compose build
docker compose run --rm daily # 生成 + 推送
docker compose run --rm daily --skip-push # 只生成
```
`output/`、`.cache/`、`logs/` 挂载到宿主机,与本地 `run-daily.ps1` 行为一致。
---
## Agent 模式(推荐)
默认 `classic` 是分区榜单模板;**推荐 Agent 模式**——导语 + 今日信号 + 精选新闻 + 榜单,全中文叙述。
```env
DAILY_REPORT_MODE=agent
CURSOR_API_KEY=cursor_...
CURSOR_MODEL=composer-2.5
DAILY_CURSOR_CWD=.
```
| 模式 | 环境变量 | 风格 |
|------|----------|------|
| `classic` | — | 分区榜单 + 模板 |
| `editor` | `DAILY_CURSOR_EDITOR=1` | 模板 + LLM 中文化 |
| **`agent`** | `DAILY_REPORT_MODE=agent` | **叙事化早报(推荐)** |
流水线:
```
Python 抓取 → Step1 趋势分析 (.trends.json) → Step2 写企微稿 (.wecom.md) → webhook 分条推送
```
- 规范见 `skills/daily-agent/SKILL.md`
- LLM 失败自动回退 `classic`,不影响 `run-daily.ps1`
- 无 `CURSOR_API_KEY` 时请用 `classic`,或配置 `DAILY_LLM_API_KEY`OpenAI 兼容 API
定时推送Windows 任务计划程序每日执行 `run-daily.ps1`,或 cron / `schtasks`。
---
## 自定义 RSS 源
国际 RSS 列表:`daily/news/feeds.py`
国内 RSS 列表:`daily/news/feeds_cn.py`
在对应文件的 `FEEDS` 列表中增删 URL 即可。常用开关(见 `.env.example`
```env
DAILY_AI_NEWS=1
DAILY_CN_AI_NEWS=1
DAILY_AI_NEWS_HOURS=72
DAILY_WECOM_AI_NEWS=10
DAILY_WECOM_CN_AI_NEWS=8
```
Skills 榜单默认爬 skills.sh 官网600+ 条);可改为第三方 feed
```env
# SKILLS_BOARD_SOURCE=feed
```
GitHub 新兴/Topic 榜需 `GITHUB_TOKEN`(见 `.env.example` 注释)。
---
## Experimental`bot/` 企微对话机器人
同仓库内的 **API 模式智能机器人**@ 机器人快查 skills、Cursor 任务、Playwright 截图)**不在 v1 开源支持范围内**,需独立 venv 与 `WECOM_BOT_ID` / `SECRET`。
详见 **[bot/README.md](bot/README.md)**。
---
## 项目结构 ## 项目结构
``` ```
skills-hot-daily/ daily-robots/
├── README.md ├── run-daily.ps1 # 生成 + 推送
├── .env.example # 早报 webhook、GitHub 等 ├── daily/ # 早报主包
├── requirements.txt # 早报 Python 依赖 ├── shared/skills_data.py # skills feed 共用数据层
├── run-daily.ps1 # 生成 + 推送一条龙 ├── skills/daily-agent/ # Agent 工作流规范
├── send-wecom.ps1 # 仅推送 ├── output/ # 生成产物gitignore
├── daily/ # 早报 Python 包 └── bot/ # experimental
│ ├── __main__.py # python -m daily [generate|push]
│ ├── config.py
│ ├── generate.py
│ ├── report_data.py # JSON 中间层
│ ├── cursor_editor.py # Cursor 编辑层
│ ├── llm_client.py
│ ├── localize.py # 仅中文化Tier A
│ ├── format_wecom.py
│ ├── webhook.py
│ ├── news/ # 国际 AI 时讯 RSS
│ │ ├── feeds.py
│ │ └── fetch.py
│ └── github/
│ ├── auth.py
│ ├── search.py
│ └── trending.py
├── output/ # YYYY-MM-DD.md / .wecom.md / .data.json / .editorial.json
├── skills/daily-editor/ # 早报 Cursor 编辑 Skill
│ └── SKILL.md
├── logs/
├── .cache/
└── bot/ # 企微 API 模式对话机器人(独立 venv
├── main.py
├── skills_service.py
└── scenarios/
``` ```
| 模块 | 配置文件 | 启动方式 |
|------|----------|----------|
| **早报推送** | 根目录 `.env``WECOM_WEBHOOK_KEY` 等) | `.\run-daily.ps1` |
| **对话 Bot** | `bot/.env``WECOM_BOT_ID` / `SECRET` 等) | `cd bot``python main.py` |
---
## 一、早报推送
```powershell
cd d:\LY\test\tech\skills-hot-daily
pip install -r requirements.txt
copy .env.example .env
.\run-daily.ps1
```
- 生成:`output/YYYY-MM-DD.md`(完整版)、`output/YYYY-MM-DD.wecom.md`(企微短版)
- 仅生成:`.\run-daily.ps1 -SkipPush`
- 仅推送:`python -m daily push output\2026-06-25.wecom.md`
**Webhook 配置**:企微群 → 群机器人 → 添加,将 `key=` 后的值写入项目根 `.env`
```env
WECOM_WEBHOOK_KEY=your-key
```
**GitHub 数据源**(见 `.env.example`
| 来源 | 说明 |
|------|------|
| GitHub Trending | `GITHUB_TRENDING_MODE=scrape``api` |
| 新兴 / Topic | Search API`GITHUB_TOKEN` |
| Release | 可选 `GITHUB_REPOS=owner/repo` |
**国际 AI 时讯**RSS`daily/news/feeds.py`
| 类别 | 覆盖 |
|------|------|
| 厂商官方 | Anthropic、OpenAI、Google、Meta、Microsoft、Mistral、Cursor 等 |
| Agent / LLM 开发者 | LangChain、LlamaIndex、Hugging Face、Copilot 等 |
| 综合科技媒体 | The Verge、TechCrunch、Ars、Wired、MIT TR 等 |
| Newsletter | Ben's Bites、Latent Space、Simon Willison、TLDR AI 等 |
| 研究 / 论文 | arXiv cs.CL/AI/LG、HF Papers |
| 社区讨论 | HN、Reddit r/LocalLLaMA / ClaudeAI / ML 等 |
环境变量:`DAILY_AI_NEWS=1` · `DAILY_CN_AI_NEWS=1` · `DAILY_AI_NEWS_HOURS=72` · `DAILY_WECOM_AI_NEWS=10` · `DAILY_WECOM_CN_AI_NEWS=8`
**国内 AI 时讯**RSS`daily/news/feeds_cn.py`
| 类别 | 覆盖 |
|------|------|
| AI 专业媒体 | 量子位、InfoQ 中文 |
| 综合科技 | 36氪、雷锋网、Google News 中文 |
| 开发者社区 | 掘金(标题 AI 关键词过滤) |
### 生成架构Tier B · Cursor 编辑层)
早报默认走 **Python 抓取 + 模板渲染**;可选开启 Cursor 做「编辑」:
```
抓取数据 → output/日期.data.json → Cursor 读 Skill 写 editorial → 模板填字 → .md / .wecom.md
```
| 文件 | 说明 |
|------|------|
| `output/YYYY-MM-DD.data.json` | 结构化榜单(供 LLM 输入) |
| `output/YYYY-MM-DD.editorial.json` | Cursor 输出的主题、速览、中文描述 |
| `skills/daily-editor/SKILL.md` | 编辑规范语气、JSON 格式) |
```env
# 开启 Tier B需 CURSOR_API_KEY 或 DAILY_LLM_API_KEY
DAILY_CURSOR_EDITOR=1
```
- 开启后:**一次 LLM 调用** 生成 `theme_line` + `highlights` + 全部中文描述
- 关闭时(默认):规则主题 + `DAILY_ZH_DESC` 仅中文化描述
- LLM 失败自动回退规则模式,不影响推送
### 生成架构Agent 工作流 · 推荐)
若觉得模板版「榜单堆砌」不友好,可改用 **Agent 三步流水线**
```
Python 抓取 → Step1 趋势分析 → Step2 叙事写稿 → Python 分条推送企微
(.trends.json) (.wecom.md)
```
| 模式 | 环境变量 | 企微版风格 |
|------|----------|------------|
| `classic`(默认) | — | 分区榜单 + 模板 |
| `editor` | `DAILY_CURSOR_EDITOR=1` | 模板 + LLM 中文化 |
| `agent` | `DAILY_REPORT_MODE=agent` | **导语 + 信号 + 精选**,全中文叙述 |
```env
DAILY_REPORT_MODE=agent
DAILY_CURSOR_CWD=d:\LY\diy\skills-hot-daily # 早报 LLM 工作目录(与 bot 的 CURSOR_CWD 独立)
```
| 文件 | 说明 |
|------|------|
| `output/YYYY-MM-DD.trends.json` | Step1 趋势分析结果 |
| `skills/daily-agent/SKILL.md` | Agent 工作流规范 |
- 完整版 `YYYY-MM-DD.md` 仍为数据表格归档;企微版由 Agent 直接写 Markdown
- Agent 失败自动回退 `classic`,不影响 `run-daily.ps1`
定时推送Windows 任务计划程序或 `/loop 1d` 执行 `run-daily.ps1`
---
## 二、可对话 Skills 助手(企业微信智能机器人)
在企微里 @ 机器人即可:
- **快查**`trending 10``hot 10``搜索 react`(本地 skills 数据,秒回)
- **截图预览**`preview` / `截图`(基于 `.env``CURSOR_CWD` 启动前端并发图)
- **通用任务**:任意自然语言需求,由 **Cursor Agent** 执行并回传结果
### 1. 创建 API 模式机器人
1. [企业微信管理后台](https://work.weixin.qq.com/) → **安全与管理****管理工具****智能机器人****创建机器人**
2. 选择 **API 模式创建****使用长连接**
3. 记录 **Bot ID****Secret**Secret 只显示一次,请立即保存)
4. 设置可见范围,将机器人 **添加到目标群** 或允许成员单聊
普通成员路径:工作台 → 智能机器人 → 手动创建 → API 模式 → 长连接
### 2. 启动本地服务
```powershell
cd d:\LY\test\tech\skills-hot-daily\bot
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
playwright install chromium
copy .env.example .env
# 编辑 .envWECOM_BOT_ID / WECOM_BOT_SECRET / CURSOR_API_KEY
python main.py
```
服务需 **常驻运行**(本机、服务器或 Docker。长连接模式下机器人进程须在线才能收消息。
### 3. 路由模式ROUTING_MODE
| 模式 | 行为 |
|------|------|
| `hybrid`(默认) | `trending`/`hot`/`搜索`/`详情` 走本地快查;其余 @ 消息交给 Cursor |
| `cursor` | 所有消息都交给 Cursor 执行 |
| `skills` | 仅本地 skills 快查(旧行为) |
**Cursor 任务示例**(群里发送):
```
@test 总结 trending top10并推荐 3 个适合前端团队的 skill
@test 对比 mattpocock/skills 和 obra/superpowers 各有哪些热门 skill
@test 帮我写一段 npx skills add 的安装说明
```
Cursor 在本机 `CURSOR_CWD` 目录下运行,默认 `d:\LY\test\tech`。复杂任务可能需要 110 分钟流式消息会显示「Cursor 正在执行任务…」。
### 4. 前端截图预览API 模式发图)
项目路径读取 `.env` 中的 **`CURSOR_CWD`**。机器人会:
1.`CURSOR_CWD` 检测 `package.json`,若有 `dev` 脚本则执行 `PREVIEW_DEV_COMMAND`(默认 `npm run dev`
2. 等待 `PREVIEW_PORT`(默认 `5173`)就绪,或用 `PREVIEW_URL` 直接访问
3. Playwright 打开页面并截图
4. 通过 API 模式 **上传图片 + 回复 image 消息** 到群
| 命令 | 说明 |
|------|------|
| `preview` / `截图` / `预览` | 访问 `http://127.0.0.1:5173/` 并截图 |
| `preview /login` | 指定路径 |
| `preview / 3000` | 指定端口 |
| `preview http://127.0.0.1:8080/` | 指定完整 URL |
**多步网页操作**(登录、点菜单、再截图)见下一节,不再写死在代码里。
`.env` 可选配置:
```env
CURSOR_CWD=d:\LY\test\tech
PREVIEW_PORT=5173
PREVIEW_URL=http://127.0.0.1:5173/
PREVIEW_DEV_COMMAND=npm run dev
PREVIEW_STARTUP_TIMEOUT=120
```
`CURSOR_CWD` 下暂无前端项目,可先手动启动 dev server或设置 `PREVIEW_URL` 指向已运行地址。
### 4b. 网页操作Playwright 步骤引擎)
支持三种方式定义操作流程,**无需改 Python 代码**
**1. 自然语言(企微里直接说)**
```
@test 访问登录页,输入账号密码,点击登录后进入主页,点击智能体管理菜单然后截图
```
账号密码从 `.env` 读取(`{{PREVIEW_LOGIN_USER}}` / `{{PREVIEW_LOGIN_PASSWORD}}`),勿在群里发密码。
**2. 场景文件 YAML**
`bot/scenarios/xiaobao-agent-manage.yaml` 示例:
```yaml
name: xiaobao-agent-manage
steps:
- goto: /login
- fill:
field: 账号
value: "{{PREVIEW_LOGIN_USER}}"
- fill:
field: 密码
value: "{{PREVIEW_LOGIN_PASSWORD}}"
- click: 登录
- wait:
url: "**/app/**"
- click: 智能体管理
- wait: 1500
- screenshot
```
触发:`@test browser xiaobao-agent-manage`
场景搜索路径:`bot/scenarios/``CURSOR_CWD/.browser-scenarios/`、环境变量 `BROWSER_SCENARIOS_DIR`
**3. 消息内 DSL**
```
browser:
goto /login
fill 账号 {{PREVIEW_LOGIN_USER}}
fill 密码 {{PREVIEW_LOGIN_PASSWORD}}
click 登录
click 智能体管理
screenshot
```
**支持的步骤**`goto` · `fill` · `click` · `wait` · `screenshot` · `press`
`.env` 登录与场景配置:
```env
PREVIEW_LOGIN_USER=test_account
PREVIEW_LOGIN_PASSWORD=your_password
# BROWSER_DEFAULT_SCENARIO=xiaobao-agent-manage
```
### 5. 支持的快查命令
| 命令 | 说明 |
|------|------|
| `trending 10` / `趋势 10` | 近期增长榜 Top N默认 10最大 30 |
| `hot 10` / `实时 10` | 实时热度榜 |
| `all 10` / `总榜 10` | 历史总安装榜 |
| `搜索 react` / `search tdd` | 关键词搜索 |
| `详情 find-skills` | 单个 skill 详情 + 安装命令 |
| `preview` / `截图` | 启动 CURSOR_CWD 前端并截图发群 |
| `帮助` | 命令列表 |
自然语言(非显式快查命令)会交给 **Cursor** 处理,例如 `@test 查 trending 并写推荐`
### 6. 本地测试(无需企微凭证)
```powershell
cd d:\LY\test\tech\skills-hot-daily\bot
python -c "from skills_service import handle_command; print(handle_command('trending 5'))"
```
---
## 其他推送方案
| 方案 | 适用场景 | 复杂度 |
|------|----------|--------|
| **群机器人 webhook** | 推送到固定群 | 低 |
| **应用消息 API** | 推送给指定成员/部门 | 中(需 corp_id、secret、agent_id |
| **邮件 + 企业微信邮箱** | 已有 SMTP | 中 |
| **PushPlus / Server酱** | 个人微信中转 | 低(第三方) |
### 应用消息 API简要
适合「推送给某个人」而非群聊。需在 [企业微信管理后台](https://work.weixin.qq.com/) 创建自建应用,调用:
`POST https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=TOKEN`
消息体支持 `text` / `markdown` / `news` 等。需先 `gettoken` 再发消息,并维护 access_token 缓存。
--- ---
## 注意事项 ## 注意事项
- Webhook **不要提交 Git**,只用环境变量 - Webhook / API Key **勿提交 Git**,只用 `.env`
- 企业微信 markdown 为**子集**(不支持完整 GitHub 表格语法时可改为文本列表 - 企微 markdown 为子集;超长报告**自动分条推送**(默认 4096 bytes/条
- 单条消息约 **4096 字节** 上限,`send-wecom.ps1` 已做截断 - 完整配置项见 [`.env.example`](.env.example)
- 开发测试:`pip install -r requirements-dev.txt` → `pytest`

View File

@@ -5,7 +5,7 @@ WECOM_BOT_SECRET=your-bot-secret
# Cursor SDK@ 机器人后的通用任务由 Cursor 执行) # Cursor SDK@ 机器人后的通用任务由 Cursor 执行)
CURSOR_API_KEY=cursor_... CURSOR_API_KEY=cursor_...
CURSOR_CWD=d:\LY\test\tech CURSOR_CWD=.
CURSOR_MODEL=composer-2.5 CURSOR_MODEL=composer-2.5
CURSOR_TIMEOUT=600 CURSOR_TIMEOUT=600

143
bot/README.md Normal file
View File

@@ -0,0 +1,143 @@
# bot/ — 企微 API 模式智能机器人
> **EXPERIMENTAL** — 本模块不在 daily-robots v1 开源支持范围内。
> 主产品为根目录 **早报推送**webhook见 [README](../README.md)。
> Bot 需独立 venv、企微 API 凭证、常驻进程;问题请自行排查或提 issue 标注 `bot`。
在企微里 @ 机器人即可:
- **快查**`trending 10``hot 10``搜索 react`(本地 skills 数据,秒回)
- **截图预览**`preview` / `截图`(基于 `CURSOR_CWD` 启动前端并发图)
- **通用任务**:任意自然语言需求,由 **Cursor Agent** 执行并回传结果
数据层与早报共用 [`shared/skills_data.py`](../shared/skills_data.py)。
---
## 1. 创建 API 模式机器人
1. [企业微信管理后台](https://work.weixin.qq.com/) → **安全与管理****管理工具****智能机器人****创建机器人**
2. 选择 **API 模式创建****使用长连接**
3. 记录 **Bot ID****Secret**Secret 只显示一次,请立即保存)
4. 设置可见范围,将机器人 **添加到目标群** 或允许成员单聊
普通成员路径:工作台 → 智能机器人 → 手动创建 → API 模式 → 长连接
---
## 2. 启动本地服务
```powershell
cd path\to\daily-robots\bot
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
playwright install chromium
copy .env.example .env
# 编辑 .envWECOM_BOT_ID / WECOM_BOT_SECRET / CURSOR_API_KEY
python main.py
```
服务需 **常驻运行**(本机、服务器或 Docker。长连接模式下机器人进程须在线才能收消息。
---
## 3. 路由模式ROUTING_MODE
| 模式 | 行为 |
|------|------|
| `hybrid`(默认) | `trending`/`hot`/`搜索`/`详情` 走本地快查;其余 @ 消息交给 Cursor |
| `cursor` | 所有消息都交给 Cursor 执行 |
| `skills` | 仅本地 skills 快查 |
**Cursor 任务示例**(群里发送):
```
@test 总结 trending top10并推荐 3 个适合前端团队的 skill
@test 对比 mattpocock/skills 和 obra/superpowers 各有哪些热门 skill
@test 帮我写一段 npx skills add 的安装说明
```
Cursor 在本机 `CURSOR_CWD` 目录下运行(见 `bot/.env`)。复杂任务可能需要 110 分钟流式消息会显示「Cursor 正在执行任务…」。
---
## 4. 前端截图预览API 模式发图)
项目路径读取 `.env` 中的 **`CURSOR_CWD`**。机器人会:
1.`CURSOR_CWD` 检测 `package.json`,若有 `dev` 脚本则执行 `PREVIEW_DEV_COMMAND`(默认 `npm run dev`
2. 等待 `PREVIEW_PORT`(默认 `5173`)就绪,或用 `PREVIEW_URL` 直接访问
3. Playwright 打开页面并截图
4. 通过 API 模式 **上传图片 + 回复 image 消息** 到群
| 命令 | 说明 |
|------|------|
| `preview` / `截图` / `预览` | 访问 `http://127.0.0.1:5173/` 并截图 |
| `preview /login` | 指定路径 |
| `preview / 3000` | 指定端口 |
| `preview http://127.0.0.1:8080/` | 指定完整 URL |
`.env` 可选配置:
```env
CURSOR_CWD=.
PREVIEW_PORT=5173
PREVIEW_URL=http://127.0.0.1:5173/
PREVIEW_DEV_COMMAND=npm run dev
PREVIEW_STARTUP_TIMEOUT=120
```
---
## 4b. 网页操作Playwright 步骤引擎)
支持三种方式定义操作流程,**无需改 Python 代码**
**1. 自然语言(企微里直接说)**
```
@test 访问登录页,输入账号密码,点击登录后进入主页,点击智能体管理菜单然后截图
```
账号密码从 `.env` 读取(`{{PREVIEW_LOGIN_USER}}` / `{{PREVIEW_LOGIN_PASSWORD}}`),勿在群里发密码。
**2. 场景文件 YAML** — 见 `scenarios/`,触发:`@test browser <场景名>`
**3. 消息内 DSL** — 以 `browser:` 开头的多行步骤
支持的步骤:`goto` · `fill` · `click` · `wait` · `screenshot` · `press`
---
## 5. 快查命令
| 命令 | 说明 |
|------|------|
| `trending 10` / `趋势 10` | 近期增长榜 Top N默认 10最大 30 |
| `hot 10` / `实时 10` | 实时热度榜 |
| `all 10` / `总榜 10` | 历史总安装榜 |
| `搜索 react` / `search tdd` | 关键词搜索 |
| `详情 find-skills` | 单个 skill 详情 + 安装命令 |
| `preview` / `截图` | 启动 CURSOR_CWD 前端并截图发群 |
| `帮助` | 命令列表 |
---
## 6. 本地测试(无需企微凭证)
```powershell
cd path\to\daily-robots\bot
python -c "from skills_service import handle_command; print(handle_command('trending 5'))"
```
---
## 配置
| 文件 | 说明 |
|------|------|
| `bot/.env` | `WECOM_BOT_ID``WECOM_BOT_SECRET``CURSOR_API_KEY` 等 |
| `.env.example` | 模板 |
与根目录 `.env`(早报 webhook**相互独立**,勿混用。

View File

@@ -3,11 +3,16 @@
from __future__ import annotations from __future__ import annotations
import os import os
import sys
from pathlib import Path from pathlib import Path
from dotenv import load_dotenv from dotenv import load_dotenv
_BOT_DIR = Path(__file__).resolve().parent _BOT_DIR = Path(__file__).resolve().parent
_REPO_ROOT = _BOT_DIR.parent
if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))
load_dotenv(_BOT_DIR / ".env") load_dotenv(_BOT_DIR / ".env")
load_dotenv(_BOT_DIR / ".env.local", override=True) load_dotenv(_BOT_DIR / ".env.local", override=True)

View File

@@ -1,32 +1,20 @@
"""skills.sh 数据查询与命令解析""" """skills.sh 快查命令解析与企微回复格式化"""
from __future__ import annotations from __future__ import annotations
import json
import logging
import re import re
import time import sys
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any, Literal from typing import Any, Literal
import certifi _REPO_ROOT = Path(__file__).resolve().parent.parent
import httpx if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))
logger = logging.getLogger(__name__) from shared.skills_data import Board, board_items, format_installs, load_feed, warm_feed_cache
FEED_URLS = [ __all__ = ["Command", "handle_command", "parse_command", "warm_feed_cache"]
# jsDelivr 在国内通常比 raw.githubusercontent.com 更稳定
"https://cdn.jsdelivr.net/gh/NeverSight/skills.sh_feed@main/data/feed.json",
"https://raw.githubusercontent.com/NeverSight/skills.sh_feed/main/data/feed.json",
]
CACHE_TTL_SECONDS = 600
CACHE_DIR = Path(__file__).resolve().parent / ".cache"
CACHE_FILE = CACHE_DIR / "feed.json"
_cache: dict[str, Any] = {"data": None, "fetched_at": 0.0}
Board = Literal["trending", "hot", "all"]
@dataclass @dataclass
@@ -37,72 +25,6 @@ class Command:
query: str = "" query: str = ""
def _fetch_json(url: str) -> dict[str, Any]:
headers = {
"User-Agent": "skills-hot-bot/1.0",
"Accept": "application/json",
}
with httpx.Client(
timeout=httpx.Timeout(20.0, connect=10.0),
verify=certifi.where(),
follow_redirects=True,
) as client:
resp = client.get(url, headers=headers)
resp.raise_for_status()
return resp.json()
def _load_disk_cache() -> dict[str, Any] | None:
if not CACHE_FILE.exists():
return None
try:
return json.loads(CACHE_FILE.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
logger.warning("读取本地缓存失败: %s", exc)
return None
def _save_disk_cache(data: dict[str, Any]) -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
def load_feed(force: bool = False) -> dict[str, Any]:
now = time.time()
if not force and _cache["data"] and now - _cache["fetched_at"] < CACHE_TTL_SECONDS:
return _cache["data"]
errors: list[str] = []
for url in FEED_URLS:
for attempt in range(3):
try:
data = _fetch_json(url)
_cache["data"] = data
_cache["fetched_at"] = now
_save_disk_cache(data)
logger.info("skills 数据已更新: %s", url)
return data
except Exception as exc:
msg = f"{url} (#{attempt + 1}): {exc}"
errors.append(msg)
logger.debug("拉取失败 %s", msg)
time.sleep(0.5 * (attempt + 1))
stale = _load_disk_cache()
if stale:
logger.warning("网络不可用,回退到本地缓存")
_cache["data"] = stale
_cache["fetched_at"] = now
return stale
raise RuntimeError(f"无法获取 skills 数据。最近错误: {errors[-1]}")
def warm_feed_cache() -> None:
"""启动时预加载,避免首条消息才触发网络请求。"""
load_feed(force=True)
def _normalize_text(text: str) -> str: def _normalize_text(text: str) -> str:
text = re.sub(r"@\S+\s*", "", text) text = re.sub(r"@\S+\s*", "", text)
return text.strip().lower() return text.strip().lower()
@@ -157,19 +79,6 @@ def parse_command(text: str) -> Command:
return Command(kind="search", query=raw, limit=5) return Command(kind="search", query=raw, limit=5)
def _format_installs(n: int | float) -> str:
if n >= 1_000_000:
return f"{n / 1_000_000:.1f}M"
if n >= 1_000:
return f"{n / 1_000:.1f}K"
return str(int(n))
def _board_items(feed: dict[str, Any], board: Board) -> list[dict[str, Any]]:
key = {"trending": "topTrending", "hot": "topHot", "all": "topAllTime"}[board]
return feed.get(key, [])
def _board_title(board: Board) -> str: def _board_title(board: Board) -> str:
return { return {
"trending": "Trending近期增长", "trending": "Trending近期增长",
@@ -180,7 +89,7 @@ def _board_title(board: Board) -> str:
def format_list(board: Board, limit: int) -> str: def format_list(board: Board, limit: int) -> str:
feed = load_feed() feed = load_feed()
items = _board_items(feed, board)[:limit] items = board_items(feed, board)[:limit]
updated = feed.get("updatedAt", "未知")[:10] updated = feed.get("updatedAt", "未知")[:10]
lines = [ lines = [
@@ -192,7 +101,7 @@ def format_list(board: Board, limit: int) -> str:
for i, item in enumerate(items, 1): for i, item in enumerate(items, 1):
title = item.get("title", "?") title = item.get("title", "?")
source = item.get("source", "?") source = item.get("source", "?")
installs = _format_installs(item.get("installs", 0)) installs = format_installs(item.get("installs", 0))
desc = item.get("description", "") desc = item.get("description", "")
if len(desc) > 80: if len(desc) > 80:
desc = desc[:77] + "..." desc = desc[:77] + "..."
@@ -241,7 +150,7 @@ def format_search(query: str, limit: int) -> str:
for i, item in enumerate(matches, 1): for i, item in enumerate(matches, 1):
title = item.get("title", "?") title = item.get("title", "?")
source = item.get("source", "?") source = item.get("source", "?")
installs = _format_installs(item.get("installs", 0)) installs = format_installs(item.get("installs", 0))
link = item.get("link", "") link = item.get("link", "")
lines.append(f"{i}. **{title}** · {installs} · `{source}`") lines.append(f"{i}. **{title}** · {installs} · `{source}`")
if link: if link:
@@ -270,7 +179,7 @@ def format_detail(name: str) -> str:
[ [
f"**{best.get('title', '?')}**", f"**{best.get('title', '?')}**",
f"`{best.get('source', '?')}`", f"`{best.get('source', '?')}`",
f"安装量:**{_format_installs(best.get('installs', 0))}**", f"安装量:**{format_installs(best.get('installs', 0))}**",
"", "",
desc, desc,
"", "",

View File

@@ -3,13 +3,11 @@
from __future__ import annotations from __future__ import annotations
import os import os
import sys
from pathlib import Path from pathlib import Path
from dotenv import load_dotenv from dotenv import load_dotenv
ROOT = Path(__file__).resolve().parent.parent ROOT = Path(__file__).resolve().parent.parent
BOT_DIR = ROOT / "bot"
OUTPUT_DIR = ROOT / "output" OUTPUT_DIR = ROOT / "output"
LOG_DIR = ROOT / "logs" LOG_DIR = ROOT / "logs"
CACHE_DIR = ROOT / ".cache" CACHE_DIR = ROOT / ".cache"
@@ -49,12 +47,6 @@ load_dotenv(ROOT / ".env")
load_dotenv(ROOT / ".env.local", override=True) load_dotenv(ROOT / ".env.local", override=True)
def ensure_bot_on_path() -> None:
bot = str(BOT_DIR)
if bot not in sys.path:
sys.path.insert(0, bot)
def _clean_env_value(raw: str | None) -> str | None: def _clean_env_value(raw: str | None) -> str | None:
if raw is None: if raw is None:
return None return None

View File

@@ -2,475 +2,58 @@
from __future__ import annotations from __future__ import annotations
import json import logging
import re
import sys import sys
import xml.etree.ElementTree as ET
from collections import defaultdict
from datetime import datetime, timezone, timedelta from datetime import datetime, timezone, timedelta
from pathlib import Path from pathlib import Path
from typing import Any
import certifi
import httpx
import logging
logger = logging.getLogger(__name__)
from daily.config import (
CACHE_DIR,
LOG_DIR,
OUTPUT_DIR,
SNAPSHOT_FILE,
ensure_bot_on_path,
env,
env_int,
full_desc_limit,
news_summary_limit,
wecom_skill_desc_limit,
)
from daily.format_wecom import build_wecom_report, finalize_wecom_skill_groups, replace_wecom_skill_sections
from daily.agent_workflow import is_agent_mode, run_agent_workflow from daily.agent_workflow import is_agent_mode, run_agent_workflow
from daily.delta import compare_depth from daily.config import LOG_DIR, OUTPUT_DIR, env
from daily.cursor_editor import ( from daily.cursor_editor import (
apply_descriptions, apply_descriptions,
is_enabled as cursor_editor_enabled, is_enabled as cursor_editor_enabled,
run_editorial, run_editorial,
theme_line_from_editorial, theme_line_from_editorial,
) )
from daily.github.auth import github_html_headers from daily.format_wecom import build_wecom_report, finalize_wecom_skill_groups, replace_wecom_skill_sections
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos from daily.github.trending import trending_data_source_note
from daily.github.trending import fetch_github_trending, trending_data_source_note
from daily.localize import LocalizeJob, localize_descriptions, needs_chinese
from daily.news.fetch import ( from daily.news.fetch import (
fetch_ai_news,
fetch_cn_ai_news,
format_cn_news_section, format_cn_news_section,
format_news_section, format_news_section,
prepare_wecom_cn_news_items, prepare_wecom_cn_news_items,
prepare_wecom_news_items, prepare_wecom_news_items,
) )
from daily.report_data import ( from daily.pipeline.collect import collect_report_context
build_full_payload, from daily.pipeline.formatters import (
build_llm_input, build_highlights,
data_json_path, fetch_latest_release_title,
save_json, format_github_repo_section,
format_skill_section,
prepare_github_item,
prepare_skill_item,
) )
from daily.skills_board import load_boards from daily.pipeline.localize import localize_descriptions_in_place
from daily.pipeline.snapshot import load_snapshot, save_snapshot
from daily.pipeline.themes import detect_theme_line, theme_clusters
from daily.report_data import build_full_payload, data_json_path, save_json
from daily.skills_group import group_skills_by_source from daily.skills_group import group_skills_by_source
ensure_bot_on_path() logger = logging.getLogger(__name__)
from skills_service import _format_installs, load_feed # noqa: E402
THEME_RULES: list[tuple[str, str, list[str]]] = [
("🎬", "AI 多媒体 / 视频", ["runcomfy", "remotion", "video", "seedance", "inpaint", "lipsync"]),
("🔧", "工程协作 / Skill 元能力", ["grill", "tdd", "architecture", "find-skills", "to-issues"]),
("📱", "飞书 / Lark", ["lark", "feishu"]),
("📣", "内容营销", ["viral", "tiktok", "instagram", "reels"]),
("🎨", "设计 / 前端", ["frontend", "design", "ui-ux", "tailwind"]),
]
def _now_cst() -> datetime: def _now_cst() -> datetime:
return datetime.now(timezone(timedelta(hours=8))) return datetime.now(timezone(timedelta(hours=8)))
def _short_desc(text: str, limit: int = 72) -> str:
text = re.sub(r"\s+", " ", text or "").strip()
if limit <= 0 or len(text) <= limit:
return text
return text[: limit - 3] + "..."
def _archive_desc(text: str) -> str:
return _short_desc(text, full_desc_limit())
def _wecom_desc(text: str, limit: int = 36) -> str:
return _short_desc(text, limit)
def _localize_descriptions_in_place(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
github_trending: list[dict[str, Any]],
github_emerging: list[dict[str, Any]],
github_topic: list[dict[str, Any]],
ai_news: dict[str, Any],
cn_ai_news: dict[str, Any] | None = None,
) -> None:
full_limit = full_desc_limit()
news_limit = news_summary_limit()
jobs: list[LocalizeJob] = []
seen_skill: set[str] = set()
for item in trending + hot:
sid = _skill_id(item)
if sid in seen_skill:
continue
seen_skill.add(sid)
desc = (item.get("description") or "").strip()
if desc:
jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
seen_repo: set[str] = set()
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
repo = item.get("repo", "")
if not repo or repo in seen_repo:
continue
seen_repo.add(repo)
desc = (item.get("description") or "").strip()
if desc:
jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
if ai_news.get("enabled"):
seen_news: set[str] = set()
for item in ai_news.get("flat") or []:
link = item.get("link", "")
if not link or link in seen_news:
continue
seen_news.add(link)
summary = (item.get("summary") or "").strip()
if summary:
jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
zh_map = localize_descriptions(jobs, archive=True)
if not zh_map and not jobs:
return
def _apply_zh(mapping: dict[str, str]) -> None:
for item in trending + hot:
key = f"skill:{_skill_id(item)}"
if key in mapping:
item["description"] = mapping[key]
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
key = f"github:{item.get('repo', '')}"
if key in mapping:
item["description"] = mapping[key]
if ai_news.get("enabled"):
for cat in ai_news.get("categories") or []:
for item in cat.get("items") or []:
key = f"news:{item.get('link', '')}"
if key in mapping:
item["summary"] = mapping[key]
for item in ai_news.get("flat") or []:
key = f"news:{item.get('link', '')}"
if key in mapping:
item["summary"] = mapping[key]
_apply_zh(zh_map)
# 仍为英文的条目再译一轮(长描述或批次失败时)
retry_jobs: list[LocalizeJob] = []
seen_skill.clear()
for item in trending + hot:
sid = _skill_id(item)
if sid in seen_skill:
continue
seen_skill.add(sid)
desc = (item.get("description") or "").strip()
if needs_chinese(desc):
retry_jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
seen_repo.clear()
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
repo = item.get("repo", "")
if not repo or repo in seen_repo:
continue
seen_repo.add(repo)
desc = (item.get("description") or "").strip()
if needs_chinese(desc):
retry_jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
if ai_news.get("enabled"):
seen_news.clear()
for item in ai_news.get("flat") or []:
link = item.get("link", "")
if not link or link in seen_news:
continue
seen_news.add(link)
summary = (item.get("summary") or "").strip()
if needs_chinese(summary):
retry_jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
if retry_jobs:
_apply_zh(localize_descriptions(retry_jobs, archive=True))
def _skill_id(item: dict[str, Any]) -> str:
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
def _load_snapshot() -> set[str]:
if not SNAPSHOT_FILE.exists():
return set()
try:
data = json.loads(SNAPSHOT_FILE.read_text(encoding="utf-8"))
return set(str(x) for x in (data.get("skill_ids") or []))
except (OSError, json.JSONDecodeError):
return set()
def _save_snapshot(feed: dict[str, Any], date_str: str) -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
ids: list[str] = []
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
sid = _skill_id(item)
if sid not in ids:
ids.append(sid)
SNAPSHOT_FILE.write_text(
json.dumps({"date": date_str, "skill_ids": ids}, ensure_ascii=False, indent=2),
encoding="utf-8",
)
def _prepare_skill_item(item: dict[str, Any], prev_ids: set[str], rank: int) -> dict[str, Any]:
badge = ""
sid = _skill_id(item)
if sid not in prev_ids and prev_ids:
badge = "🆕"
elif rank == 1:
badge = "👑"
installs_fmt = item.get("installs_fmt") or _format_installs(item.get("installs", 0))
title = item.get("source", "?") if item.get("cluster") else item.get("title", "?")
desc = item.get("wecom_desc") or item.get("description") or item.get("cluster_titles") or ""
limit = wecom_skill_desc_limit()
desc_short = desc if item.get("wecom_desc") or limit <= 0 else _wecom_desc(desc, limit)
return {
"title": title,
"source": item.get("source", "?"),
"installs_fmt": installs_fmt,
"link": item.get("link", ""),
"desc_short": desc_short,
"badge": badge,
"cluster": bool(item.get("cluster")),
"cluster_count": item.get("cluster_count"),
"cluster_titles": item.get("cluster_titles"),
}
def _detect_theme_line(feed: dict[str, Any]) -> str:
scores: dict[str, int] = defaultdict(int)
for board in ("topTrending", "topHot"):
for rank, item in enumerate(feed.get(board, [])[:10], 1):
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, label, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
scores[label] += max(1, 11 - rank)
break
if not scores:
return "**今日主题**Agent Skills 生态持续活跃"
return f"**今日主题**{max(scores.items(), key=lambda x: x[1])[0]}"
def _build_highlights(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
github_trending: list[dict[str, Any]],
github_emerging: list[dict[str, Any]],
ai_news: dict[str, Any] | None = None,
cn_ai_news: dict[str, Any] | None = None,
) -> list[str]:
points: list[str] = []
if ai_news and ai_news.get("enabled"):
top_news = prepare_wecom_news_items(ai_news)
if top_news:
n0 = top_news[0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🌍 AI 时讯 [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
elif ai_news.get("flat"):
n0 = ai_news["flat"][0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(f"🌍 AI 时讯 [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}")
if cn_ai_news and cn_ai_news.get("enabled"):
top_cn = prepare_wecom_cn_news_items(cn_ai_news)
if top_cn:
n0 = top_cn[0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🇨🇳 国内 AI [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
elif cn_ai_news.get("flat"):
n0 = cn_ai_news["flat"][0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🇨🇳 国内 AI [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
if trending:
t0 = trending[0]
points.append(f"📈 Skills 榜首 **{t0.get('title')}**{_format_installs(t0.get('installs', 0))}")
if github_trending:
g0 = github_trending[0]
stars = g0.get("stars_today_fmt", "")
total = g0.get("total_stars_fmt", "")
star_hint = f"+{stars} today · " if stars else (f"{total} · " if total else "")
points.append(f"🐙 GitHub Trending [{g0['repo']}]({g0['url']}){star_hint}{g0.get('language', '')}")
if github_emerging:
e0 = github_emerging[0]
points.append(f"🌱 新兴 [{e0['repo']}]({e0['url']})(⭐ {e0.get('total_stars_fmt', '?')}")
elif hot:
h0 = hot[0]
points.append(f"🔥 Skills Hot 榜首 **{h0.get('title')}**1H {_format_installs(h0.get('installs', 0))}")
while len(points) < 3 and len(trending) > len(points):
item = trending[len(points)]
points.append(f"✨ **{item.get('title')}** · `{item.get('source')}`")
return points[:3]
def _prepare_github_item(item: dict[str, Any]) -> dict[str, Any]:
return {**item, "desc_short": _wecom_desc(item.get("description", ""), 40)}
def _fetch_latest_release_title(repo: str) -> str | None:
atom_url = f"https://github.com/{repo}/releases.atom"
try:
with httpx.Client(
timeout=12.0,
verify=certifi.where(),
follow_redirects=True,
headers=github_html_headers(),
) as client:
resp = client.get(atom_url)
if resp.status_code != 200:
return None
root = ET.fromstring(resp.text)
ns = {"a": "http://www.w3.org/2005/Atom"}
entry = root.find("a:entry", ns)
if entry is None:
return None
title = entry.find("a:title", ns)
return title.text.strip() if title is not None and title.text else None
except Exception:
return None
def _theme_clusters(feed: dict[str, Any], limit: int = 5) -> list[tuple[str, list[str]]]:
buckets: dict[str, list[str]] = defaultdict(list)
seen: set[str] = set()
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
item_id = _skill_id(item)
if item_id in seen:
continue
seen.add(item_id)
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, theme, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
label = f"**{item.get('title')}** (`{item.get('source')}`)"
if label not in buckets[theme]:
buckets[theme].append(label)
break
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]
def _format_github_repo_section(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
lines: list[str] = []
for i, repo in enumerate(repos, 1):
lang = repo.get("language") or ""
stars_today = repo.get("stars_today_fmt") or ""
total = repo.get("total_stars_fmt") or ""
created = repo.get("created_at") or ""
meta_parts = [lang]
if stars_today:
meta_parts.append(f"+{stars_today} today")
if total:
meta_parts.append(f"总 ⭐ {total}")
if show_created and created:
meta_parts.append(f"创建于 {created}")
lines.append(f"{i}. **[{repo['repo']}]({repo['url']})** · {' · '.join(meta_parts)}")
desc = _archive_desc(repo.get("description", ""))
if desc:
lines.append(f" - {desc}")
lines.append("")
return lines
def _format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) -> list[str]:
lines: list[str] = []
for i, item in enumerate(items, 1):
skill_id = item.get("id") or f"{item.get('source', '?')}/{item.get('title', '?')}"
link = item.get("link", "")
installs = _format_installs(item.get("installs", 0))
meta = f"1H {installs}" if hot else f"总安装 {installs}"
if link:
lines.append(f"{i}. **[{skill_id}]({link})** · {meta}")
else:
lines.append(f"{i}. **{skill_id}** · {meta}")
desc = _archive_desc(item.get("description", ""))
if desc:
lines.append(f" - {desc}")
lines.append("")
return lines
def generate_report() -> tuple[str, str, Path, Path]: def generate_report() -> tuple[str, str, Path, Path]:
trending_n = env_int("DAILY_TRENDING_LIMIT", 150) ctx = collect_report_context(_now_cst())
hot_n = max(env_int("DAILY_HOT_LIMIT", 150), compare_depth())
compare_n = compare_depth()
skill_pool = max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
wecom_trending = env_int("DAILY_WECOM_TRENDING", 10)
wecom_hot = env_int("DAILY_WECOM_HOT", 10)
github_limit = env_int("DAILY_GITHUB_TRENDING_LIMIT", 10)
wecom_github = env_int("DAILY_WECOM_GITHUB_TRENDING", env_int("DAILY_WECOM_REPOS", 10))
github_fetch_n = max(github_limit, compare_n, wecom_github)
emerging_limit = env_int("DAILY_GITHUB_EMERGING_LIMIT", 10)
wecom_emerging = env_int("DAILY_WECOM_GITHUB_EMERGING", 10)
emerging_fetch_n = max(emerging_limit, compare_n, wecom_emerging)
topic_limit = env_int("DAILY_GITHUB_TOPIC_LIMIT", 10)
wecom_topic = env_int("DAILY_WECOM_GITHUB_TOPIC", 10)
topic_fetch_n = max(topic_limit, compare_n, wecom_topic)
feed = load_feed(force=True)
prev_ids = _load_snapshot()
now = _now_cst() now = _now_cst()
date_str = now.strftime("%Y-%m-%d") prev_ids = load_snapshot()
time_str = now.strftime("%H:%M") + " (UTC+8)"
updated = (feed.get("updatedAt") or "")[:10]
trending, hot = load_boards(feed, trending_limit=trending_n, hot_limit=hot_n)
github_trending = fetch_github_trending(github_fetch_n)
seen_repos = {r["repo"] for r in github_trending}
github_emerging = fetch_emerging_repos(emerging_fetch_n, exclude=seen_repos)
seen_repos.update(r["repo"] for r in github_emerging)
topic_name, github_topic = fetch_topic_hot_repos(topic_fetch_n, exclude=seen_repos)
ai_news = fetch_ai_news()
cn_ai_news = fetch_cn_ai_news()
wecom_limits = {
"trending": wecom_trending,
"hot": wecom_hot,
"trending_pool": skill_pool,
"hot_pool": skill_pool,
"github": wecom_github,
"emerging": wecom_emerging,
"topic": wecom_topic,
"ai_news": env_int("DAILY_WECOM_AI_NEWS", 10),
"cn_ai_news": env_int("DAILY_WECOM_CN_AI_NEWS", 8),
}
llm_input = build_llm_input(
date_str=date_str,
updated=updated,
trending=trending,
hot=hot,
github_trending=github_trending,
github_emerging=github_emerging,
github_topic=github_topic,
topic_name=topic_name,
ai_news=ai_news,
cn_ai_news=cn_ai_news,
wecom_limits=wecom_limits,
)
save_json( save_json(
data_json_path(date_str), data_json_path(ctx.date_str),
build_full_payload( build_full_payload(
llm_input, ctx.llm_input,
meta={ meta={
"generated_at": now.isoformat(), "generated_at": now.isoformat(),
"report_mode": "agent" if is_agent_mode() else "classic", "report_mode": "agent" if is_agent_mode() else "classic",
@@ -482,10 +65,10 @@ def generate_report() -> tuple[str, str, Path, Path]:
agent_wecom: str | None = None agent_wecom: str | None = None
if is_agent_mode(): if is_agent_mode():
agent_wecom = run_agent_workflow( agent_wecom = run_agent_workflow(
llm_input, ctx.llm_input,
date_str=date_str, date_str=ctx.date_str,
time_str=time_str, time_str=ctx.time_str,
updated=updated, updated=ctx.updated,
) )
if not agent_wecom: if not agent_wecom:
logger.warning("Agent 工作流失败,回退 classic 模式") logger.warning("Agent 工作流失败,回退 classic 模式")
@@ -493,79 +76,88 @@ def generate_report() -> tuple[str, str, Path, Path]:
editorial_theme: str | None = None editorial_theme: str | None = None
editorial_highlights: list[str] | None = None editorial_highlights: list[str] | None = None
if agent_wecom is None: if agent_wecom is None:
editorial = run_editorial(llm_input, date_str=date_str) editorial = run_editorial(ctx.llm_input, date_str=ctx.date_str)
if editorial: if editorial:
apply_descriptions( apply_descriptions(
trending=trending, trending=ctx.trending,
hot=hot, hot=ctx.hot,
github_trending=github_trending, github_trending=ctx.github_trending,
github_emerging=github_emerging, github_emerging=ctx.github_emerging,
github_topic=github_topic, github_topic=ctx.github_topic,
ai_news=ai_news, ai_news=ctx.ai_news,
cn_ai_news=cn_ai_news, cn_ai_news=ctx.cn_ai_news,
descriptions=editorial.get("descriptions") or {}, descriptions=editorial.get("descriptions") or {},
) )
editorial_theme = theme_line_from_editorial(editorial) or None editorial_theme = theme_line_from_editorial(editorial) or None
hl = editorial.get("highlights") or [] hl = editorial.get("highlights") or []
editorial_highlights = hl if hl else None editorial_highlights = hl if hl else None
# 完整版归档:中文化 + 加长摘要(已是中文的条目会跳过翻译) localize_descriptions_in_place(
_localize_descriptions_in_place( ctx.trending,
trending, hot, github_trending, github_emerging, github_topic, ai_news, cn_ai_news ctx.hot,
ctx.github_trending,
ctx.github_emerging,
ctx.github_topic,
ctx.ai_news,
ctx.cn_ai_news,
) )
themes = _theme_clusters(feed) limits = ctx.limits
themes = theme_clusters(ctx.feed)
lines = [ lines = [
f"# 早报 · {date_str}", f"# 早报 · {ctx.date_str}",
"", "",
f"> 生成时间:{now.strftime('%Y-%m-%d %H:%M')} (UTC+8) ", f"> 生成时间:{now.strftime('%Y-%m-%d %H:%M')} (UTC+8) ",
f"> skills 数据更新:{updated} ", f"> skills 数据更新:{ctx.updated} ",
"> 数据来源:[skills.sh/trending](https://skills.sh/trending) · [skills.sh/hot](https://skills.sh/hot) · 国际/国内 AI RSS", "> 数据来源:[skills.sh/trending](https://skills.sh/trending) · [skills.sh/hot](https://skills.sh/hot) · 国际/国内 AI RSS",
"", "",
"---", "---",
"", "",
f"## 一、Skills Trending Top {trending_n}", f"## 一、Skills Trending Top {limits.trending_n}",
"", "",
*_format_skill_section(trending), *format_skill_section(ctx.trending),
"---", "---",
"", "",
f"## 二、Skills Hot Top {hot_n}", f"## 二、Skills Hot Top {limits.hot_n}",
"", "",
*_format_skill_section(hot, hot=True), *format_skill_section(ctx.hot, hot=True),
"", "",
"---", "---",
"", "",
f"## 三、GitHub Trending Top {github_limit}", f"## 三、GitHub Trending Top {limits.github_limit}",
"", "",
trending_data_source_note(), trending_data_source_note(),
"", "",
] ]
if github_trending: if ctx.github_trending:
lines.extend(_format_github_repo_section(github_trending)) lines.extend(format_github_repo_section(ctx.github_trending))
else: else:
lines.append("*GitHub Trending 获取失败,请检查网络或配置 GITHUB_TOKEN。*") lines.append("*GitHub Trending 获取失败,请检查网络或配置 GITHUB_TOKEN。*")
lines.append("") lines.append("")
lines.extend(["---", "", f"## 四、新兴项目 Top {emerging_limit}", "", "> 数据来源GitHub Search API需 `GITHUB_TOKEN`", ""]) lines.extend(
if github_emerging: ["---", "", f"## 四、新兴项目 Top {limits.emerging_limit}", "", "> 数据来源GitHub Search API需 `GITHUB_TOKEN`", ""]
lines.extend(_format_github_repo_section(github_emerging, show_created=True)) )
if ctx.github_emerging:
lines.extend(format_github_repo_section(ctx.github_emerging, show_created=True))
else: else:
lines.append("*新兴项目获取失败或未配置 GITHUB_TOKEN。*") lines.append("*新兴项目获取失败或未配置 GITHUB_TOKEN。*")
lines.append("") lines.append("")
lines.extend(["---", "", f"## 五、Topic `{topic_name}` Top {topic_limit}", "", "> 数据来源GitHub Search API需 `GITHUB_TOKEN`", ""]) lines.extend(
if github_topic: ["---", "", f"## 五、Topic `{ctx.topic_name}` Top {limits.topic_limit}", "", "> 数据来源GitHub Search API需 `GITHUB_TOKEN`", ""]
lines.extend(_format_github_repo_section(github_topic)) )
if ctx.github_topic:
lines.extend(format_github_repo_section(ctx.github_topic))
else: else:
lines.append(f"*Topic `{topic_name}` 热点获取失败或未配置 GITHUB_TOKEN。*") lines.append(f"*Topic `{ctx.topic_name}` 热点获取失败或未配置 GITHUB_TOKEN。*")
lines.append("") lines.append("")
section_no = 6 section_no = 6
lines.extend(format_news_section(ai_news, section_no=section_no)) lines.extend(format_news_section(ctx.ai_news, section_no=section_no))
section_no += 1 section_no += 1
lines.extend(format_cn_news_section(cn_ai_news, section_no=section_no)) lines.extend(format_cn_news_section(ctx.cn_ai_news, section_no=section_no))
section_no += 1 section_no += 1
watch = (env("GITHUB_REPOS") or "").strip() watch = (env("GITHUB_REPOS") or "").strip()
@@ -573,7 +165,7 @@ def generate_report() -> tuple[str, str, Path, Path]:
lines.extend(["---", "", f"## {section_no}、关注仓库 Release", ""]) lines.extend(["---", "", f"## {section_no}、关注仓库 Release", ""])
section_no += 1 section_no += 1
for repo in [r.strip() for r in watch.split(",") if r.strip()]: for repo in [r.strip() for r in watch.split(",") if r.strip()]:
release = _fetch_latest_release_title(repo) release = fetch_latest_release_title(repo)
lines.append(f"- **{repo}**{release or '暂无 release'}") lines.append(f"- **{repo}**{release or '暂无 release'}")
lines.append("") lines.append("")
@@ -584,8 +176,8 @@ def generate_report() -> tuple[str, str, Path, Path]:
lines.append(f"- {ex}") lines.append(f"- {ex}")
lines.append("") lines.append("")
pick_src = trending[0].get("source", "") if trending else "" pick_src = ctx.trending[0].get("source", "") if ctx.trending else ""
pick_name = trending[0].get("title", "") if trending else "" pick_name = ctx.trending[0].get("title", "") if ctx.trending else ""
pick_command = ( pick_command = (
f"npx skills add {pick_src}/{pick_name}" f"npx skills add {pick_src}/{pick_name}"
if pick_src and pick_name if pick_src and pick_name
@@ -593,56 +185,69 @@ def generate_report() -> tuple[str, str, Path, Path]:
) )
lines.extend(["---", "", "## 安装示例", "", "```bash"]) lines.extend(["---", "", "## 安装示例", "", "```bash"])
for item in trending[:4]: for item in ctx.trending[:4]:
src, name = item.get("source", ""), item.get("title", "") src, name = item.get("source", ""), item.get("title", "")
if src and name: if src and name:
lines.append(f"npx skills add {src}/{name}") lines.append(f"npx skills add {src}/{name}")
lines.extend(["```", "", f"*企微短版见 `output/{date_str}.wecom.md`*"]) lines.extend(["```", "", f"*企微短版见 `output/{ctx.date_str}.wecom.md`*"])
markdown = "\n".join(lines) markdown = "\n".join(lines)
if agent_wecom: if agent_wecom:
gt = group_skills_by_source(trending, limit=wecom_trending, pool_size=skill_pool) gt = group_skills_by_source(
gh = group_skills_by_source(hot, limit=wecom_hot, pool_size=skill_pool) ctx.trending, limit=limits.wecom_trending, pool_size=limits.skill_pool
)
gh = group_skills_by_source(ctx.hot, limit=limits.wecom_hot, pool_size=limits.skill_pool)
wecom_md = replace_wecom_skill_sections(agent_wecom, trending=gt, hot=gh) wecom_md = replace_wecom_skill_sections(agent_wecom, trending=gt, hot=gh)
else: else:
wecom_md = build_wecom_report( wecom_md = build_wecom_report(
date_str=date_str, date_str=ctx.date_str,
time_str=time_str, time_str=ctx.time_str,
updated=updated, updated=ctx.updated,
highlights=editorial_highlights highlights=editorial_highlights
or _build_highlights(trending, hot, github_trending, github_emerging, ai_news, cn_ai_news), or build_highlights(
theme_line=editorial_theme or _detect_theme_line(feed), ctx.trending,
ai_news=prepare_wecom_news_items(ai_news), ctx.hot,
cn_ai_news=prepare_wecom_cn_news_items(cn_ai_news), ctx.github_trending,
ctx.github_emerging,
ctx.ai_news,
ctx.cn_ai_news,
),
theme_line=editorial_theme or detect_theme_line(ctx.feed),
ai_news=prepare_wecom_news_items(ctx.ai_news),
cn_ai_news=prepare_wecom_cn_news_items(ctx.cn_ai_news),
trending=[ trending=[
_prepare_skill_item(item, prev_ids, r) prepare_skill_item(item, prev_ids, r)
for r, item in enumerate( for r, item in enumerate(
finalize_wecom_skill_groups( finalize_wecom_skill_groups(
group_skills_by_source(trending, limit=wecom_trending, pool_size=skill_pool) group_skills_by_source(
ctx.trending, limit=limits.wecom_trending, pool_size=limits.skill_pool
)
), ),
1, 1,
) )
], ],
hot=[ hot=[
_prepare_skill_item(item, prev_ids, r) prepare_skill_item(item, prev_ids, r)
for r, item in enumerate( for r, item in enumerate(
finalize_wecom_skill_groups( finalize_wecom_skill_groups(
group_skills_by_source(hot, limit=wecom_hot, pool_size=skill_pool) group_skills_by_source(
ctx.hot, limit=limits.wecom_hot, pool_size=limits.skill_pool
)
), ),
1, 1,
) )
], ],
repos=[_prepare_github_item(item) for item in github_trending[:wecom_github]], repos=[prepare_github_item(item) for item in ctx.github_trending[: limits.wecom_github]],
emerging=[_prepare_github_item(item) for item in github_emerging[:wecom_emerging]], emerging=[prepare_github_item(item) for item in ctx.github_emerging[: limits.wecom_emerging]],
topic_name=topic_name, topic_name=ctx.topic_name,
topic_repos=[_prepare_github_item(item) for item in github_topic[:wecom_topic]], topic_repos=[prepare_github_item(item) for item in ctx.github_topic[: limits.wecom_topic]],
pick_command=pick_command, pick_command=pick_command,
) )
_save_snapshot(feed, date_str) save_snapshot(ctx.feed, ctx.date_str)
OUTPUT_DIR.mkdir(parents=True, exist_ok=True) OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
out_md = OUTPUT_DIR / f"{date_str}.md" out_md = OUTPUT_DIR / f"{ctx.date_str}.md"
out_wecom = OUTPUT_DIR / f"{date_str}.wecom.md" out_wecom = OUTPUT_DIR / f"{ctx.date_str}.wecom.md"
out_md.write_text(markdown, encoding="utf-8") out_md.write_text(markdown, encoding="utf-8")
out_wecom.write_text(wecom_md, encoding="utf-8") out_wecom.write_text(wecom_md, encoding="utf-8")
return markdown, wecom_md, out_md, out_wecom return markdown, wecom_md, out_md, out_wecom

View File

@@ -74,11 +74,14 @@ def _cursor_chat(system: str, user: str) -> str:
api_key = (env("CURSOR_API_KEY") or "").strip() api_key = (env("CURSOR_API_KEY") or "").strip()
if not api_key: if not api_key:
return "" return ""
import sys
from daily.config import ROOT
from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions from cursor_sdk import Agent, AgentOptions, CursorAgentError, LocalAgentOptions
from daily.config import ensure_bot_on_path _bot = str(ROOT / "bot")
if _bot not in sys.path:
ensure_bot_on_path() sys.path.insert(0, _bot)
try: try:
from bridge_manager import warm_cursor_bridge from bridge_manager import warm_cursor_bridge
except ImportError: except ImportError:

View File

@@ -22,7 +22,7 @@ from daily.news.feeds_cn import CN_AI_TITLE_KEYWORDS, CN_NEWS_CATEGORIES
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)" USER_AGENT = "Mozilla/5.0 (compatible; daily-robots/1.0; +https://skills.sh)"
BROWSER_USER_AGENT = ( BROWSER_USER_AGENT = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) " "AppleWebKit/537.36 (KHTML, like Gecko) "
@@ -286,7 +286,7 @@ def _fetch_one(
client: httpx.Client, client: httpx.Client,
category: NewsCategory, category: NewsCategory,
feed: NewsFeed, feed: NewsFeed,
) -> list[dict[str, Any]]: ) -> tuple[list[dict[str, Any]], bool]:
last_exc: Exception | None = None last_exc: Exception | None = None
for url in _reddit_fetch_urls(feed.url): for url in _reddit_fetch_urls(feed.url):
try: try:
@@ -294,12 +294,12 @@ def _fetch_one(
resp = client.get(url, headers=headers) resp = client.get(url, headers=headers)
resp.raise_for_status() resp.raise_for_status()
entries = _parse_feed(resp.text, feed.name, category) entries = _parse_feed(resp.text, feed.name, category)
return _filter_ai_entries(entries, ai_filter=feed.ai_filter) return _filter_ai_entries(entries, ai_filter=feed.ai_filter), True
except Exception as exc: except Exception as exc:
last_exc = exc last_exc = exc
continue continue
logger.warning("RSS fetch failed [%s] %s: %s", feed.name, feed.url, last_exc) logger.warning("RSS fetch failed [%s] %s: %s", feed.name, feed.url, last_exc)
return [] return [], False
def _dedupe_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]: def _dedupe_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
@@ -344,7 +344,12 @@ def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
tasks.append((category, feed)) tasks.append((category, feed))
raw_by_category: dict[str, list[dict[str, Any]]] = {c.id: [] for c in categories} raw_by_category: dict[str, list[dict[str, Any]]] = {c.id: [] for c in categories}
stats = {"feeds_total": len(tasks), "feeds_ok": 0, "items_raw": 0} stats: dict[str, Any] = {
"feeds_total": len(tasks),
"feeds_ok": 0,
"items_raw": 0,
"feeds_failed": [],
}
with httpx.Client(timeout=15.0, verify=certifi.where(), follow_redirects=True, headers=headers) as client: with httpx.Client(timeout=15.0, verify=certifi.where(), follow_redirects=True, headers=headers) as client:
fast_tasks = [t for t in tasks if not t[1].slow] fast_tasks = [t for t in tasks if not t[1].slow]
@@ -358,19 +363,24 @@ def _fetch_news(categories: tuple[NewsCategory, ...]) -> dict[str, Any]:
for future in as_completed(futures): for future in as_completed(futures):
cat_id, feed_name = futures[future] cat_id, feed_name = futures[future]
try: try:
entries = future.result() entries, ok = future.result()
except Exception as exc: except Exception as exc:
logger.warning("RSS 任务异常 [%s]: %s", feed_name, exc) logger.warning("RSS 任务异常 [%s]: %s", feed_name, exc)
stats["feeds_failed"].append(feed_name)
continue continue
if entries: if ok:
stats["feeds_ok"] += 1 stats["feeds_ok"] += 1
else:
stats["feeds_failed"].append(feed_name)
stats["items_raw"] += len(entries) stats["items_raw"] += len(entries)
raw_by_category[cat_id].extend(entries[:per_feed]) raw_by_category[cat_id].extend(entries[:per_feed])
for cat, feed in slow_tasks: for cat, feed in slow_tasks:
entries = _fetch_one(client, cat, feed) entries, ok = _fetch_one(client, cat, feed)
if entries: if ok:
stats["feeds_ok"] += 1 stats["feeds_ok"] += 1
else:
stats["feeds_failed"].append(feed.name)
stats["items_raw"] += len(entries) stats["items_raw"] += len(entries)
raw_by_category[cat.id].extend(entries[:per_feed]) raw_by_category[cat.id].extend(entries[:per_feed])
if _is_reddit_url(feed.url): if _is_reddit_url(feed.url):
@@ -467,6 +477,12 @@ def _format_news_section(
f"> 近 **{hours}h** · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用", f"> 近 **{hours}h** · {news.get('stats', {}).get('feeds_ok', 0)}/{news.get('stats', {}).get('feeds_total', 0)} 源可用",
"", "",
] ]
failed = news.get("stats", {}).get("feeds_failed") or []
if failed:
preview = "".join(failed[:5])
suffix = "" if len(failed) > 5 else ""
lines.append(f"> ⚠️ {len(failed)} 个源抓取失败:{preview}{suffix}")
lines.append("")
if not categories: if not categories:
lines.append("*暂无可用条目(网络/RSS 源异常或时间窗口内无更新)。*") lines.append("*暂无可用条目(网络/RSS 源异常或时间窗口内无更新)。*")
@@ -497,12 +513,20 @@ def _format_news_section(
return lines return lines
def prepare_wecom_news_items(news: dict[str, Any]) -> list[dict[str, Any]]: def _sorted_flat(news: dict[str, Any]) -> list[dict[str, Any]]:
flat = list(news.get("flat") or [])
flat.sort(
key=lambda item: (float(item.get("score") or 0), _sort_key(item)[1]),
reverse=True,
)
return flat
def prepare_wecom_news_items(news: dict[str, Any], *, limit: int | None = None) -> list[dict[str, Any]]:
if not news.get("enabled"): if not news.get("enabled"):
return [] return []
limit = _wecom_limit() pick_limit = limit or _wecom_limit()
flat = _dedupe_items(news.get("flat") or []) flat = _sorted_flat(news)
flat.sort(key=_sort_key, reverse=True)
preferred = ("media", "newsletter", "official", "community", "research", "developer") preferred = ("media", "newsletter", "official", "community", "research", "developer")
picked: list[dict[str, Any]] = [] picked: list[dict[str, Any]] = []
seen: set[str] = set() seen: set[str] = set()
@@ -513,12 +537,12 @@ def prepare_wecom_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
continue continue
picked.append(item) picked.append(item)
seen.add(link) seen.add(link)
if len(picked) >= limit: if len(picked) >= pick_limit:
break break
if len(picked) >= limit: if len(picked) >= pick_limit:
break break
items: list[dict[str, Any]] = [] items: list[dict[str, Any]] = []
for item in picked[:limit]: for item in picked[:pick_limit]:
items.append( items.append(
{ {
"title": item.get("title", "?"), "title": item.get("title", "?"),
@@ -526,17 +550,17 @@ def prepare_wecom_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
"source_name": item.get("source_name", "?"), "source_name": item.get("source_name", "?"),
"published_fmt": item.get("published_fmt", ""), "published_fmt": item.get("published_fmt", ""),
"desc_short": _clean_text(item.get("summary", ""), 36), "desc_short": _clean_text(item.get("summary", ""), 36),
"score": item.get("score"),
} }
) )
return items return items
def prepare_wecom_cn_news_items(news: dict[str, Any]) -> list[dict[str, Any]]: def prepare_wecom_cn_news_items(news: dict[str, Any], *, limit: int | None = None) -> list[dict[str, Any]]:
if not news.get("enabled"): if not news.get("enabled"):
return [] return []
limit = _wecom_cn_limit() pick_limit = limit or _wecom_cn_limit()
flat = _dedupe_items(news.get("flat") or []) flat = _sorted_flat(news)
flat.sort(key=_sort_key, reverse=True)
preferred = ("media", "tech", "dev") preferred = ("media", "tech", "dev")
picked: list[dict[str, Any]] = [] picked: list[dict[str, Any]] = []
seen_links: set[str] = set() seen_links: set[str] = set()
@@ -551,23 +575,23 @@ def prepare_wecom_cn_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
picked.append(item) picked.append(item)
seen_links.add(link) seen_links.add(link)
seen_sources.add(source) seen_sources.add(source)
if len(picked) >= limit: if len(picked) >= pick_limit:
break break
if len(picked) >= limit: if len(picked) >= pick_limit:
break break
if len(picked) < limit: if len(picked) < pick_limit:
for item in flat: for item in flat:
link = _normalize_link(item.get("link", "")) link = _normalize_link(item.get("link", ""))
if not link or link in seen_links: if not link or link in seen_links:
continue continue
picked.append(item) picked.append(item)
seen_links.add(link) seen_links.add(link)
if len(picked) >= limit: if len(picked) >= pick_limit:
break break
items: list[dict[str, Any]] = [] items: list[dict[str, Any]] = []
for item in picked[:limit]: for item in picked[:pick_limit]:
items.append( items.append(
{ {
"title": item.get("title", "?"), "title": item.get("title", "?"),
@@ -575,6 +599,7 @@ def prepare_wecom_cn_news_items(news: dict[str, Any]) -> list[dict[str, Any]]:
"source_name": item.get("source_name", "?"), "source_name": item.get("source_name", "?"),
"published_fmt": item.get("published_fmt", ""), "published_fmt": item.get("published_fmt", ""),
"desc_short": _clean_text(item.get("summary", ""), 36), "desc_short": _clean_text(item.get("summary", ""), 36),
"score": item.get("score"),
} }
) )
return items return items

171
daily/news/rank.py Normal file
View File

@@ -0,0 +1,171 @@
"""RSS 新闻去重、打分与排序。"""
from __future__ import annotations
import logging
import re
from datetime import datetime, timezone
from typing import Any
from daily.config import env_int
from daily.delta import find_previous_data
from daily.news.fetch import _entry_datetime, _normalize_link, _normalize_title
logger = logging.getLogger(__name__)
_WS = re.compile(r"\s+")
_TOKEN = re.compile(r"[\w]{2,}", re.UNICODE)
_CJK = re.compile(r"[\u4e00-\u9fff]")
CATEGORY_TIER: dict[str, int] = {
"official": 20,
"developer": 16,
"research": 14,
"media": 12,
"newsletter": 10,
"community": 8,
"tech": 12,
"dev": 10,
}
def title_similarity_threshold() -> float:
raw = env_int("DAILY_NEWS_TITLE_SIM", 55)
return max(0, min(raw, 95)) / 100.0
def _title_tokens(title: str) -> set[str]:
normalized = _normalize_title(title)
tokens = set(_TOKEN.findall(normalized))
cjk = "".join(_CJK.findall(title))
for i in range(max(0, len(cjk) - 1)):
tokens.add(cjk[i : i + 2])
if not tokens and normalized:
tokens.add(normalized)
return tokens
def jaccard_similarity(left: set[str], right: set[str]) -> float:
if not left or not right:
return 0.0
inter = len(left & right)
union = len(left | right)
return inter / union if union else 0.0
def fuzzy_dedupe_by_title(
items: list[dict[str, Any]],
*,
threshold: float | None = None,
) -> list[dict[str, Any]]:
"""按标题相似度合并重复报道,保留 score 更高(或更靠前)的条目。"""
if not items:
return []
limit = threshold if threshold is not None else title_similarity_threshold()
kept: list[dict[str, Any]] = []
kept_tokens: list[set[str]] = []
for item in items:
tokens = _title_tokens(str(item.get("title") or ""))
duplicate_idx: int | None = None
for idx, existing_tokens in enumerate(kept_tokens):
if jaccard_similarity(tokens, existing_tokens) >= limit:
duplicate_idx = idx
break
if duplicate_idx is None:
kept.append(item)
kept_tokens.append(tokens)
continue
existing = kept[duplicate_idx]
if float(item.get("score") or 0) > float(existing.get("score") or 0):
kept[duplicate_idx] = item
kept_tokens[duplicate_idx] = tokens
return kept
def _freshness_points(item: dict[str, Any], *, now: datetime | None = None) -> int:
now = now or datetime.now(timezone.utc)
dt = _entry_datetime(item)
if dt is None:
return 4
age_hours = max(0.0, (now - dt).total_seconds() / 3600.0)
if age_hours <= 6:
return 30
if age_hours <= 24:
return 22
if age_hours <= 72:
return 8
return 0
def _source_tier(item: dict[str, Any]) -> int:
category_id = str(item.get("category_id") or "")
return CATEGORY_TIER.get(category_id, 8)
def _novelty_points(item: dict[str, Any], yesterday_links: set[str]) -> int:
link = _normalize_link(str(item.get("link") or ""))
if link and link in yesterday_links:
return -20
return 5
def score_news_item(
item: dict[str, Any],
*,
yesterday_links: set[str] | None = None,
now: datetime | None = None,
) -> int:
links = yesterday_links or set()
total = _source_tier(item) + _freshness_points(item, now=now) + _novelty_points(item, links)
return max(0, total)
def load_yesterday_news_links(date_str: str) -> set[str]:
baseline = find_previous_data(date_str)
if not baseline:
return set()
_, data = baseline
links: set[str] = set()
for key in ("ai_news", "cn_ai_news"):
for item in data.get(key) or []:
if not isinstance(item, dict):
continue
link = _normalize_link(str(item.get("link") or ""))
if link:
links.add(link)
return links
def apply_news_ranking(news: dict[str, Any], *, date_str: str) -> dict[str, Any]:
"""对 categories / flat 打分、模糊去重并写回 score 字段。"""
if not news.get("enabled"):
return news
yesterday_links = load_yesterday_news_links(date_str)
threshold = title_similarity_threshold()
now = datetime.now(timezone.utc)
categories = news.get("categories") or []
for category in categories:
items = list(category.get("items") or [])
for item in items:
item["score"] = score_news_item(item, yesterday_links=yesterday_links, now=now)
items.sort(key=lambda x: float(x.get("score") or 0), reverse=True)
category["items"] = fuzzy_dedupe_by_title(items, threshold=threshold)
flat: list[dict[str, Any]] = []
for category in categories:
flat.extend(category.get("items") or [])
flat.sort(key=lambda x: float(x.get("score") or 0), reverse=True)
news["flat"] = fuzzy_dedupe_by_title(flat, threshold=threshold)
stats = dict(news.get("stats") or {})
stats["ranked_items"] = len(news["flat"])
stats["yesterday_links"] = len(yesterday_links)
news["stats"] = stats
logger.info(
"新闻排序完成:%d 条 flat昨日链接基准 %d",
len(news["flat"]),
len(yesterday_links),
)
return news

View File

@@ -0,0 +1 @@
"""早报生成流水线子模块。"""

145
daily/pipeline/collect.py Normal file
View File

@@ -0,0 +1,145 @@
"""抓取与结构化输入组装。"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime
from typing import Any
from daily.agent_workflow import is_agent_mode
from daily.config import env_int
from daily.delta import compare_depth
from daily.github.search import fetch_emerging_repos, fetch_topic_hot_repos
from daily.github.trending import fetch_github_trending
from daily.news.fetch import fetch_ai_news, fetch_cn_ai_news
from daily.news.rank import apply_news_ranking
from daily.report_data import build_llm_input
from daily.skills_board import load_boards
from shared.skills_data import load_feed
@dataclass
class ReportLimits:
trending_n: int
hot_n: int
skill_pool: int
wecom_trending: int
wecom_hot: int
github_limit: int
wecom_github: int
emerging_limit: int
wecom_emerging: int
topic_limit: int
wecom_topic: int
@dataclass
class ReportContext:
feed: dict[str, Any]
date_str: str
time_str: str
updated: str
trending: list[dict[str, Any]]
hot: list[dict[str, Any]]
github_trending: list[dict[str, Any]]
github_emerging: list[dict[str, Any]]
github_topic: list[dict[str, Any]]
topic_name: str
ai_news: dict[str, Any]
cn_ai_news: dict[str, Any]
limits: ReportLimits
wecom_limits: dict[str, int]
llm_input: dict[str, Any]
def resolve_limits() -> ReportLimits:
compare_n = compare_depth()
trending_n = env_int("DAILY_TRENDING_LIMIT", 150)
hot_n = max(env_int("DAILY_HOT_LIMIT", 150), compare_n)
skill_pool = max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
wecom_trending = env_int("DAILY_WECOM_TRENDING", 10)
wecom_hot = env_int("DAILY_WECOM_HOT", 10)
github_limit = env_int("DAILY_GITHUB_TRENDING_LIMIT", 10)
wecom_github = env_int("DAILY_WECOM_GITHUB_TRENDING", env_int("DAILY_WECOM_REPOS", 10))
emerging_limit = env_int("DAILY_GITHUB_EMERGING_LIMIT", 10)
wecom_emerging = env_int("DAILY_WECOM_GITHUB_EMERGING", 10)
topic_limit = env_int("DAILY_GITHUB_TOPIC_LIMIT", 10)
wecom_topic = env_int("DAILY_WECOM_GITHUB_TOPIC", 10)
return ReportLimits(
trending_n=trending_n,
hot_n=hot_n,
skill_pool=skill_pool,
wecom_trending=wecom_trending,
wecom_hot=wecom_hot,
github_limit=github_limit,
wecom_github=wecom_github,
emerging_limit=emerging_limit,
wecom_emerging=wecom_emerging,
topic_limit=topic_limit,
wecom_topic=wecom_topic,
)
def collect_report_context(now: datetime) -> ReportContext:
limits = resolve_limits()
compare_n = compare_depth()
github_fetch_n = max(limits.github_limit, compare_n, limits.wecom_github)
emerging_fetch_n = max(limits.emerging_limit, compare_n, limits.wecom_emerging)
topic_fetch_n = max(limits.topic_limit, compare_n, limits.wecom_topic)
feed = load_feed(force=True)
date_str = now.strftime("%Y-%m-%d")
time_str = now.strftime("%H:%M") + " (UTC+8)"
updated = (feed.get("updatedAt") or "")[:10]
trending, hot = load_boards(feed, trending_limit=limits.trending_n, hot_limit=limits.hot_n)
github_trending = fetch_github_trending(github_fetch_n)
seen_repos = {r["repo"] for r in github_trending}
github_emerging = fetch_emerging_repos(emerging_fetch_n, exclude=seen_repos)
seen_repos.update(r["repo"] for r in github_emerging)
topic_name, github_topic = fetch_topic_hot_repos(topic_fetch_n, exclude=seen_repos)
ai_news = apply_news_ranking(fetch_ai_news(), date_str=date_str)
cn_ai_news = apply_news_ranking(fetch_cn_ai_news(), date_str=date_str)
wecom_limits = {
"trending": limits.wecom_trending,
"hot": limits.wecom_hot,
"trending_pool": limits.skill_pool,
"hot_pool": limits.skill_pool,
"github": limits.wecom_github,
"emerging": limits.wecom_emerging,
"topic": limits.wecom_topic,
"ai_news": env_int("DAILY_WECOM_AI_NEWS", 10),
"cn_ai_news": env_int("DAILY_WECOM_CN_AI_NEWS", 8),
}
llm_input = build_llm_input(
date_str=date_str,
updated=updated,
trending=trending,
hot=hot,
github_trending=github_trending,
github_emerging=github_emerging,
github_topic=github_topic,
topic_name=topic_name,
ai_news=ai_news,
cn_ai_news=cn_ai_news,
wecom_limits=wecom_limits,
agent_mode=is_agent_mode(),
)
return ReportContext(
feed=feed,
date_str=date_str,
time_str=time_str,
updated=updated,
trending=trending,
hot=hot,
github_trending=github_trending,
github_emerging=github_emerging,
github_topic=github_topic,
topic_name=topic_name,
ai_news=ai_news,
cn_ai_news=cn_ai_news,
limits=limits,
wecom_limits=wecom_limits,
llm_input=llm_input,
)

View File

@@ -0,0 +1,180 @@
"""归档 / 企微格式化与摘要构建。"""
from __future__ import annotations
import re
import xml.etree.ElementTree as ET
from typing import Any
import certifi
import httpx
from daily.config import full_desc_limit, wecom_skill_desc_limit
from daily.github.auth import github_html_headers
from daily.news.fetch import prepare_wecom_cn_news_items, prepare_wecom_news_items
from shared.skills_data import format_installs
from daily.pipeline.snapshot import skill_id
def short_desc(text: str, limit: int = 72) -> str:
text = re.sub(r"\s+", " ", text or "").strip()
if limit <= 0 or len(text) <= limit:
return text
return text[: limit - 3] + "..."
def archive_desc(text: str) -> str:
return short_desc(text, full_desc_limit())
def wecom_desc(text: str, limit: int = 36) -> str:
return short_desc(text, limit)
def prepare_skill_item(item: dict[str, Any], prev_ids: set[str], rank: int) -> dict[str, Any]:
badge = ""
sid = skill_id(item)
if sid not in prev_ids and prev_ids:
badge = "🆕"
elif rank == 1:
badge = "👑"
installs_fmt = item.get("installs_fmt") or format_installs(item.get("installs", 0))
title = item.get("source", "?") if item.get("cluster") else item.get("title", "?")
desc = item.get("wecom_desc") or item.get("description") or item.get("cluster_titles") or ""
limit = wecom_skill_desc_limit()
desc_short = desc if item.get("wecom_desc") or limit <= 0 else wecom_desc(desc, limit)
return {
"title": title,
"source": item.get("source", "?"),
"installs_fmt": installs_fmt,
"link": item.get("link", ""),
"desc_short": desc_short,
"badge": badge,
"cluster": bool(item.get("cluster")),
"cluster_count": item.get("cluster_count"),
"cluster_titles": item.get("cluster_titles"),
}
def prepare_github_item(item: dict[str, Any]) -> dict[str, Any]:
return {**item, "desc_short": wecom_desc(item.get("description", ""), 40)}
def build_highlights(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
github_trending: list[dict[str, Any]],
github_emerging: list[dict[str, Any]],
ai_news: dict[str, Any] | None = None,
cn_ai_news: dict[str, Any] | None = None,
) -> list[str]:
points: list[str] = []
if ai_news and ai_news.get("enabled"):
top_news = prepare_wecom_news_items(ai_news)
if top_news:
n0 = top_news[0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🌍 AI 时讯 [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
elif ai_news.get("flat"):
n0 = ai_news["flat"][0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(f"🌍 AI 时讯 [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}")
if cn_ai_news and cn_ai_news.get("enabled"):
top_cn = prepare_wecom_cn_news_items(cn_ai_news)
if top_cn:
n0 = top_cn[0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🇨🇳 国内 AI [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
elif cn_ai_news.get("flat"):
n0 = cn_ai_news["flat"][0]
pub = f" · {n0['published_fmt']}" if n0.get("published_fmt") else ""
points.append(
f"🇨🇳 国内 AI [{n0['title']}]({n0['link']})`{n0.get('source_name', '?')}`{pub}"
)
if trending:
t0 = trending[0]
points.append(f"📈 Skills 榜首 **{t0.get('title')}**{format_installs(t0.get('installs', 0))}")
if github_trending:
g0 = github_trending[0]
stars = g0.get("stars_today_fmt", "")
total = g0.get("total_stars_fmt", "")
star_hint = f"+{stars} today · " if stars else (f"{total} · " if total else "")
points.append(f"🐙 GitHub Trending [{g0['repo']}]({g0['url']}){star_hint}{g0.get('language', '')}")
if github_emerging:
e0 = github_emerging[0]
points.append(f"🌱 新兴 [{e0['repo']}]({e0['url']})(⭐ {e0.get('total_stars_fmt', '?')}")
elif hot:
h0 = hot[0]
points.append(f"🔥 Skills Hot 榜首 **{h0.get('title')}**1H {format_installs(h0.get('installs', 0))}")
while len(points) < 3 and len(trending) > len(points):
item = trending[len(points)]
points.append(f"✨ **{item.get('title')}** · `{item.get('source')}`")
return points[:3]
def format_github_repo_section(repos: list[dict[str, Any]], *, show_created: bool = False) -> list[str]:
lines: list[str] = []
for i, repo in enumerate(repos, 1):
lang = repo.get("language") or ""
stars_today = repo.get("stars_today_fmt") or ""
total = repo.get("total_stars_fmt") or ""
created = repo.get("created_at") or ""
meta_parts = [lang]
if stars_today:
meta_parts.append(f"+{stars_today} today")
if total:
meta_parts.append(f"总 ⭐ {total}")
if show_created and created:
meta_parts.append(f"创建于 {created}")
lines.append(f"{i}. **[{repo['repo']}]({repo['url']})** · {' · '.join(meta_parts)}")
desc = archive_desc(repo.get("description", ""))
if desc:
lines.append(f" - {desc}")
lines.append("")
return lines
def format_skill_section(items: list[dict[str, Any]], *, hot: bool = False) -> list[str]:
lines: list[str] = []
for i, item in enumerate(items, 1):
sid = item.get("id") or f"{item.get('source', '?')}/{item.get('title', '?')}"
link = item.get("link", "")
installs = format_installs(item.get("installs", 0))
meta = f"1H {installs}" if hot else f"总安装 {installs}"
if link:
lines.append(f"{i}. **[{sid}]({link})** · {meta}")
else:
lines.append(f"{i}. **{sid}** · {meta}")
desc = archive_desc(item.get("description", ""))
if desc:
lines.append(f" - {desc}")
lines.append("")
return lines
def fetch_latest_release_title(repo: str) -> str | None:
atom_url = f"https://github.com/{repo}/releases.atom"
try:
with httpx.Client(
timeout=12.0,
verify=certifi.where(),
follow_redirects=True,
headers=github_html_headers(),
) as client:
resp = client.get(atom_url)
if resp.status_code != 200:
return None
root = ET.fromstring(resp.text)
ns = {"a": "http://www.w3.org/2005/Atom"}
entry = root.find("a:entry", ns)
if entry is None:
return None
title = entry.find("a:title", ns)
return title.text.strip() if title is not None and title.text else None
except Exception:
return None

116
daily/pipeline/localize.py Normal file
View File

@@ -0,0 +1,116 @@
"""归档内容中文化。"""
from __future__ import annotations
from typing import Any
from daily.config import full_desc_limit, news_summary_limit
from daily.localize import LocalizeJob, localize_descriptions, needs_chinese
from daily.pipeline.snapshot import skill_id
def localize_descriptions_in_place(
trending: list[dict[str, Any]],
hot: list[dict[str, Any]],
github_trending: list[dict[str, Any]],
github_emerging: list[dict[str, Any]],
github_topic: list[dict[str, Any]],
ai_news: dict[str, Any],
cn_ai_news: dict[str, Any] | None = None,
) -> None:
full_limit = full_desc_limit()
news_limit = news_summary_limit()
jobs: list[LocalizeJob] = []
seen_skill: set[str] = set()
for item in trending + hot:
sid = skill_id(item)
if sid in seen_skill:
continue
seen_skill.add(sid)
desc = (item.get("description") or "").strip()
if desc:
jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
seen_repo: set[str] = set()
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
repo = item.get("repo", "")
if not repo or repo in seen_repo:
continue
seen_repo.add(repo)
desc = (item.get("description") or "").strip()
if desc:
jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
if ai_news.get("enabled"):
seen_news: set[str] = set()
for item in ai_news.get("flat") or []:
link = item.get("link", "")
if not link or link in seen_news:
continue
seen_news.add(link)
summary = (item.get("summary") or "").strip()
if summary:
jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
zh_map = localize_descriptions(jobs, archive=True)
if not zh_map and not jobs:
return
def _apply_zh(mapping: dict[str, str]) -> None:
for item in trending + hot:
key = f"skill:{skill_id(item)}"
if key in mapping:
item["description"] = mapping[key]
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
key = f"github:{item.get('repo', '')}"
if key in mapping:
item["description"] = mapping[key]
if ai_news.get("enabled"):
for cat in ai_news.get("categories") or []:
for item in cat.get("items") or []:
key = f"news:{item.get('link', '')}"
if key in mapping:
item["summary"] = mapping[key]
for item in ai_news.get("flat") or []:
key = f"news:{item.get('link', '')}"
if key in mapping:
item["summary"] = mapping[key]
_apply_zh(zh_map)
retry_jobs: list[LocalizeJob] = []
seen_skill.clear()
for item in trending + hot:
sid = skill_id(item)
if sid in seen_skill:
continue
seen_skill.add(sid)
desc = (item.get("description") or "").strip()
if needs_chinese(desc):
retry_jobs.append(LocalizeJob(f"skill:{sid}", desc, full_limit))
seen_repo.clear()
for repo_list in (github_trending, github_emerging, github_topic):
for item in repo_list:
repo = item.get("repo", "")
if not repo or repo in seen_repo:
continue
seen_repo.add(repo)
desc = (item.get("description") or "").strip()
if needs_chinese(desc):
retry_jobs.append(LocalizeJob(f"github:{repo}", desc, full_limit))
if ai_news.get("enabled"):
seen_news.clear()
for item in ai_news.get("flat") or []:
link = item.get("link", "")
if not link or link in seen_news:
continue
seen_news.add(link)
summary = (item.get("summary") or "").strip()
if needs_chinese(summary):
retry_jobs.append(LocalizeJob(f"news:{link}", summary, news_limit))
if retry_jobs:
_apply_zh(localize_descriptions(retry_jobs, archive=True))

View File

@@ -0,0 +1,36 @@
"""Skills 快照读写。"""
from __future__ import annotations
import json
from typing import Any
from daily.config import CACHE_DIR, SNAPSHOT_FILE
def skill_id(item: dict[str, Any]) -> str:
return str(item.get("id") or f"{item.get('source')}/{item.get('title')}")
def load_snapshot() -> set[str]:
if not SNAPSHOT_FILE.exists():
return set()
try:
data = json.loads(SNAPSHOT_FILE.read_text(encoding="utf-8"))
return set(str(x) for x in (data.get("skill_ids") or []))
except (OSError, json.JSONDecodeError):
return set()
def save_snapshot(feed: dict[str, Any], date_str: str) -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
ids: list[str] = []
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
sid = skill_id(item)
if sid not in ids:
ids.append(sid)
SNAPSHOT_FILE.write_text(
json.dumps({"date": date_str, "skill_ids": ids}, ensure_ascii=False, indent=2),
encoding="utf-8",
)

53
daily/pipeline/themes.py Normal file
View File

@@ -0,0 +1,53 @@
"""主题检测与聚类。"""
from __future__ import annotations
from collections import defaultdict
from typing import Any
THEME_RULES: list[tuple[str, str, list[str]]] = [
("🎬", "AI 多媒体 / 视频", ["runcomfy", "remotion", "video", "seedance", "inpaint", "lipsync"]),
("🔧", "工程协作 / Skill 元能力", ["grill", "tdd", "architecture", "find-skills", "to-issues"]),
("📱", "飞书 / Lark", ["lark", "feishu"]),
("📣", "内容营销", ["viral", "tiktok", "instagram", "reels"]),
("🎨", "设计 / 前端", ["frontend", "design", "ui-ux", "tailwind"]),
]
def detect_theme_line(feed: dict[str, Any]) -> str:
scores: dict[str, int] = defaultdict(int)
for board in ("topTrending", "topHot"):
for rank, item in enumerate(feed.get(board, [])[:10], 1):
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, label, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
scores[label] += max(1, 11 - rank)
break
if not scores:
return "**今日主题**Agent Skills 生态持续活跃"
return f"**今日主题**{max(scores.items(), key=lambda x: x[1])[0]}"
def theme_clusters(feed: dict[str, Any], limit: int = 5) -> list[tuple[str, list[str]]]:
from daily.pipeline.snapshot import skill_id
buckets: dict[str, list[str]] = defaultdict(list)
seen: set[str] = set()
for board in ("topTrending", "topHot"):
for item in feed.get(board, [])[:20]:
item_id = skill_id(item)
if item_id in seen:
continue
seen.add(item_id)
haystack = " ".join(
[item.get("title", ""), item.get("source", ""), item.get("description", "")]
).lower()
for _icon, theme, keywords in THEME_RULES:
if any(k in haystack for k in keywords):
label = f"**{item.get('title')}** (`{item.get('source')}`)"
if label not in buckets[theme]:
buckets[theme].append(label)
break
return [(theme, examples[:limit]) for theme, examples in buckets.items() if examples]

View File

@@ -61,33 +61,47 @@ def _slim_news_items(
prepare=prepare_wecom_news_items, prepare=prepare_wecom_news_items,
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
items: list[dict[str, Any]] = [] items: list[dict[str, Any]] = []
for item in prepare(ai_news): for item in prepare(ai_news, limit=limit):
items.append( payload = {
{ "link": item.get("link", ""),
"link": item.get("link", ""), "title": item.get("title", ""),
"title": item.get("title", ""), "source_name": item.get("source_name", ""),
"source_name": item.get("source_name", ""), "published_fmt": item.get("published_fmt", ""),
"published_fmt": item.get("published_fmt", ""), "summary": item.get("desc_short") or "",
"summary": item.get("desc_short") or "", }
} if item.get("score") is not None:
) payload["score"] = item.get("score")
items.append(payload)
if len(items) >= limit: if len(items) >= limit:
break break
if items: if items:
return items return items
for item in (ai_news.get("flat") or [])[:limit]: flat = sorted(
items.append( ai_news.get("flat") or [],
{ key=lambda row: float(row.get("score") or 0),
"link": item.get("link", ""), reverse=True,
"title": item.get("title", ""), )
"source_name": item.get("source_name", ""), for item in flat[:limit]:
"published_fmt": item.get("published_fmt", ""), payload = {
"summary": item.get("summary", ""), "link": item.get("link", ""),
} "title": item.get("title", ""),
) "source_name": item.get("source_name", ""),
"published_fmt": item.get("published_fmt", ""),
"summary": item.get("summary", ""),
}
if item.get("score") is not None:
payload["score"] = item.get("score")
items.append(payload)
return items return items
def _news_pool_limit(wecom_limit: int, *, env_key: str, default_pool: int, agent_mode: bool) -> int:
if not agent_mode:
return wecom_limit
pool = env_int(env_key, default_pool)
return max(wecom_limit, pool)
def _wecom_skill_pool() -> int: def _wecom_skill_pool() -> int:
return max(10, env_int("DAILY_WECOM_SKILL_POOL", 200)) return max(10, env_int("DAILY_WECOM_SKILL_POOL", 200))
@@ -105,10 +119,21 @@ def build_llm_input(
ai_news: dict[str, Any], ai_news: dict[str, Any],
cn_ai_news: dict[str, Any], cn_ai_news: dict[str, Any],
wecom_limits: dict[str, int], wecom_limits: dict[str, int],
agent_mode: bool = False,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""供 Cursor 编辑的精简 JSON不含完整 markdown""" """供 Cursor 编辑的精简 JSON不含完整 markdown"""
news_limit = wecom_limits.get("ai_news", 10) news_limit = _news_pool_limit(
cn_news_limit = wecom_limits.get("cn_ai_news", 8) wecom_limits.get("ai_news", 10),
env_key="DAILY_AGENT_NEWS_POOL",
default_pool=40,
agent_mode=agent_mode,
)
cn_news_limit = _news_pool_limit(
wecom_limits.get("cn_ai_news", 8),
env_key="DAILY_AGENT_CN_NEWS_POOL",
default_pool=30,
agent_mode=agent_mode,
)
depth = compare_depth() depth = compare_depth()
trend_cmp = trending[:depth] trend_cmp = trending[:depth]
hot_cmp = hot[:depth] hot_cmp = hot[:depth]

View File

@@ -15,7 +15,7 @@ logger = logging.getLogger(__name__)
Board = Literal["trending", "hot"] Board = Literal["trending", "hot"]
SKILLS_SITE = "https://www.skills.sh" SKILLS_SITE = "https://www.skills.sh"
USER_AGENT = "Mozilla/5.0 (compatible; skills-hot-daily/1.0; +https://skills.sh)" USER_AGENT = "Mozilla/5.0 (compatible; daily-robots/1.0; +https://skills.sh)"
_SKILL_RE = re.compile( _SKILL_RE = re.compile(
r'\{"source":"(?P<source>[^"]+)","skillId":"(?P<skill_id>[^"]+)",' r'\{"source":"(?P<source>[^"]+)","skillId":"(?P<skill_id>[^"]+)",'

14
docker-compose.yml Normal file
View File

@@ -0,0 +1,14 @@
services:
daily:
build: .
env_file: .env
environment:
TZ: Asia/Shanghai
volumes:
- ./output:/app/output
- ./.cache:/app/.cache
- ./logs:/app/logs
# Examples:
# docker compose run --rm daily
# docker compose run --rm daily --skip-push
# docker compose run --rm daily --force

View File

@@ -0,0 +1,35 @@
<!-- 示例Agent 模式企微早报节选,日期 2026-07-03 -->
📰 **早报 · 2026-07-03**
> ⏱ 09:30 (UTC+8) · 数据截至 2026-07-02
> remotion-render 以 21549 次安装稳坐 Skills Trending 榜首halt-catch-fire 五件套把「TSX 直出 MP4」推到台前。GitHub 侧 openclaw 仍以 381.5K star 居 Trending 第一;新闻头条则是 OpenAI 拟向特朗普政府让渡 AI 繁荣 5% 分成——多媒体 skill 装机、本地 Agent 开源栈与 AI 治理议题同日撞屏。
🎯 **代码出片霸榜Agent基建共振**
💡 **今日信号**
> 🔥 Skills Hotxixu-me/skills 12 件套新占 Top10 前十,含 xdrop、openclaw-secure-linux-cloud 等开发者工具簇
> 📦 Skills较昨日 Trending Top15 新入 7 条,飞书 lark-approval、lark-wiki 等办公 skill 批量涌入
> 🐙 GitHubaffaan-m/ECC 225.2K star 新入 Trending #2面向 Claude Code/Codex/Cursor 的 Agent harness 优化成显学
> 🌱 GitHublangflow、dify 新入 Trending Top10可视化 Agent 工作流平台与 DeepSpec 推测解码基建各据一角
> 📰 时讯Anthropic 搁置已久的 Fable 5 获准回归,模型内容政策与 Zuckerberg「Agent 进展慢于预期」形成对照
📦 **今日首推**
`npx skills add halt-catch-fire/skills/remotion-render`
> Trending #1、21549 安装5 skill 集群覆盖 remotion-render 到 ai-video-generation把 React/Remotion 组件程序化渲染 MP4 做成默认可装工作流。
🌍 **国际 AI · 精选 10**
1. [OpenAI拟向特朗普政府让渡AI繁荣5%分成](https://www.theverge.com/ai-artificial-intelligence/960588/openai-government-5-percent-stake-trump) — The Verge 头条:头部模型公司探讨向政府让渡 AI 收益分成,商业化与治理边界被推上舆论前台
2. [Anthropic搁置已久的Fable 5获准回归](https://www.theverge.com/ai-artificial-intelligence/958964/anthropic-claude-fable-5-is-back) — 经数周协商后,此前被搁置的 Claude Fable 5 内容政策模型重新获准上线
3. [Zuckerberg称AI Agent进展慢于预期](https://techcrunch.com/2026/07/02/mark-zuckerberg-tells-staff-that-ai-agents-havent-progressed-as-quickly-as-hed-hoped/) — Meta 内部会议透露Agent 落地节奏未达此前预期
🇨🇳 **国内 AI · 精选 8**
1. [让Agent越用越强AReaL 2.0开源打造面向自演进智能体的RL基础设施](https://www.qbitai.com/2026/07/442134.html) — 量子位:面向自演进 Agent 的 RL 训练基建开源
2. [Agent Loop 深度调研:把决定权交给模型的一次换代,为什么发生在现在](https://juejin.cn/post/7657771483503018036) — 掘金长文梳理 Agent Loop 架构演进
📈 **Skills Trending Top 9**
1. [**halt-catch-fire/skills** · 5 skills · **21.5K21.5K**](https://www.skills.sh/halt-catch-fire/skills/remotion-render)
> 用 inference.sh 将 Remotion 组件代码直接渲染为 MP4可配分辨率帧率时长适合程序化视
2. [**find-skills**](https://www.skills.sh/vercel-labs/skills/find-skills) · `vercel-labs/skills` · **18.0K**
> 用户询问如何做某事或寻找能力扩展时,帮助发现并安装合适的 Agent Skill。
<!-- 完整版还包含 Hot 榜、GitHub Trending/新兴/Topic 等分区,见本地 output/ -->

3
pytest.ini Normal file
View File

@@ -0,0 +1,3 @@
[pytest]
testpaths = tests
pythonpath = .

4
requirements-dev.txt Normal file
View File

@@ -0,0 +1,4 @@
python-dotenv>=1.0.0
httpx>=0.27.0
certifi>=2024.0.0
pytest>=8.0

View File

@@ -3,16 +3,21 @@
# .\run-daily.ps1 # .\run-daily.ps1
# .\run-daily.ps1 -SkipPush # .\run-daily.ps1 -SkipPush
# .\run-daily.ps1 -SkipGenerate # .\run-daily.ps1 -SkipGenerate
# .\run-daily.ps1 -Force # bypass duplicate-run lock
param( param(
[switch]$SkipPush, [switch]$SkipPush,
[switch]$SkipGenerate [switch]$SkipGenerate,
[switch]$Force
) )
$ErrorActionPreference = "Stop" $ErrorActionPreference = "Stop"
$Root = Split-Path -Parent $MyInvocation.MyCommand.Path $Root = Split-Path -Parent $MyInvocation.MyCommand.Path
Set-Location $Root Set-Location $Root
$LockFile = Join-Path $Root ".cache\run-daily.lock"
$LockMaxMinutes = 30
function Import-DotEnvFile { function Import-DotEnvFile {
param([string]$Path) param([string]$Path)
if (-not (Test-Path $Path)) { return } if (-not (Test-Path $Path)) { return }
@@ -29,30 +34,61 @@ function Import-DotEnvFile {
} }
} }
Import-DotEnvFile (Join-Path $Root ".env") function Test-RunDailyLock {
Import-DotEnvFile (Join-Path $Root ".env.local") if (-not (Test-Path $LockFile)) { return $false }
$age = (Get-Date) - (Get-Item $LockFile).LastWriteTime
return $age.TotalMinutes -lt $LockMaxMinutes
}
$python = "python" function Set-RunDailyLock {
$date = Get-Date -Format "yyyy-MM-dd" $dir = Split-Path $LockFile -Parent
$reportWecom = Join-Path $Root "output\$date.wecom.md" if (-not (Test-Path $dir)) {
New-Item -ItemType Directory -Path $dir -Force | Out-Null
}
Set-Content -Path $LockFile -Value (Get-Date -Format "o") -Encoding UTF8
}
if (-not $SkipGenerate) { function Clear-RunDailyLock {
Write-Host "Generating daily report: $date" if (Test-Path $LockFile) {
& $python -m daily generate Remove-Item $LockFile -Force -ErrorAction SilentlyContinue
if ($LASTEXITCODE -ne 0) {
throw "daily generate failed with exit code $LASTEXITCODE"
} }
} }
if (-not $SkipPush) { if (-not $Force -and (Test-RunDailyLock)) {
if (-not (Test-Path $reportWecom)) { Write-Host "Skip: run-daily already ran within ${LockMaxMinutes} minutes (lock: $LockFile). Use -Force to override."
throw "Report not found: $reportWecom" exit 0
}
Write-Host "Pushing to WeCom webhook..."
& $python -m daily push $reportWecom
if ($LASTEXITCODE -ne 0) {
throw "daily push failed with exit code $LASTEXITCODE"
}
} }
Write-Host "Done: $date" Set-RunDailyLock
try {
Import-DotEnvFile (Join-Path $Root ".env")
Import-DotEnvFile (Join-Path $Root ".env.local")
$python = "python"
$date = Get-Date -Format "yyyy-MM-dd"
$reportWecom = Join-Path $Root "output\$date.wecom.md"
if (-not $SkipGenerate) {
Write-Host "Generating daily report: $date"
& $python -m daily generate
if ($LASTEXITCODE -ne 0) {
throw "daily generate failed with exit code $LASTEXITCODE"
}
}
if (-not $SkipPush) {
if (-not (Test-Path $reportWecom)) {
throw "Report not found: $reportWecom"
}
Write-Host "Pushing to WeCom webhook..."
& $python -m daily push $reportWecom
if ($LASTEXITCODE -ne 0) {
throw "daily push failed with exit code $LASTEXITCODE"
}
}
Write-Host "Done: $date"
}
finally {
Clear-RunDailyLock
}

106
run-daily.sh Normal file
View File

@@ -0,0 +1,106 @@
#!/usr/bin/env bash
# Daily report: generate + push to WeCom webhook
# Usage:
# ./run-daily.sh
# ./run-daily.sh --skip-push
# ./run-daily.sh --skip-generate
# ./run-daily.sh --force
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$ROOT"
LOCK_FILE="$ROOT/.cache/run-daily.lock"
LOCK_MAX_MINUTES=30
SKIP_PUSH=0
SKIP_GENERATE=0
FORCE=0
while [[ $# -gt 0 ]]; do
case "$1" in
--skip-push) SKIP_PUSH=1 ;;
--skip-generate) SKIP_GENERATE=1 ;;
--force) FORCE=1 ;;
-h|--help)
echo "Usage: $0 [--skip-push] [--skip-generate] [--force]"
exit 0
;;
*)
echo "Unknown option: $1" >&2
exit 1
;;
esac
shift
done
load_dotenv() {
local file="$1"
[[ -f "$file" ]] || return 0
while IFS= read -r line || [[ -n "$line" ]]; do
[[ "$line" =~ ^[[:space:]]*# ]] && continue
[[ "$line" != *"="* ]] && continue
local name="${line%%=*}"
local value="${line#*=}"
name="${name#"${name%%[![:space:]]*}"}"
name="${name%"${name##*[![:space:]]}"}"
value="${value#"${value%%[![:space:]]*}"}"
value="${value%"${value##*[![:space:]]}"}"
value="${value%\"}"
value="${value#\"}"
value="${value%\'}"
value="${value#\'}"
if [[ -n "$name" && -n "$value" ]]; then
export "$name=$value"
fi
done < "$file"
}
lock_active() {
[[ -f "$LOCK_FILE" ]] || return 1
local now lock_ts age_minutes
now="$(date +%s)"
lock_ts="$(date -r "$LOCK_FILE" +%s 2>/dev/null || stat -c %Y "$LOCK_FILE" 2>/dev/null || echo 0)"
age_minutes=$(( (now - lock_ts) / 60 ))
[[ "$age_minutes" -lt "$LOCK_MAX_MINUTES" ]]
}
set_lock() {
mkdir -p "$(dirname "$LOCK_FILE")"
date -Iseconds > "$LOCK_FILE"
}
clear_lock() {
rm -f "$LOCK_FILE"
}
if [[ "$FORCE" -eq 0 ]] && lock_active; then
echo "Skip: run-daily already ran within ${LOCK_MAX_MINUTES} minutes (lock: $LOCK_FILE). Use --force to override."
exit 0
fi
set_lock
trap clear_lock EXIT
load_dotenv "$ROOT/.env"
load_dotenv "$ROOT/.env.local"
PYTHON="${PYTHON:-python}"
DATE="$(TZ=Asia/Shanghai date +%Y-%m-%d)"
REPORT_WECOM="$ROOT/output/${DATE}.wecom.md"
if [[ "$SKIP_GENERATE" -eq 0 ]]; then
echo "Generating daily report: $DATE"
"$PYTHON" -m daily generate
fi
if [[ "$SKIP_PUSH" -eq 0 ]]; then
if [[ ! -f "$REPORT_WECOM" ]]; then
echo "Report not found: $REPORT_WECOM" >&2
exit 1
fi
echo "Pushing to WeCom webhook..."
"$PYTHON" -m daily push "$REPORT_WECOM"
fi
echo "Done: $DATE"

10
shared/__init__.py Normal file
View File

@@ -0,0 +1,10 @@
"""Repo-root shared modules (skills feed data, etc.)."""
from shared.skills_data import (
Board,
format_installs,
load_feed,
warm_feed_cache,
)
__all__ = ["Board", "format_installs", "load_feed", "warm_feed_cache"]

109
shared/skills_data.py Normal file
View File

@@ -0,0 +1,109 @@
"""skills.sh feed.json 拉取、缓存与榜单数据。"""
from __future__ import annotations
import json
import logging
import time
from pathlib import Path
from typing import Any, Literal
import certifi
import httpx
logger = logging.getLogger(__name__)
ROOT = Path(__file__).resolve().parent.parent
FEED_URLS = [
"https://cdn.jsdelivr.net/gh/NeverSight/skills.sh_feed@main/data/feed.json",
"https://raw.githubusercontent.com/NeverSight/skills.sh_feed/main/data/feed.json",
]
CACHE_TTL_SECONDS = 600
CACHE_DIR = ROOT / ".cache"
CACHE_FILE = CACHE_DIR / "skills-feed.json"
USER_AGENT = "Mozilla/5.0 (compatible; daily-robots/1.0; +https://skills.sh)"
_cache: dict[str, Any] = {"data": None, "fetched_at": 0.0}
Board = Literal["trending", "hot", "all"]
BOARD_FEED_KEYS = {
"trending": "topTrending",
"hot": "topHot",
"all": "topAllTime",
}
def _fetch_json(url: str) -> dict[str, Any]:
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
with httpx.Client(
timeout=httpx.Timeout(20.0, connect=10.0),
verify=certifi.where(),
follow_redirects=True,
) as client:
resp = client.get(url, headers=headers)
resp.raise_for_status()
return resp.json()
def _load_disk_cache() -> dict[str, Any] | None:
if not CACHE_FILE.exists():
return None
try:
return json.loads(CACHE_FILE.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
logger.warning("读取 skills feed 本地缓存失败: %s", exc)
return None
def _save_disk_cache(data: dict[str, Any]) -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
def load_feed(force: bool = False) -> dict[str, Any]:
now = time.time()
if not force and _cache["data"] and now - _cache["fetched_at"] < CACHE_TTL_SECONDS:
return _cache["data"]
errors: list[str] = []
for url in FEED_URLS:
for attempt in range(3):
try:
data = _fetch_json(url)
_cache["data"] = data
_cache["fetched_at"] = now
_save_disk_cache(data)
logger.info("skills 数据已更新: %s", url)
return data
except Exception as exc:
msg = f"{url} (#{attempt + 1}): {exc}"
errors.append(msg)
logger.debug("拉取失败 %s", msg)
time.sleep(0.5 * (attempt + 1))
stale = _load_disk_cache()
if stale:
logger.warning("网络不可用,回退到本地 skills feed 缓存")
_cache["data"] = stale
_cache["fetched_at"] = now
return stale
raise RuntimeError(f"无法获取 skills 数据。最近错误: {errors[-1]}")
def warm_feed_cache() -> None:
"""启动时预加载,避免首条消息才触发网络请求。"""
load_feed(force=True)
def format_installs(n: int | float) -> str:
if n >= 1_000_000:
return f"{n / 1_000_000:.1f}M"
if n >= 1_000:
return f"{n / 1_000:.1f}K"
return str(int(n))
def board_items(feed: dict[str, Any], board: Board) -> list[dict[str, Any]]:
return feed.get(BOARD_FEED_KEYS[board], [])

50
tests/conftest.py Normal file
View File

@@ -0,0 +1,50 @@
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
FIXTURES = Path(__file__).parent / "fixtures"
@pytest.fixture
def skills_feed() -> dict:
return json.loads((FIXTURES / "skills-feed.json").read_text(encoding="utf-8"))
@pytest.fixture
def fixed_cst():
return datetime(2026, 7, 3, 9, 30, tzinfo=timezone(timedelta(hours=8)))
@pytest.fixture
def isolated_output(tmp_path, monkeypatch):
import daily.config as config
import daily.report_data as report_data
out = tmp_path / "output"
cache = tmp_path / "cache"
logs = tmp_path / "logs"
out.mkdir()
cache.mkdir()
logs.mkdir()
monkeypatch.setattr(config, "OUTPUT_DIR", out)
monkeypatch.setattr(report_data, "OUTPUT_DIR", out)
monkeypatch.setattr(config, "CACHE_DIR", cache)
monkeypatch.setattr(config, "LOG_DIR", logs)
monkeypatch.setattr(config, "SNAPSHOT_FILE", cache / "last-report.json")
return out
@pytest.fixture
def offline_generate_env(monkeypatch):
monkeypatch.setenv("DAILY_AI_NEWS", "0")
monkeypatch.setenv("DAILY_CN_AI_NEWS", "0")
monkeypatch.setenv("DAILY_ZH_DESC", "0")
monkeypatch.setenv("SKILLS_BOARD_SOURCE", "feed")
monkeypatch.setenv("DAILY_REPORT_MODE", "classic")
monkeypatch.setenv("DAILY_CURSOR_EDITOR", "0")
monkeypatch.setenv("DAILY_TRENDING_LIMIT", "10")
monkeypatch.setenv("DAILY_HOT_LIMIT", "10")

12
tests/fixtures/sample-rss.xml vendored Normal file
View File

@@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>Sample AI Feed</title>
<item>
<title>Sample AI headline for smoke tests</title>
<link>https://example.com/ai-news/1</link>
<pubDate>Wed, 02 Jul 2026 08:00:00 GMT</pubDate>
<description>A minimal RSS item used by offline tests.</description>
</item>
</channel>
</rss>

32
tests/fixtures/skills-feed.json vendored Normal file
View File

@@ -0,0 +1,32 @@
{
"updatedAt": "2026-07-02T12:00:00Z",
"topTrending": [
{
"id": "vercel-labs/skills/find-skills",
"title": "find-skills",
"source": "vercel-labs/skills",
"installs": 18000,
"link": "https://www.skills.sh/vercel-labs/skills/find-skills",
"description": "Help users discover and install Agent Skills."
},
{
"id": "halt-catch-fire/skills/remotion-render",
"title": "remotion-render",
"source": "halt-catch-fire/skills",
"installs": 21549,
"link": "https://www.skills.sh/halt-catch-fire/skills/remotion-render",
"description": "Render videos from React/Remotion component code."
}
],
"topHot": [
{
"id": "vercel-labs/skills/find-skills",
"title": "find-skills",
"source": "vercel-labs/skills",
"installs": 243,
"link": "https://www.skills.sh/vercel-labs/skills/find-skills",
"description": "Help users discover and install Agent Skills."
}
],
"topAllTime": []
}

View File

@@ -0,0 +1,55 @@
from __future__ import annotations
import json
def test_analyze_trends_parses_json(monkeypatch, isolated_output):
import daily.agent_workflow as agent
import daily.config as config
monkeypatch.setattr(config, "OUTPUT_DIR", isolated_output)
monkeypatch.setattr(agent, "OUTPUT_DIR", isolated_output)
monkeypatch.setattr(
agent,
"llm_chat",
lambda system, user: json.dumps(
{
"headline": "Skills 视频工具升温",
"opening": "今日 remotion 相关技能继续走强。",
"themes": [{"name": "视频", "summary": "Remotion 生态活跃"}],
"top_picks": [],
"signals": [],
},
ensure_ascii=False,
),
)
llm_input = {"date": "2026-07-03", "skills_trending": []}
trends = agent.analyze_trends(llm_input, date_str="2026-07-03")
assert trends is not None
assert trends["headline"] == "Skills 视频工具升温"
assert (isolated_output / "2026-07-03.trends.json").exists()
def test_write_wecom_report_extracts_markdown_block(monkeypatch):
import daily.agent_workflow as agent
monkeypatch.setattr(
agent,
"llm_chat",
lambda system, user: "```markdown\n📰 **早报 · 2026-07-03**\n\n正文\n```",
)
md = agent.write_wecom_report(
{"date": "2026-07-03"},
{"headline": "test", "opening": "open"},
date_str="2026-07-03",
time_str="09:30 (UTC+8)",
updated="2026-07-02",
)
assert md is not None
assert md.startswith("📰")
assert "正文" in md

67
tests/test_delta.py Normal file
View File

@@ -0,0 +1,67 @@
from __future__ import annotations
import json
import pytest
def test_build_movement_context_detects_new_skill(isolated_output, monkeypatch):
import daily.config as config
import daily.delta as delta
monkeypatch.setattr(config, "OUTPUT_DIR", isolated_output)
monkeypatch.setattr(delta, "OUTPUT_DIR", isolated_output)
prev_path = isolated_output / "2026-07-02.data.json"
prev_path.write_text(
json.dumps(
{
"data": {
"date": "2026-07-02",
"movement_baseline": {
"skills_trending": [{"id": "a/old", "title": "old"}],
"skills_hot": [],
"github_trending": [],
"github_emerging": [],
"github_topic": [],
},
}
},
ensure_ascii=False,
),
encoding="utf-8",
)
trending = [
{"id": "a/new", "title": "new-skill", "source": "a", "installs": 10, "link": "https://x", "description": ""},
{"id": "a/old", "title": "old", "source": "a", "installs": 9, "link": "https://y", "description": ""},
]
movement = delta.build_movement_context(
date_str="2026-07-03",
trending=trending,
hot=[],
github_trending=[],
github_emerging=[],
github_topic=[],
)
assert movement["baseline_date"] == "2026-07-02"
assert len(movement["skills_trending_moves"]) == 1
assert movement["skills_trending_moves"][0]["id"] == "a/new"
def test_find_previous_data_skips_missing_days(isolated_output, monkeypatch):
import daily.config as config
import daily.delta as delta
monkeypatch.setattr(config, "OUTPUT_DIR", isolated_output)
monkeypatch.setattr(delta, "OUTPUT_DIR", isolated_output)
(isolated_output / "2026-07-01.data.json").write_text(
json.dumps({"data": {"date": "2026-07-01", "skills_trending": []}}),
encoding="utf-8",
)
found = delta.find_previous_data("2026-07-03")
assert found is not None
assert found[0] == "2026-07-01"

92
tests/test_news_rank.py Normal file
View File

@@ -0,0 +1,92 @@
from __future__ import annotations
from datetime import datetime, timezone
from daily.news.rank import (
apply_news_ranking,
fuzzy_dedupe_by_title,
jaccard_similarity,
score_news_item,
_title_tokens,
)
def test_jaccard_similarity_detects_duplicate_headlines():
left = _title_tokens("OpenAI launches GPT-5 with new reasoning")
right = _title_tokens("OpenAI launches GPT 5 with reasoning upgrade")
assert jaccard_similarity(left, right) >= 0.55
def test_fuzzy_dedupe_keeps_higher_score():
items = [
{"title": "OpenAI launches GPT-5", "link": "https://a.example/1", "score": 20},
{"title": "OpenAI launches GPT 5 today", "link": "https://b.example/2", "score": 45},
]
deduped = fuzzy_dedupe_by_title(items, threshold=0.55)
assert len(deduped) == 1
assert deduped[0]["link"] == "https://b.example/2"
def test_score_news_item_prefers_fresh_official_and_penalizes_yesterday():
now = datetime(2026, 7, 3, 12, 0, tzinfo=timezone.utc)
fresh = {
"title": "Claude update",
"link": "https://anthropic.com/news/claude",
"category_id": "official",
"published": "Thu, 03 Jul 2026 10:00:00 GMT",
}
stale_seen = {
"title": "Old story",
"link": "https://example.com/old",
"category_id": "media",
"published": "Mon, 30 Jun 2026 10:00:00 GMT",
}
fresh_score = score_news_item(fresh, yesterday_links=set(), now=now)
stale_score = score_news_item(
stale_seen,
yesterday_links={"https://example.com/old"},
now=now,
)
assert fresh_score > stale_score
def test_apply_news_ranking_sets_flat_scores(monkeypatch, tmp_path):
import daily.config as config
import daily.news.rank as rank
out = tmp_path / "output"
out.mkdir()
monkeypatch.setattr(config, "OUTPUT_DIR", out)
monkeypatch.setattr(rank, "load_yesterday_news_links", lambda _date: set())
news = {
"enabled": True,
"categories": [
{
"id": "official",
"name": "厂商官方",
"icon": "🏢",
"items": [
{
"title": "OpenAI ships new model",
"link": "https://openai.com/a",
"category_id": "official",
"published": "Thu, 03 Jul 2026 08:00:00 GMT",
},
{
"title": "OpenAI ships a new model today",
"link": "https://techcrunch.com/a",
"category_id": "media",
"published": "Thu, 03 Jul 2026 07:00:00 GMT",
},
],
}
],
"flat": [],
"stats": {},
}
ranked = apply_news_ranking(news, date_str="2026-07-03")
assert ranked["flat"]
assert all("score" in item for item in ranked["flat"])
assert len(ranked["flat"]) == 1

54
tests/test_rss_parse.py Normal file
View File

@@ -0,0 +1,54 @@
from __future__ import annotations
from pathlib import Path
import pytest
FIXTURES = Path(__file__).parent / "fixtures"
def test_parse_sample_rss_fixture():
from daily.news.feeds import NEWS_CATEGORIES
from daily.news.fetch import _parse_feed
category = NEWS_CATEGORIES[0]
feed_name = category.feeds[0].name
xml = (FIXTURES / "sample-rss.xml").read_text(encoding="utf-8")
items = _parse_feed(xml, feed_name, category)
assert len(items) == 1
assert items[0]["title"] == "Sample AI headline for smoke tests"
assert items[0]["link"] == "https://example.com/ai-news/1"
assert items[0]["source_name"] == feed_name
assert "minimal RSS item" in items[0]["summary"]
def test_fetch_ai_news_offline(monkeypatch):
from daily.news import fetch as news_fetch
sample = {
"title": "Sample AI headline for smoke tests",
"link": "https://example.com/ai-news/1",
"summary": "A minimal RSS item used by offline tests.",
"published": "Wed, 02 Jul 2026 08:00:00 GMT",
"source_name": "OpenAI",
"category_id": "official",
"category_name": "厂商官方",
"category_icon": "🏢",
}
monkeypatch.setattr(
news_fetch,
"_fetch_news",
lambda categories: {
"enabled": True,
"hours": 72,
"categories": [{"id": "official", "name": "厂商官方", "icon": "🏢", "items": [sample]}],
"flat": [sample],
"stats": {"feeds_total": 1, "feeds_ok": 1, "items_raw": 1, "feeds_failed": []},
},
)
payload = news_fetch.fetch_ai_news()
assert payload["enabled"] is True
assert payload["flat"][0]["title"].startswith("Sample AI")

67
tests/test_smoke.py Normal file
View File

@@ -0,0 +1,67 @@
from __future__ import annotations
import json
from pathlib import Path
import pytest
FIXTURES = Path(__file__).parent / "fixtures"
def _sample_github_repo() -> dict:
return {
"repo": "owner/sample",
"url": "https://github.com/owner/sample",
"description": "Sample repository for smoke tests.",
"language": "Python",
"stars_today_fmt": "120",
"total_stars_fmt": "1.2K",
"created_at": "2026-06-01",
}
def test_format_installs():
from shared.skills_data import format_installs
assert format_installs(999) == "999"
assert format_installs(18000) == "18.0K"
assert format_installs(2_500_000) == "2.5M"
def test_generate_report_offline(
skills_feed,
fixed_cst,
isolated_output,
offline_generate_env,
monkeypatch,
):
import daily.generate as generate
import daily.pipeline.collect as collect
monkeypatch.setattr(collect, "load_feed", lambda force=False: skills_feed)
monkeypatch.setattr(collect, "fetch_github_trending", lambda n: [_sample_github_repo()])
monkeypatch.setattr(collect, "fetch_emerging_repos", lambda n, exclude=None: [])
monkeypatch.setattr(collect, "fetch_topic_hot_repos", lambda n, exclude=None: ("llm", []))
monkeypatch.setattr(generate, "_now_cst", lambda: fixed_cst)
_, wecom_md, out_md, out_wecom = generate.generate_report()
assert out_md.exists()
assert out_wecom.exists()
assert "早报" in out_md.read_text(encoding="utf-8")
assert wecom_md.strip()
assert "📰" in wecom_md
data_path = isolated_output / "2026-07-03.data.json"
assert data_path.exists()
payload = json.loads(data_path.read_text(encoding="utf-8"))
assert payload["meta"]["report_mode"] == "classic"
assert payload["data"]["date"] == "2026-07-03"
assert len(payload["data"]["skills_trending"]) >= 1
assert len(payload["data"]["skills_hot"]) >= 1
def test_rss_fixture_is_well_formed():
xml = (FIXTURES / "sample-rss.xml").read_text(encoding="utf-8")
assert "Sample AI headline" in xml
assert "<item>" in xml

34
tests/test_wecom_split.py Normal file
View File

@@ -0,0 +1,34 @@
from __future__ import annotations
from daily.wecom_split import split_wecom_messages
def test_split_wecom_messages_keeps_short_text():
text = "📰 **早报 · 2026-07-03**\n\n短内容"
parts = split_wecom_messages(text, limit=4096)
assert parts == [text]
def test_split_wecom_messages_adds_part_footer():
section = "📰 **早报**\n\n" + ("正文行\n" * 200)
parts = split_wecom_messages(section, limit=600)
assert len(parts) > 1
assert all(len(part.encode("utf-8")) <= 600 for part in parts)
assert parts[0].endswith("1/" + str(len(parts)))
def test_split_sections_on_emoji_headers():
from daily.wecom_split import _split_sections
text = "\n\n".join(
[
"📰 **早报 · 2026-07-03**",
"💡 **今日速览**\n- item one\n- item two",
"🌍 **国际 AI 时讯**\n1. [Headline](https://example.com)",
]
)
sections = _split_sections(text)
assert len(sections) == 3
assert sections[0].startswith("📰")
assert sections[1].startswith("💡")
assert sections[2].startswith("🌍")

View File

@@ -1,14 +0,0 @@
import re
import certifi
import httpx
url = "https://skills.sh/vercel-labs/skills/find-skills"
r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20, verify=certifi.where())
chunks = re.findall(r'self\.__next_f\.push\(\[1,"(.*?)"\]\)', r.text, re.DOTALL)
blob = "\n".join(chunks).encode("utf-8").decode("unicode_escape", errors="ignore")
for needle in ("description", "Helps users", "SKILL.md", "summary"):
print(needle, blob.count(needle))
idx = blob.find("Helps users")
if idx >= 0:
print(blob[idx : idx + 300])

Binary file not shown.