daily-robots/docs/superpowers/plans/2026-07-14-wecom-diversity-dedup.md

企微早报多样性与去重 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: 实现企微早报硬去重多样性：五榜周去重（深池补满）、首推与昨日相同则改推（月去重）、叙事轴代码互斥、取消新闻「放宽窗口」凑数；且 movement_baseline 与 wecom_shown_keys 严格分离。

Architecture: 在 daily generate 管线加代码选择器：board_select 为 full/delta 唯一列表主人；周历史只读写 data.wecom_shown_keys（post-render）；movement_baseline 仍为 raw Top compare；featured_resolve 先定人再 research；pick_narrative_axis 代码选轴注入 Agent Step1；新闻关 backfill + 剥「放宽」前缀。

Tech Stack: Python 3.13+、现有 unittest/pytest、daily/delta.py / format_wecom.py / featured_pick.py / news/fetch.py、output/*.data.json

Spec: docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md（Status: APPROVED）

Global Constraints

• movement_baseline 禁止被展示历史覆写；周去重只读 wecom_shown_keys
• full/delta 唯一列表主人 = board_select（delta 的 pad 共用同一套 shown 历史）
• DAILY_BOARD_DEDUP_DAYS 默认 7；与 pad lookback 对齐且数据源同一
• DAILY_FEATURED_DEDUP_DAYS 默认 30
• DAILY_NARRATIVE_AXIS_DAYS 默认 3；轴枚举固定 7 个（见 Task 5）
• DAILY_NEWS_BACKFILL 默认 0（禁止旧闻凑数）
• research 补新闻年龄上限 = DAILY_AI_NEWS_HOURS（不得变相 48h 放宽）
• 不做：语义「同一类」、同日 Trending↔Hot 互斥、FEATURED_FORCE、关键短语硬匹配
• Commit 信息正文用中文（若本任务含 Commit 步）

---

File Structure

文件	职责
daily/config.py	新 env：board_dedup_days、board_pool_size、featured_dedup_days、theme_ban_days、narrative_axis_days、news_backfill_enabled
daily/board_history.py	新建 — load_recent_shown_keys / extract_shown_keys / attach_wecom_shown_keys（读写 data.wecom_shown_keys）
daily/board_select.py	新建 — board_select(...) 周过滤+深池
daily/delta.py	load_recent_board_keys 改为委托 load_recent_shown_keys（保留函数名兼容）；不改 build_movement_baseline
daily/format_wecom.py	pad 使用 shown keys；可选返回最终展示 items 供写回
daily/featured_pick.py	featured_identity_key、featured_resolve、先定人再 research
daily/narrative_axis.py	新建 — NARRATIVE_AXES、pick_narrative_axis、load_recent_axes
daily/news/fetch.py	_apply_pushed_dedup_with_backfill 尊重 news_backfill_enabled()；默认不塞回
daily/news/research.py + skills/daily-ai-news-research/SKILL.md	禁放宽文案；补入不超时窗
daily/text_utils.py 或 daily/news/sanitize.py	strip_news_relax_prefix(desc)
daily/agent_workflow.py	Step1 注入 axis + 近 7 日 theme 软禁；强制覆写冲突轴
daily/generate.py	串联：select → featured → editorial → render → persist shown/key/axis
daily/report_data.py	data.json 可携 wecom_shown_keys / featured_pick_key / narrative_axis（写回可由 generate 合并）
.env.example	文档化新变量
skills/daily-agent/SKILL.md	narrative_axis 必填且等于输入指定轴
tests/test_board_select.py	新建
tests/test_board_history.py	新建
tests/test_featured_resolve.py	新建
tests/test_narrative_axis.py	新建
tests/test_news_relax.py	新建
tests/test_wecom_delta.py	回归：pad 不读 movement 当展示史

---

Task 1: Config + shown-keys 历史层

Files:

• Modify: daily/config.py（文件末尾追加）
• Create: daily/board_history.py
• Modify: daily/delta.py（load_recent_board_keys 改委托）
• Test: tests/test_board_history.py
• Modify: .env.example

Interfaces:

• Consumes: OUTPUT_DIR、现有 delta.skill_id / repo key 约定

• Produces:

  • board_dedup_days() -> int（默认 7）
  • board_pool_size() -> int（默认 max(50, env WECOM_SKILL_POOL)）
  • featured_dedup_days() -> int（默认 30）
  • theme_ban_days() -> int（默认 7）
  • narrative_axis_days() -> int（默认 3）
  • news_backfill_enabled() -> bool（默认 False；env DAILY_NEWS_BACKFILL）
  • extract_shown_keys(board: str, items: list[dict]) -> list[str]
  • load_recent_shown_keys(date_str: str, *, lookback_days: int | None = None) -> dict[str, set[str]]
  • merge_wecom_shown_into_data(data: dict, shown: dict[str, list[str]]) -> dict

• Step 1: Write the failing test

# tests/test_board_history.py
from __future__ import annotations

import json
import os
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch

from daily.board_history import extract_shown_keys, load_recent_shown_keys, merge_wecom_shown_into_data
from daily.config import board_dedup_days, news_backfill_enabled


class ConfigDiversityTests(unittest.TestCase):
    def test_board_dedup_days_default(self):
        with patch.dict(os.environ, {}, clear=True):
            self.assertEqual(board_dedup_days(), 7)

    def test_news_backfill_default_off(self):
        with patch.dict(os.environ, {}, clear=True):
            self.assertFalse(news_backfill_enabled())


class ShownKeysTests(unittest.TestCase):
    def test_extract_github_repo_keys(self):
        items = [{"repo": "a/b"}, {"repo": "c/d"}]
        self.assertEqual(extract_shown_keys("github_trending", items), ["a/b", "c/d"])

    def test_load_recent_reads_wecom_shown_not_baseline(self):
        with tempfile.TemporaryDirectory() as tmp:
            out = Path(tmp)
            # 前日：shown 只有 x/y；baseline raw 含 a/b —— 周去重只能看到 x/y
            payload = {
                "data": {
                    "date": "2026-07-13",
                    "movement_baseline": {
                        "github_trending": [{"repo": "a/b"}, {"repo": "x/y"}],
                    },
                    "wecom_shown_keys": {"github_trending": ["x/y"]},
                }
            }
            (out / "2026-07-13.data.json").write_text(
                json.dumps(payload, ensure_ascii=False), encoding="utf-8"
            )
            with patch("daily.board_history.OUTPUT_DIR", out):
                keys = load_recent_shown_keys("2026-07-14", lookback_days=7)
            self.assertEqual(keys["github_trending"], {"x/y"})
            self.assertNotIn("a/b", keys["github_trending"])

    def test_merge_shown_does_not_touch_baseline(self):
        data = {
            "movement_baseline": {"github_trending": [{"repo": "raw/one"}]},
        }
        merged = merge_wecom_shown_into_data(
            data, {"github_trending": ["shown/one"]}
        )
        self.assertEqual(
            merged["movement_baseline"]["github_trending"][0]["repo"], "raw/one"
        )
        self.assertEqual(merged["wecom_shown_keys"]["github_trending"], ["shown/one"])

• Step 2: Run test to verify it fails

Run: pytest tests/test_board_history.py -v
Expected: FAIL（模块/函数不存在）

• Step 3: Implement config + board_history + delta 委托

在 daily/config.py 追加：

def board_dedup_days() -> int:
    return max(1, env_int("DAILY_BOARD_DEDUP_DAYS", 7))

def board_pool_size() -> int:
    fallback = env_int("DAILY_WECOM_SKILL_POOL", 50)
    return max(1, env_int("DAILY_BOARD_POOL_SIZE", max(50, fallback)))

def featured_dedup_days() -> int:
    return max(1, env_int("DAILY_FEATURED_DEDUP_DAYS", 30))

def theme_ban_days() -> int:
    return max(1, env_int("DAILY_THEME_BAN_DAYS", 7))

def narrative_axis_days() -> int:
    return max(1, env_int("DAILY_NARRATIVE_AXIS_DAYS", 3))

def news_backfill_enabled() -> bool:
    return env_bool("DAILY_NEWS_BACKFILL", False)

新建 daily/board_history.py：实现 BOARD_KEYS 与 delta.RECENT_BOARD_KEYS 同五榜；Skills 用 delta.skill_id；GitHub 用 repo；load_recent_shown_keys 只读各日 data.wecom_shown_keys，缺省空集，读写失败打 log 后当空集。

修改 daily/delta.py 的 load_recent_board_keys：改为

def load_recent_board_keys(date_str: str, *, lookback_days: int | None = None) -> dict[str, set[str]]:
    from daily.board_history import load_recent_shown_keys
    from daily.config import board_dedup_days
    days = lookback_days if lookback_days is not None else board_dedup_days()
    return load_recent_shown_keys(date_str, lookback_days=days)

删除（或不再走）原「从 movement_baseline 抽 keys」逻辑，避免 pad 继续把 raw Top 当展示史。

.env.example 追加注释块：

# 多样性 / 去重（见 docs/superpowers/specs/2026-07-14-wecom-diversity-dedup-design.md）
# DAILY_BOARD_DEDUP_DAYS=7
# DAILY_BOARD_POOL_SIZE=50
# DAILY_FEATURED_DEDUP_DAYS=30
# DAILY_THEME_BAN_DAYS=7
# DAILY_NARRATIVE_AXIS_DAYS=3
DAILY_NEWS_BACKFILL=0

• Step 4: Run tests

Run: pytest tests/test_board_history.py tests/test_wecom_delta.py -v
Expected: test_board_history PASS；既有 delta 测试若依赖「baseline 即 recent」行为，按 Task 1 语义改断言为 shown_keys（本 Task 内修回归，勿留红）。

• Step 5: Commit

git add daily/config.py daily/board_history.py daily/delta.py .env.example tests/test_board_history.py tests/test_wecom_delta.py
git commit -m "feat: 拆分 wecom_shown_keys 与 movement_baseline 历史层"

---

Task 2: board_select 周去重 + 深池

Files:

• Create: daily/board_select.py
• Test: tests/test_board_select.py

Interfaces:

• Consumes: extract_shown_keys / skill_id；group_skills_by_source（Skills 板）

• Produces:

  • board_select(*, board: str, items: list[dict], recent_keys: set[str], limit: int, pool_size: int, kind: Literal["skill","github"]) -> list[dict]
  • 日志短榜：board_short:{board}:{n}（logging.getLogger(__name__).info）

• Step 1: Write the failing test

# tests/test_board_select.py
from __future__ import annotations

import unittest

from daily.board_select import board_select


def _gh(repo: str) -> dict:
    return {"repo": repo, "description": repo}


class BoardSelectTests(unittest.TestCase):
    def test_filters_recent_and_keeps_order(self):
        pool = [_gh(f"o/r{i}") for i in range(20)]
        recent = {"o/r0", "o/r1", "o/r2"}
        out = board_select(
            board="github_trending",
            items=pool,
            recent_keys=recent,
            limit=5,
            pool_size=20,
            kind="github",
        )
        keys = [x["repo"] for x in out]
        self.assertEqual(keys, ["o/r3", "o/r4", "o/r5", "o/r6", "o/r7"])

    def test_deep_pool_fills_after_filter(self):
        pool = [_gh(f"o/r{i}") for i in range(8)]
        recent = {f"o/r{i}" for i in range(6)}  # 前 6 全封
        out = board_select(
            board="github_emerging",
            items=pool,
            recent_keys=recent,
            limit=5,
            pool_size=8,
            kind="github",
        )
        self.assertEqual([x["repo"] for x in out], ["o/r6", "o/r7"])  # 短榜

    def test_skill_uses_skill_id(self):
        items = [
            {"id": "a/b/s1", "source": "a/b", "title": "s1", "installs": 10},
            {"id": "c/d/s2", "source": "c/d", "title": "s2", "installs": 9},
        ]
        out = board_select(
            board="skills_trending",
            items=items,
            recent_keys={"a/b/s1"},
            limit=10,
            pool_size=50,
            kind="skill",
        )
        self.assertEqual([x["id"] for x in out], ["c/d/s2"])

• Step 2: Run test to verify it fails

Run: pytest tests/test_board_select.py -v
Expected: FAIL

• Step 3: Implement board_select

# daily/board_select.py — 核心逻辑示意
def board_select(*, board, items, recent_keys, limit, pool_size, kind):
    if kind == "skill":
        from daily.skills_group import group_skills_by_source
        pool = group_skills_by_source(items, limit=pool_size, pool_size=pool_size)
        def key_fn(x): return skill_id(x)
    else:
        pool = items[: max(pool_size, limit)]
        def key_fn(x): return str(x.get("repo") or "")
    out = []
    for item in pool:
        k = key_fn(item)
        if not k or k in recent_keys:
            continue
        out.append(item)
        if len(out) >= limit:
            break
    if len(out) < limit:
        logger.info("board_short:%s:%s", board, len(out))
    return out

Skills：输入可为未 group 的 raw；函数内 group。GitHub：输入为 repo 列表。

• Step 4: Run tests — expect PASS

Run: pytest tests/test_board_select.py -v

• Step 5: Commit

git add daily/board_select.py tests/test_board_select.py
git commit -m "feat: 实现 board_select 周去重与深池补满"

---

Task 3: 接入 generate / format_wecom（唯一列表主人 + post-render 写回）

Files:

• Modify: daily/generate.py（选榜、传入 pad、渲染后 merge shown）
• Modify: daily/format_wecom.py（delta pad 已通过改写后的 load_recent_board_keys 读 shown；确保传入的 *_pad 池已是 board_select 深池结果）
• Modify: daily/report_data.py（可选：llm_input 切片改为 board_select 后列表，避免 Agent 看见未去重 Top）
• Test: tests/test_board_history.py 增补「shown ≠ baseline 推导」集成断言；tests/test_wecom_delta.py pad 用例

Interfaces:

• Consumes: Task1–2

• Produces: 每次成功 generate 后 output/{date}.data.json 含 data.wecom_shown_keys

• Step 1: Write / extend failing integration test

def test_persist_shown_keys_differs_from_baseline_keys(self):
    # 构造：raw trending 前 3 名本周已 shown；board_select 选出 3..；
    # movement_baseline 仍含 0..compare_depth
    # 断言 data["wecom_shown_keys"]["github_trending"] 与 baseline repos 集合不等
    ...

（可用临时 OUTPUT_DIR + 调用抽取出的 persist 辅助，或测 merge_wecom_shown_into_data + board_select 组合。）

• Step 2: Run — expect FAIL（generate 尚未写 shown）

• Step 3: Wire generate

在 generate_report 中，在组装 wecom 榜之前：

1. recent = load_recent_shown_keys(date_str)
2. 对五榜分别 board_select(...) 得到 selected_*（limit=wecom_*，pool=board_pool_size()）
3. full：渲染用 selected_*
4. delta：trending_pad/github_*_pad = 同规则更大 pool 的 select 结果（或 raw 深池再 select）；replace_wecom_board_sections(..., pad=True) 内部 recent 已是 shown
5. 渲染后根据最终写入正文的 items（full=selected；delta=函数返回或并行计算最终列表）调用 extract_shown_keys，merge_wecom_shown_into_data，写回 data.json（在现有 save_json 路径合并字段）

注意：movement_baseline 仍用 raw compare 切片构建（report_data.build_llm_input 现逻辑保留）。

若 build_llm_input 当前把未过滤 Top 塞进 Agent：改为传入 selected_*（或另字段 boards_for_wecom），避免 opening 引用已周封杀的榜首。

辅助：在 format_wecom 增加 resolve_wecom_board_items(...) 返回最终 items dict，供写回与 featured 池 A 共用，避免正文与 history 分叉。

• Step 4: Run tests

Run: pytest tests/test_board_select.py tests/test_board_history.py tests/test_wecom_delta.py -v
Expected: PASS

• Step 5: Commit

git add daily/generate.py daily/format_wecom.py daily/report_data.py tests/
git commit -m "feat: generate 以 board_select 为唯一列表主人并写回 shown keys"

---

Task 4: featured_resolve（先定人再 research）

Files:

• Modify: daily/featured_pick.py
• Modify: daily/generate.py（调用顺序）
• Test: tests/test_featured_resolve.py

Interfaces:

• Consumes: 最终展示 items（池 A）、raw 深池（池 B）、近 30 日 featured_pick_key

• Produces:

  • featured_identity_key(featured: dict) -> str（type==skill → id；github → repo；兜底 url path）
  • load_recent_featured_keys(date_str, days) -> set[str]
  • featured_resolve(*, date_str, candidate: dict | None, pool_a: list[dict], pool_b: list[dict], rng: random.Random | None) -> tuple[dict | None, str | None]
    返回 (resolved_seed_or_featured_stub, identity_key)；不含完整 why（why 由后续 research 写）
  • 改 apply_featured_pick：先 resolve 身份（若与昨日冲突则换候选写入 query），再 research_featured_pick

• Step 1: Failing tests

# tests/test_featured_resolve.py
def test_same_as_yesterday_picks_from_pool_a(self):
    yesterday_key = "headroomlabs-ai/headroom"
    pool_a = [
        {"repo": "headroomlabs-ai/headroom", "board": "github_topic"},
        {"repo": "ollama/ollama", "board": "github_trending"},
    ]
    rng = random.Random(0)
    resolved, key = featured_resolve(
        date_str="2026-07-14",
        candidate={"type": "github", "url": "https://github.com/headroomlabs-ai/headroom", "title": "headroom"},
        pool_a=pool_a,
        pool_b=[],
        recent_featured={yesterday_key},
        yesterday_key=yesterday_key,
        rng=rng,
    )
    self.assertNotEqual(key, yesterday_key)
    self.assertEqual(key, "ollama/ollama")

def test_pool_a_before_pool_b(self):
    ...

def test_exhausted_keeps_original(self):
    ...

• Step 2: Run — FAIL

• Step 3: Implement

featured_resolve：若无 candidate 或与 yesterday_key 不同 → 原样返回。
冲突时：过滤 recent_featured | {yesterday_key}，先从 pool_a 建可选项（每项抽 identity），rng.choice；空则 pool_b；仍空 log featured_fallback_exhausted 并保留原 candidate。

apply_featured_pick / generate 流程：

1. 解析 env 得到初始 query/candidate
2. featured_resolve（此时池 A 已是 board_select 结果）
3. 若换人：用新 repo/skill 构造 config，再 research_featured_pick
4. 写入 llm_input["featured_pick"] 与之后 data.featured_pick_key

随机默认：random.Random(int(hashlib.sha256(f"{date_str}:featured".encode()).hexdigest()[:16], 16))

• Step 4: pytest PASS

• Step 5: Commit

git add daily/featured_pick.py daily/generate.py tests/test_featured_resolve.py
git commit -m "feat: 首推与昨日冲突时改推并保证一月不重复"

---

Task 5: narrative_axis 硬互斥 + Step1 软禁 theme

Files:

• Create: daily/narrative_axis.py
• Modify: daily/agent_workflow.py（analyze_trends）
• Modify: skills/daily-agent/SKILL.md
• Modify: daily/generate.py（落盘 narrative_axis；注入 llm_input）
• Test: tests/test_narrative_axis.py

Interfaces:

• Produces:

  • NARRATIVE_AXES: tuple[str, ...] = ("政策监管", "模型发布", "工具链/Agent", "芯片算力", "开源生态", "应用落地", "安全/诉讼")
  • pick_narrative_axis(used: set[str], *, rng: random.Random | None = None) -> str
  • load_recent_axes(date_str, days) -> list[str]（近 N 日 data.narrative_axis）
  • enforce_narrative_axis(trends: dict, axis: str) -> dict（强制 trends["narrative_axis"]=axis）

• Step 1: Failing tests

def test_pick_excludes_used(self):
    used = {"政策监管", "模型发布", "工具链/Agent"}
    for _ in range(20):
        axis = pick_narrative_axis(used, rng=random.Random(1))
        self.assertNotIn(axis, used)

def test_enforce_overwrites_llm(self):
    trends = {"narrative_axis": "开源生态", "opening": "..."}
    out = enforce_narrative_axis(trends, "芯片算力")
    self.assertEqual(out["narrative_axis"], "芯片算力")

• Step 2: FAIL → Step 3 implement

analyze_trends：计算 axis = pick_narrative_axis(set(load_recent_axes(...)))；把 required_narrative_axis 与近 theme_ban_days 的 theme/opening 摘要列表注入 system prompt；要求 JSON 含 narrative_axis 且必须等于 required。解析后 enforce_narrative_axis。
generate 将 axis 写入 data.json。

SKILL.md Step1 schema 增加 narrative_axis 字段说明。

• Step 4–5: pytest + commit

git commit -m "feat: 代码选定叙事轴并注入 Agent 开场约束"

---

Task 6: 取消新闻「放宽窗口」

Files:

• Modify: daily/news/fetch.py（_apply_pushed_dedup_with_backfill）
• Modify: daily/news/research.py（确认只 filter_unpushed；不足不拉超窗）
• Create: daily/news/sanitize.py（或放入 text_utils）— strip_relax_window_prefix(text: str) -> str
• Modify: skills/daily-ai-news-research/SKILL.md（删除「放宽至 48h 并注明」；改为不足则少返回、禁止标注）
• Modify: daily/generate.py / news finalize（对 desc_short 剥前缀）
• Test: tests/test_news_relax.py；扩展 tests/test_news_fetch_window.py

Interfaces:

• news_backfill_enabled() == False 时：_apply_pushed_dedup_with_backfill 等价于只返回 filter_unpushed_items(...)[:limit]，不再从 picked 塞回

• strip_relax_window_prefix：去掉开头的 放宽窗口[:：]? / 放宽至[^：:]*[:：]

• Step 1: Failing tests

def test_backfill_disabled_does_not_reinsert_pushed(self):
    # fresh 不足 limit；picked 含已推；BACKFILL=0 → 结果不含已推 link
    ...

def test_strip_relax_prefix(self):
    self.assertEqual(
        strip_relax_window_prefix("放宽窗口：苹果起诉 OpenAI"),
        "苹果起诉 OpenAI",
    )

• Step 2–4: 实现并跑 pytest tests/test_news_relax.py tests/test_news_fetch_window.py -v

Research 路径：SKUILL 改完后，代码侧对 items 统一 strip；不足时 log news_short:{n}，接受短列表。

• Step 5: Commit

git commit -m "fix: 关闭新闻放宽凑数并剥离放宽窗口文案"

---

Task 7: 端到端回归与文档对齐

Files:

• Modify: 如有遗漏的 .env.example / SKILL

• Test: 全量相关测试

• Step 1: 跑全套

Run:

pytest tests/test_board_history.py tests/test_board_select.py tests/test_featured_resolve.py tests/test_narrative_axis.py tests/test_news_relax.py tests/test_news_fetch_window.py tests/test_wecom_delta.py tests/test_featured_pick.py -v

Expected: 全部 PASS

• Step 2: Spec 对照清单（人工）

Spec 要求	任务
wecom_shown_keys ≠ movement_baseline	T1, T3
board_select 唯一主人 + delta pad 共用 shown	T2, T3
首推月去重 A→B、先定人再 why	T4
narrative_axis 硬保证	T5
禁放宽 backfill + 剥前缀 + hours 窗	T6
成功标准可测	各测覆盖

• Step 3: Commit（若有收尾文档）

git add -u
git commit -m "test: 多样性去重全链路回归通过"

---

Spec Coverage Self-Review

Spec 节	计划任务
1.1 双基准分离	T1
1.2 唯一列表主人	T2–T3
1.3 真相表	T1, T3–T5 落盘字段
2.1 board_select	T2
2.2 featured_resolve	T4
3.1 narrative_axis + 软 theme	T5
3.2 取消放宽	T6
4.x 配置/降级/测试	T1–T7

无 TBD；Commit 信息均为中文描述体。

Execution Handoff

Plan complete and saved to docs/superpowers/plans/2026-07-14-wecom-diversity-dedup.md.

两种执行方式：

1. Subagent-Driven（推荐） — 每任务新开子代理，任务间审查
2. Inline Execution — 本会话按 executing-plans 连续做完，设检查点

要哪个？