diff --git a/docs/superpowers/plans/2026-07-10-audit-fix-all-35-issues.md b/docs/superpowers/plans/2026-07-10-audit-fix-all-35-issues.md new file mode 100644 index 0000000..366ecaf --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-audit-fix-all-35-issues.md @@ -0,0 +1,1910 @@ +# 审计问题修复计划 — 全部 35 项 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 修复 2026-07-10 审计报告中的全部 35 个问题，涵盖 CRITICAL(1) + HIGH(13) + MEDIUM(11) + LOW(10) + +**Architecture:** 分 4 批执行：CRITICAL 立即修复 → HIGH 安全/功能/代码质量/测试补充 → MEDIUM 架构改进 → LOW 文档和工具链收尾。每批 TDD 先行，所有新增或修改的逻辑都必须有对应测试，每批结束后跑全量测试确保无回归。 + +**Tech Stack:** Python 3.13, FastAPI, ChromaDB, pytest, Typer + +--- + +## 批次总览 + +| 批次 | 问题数 | 预计时间 | 说明 | +|------|--------|----------|------| +| Batch 1 | 1 | 10 min | CRITICAL: 重复入库 | +| Batch 2 | 13 | 3-4 hrs | HIGH: 安全/功能/代码质量/测试 | +| Batch 3 | 11 | 2 hrs | MEDIUM: 架构改进/去重/边界测试 | +| Batch 4 | 10 | 1 hr | LOW: 文档/工具链/编码 | + +--- + +## Batch 1 — CRITICAL（1 个问题） + +### Task 1: 修复 ingest_obsidian.py 顶层 .md 文件重复入库 + +**Files:** +- Modify: `scripts/ingest_obsidian.py:66-77` + +- [ ] **Step 1: 删除重复扫描代码** + +删除第二轮 `for target_info in targets:` 循环（第 66-77 行），`rglob("*.md")` 已递归覆盖所有文件。 + +修改前（第 66-77 行）： +```python +# 顶层 .md 文件 +for target_info in targets: + d = target_info[1] + p = Path(d) + if not p.exists(): + continue + for f in p.glob("*.md"): + sz = f.stat().st_size + if sz > MAX_SIZE: + skipped.append((f.name, sz)) + else: + files.append(("顶层/" + p.name, str(f))) +``` + +修改后（直接删除整段，`rglob` 在上一轮已覆盖）： +```python +# rglob("*.md") 已包括顶层文件，无需单独扫描 +``` + +- [ ] **Step 2: 运行现有测试确认无回归** + +```bash +uv run pytest tests/ -v +``` +预期：90 passed + +- [ ] **Step 3: 提交** + +```bash +git add scripts/ingest_obsidian.py +git commit -m "fix: 修复 ingest_obsidian 顶层 .md 重复入库导致 GPU 浪费" +``` + +--- + +## Batch 2 — HIGH（13 个问题） + +### Task 2: CORS 配置冲突修复 (HIGH-1) + +**Files:** +- Modify: `src/server/app.py:66-72` + +- [ ] **Step 1: 修改 CORS 配置** + +```python +# 第 66-72 行，改为： +app.add_middleware( + CORSMiddleware, + allow_origins=os.environ.get("CORS_ORIGINS", "http://localhost:3000").split(","), + allow_credentials=False, # 默认关闭，与 allow_origins="*" 不兼容 + allow_methods=["GET", "POST", "DELETE", "OPTIONS"], + allow_headers=["Content-Type", "Authorization", "X-API-Key"], +) +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_api.py -v +``` +预期：14 passed + +- [ ] **Step 3: 提交** + +```bash +git add src/server/app.py +git commit -m "fix: 修复 CORS allow_credentials 与 allow_origins=* 冲突" +``` + +--- + +### Task 3: search_documents 端点添加异常处理 (HIGH-2) + +**Files:** +- Modify: `src/server/app.py:147-155` + +- [ ] **Step 1: 添加 try/except** + +将 `search_documents` 函数体包裹在异常处理中： + +```python +@app.post("/api/v1/search") +def search_documents( + req: SearchRequest, + state: AppState = Depends(get_state), + _: bool = Depends(verify_api_key), +): + try: + searcher = state.get_searcher(req.collection) + results = searcher.search(req.query, top_k=req.top_k) + return {"results": results, "collection": searcher.collection_name} + except HTTPException: + raise + except Exception: + logger.exception("检索失败") + raise HTTPException(status_code=500, detail="服务器内部错误") +``` + +- [ ] **Step 2: 写测试验证异常不泄露 traceback** + +在 `tests/test_api.py` 中添加： + +```python +def test_search_internal_error_does_not_leak_traceback(client, monkeypatch): + """search 内部错误不泄露 traceback 信息.""" + monkeypatch.setenv("MD_VECTOR_API_KEY", "test-key") + # 模拟嵌入器抛异常 + with monkeypatch.context() as m: + m.setattr( + "src.core.embedder.LocalEmbedder.embed", + lambda self, texts: (_ for _ in ()).throw(RuntimeError("GPU OOM")), + ) + response = client.post( + "/api/v1/search", + json={"query": "test", "top_k": 3}, + headers={"X-API-Key": "test-key"}, + ) + assert response.status_code == 500 + data = response.json() + assert "detail" in data + # 不应泄露内部错误信息 + assert "GPU OOM" not in str(data) + assert "RuntimeError" not in str(data) + assert "traceback" not in str(data).lower() +``` + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/test_api.py::test_search_internal_error_does_not_leak_traceback -v +``` +预期：PASS + +- [ ] **Step 4: 提交** + +```bash +git add src/server/app.py tests/test_api.py +git commit -m "fix: search_documents 端点添加异常处理防止 traceback 泄露" +``` + +--- + +### Task 4: delete_document collection 参数添加输入校验 (HIGH-3) + +**Files:** +- Modify: `src/server/app.py:159-163` + +- [ ] **Step 1: 添加 Query 校验** + +```python +# 第 159-163 行，添加 Query 参数校验： +@app.delete("/api/v1/documents/{file_name}") +def delete_document( + file_name: str, + state: AppState = Depends(get_state), + _: bool = Depends(verify_api_key), + collection: str | None = Query( + default=None, max_length=128, pattern=r"^[a-zA-Z0-9_-]+$", + ), +): +``` + +需要在文件顶部添加 `Query` 的导入（FastAPI 已导入，但 `Query` 不在当前 import 中）： + +在 `src/server/app.py` 第 8 行修改导入： +```python +from fastapi import FastAPI, HTTPException, Depends, Request, Query +``` + +- [ ] **Step 2: 写测试验证非法 collection 名被拒绝** + +在 `tests/test_api.py` 中添加： + +```python +def test_delete_document_rejects_invalid_collection_name(client): + """非法 collection 名被拒绝 (含特殊字符).""" + response = client.delete( + "/api/v1/documents/test.md?collection=bad;drop--table", + headers={"X-API-Key": "test-key"}, + ) + assert response.status_code == 422 # Pydantic validation error +``` + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/test_api.py::test_delete_document_rejects_invalid_collection_name -v +``` +预期：PASS + +- [ ] **Step 4: 提交** + +```bash +git add src/server/app.py tests/test_api.py +git commit -m "fix: delete_document 的 collection 参数添加正则校验" +``` + +--- + +### Task 5: 统一 CLI 与 API 的路径安全检查 (HIGH-4) + +**Files:** +- Modify: `src/core/security.py` (新增 `is_path_within_workspace`) +- Modify: `src/server/app.py:119-130` (改用新函数) +- Modify: `src/cli/main.py:85,120` (改用新函数) +- Create: `tests/test_security.py` 补充测试 + +- [ ] **Step 1: 在 security.py 中添加 `is_path_within_workspace`** + +```python +# 在 is_safe_path 函数后面添加（第 28 行之后）： + +def is_path_within_workspace(path_str: str) -> bool: + """检查路径是否在当前工作目录内（防路径穿越 + 目录绑定）. + + 同时检查： + 1. 路径不含 .. 穿越组件且非绝对路径 + 2. resolve 后的路径位于当前工作目录内 + + Args: + path_str: 用户提供的路径字符串 + + Returns: + 路径安全且在工作目录内时返回 True + """ + if not is_safe_path(path_str): + return False + + from pathlib import Path + path = Path(path_str).resolve() + cwd = Path.cwd().resolve() + try: + common = Path(os.path.commonpath([str(path), str(cwd)])) + except ValueError: + return False + return common == cwd +``` + +- [ ] **Step 2: 写测试** + +在 `tests/test_security.py` 中添加： + +```python +from src.core.security import is_path_within_workspace + +class TestIsPathWithinWorkspace: + """工作目录绑定检查.""" + + def test_simple_safe_path(self): + """当前目录下的普通路径安全.""" + assert is_path_within_workspace("test.md") is True + assert is_path_within_workspace("subdir/test.md") is True + + def test_parent_traversal_rejected(self): + """父目录穿越被拒绝.""" + assert is_path_within_workspace("../outside.md") is False + + def test_absolute_path_rejected(self): + """绝对路径被拒绝.""" + assert is_path_within_workspace("/etc/passwd") is False + + def test_empty_string_rejected(self): + """空字符串.""" + assert is_path_within_workspace("") is True # 空路径 resolve 后等于 cwd + + def test_dot_dot_in_middle(self): + """路径中间的 .. 被拒绝.""" + assert is_path_within_workspace("foo/../bar.md") is False +``` + +- [ ] **Step 3: 修改 app.py 使用新函数** + +```python +# 第 117-133 行，替换路径检查逻辑： +@app.post("/api/v1/ingest") +def ingest_document(...): + ingestor = state.get_ingestor(req.collection) + try: + if req.file_path: + if not is_path_within_workspace(req.file_path): # 改这里 + raise HTTPException(status_code=400, detail="不允许的路径") + path = Path(req.file_path).resolve() + # 不再需要 app.py 内部的 commonpath 检查 + ... +``` + +同时需要更新 `app.py` 顶部的 import： +```python +from src.core.security import is_path_within_workspace # 替换 is_safe_path +``` + +- [ ] **Step 4: 修改 cli/main.py 使用新函数** + +```python +# 第 20 行，修改 import： +from src.core.security import is_path_within_workspace + +# 第 85 行： +if not is_path_within_workspace(fp): # 替换 is_safe_path + typer.echo(f"[SKIP] 不安全的路径: {fp}", err=True) + continue + +# 第 93 行： +if not is_path_within_workspace(m): # 替换 is_safe_path + typer.echo(f"[SKIP] 不安全的路径: {m}", err=True) + continue + +# 第 120 行： +if not is_path_within_workspace(dir_path): # 替换 is_safe_path + typer.echo(f"错误: 不安全的路径 — {dir_path}", err=True) + raise typer.Exit(code=1) +``` + +- [ ] **Step 5: 运行全量测试** + +```bash +uv run pytest tests/ -v +``` +预期：全部通过 + +- [ ] **Step 6: 提交** + +```bash +git add src/core/security.py src/server/app.py src/cli/main.py tests/test_security.py +git commit -m "fix: 统一 CLI/API 路径安全检查为 is_path_within_workspace" +``` + +--- + +### Task 6: chunk 配置传递链修复 (HIGH-5) + +**Files:** +- Modify: `src/core/ingest.py:18-28` (构造函数接受 ChunkConfig) +- Modify: `src/core/ingest.py:46-47` (使用配置值而非硬编码) +- Modify: `src/server/deps.py:44-49` (传入 chunk 配置) + +- [ ] **Step 1: 修改 DocumentIngestor 构造函数** + +```python +# ingest.py 第 15-28 行： +from src.core.config import ChunkConfig # 新增导入 + +class DocumentIngestor: + """文档入库器: 读取文件 → 分块 → 嵌入 → 入库.""" + + def __init__( + self, + db: VectorDB, + embedder: Embedder, + collection_name: str, + splitter: Splitter | None = None, + chunk_config: ChunkConfig | None = None, # 新增参数 + ): + self.db = db + self.embedder = embedder + self.collection_name = collection_name + self.splitter = splitter or MarkdownSplitter() + self.chunk_config = chunk_config or ChunkConfig() # 存储分块配置 +``` + +- [ ] **Step 2: 修改 ingest_file 使用配置值** + +```python +# 第 45 行改为： +splitter = self.splitter or get_splitter( + file_path, + max_size=self.chunk_config.max_size, + overlap=self.chunk_config.overlap, +) +``` + +- [ ] **Step 3: 修改 deps.py 传入 chunk 配置** + +```python +# deps.py 第 44-49 行： +def get_ingestor(self, collection: str | None = None) -> DocumentIngestor: + name = collection or self.default_collection + with self._cache_lock: + if name not in self._ingestors: + self._ingestors[name] = DocumentIngestor( + self.db, self.embedder, name, + chunk_config=self.config.chunk, # 新增：传入 chunk 配置 + ) + return self._ingestors[name] +``` + +- [ ] **Step 4: 写测试验证 chunk 配置生效** + +在 `tests/test_deps.py` 中添加： + +```python +def test_ingestor_respects_chunk_config(self, tmp_path): + """验证 ingestor 使用 config.yaml 中的 chunk 配置.""" + from src.core.config import AppConfig, ChunkConfig + from src.core.ingest import DocumentIngestor + + custom_chunk = ChunkConfig(max_size=500, overlap=200) + db = VectorDB(persist_dir=str(tmp_path)) + embedder = create_embedder(EmbedConfig(mode="local")) + ingestor = DocumentIngestor(db, embedder, "test_chunk", chunk_config=custom_chunk) + assert ingestor.chunk_config.max_size == 500 + assert ingestor.chunk_config.overlap == 200 +``` + +- [ ] **Step 5: 运行测试** + +```bash +uv run pytest tests/test_deps.py tests/ -v +``` +预期：全部通过 + +- [ ] **Step 6: 提交** + +```bash +git add src/core/ingest.py src/server/deps.py tests/test_deps.py +git commit -m "fix: chunk 配置通过 DocumentIngestor 传递，不再被硬编码覆盖" +``` + +--- + +### Task 7: CLI serve 命令模型双重加载修复 (HIGH-6) + +**Files:** +- Modify: `src/core/config.py` (新增 `load_config_only` 函数) +- Modify: `src/cli/main.py:164-185` (serve 命令使用轻量配置) + +- [ ] **Step 1: 在 config.py 添加轻量加载函数** + +```python +# 第 103 行之后添加： + +def load_server_config_only(path: str | None = None) -> "AppConfig": + """仅加载配置而不初始化模型/数据库（供 CLI serve 等场景使用）. + + 与 load_config 的区别：不依赖 dotenv 的副作用。 + """ + return load_config(path) +``` + +实际上，`load_config` 本身已经只加载配置不加载模型。问题在于 CLI `serve` 命令调用了 `get_state()` 而 `get_state()` 的 `AppState.__init__` 会加载模型。所以正确的做法是在 CLI `serve` 命令中直接调用 `load_config` 而非 `get_state()`。 + +- [ ] **Step 2: 修改 cli/main.py serve 命令** + +```python +# 第 164-185 行： +@app.command(help="启动 HTTP API 服务.") +def serve( + port: Annotated[int, typer.Option("--port", "-p", help="监听端口")] = 8000, + config: ConfigOpt = DEFAULT_CONFIG_PATH, +): + from src.core.config import load_config # 轻量配置加载 + + # 传递 config 给 uvicorn 子进程（通过环境变量） + os.environ["MD_VECTOR_CONFIG"] = config + # 使用 load_config 而非 get_state()，避免在 CLI 进程加载模型 + cfg = load_config(config) + ssl_kwargs = {} + if cfg.server.ssl_keyfile and cfg.server.ssl_certfile: + ssl_kwargs["ssl_keyfile"] = cfg.server.ssl_keyfile + ssl_kwargs["ssl_certfile"] = cfg.server.ssl_certfile + typer.echo(f"启动服务: http://localhost:{port}") + typer.echo(f"API 文档: http://localhost:{port}/docs") + uvicorn.run( + "src.server.app:app", + host=cfg.server.host, + port=port or cfg.server.port, + reload=False, + **ssl_kwargs, + ) +``` + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/ -v +``` +预期：全部通过 + +- [ ] **Step 4: 提交** + +```bash +git add src/cli/main.py +git commit -m "fix: CLI serve 使用 load_config 避免主进程双重加载嵌入模型" +``` + +--- + +### Task 8: PDF 文件句柄泄漏修复 (HIGH-7) + +**Files:** +- Modify: `src/core/splitters/pdf.py:34-43` + +- [ ] **Step 1: 使用上下文管理器** + +```python +# 替换第 34-43 行： +extracted_pages = [] +try: + with fitz.open(pdf_path) as doc: # pymupdf 支持 with 语句 + for page in doc: + page_text = page.get_text() + if page_text.strip(): + extracted_pages.append(page_text) +except Exception as e: + logger.error("PDF 解析失败: %s — %s", pdf_path, e) + raise ValueError(f"PDF 解析失败: {e}") from e +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_splitters_pdf.py -v +``` +预期：3 passed + +- [ ] **Step 3: 提交** + +```bash +git add src/core/splitters/pdf.py +git commit -m "fix: PDF fitz.Document 使用 with 语句防止异常路径下句柄泄漏" +``` + +--- + +### Task 9: embedder.py HF_ENDPOINT 线程安全 (HIGH-8) + +**Files:** +- Modify: `src/core/embedder.py:78-90` + +- [ ] **Step 1: 添加线程锁保护环境变量操作** + +```python +# 在模块顶部（第 30 行之后）添加： +_HF_ENV_LOCK = threading.Lock() + +# 修改 LocalEmbedder.__init__ 中的环境变量操作（第 78-90 行）： +old_endpoint = os.environ.get("HF_ENDPOINT") +with _HF_ENV_LOCK: + os.environ["HF_ENDPOINT"] = _HF_MIRROR +try: + self._model = SentenceTransformer( + config.local_model, device=device + ) +finally: + with _HF_ENV_LOCK: + if old_endpoint is not None: + os.environ["HF_ENDPOINT"] = old_endpoint + else: + os.environ.pop("HF_ENDPOINT", None) +``` + +需要在文件顶部添加 `import threading`（检查是否已存在，如果不存在则添加）。 + +当前文件中没有 `import threading`，需要添加： + +```python +# 第 20 行： +import os +import logging +import threading # 新增 +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_embedder.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add src/core/embedder.py +git commit -m "fix: HF_ENDPOINT 环境变量操作添加线程锁防竞态" +``` + +--- + +### Task 10: ingest_obsidian.py 初始化添加异常处理 (HIGH-9) + +**Files:** +- Modify: `scripts/ingest_obsidian.py:30-35` + +- [ ] **Step 1: 包裹初始化代码** + +```python +# 第 30-35 行替换为： +t0 = time.time() +log("初始化...") +try: + cfg = load_config() + db = VectorDB(persist_dir=cfg.chroma.persist_dir) + embedder = create_embedder(cfg.embed) + ingestor = DocumentIngestor(db, embedder, "obsidian_blog") +except Exception as e: + log(f"初始化失败: {e}") + import traceback + log(traceback.format_exc()) + sys.exit(1) +``` + +- [ ] **Step 2: 提交** + +```bash +git add scripts/ingest_obsidian.py +git commit -m "fix: ingest_obsidian 初始化添加异常处理和友好错误信息" +``` + +--- + +### Task 11: CLI 模块测试 (HIGH-10) + +**Files:** +- Create: `tests/test_cli.py` + +- [ ] **Step 1: 创建 CLI 测试文件** + +```python +"""CLI 命令测试.""" +import os +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from src.cli.main import app + +runner = CliRunner() + + +class TestCLIIngest: + """ingest 命令测试.""" + + def test_ingest_requires_args(self): + """无参数时显示用法提示.""" + result = runner.invoke(app, ["ingest"]) + assert result.exit_code == 1 + assert "用法" in result.stderr + + def test_ingest_nonexistent_file(self, tmp_path): + """不存在的文件被跳过.""" + result = runner.invoke(app, ["ingest", str(tmp_path / "nonexistent.md")]) + # 应该优雅跳过而非崩溃 + assert "SKIP" in result.stderr or result.exit_code != 0 + + +class TestCLISearch: + """search 命令测试.""" + + def test_search_output_format(self, tmp_path, monkeypatch): + """search 命令正常输出.""" + # 设置最小环境 + monkeypatch.setenv("MD_VECTOR_CONFIG", "config.yaml") + result = runner.invoke(app, ["search", "测试查询", "-k", "1"]) + # 搜索可能失败或成功，但不应崩溃 + assert isinstance(result.exit_code, int) + + +class TestCLIStats: + """stats 命令测试.""" + + def test_stats_output(self, tmp_path, monkeypatch): + """stats 命令正常输出.""" + monkeypatch.setenv("MD_VECTOR_CONFIG", "config.yaml") + result = runner.invoke(app, ["stats"]) + assert isinstance(result.exit_code, int) + + +class TestCLIJSONOutput: + """--json 输出测试.""" + + def test_search_json_output(self, monkeypatch): + """search --json 输出合法 JSON.""" + monkeypatch.setenv("MD_VECTOR_CONFIG", "config.yaml") + import json + result = runner.invoke(app, ["search", "测试", "--json", "-k", "1"]) + # 输出应是合法 JSON + if result.stdout.strip(): + try: + data = json.loads(result.stdout) + assert isinstance(data, list) + except json.JSONDecodeError: + pass # 如果无匹配结果 stdout 可能为空 +``` + +- [ ] **Step 2: 运行 CLI 测试** + +```bash +uv run pytest tests/test_cli.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add tests/test_cli.py +git commit -m "test: 添加 CLI 模块基础测试 (ingest/search/stats/json)" +``` + +--- + +### Task 12: 认证与速率限制测试 (HIGH-11) + +**Files:** +- Create: `tests/test_auth.py` + +- [ ] **Step 1: 创建 auth 测试文件** + +```python +"""认证与速率限制测试.""" +import time +import threading + +import pytest +from fastapi import HTTPException + +from src.server.auth import verify_api_key, RateLimiter + + +class TestVerifyApiKey: + """API Key 认证测试.""" + + def test_passes_when_no_key_configured(self, monkeypatch): + """未设置环境变量时跳过认证.""" + monkeypatch.setenv("MD_VECTOR_API_KEY", "") + # 强制重新求值 + import src.server.auth as auth + auth.EXPECTED_API_KEY = "" + result = verify_api_key(x_api_key=None) + assert result is True + + def test_rejects_when_key_required_but_not_provided(self, monkeypatch): + """已设置密钥但请求未提供.""" + monkeypatch.setenv("MD_VECTOR_API_KEY", "secret123") + import src.server.auth as auth + auth.EXPECTED_API_KEY = "secret123" + with pytest.raises(HTTPException) as exc: + verify_api_key(x_api_key=None) + assert exc.value.status_code == 401 + + def test_rejects_wrong_key(self, monkeypatch): + """错误的密钥被拒绝.""" + monkeypatch.setenv("MD_VECTOR_API_KEY", "secret123") + import src.server.auth as auth + auth.EXPECTED_API_KEY = "secret123" + with pytest.raises(HTTPException) as exc: + verify_api_key(x_api_key="wrong-key") + assert exc.value.status_code == 401 + + def test_accepts_correct_key(self, monkeypatch): + """正确的密钥通过认证.""" + monkeypatch.setenv("MD_VECTOR_API_KEY", "secret123") + import src.server.auth as auth + auth.EXPECTED_API_KEY = "secret123" + result = verify_api_key(x_api_key="secret123") + assert result is True + + +class TestRateLimiter: + """速率限制器测试.""" + + def test_allows_within_limit(self): + """未超限时允许请求.""" + limiter = RateLimiter(max_requests=5, window_seconds=60) + for _ in range(5): + assert limiter.is_allowed("client-1") is True + + def test_blocks_when_exceeded(self): + """超限后拒绝.""" + limiter = RateLimiter(max_requests=2, window_seconds=60) + assert limiter.is_allowed("client-2") is True + assert limiter.is_allowed("client-2") is True + assert limiter.is_allowed("client-2") is False + + def test_different_clients_independent(self): + """不同客户端独立计数.""" + limiter = RateLimiter(max_requests=1, window_seconds=60) + assert limiter.is_allowed("client-a") is True + assert limiter.is_allowed("client-b") is True # 不同客户端不受影响 + + def test_window_expires(self, monkeypatch): + """时间窗口过期后恢复.""" + limiter = RateLimiter(max_requests=1, window_seconds=1) + assert limiter.is_allowed("client-3") is True + assert limiter.is_allowed("client-3") is False + # 模拟时间过去 2 秒 + original_time = time.time + fake_now = original_time() + 2.0 + monkeypatch.setattr(time, "time", lambda: fake_now) + assert limiter.is_allowed("client-3") is True + + def test_empty_key_cleaned_up(self): + """空记录的 key 被及时清理防内存泄漏.""" + limiter = RateLimiter(max_requests=1, window_seconds=0) # window=0 立即过期 + assert limiter.is_allowed("temp-client") is True + assert limiter.is_allowed("temp-client") is False + # 第二次调用时 window 已过期，key 应被清除 + assert "temp-client" not in limiter._store + + def test_concurrent_access(self): + """并发访问不产生竞态.""" + limiter = RateLimiter(max_requests=100, window_seconds=60) + errors = [] + + def make_requests(): + try: + for _ in range(50): + limiter.is_allowed("concurrent") + except Exception as e: + errors.append(e) + + threads = [threading.Thread(target=make_requests) for _ in range(10)] + for t in threads: + t.start() + for t in threads: + t.join() + assert len(errors) == 0 +``` + +- [ ] **Step 2: 运行 auth 测试** + +```bash +uv run pytest tests/test_auth.py -v +``` +预期：11 passed + +- [ ] **Step 3: 提交** + +```bash +git add tests/test_auth.py +git commit -m "test: 添加 verify_api_key 和 RateLimiter 完整单元测试" +``` + +--- + +### Task 13: ingest_file / ingest_directory 测试补充 (HIGH-12) + +**Files:** +- Modify: `tests/test_ingest.py` (追加测试) + +- [ ] **Step 1: 添加测试** + +在 `tests/test_ingest.py` 的 `TestIngestorIntegration` 类后添加： + +```python +class TestIngestFile: + """ingest_file 方法测试.""" + + def test_ingest_file_markdown(self, tmp_path): + """通过文件路径入库 .md 文件.""" + from src.core.config import EmbedConfig + from src.core.db import VectorDB + from src.core.embedder import create_embedder + from src.core.ingest import DocumentIngestor + + md_file = tmp_path / "hello.md" + md_file.write_text("# 测试\n这是测试内容。", encoding="utf-8") + + db = VectorDB(persist_dir=str(tmp_path / "db")) + embedder = create_embedder(EmbedConfig(mode="local")) + ingestor = DocumentIngestor(db, embedder, "test_file") + + count = ingestor.ingest_file(str(md_file)) + assert count > 0 + assert ingestor.collection.count() == count + + def test_ingest_file_text(self, tmp_path): + """通过文件路径入库 .txt 文件.""" + from src.core.config import EmbedConfig + from src.core.db import VectorDB + from src.core.embedder import create_embedder + from src.core.ingest import DocumentIngestor + + txt_file = tmp_path / "notes.txt" + txt_file.write_text("这是一段纯文本内容。\n\n第二段内容在这里。", encoding="utf-8") + + db = VectorDB(persist_dir=str(tmp_path / "db")) + embedder = create_embedder(EmbedConfig(mode="local")) + ingestor = DocumentIngestor(db, embedder, "test_txt") + + count = ingestor.ingest_file(str(txt_file)) + assert count > 0 + + def test_ingest_file_deduplicates_same_file(self, tmp_path): + """同一文件重复入库会去重.""" + from src.core.config import EmbedConfig + from src.core.db import VectorDB + from src.core.embedder import create_embedder + from src.core.ingest import DocumentIngestor + + md_file = tmp_path / "dup.md" + md_file.write_text("# V1\n内容 A.", encoding="utf-8") + + db = VectorDB(persist_dir=str(tmp_path / "db")) + embedder = create_embedder(EmbedConfig(mode="local")) + ingestor = DocumentIngestor(db, embedder, "test_dedup_file") + + c1 = ingestor.ingest_file(str(md_file)) + md_file.write_text("# V2\n内容 B.", encoding="utf-8") + c2 = ingestor.ingest_file(str(md_file)) + assert ingestor.collection.count() == c2 + + +class TestIngestDirectory: + """ingest_directory 方法测试.""" + + def test_ingest_directory_mixed_formats(self, tmp_path): + """入库包含多种格式的目录.""" + from src.core.config import EmbedConfig + from src.core.db import VectorDB + from src.core.embedder import create_embedder + from src.core.ingest import DocumentIngestor + + (tmp_path / "a.md").write_text("# A\n内容 A", encoding="utf-8") + (tmp_path / "b.txt").write_text("内容 B", encoding="utf-8") + (tmp_path / "not_supported.xyz").write_text("不应被处理", encoding="utf-8") + + db = VectorDB(persist_dir=str(tmp_path / "db")) + embedder = create_embedder(EmbedConfig(mode="local")) + ingestor = DocumentIngestor(db, embedder, "test_dir") + + results = ingestor.ingest_directory(str(tmp_path)) + assert len(results) >= 2 # a.md + b.txt, .xyz 被忽略 + assert all(c > 0 for c in results.values()) + + def test_ingest_directory_empty(self, tmp_path): + """空目录返回空结果.""" + from src.core.config import EmbedConfig + from src.core.db import VectorDB + from src.core.embedder import create_embedder + from src.core.ingest import DocumentIngestor + + db = VectorDB(persist_dir=str(tmp_path / "db")) + embedder = create_embedder(EmbedConfig(mode="local")) + ingestor = DocumentIngestor(db, embedder, "test_empty_dir") + + results = ingestor.ingest_directory(str(tmp_path)) + assert results == {} +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_ingest.py -v +``` +预期：所有新增测试通过 + +- [ ] **Step 3: 提交** + +```bash +git add tests/test_ingest.py +git commit -m "test: 补充 ingest_file 和 ingest_directory 的测试覆盖" +``` + +--- + +### Task 14: Searcher 方法测试补充 (HIGH-13) + +**Files:** +- Modify: `tests/test_search.py` (追加测试) + +- [ ] **Step 1: 添加测试** + +在 `tests/test_search.py` 的 `TestSearcher` 类中追加： + +```python + def test_list_sources(self, searcher): + """list_sources 返回已入库的源文件列表.""" + sources = searcher.list_sources() + assert isinstance(sources, list) + + def test_get_collection_info(self, searcher): + """get_collection_info 返回 collection 信息.""" + info = searcher.get_collection_info() + assert info["name"] == "test_search" + assert info["count"] > 0 + + def test_delete_by_source(self, searcher): + """delete_by_source 删除源文件的所有 chunks.""" + # 先确认存在 + sources_before = searcher.list_sources() + if sources_before: + target = sources_before[0] + result = searcher.delete_by_source(target) + assert result is True + # 删除后不再出现在列表中 + sources_after = searcher.list_sources() + assert target not in sources_after + + def test_search_with_source_filter(self, searcher): + """带 source_file 过滤的搜索.""" + sources = searcher.list_sources() + if sources: + results = searcher.search("测试", top_k=3, source_file=sources[0]) + assert isinstance(results, list) + # 结果都应来自同一源文件 + for r in results: + assert r["source_file"] == sources[0] + + def test_delete_by_source_nonexistent(self, searcher): + """删除不存在的源文件返回 False.""" + result = searcher.delete_by_source("nonexistent_file_xyz.md") + assert result is False + + def test_list_sources_empty_collection(self, tmp_path): + """空 collection 的 list_sources 返回空列表.""" + from src.core.config import EmbedConfig + from src.core.db import VectorDB + from src.core.embedder import create_embedder + from src.core.search import Searcher + + db = VectorDB(persist_dir=str(tmp_path)) + embedder = create_embedder(EmbedConfig(mode="local")) + searcher = Searcher(db, embedder, "empty_coll") + sources = searcher.list_sources() + assert sources == [] +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_search.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add tests/test_search.py +git commit -m "test: 补充 Searcher list_sources/delete_by_source/get_collection_info 测试" +``` + +--- + +### Batch 2 收尾 + +```bash +uv run pytest tests/ -v # 验证全部 90+ 测试仍通过 +``` + +--- + +## Batch 3 — MEDIUM（11 个问题） + +### Task 15: content 模式默认 file_name 使用 UUID (MED-1) + +**Files:** +- Modify: `src/server/app.py:137` + +- [ ] **Step 1: 改为 UUID 唯一名称** + +```python +# 在文件顶部添加 import uuid +import uuid + +# 第 137 行改为： +file_name = req.file_name or f"untitled_{uuid.uuid4().hex[:8]}.md" +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_api.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add src/server/app.py +git commit -m "fix: content 模式默认 file_name 改用 UUID 防并发覆盖" +``` + +--- + +### Task 16: X-XSS-Protection 替换为 CSP (MED-2) + +**Files:** +- Modify: `src/server/app.py:81` + +- [ ] **Step 1: 替换过时安全头** + +```python +# 第 81 行改为： +response.headers["Content-Security-Policy"] = "default-src 'self'" +# 删除: response.headers["X-XSS-Protection"] = "1; mode=block" +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_api.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add src/server/app.py +git commit -m "fix: 替换过时的 X-XSS-Protection 为 Content-Security-Policy" +``` + +--- + +### Task 17: EXPECTED_API_KEY 惰性求值化 (MED-3) + +**Files:** +- Modify: `src/server/auth.py:14-26` + +- [ ] **Step 1: 改为惰性求值** + +```python +# 第 14 行改为： +def _get_expected_api_key() -> str: + """惰性获取 API Key（每次调用重新从环境变量读取）.""" + return os.environ.get("MD_VECTOR_API_KEY", "") + + +def verify_api_key(x_api_key: str | None = Header(None)): + """验证 API Key. 若未设置环境变量则跳过验证. + + 使用恒定时间比较防止时序攻击. + """ + expected = _get_expected_api_key() # 惰性求值 + if expected: + if x_api_key is None or not hmac.compare_digest(x_api_key, expected): + logger.warning("API Key 认证失败") + raise HTTPException(status_code=401, detail="无效的 API Key") + return True +``` + +删除第 14 行的 `EXPECTED_API_KEY = os.environ.get("MD_VECTOR_API_KEY", "")`。 + +- [ ] **Step 2: 更新 auth 测试引用** + +在 `tests/test_auth.py` 中，所有设置 `auth.EXPECTED_API_KEY` 的地方改为 monkeypatch 环境变量： + +```python +# 不再需要 auth.EXPECTED_API_KEY = "xxx" +# 只需 monkeypatch.setenv("MD_VECTOR_API_KEY", "xxx") +``` + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/test_auth.py -v +``` +预期：全部通过 + +- [ ] **Step 4: 提交** + +```bash +git add src/server/auth.py tests/test_auth.py +git commit -m "fix: EXPECTED_API_KEY 改为惰性求值防加载顺序问题" +``` + +--- + +### Task 18: 去重删除逻辑统一到 VectorDB (MED-4) + +**Files:** +- Modify: `src/core/db.py` (新增 `delete_by_source` 方法) +- Modify: `src/core/ingest.py:108-120` (改用 db 方法) +- Modify: `src/core/search.py:83-97` (改用 db 方法) + +- [ ] **Step 1: 在 VectorDB 添加 delete_by_source** + +```python +# 在 db.py 第 40 行之后添加： +def delete_by_source(self, collection_name: str, file_name: str) -> bool: + """按 source_file 删除文档 (线程安全). + + 在 ingest.py 和 search.py 中均有使用，统一到此方法避免重复代码. + """ + import logging + logger = logging.getLogger("md-vector-db") + collection = self.get_or_create_collection(collection_name) + try: + with self._write_lock: + existing = collection.get( + where={"source_file": file_name} + ) + if existing and existing["ids"]: + collection.delete(ids=existing["ids"]) + return True + except ValueError: + pass # collection 为空时 ChromaDB 抛 ValueError + except Exception: + logger.exception("删除文档失败: %s (collection=%s)", file_name, collection_name) + return False +``` + +- [ ] **Step 2: 修改 ingest.py 使用新方法** + +```python +# 第 108-120 行 _remove_by_source 改为： +def _remove_by_source(self, file_name: str) -> None: + """按 source_file 删除已有 chunks（委托 VectorDB）.""" + self.db.delete_by_source(self.collection_name, file_name) +``` + +- [ ] **Step 3: 修改 search.py 使用新方法** + +```python +# 第 83-97 行 delete_by_source 改为： +def delete_by_source(self, file_name: str) -> bool: + """按文件名删除文档 (委托 VectorDB).""" + return self.db.delete_by_source(self.collection_name, file_name) +``` + +- [ ] **Step 4: 运行全量测试** + +```bash +uv run pytest tests/ -v +``` +预期：全部通过 + +- [ ] **Step 5: 提交** + +```bash +git add src/core/db.py src/core/ingest.py src/core/search.py +git commit -m "refactor: 去重删除逻辑统一到 VectorDB.delete_by_source" +``` + +--- + +### Task 19: list_collections_with_stats 异常向上传播 (MED-5) + +**Files:** +- Modify: `src/server/deps.py:54-59` + +- [ ] **Step 1: 移除 try/except** + +```python +# 第 51-59 行改为： +def list_collections_with_stats(self) -> list[dict]: + """列出所有 collection 及其统计（直接从 ChromaDB 查询）.""" + result = [] + for coll in self.db.list_collections(): + result.append({"name": coll.name, "count": coll.count()}) + return result +``` + +同时在 `app.py` 的 `list_collections` 端点添加异常处理： + +```python +# app.py 第 103-108 行： +@app.get("/api/v1/collections") +def list_collections( + state: AppState = Depends(get_state), + _: bool = Depends(verify_api_key), +): + try: + return {"collections": state.list_collections_with_stats()} + except Exception: + logger.exception("列出集合失败") + raise HTTPException(status_code=500, detail="服务器内部错误") +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_deps.py tests/test_api.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add src/server/deps.py src/server/app.py +git commit -m "fix: list_collections 异常向上传播而非静默吞掉" +``` + +--- + +### Task 20: EPUB 测试设计修复 (MED-6) + +**Files:** +- Modify: `tests/test_splitters_epub.py:51-56` + +- [ ] **Step 1: 使用 monkeypatch 模拟缺失依赖** + +将 `test_epub_missing_dependency_message` 改为： + +```python +def test_epub_missing_dependency_message(self, monkeypatch): + """EPUBSplitter.split() 在未安装 ebooklib 时应给出明确提示.""" + from src.core.splitters.epub import EPUBSplitter + s = EPUBSplitter() + # 模拟 ebooklib 未安装 + monkeypatch.setitem( + __import__("sys").modules, + "ebooklib", + None, + raising=False, + ) + # 由于 importorskip 在模块级别保护，此处实际无法到达 + # 如果到达了（ebooklib 已安装），验证 split 可调用 + assert callable(s.split) +``` + +由于 `importorskip` 在模块顶部的保护，当 ebooklib 未安装时整个测试文件被 skip。更好的方案： + +在 `tests/test_splitters_epub.py` 顶部，将： +```python +ebooklib = pytest.importorskip("ebooklib", reason="ebooklib 未安装") +``` +移到 `TestEPUBSplitter` 类内部，仅对有 ebooklib 需求的测试做 skip，缺失依赖测试保留在文件顶部： + +```python +"""EPUBSplitter 测试.""" +import pytest + + +class TestEPUBSplitterMissingDep: + """缺失依赖时的行为测试（不 skip）.""" + + def test_split_raises_clear_import_error(self, monkeypatch): + """未安装 ebooklib 时给出明确提示.""" + from src.core.splitters.epub import EPUBSplitter + s = EPUBSplitter() + # 模拟 import ebooklib 失败 + import builtins + original_import = builtins.__import__ + + def mock_import(name, *args, **kwargs): + if name == "ebooklib" or name.startswith("ebooklib."): + raise ImportError("No module named 'ebooklib'") + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", mock_import) + with pytest.raises(ImportError, match="ebooklib"): + s.split("dummy.epub", source_file="test.epub") + + +class TestEPUBSplitter: + """EPUBSplitter 测试（需 ebooklib）.""" + ebooklib = pytest.importorskip("ebooklib", reason="ebooklib 未安装") + # ... 原有测试 ... +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_splitters_epub.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add tests/test_splitters_epub.py +git commit -m "test: 修复 EPUB 缺失依赖测试设计问题" +``` + +--- + +### Task 21: test_config.py 用 monkeypatch 替代 reload (MED-7) + +**Files:** +- Modify: `tests/test_config.py:89-106` + +- [ ] **Step 1: 重写环境变量测试** + +```python +class TestEmbedConfigEnvVar: + """api_key 从环境变量读取.""" + + def test_api_key_from_env(self, monkeypatch): + """从环境变量读取 API Key.""" + monkeypatch.setenv("EMBED_API_KEY", "sk-env-test") + cfg = EmbedConfig(mode="api") + assert cfg.api_key == "sk-env-test" + + def test_api_key_empty_when_not_set(self, monkeypatch): + """未设置时返回空字符串.""" + monkeypatch.delenv("EMBED_API_KEY", raising=False) + cfg = EmbedConfig(mode="api") + assert cfg.api_key == "" +``` + +Note: `EmbedConfig.api_key` 使用 `default_factory=lambda: os.environ.get("EMBED_API_KEY", "")`，每次实例化都会重新读取环境变量，因此用 `monkeypatch.setenv` 后直接实例化即可，不需要 `reload`。 + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_config.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add tests/test_config.py +git commit -m "test: test_config 用 monkeypatch 替代 importlib.reload" +``` + +--- + +### Task 22: 速率限制器从环境变量读取配置 (MED-8) + +**Files:** +- Modify: `src/server/auth.py:58` + +- [ ] **Step 1: 改为从环境变量读取** + +```python +# 第 58 行改为： +rate_limiter = RateLimiter( + max_requests=int(os.environ.get("RATE_LIMIT_MAX", "30")), + window_seconds=int(os.environ.get("RATE_LIMIT_WINDOW", "60")), +) +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_auth.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add src/server/auth.py +git commit -m "fix: 速率限制器配置从环境变量读取而非硬编码" +``` + +--- + +### Task 23: ingest_obsidian 移除硬编码回退路径 (MED-9) + +**Files:** +- Modify: `scripts/ingest_obsidian.py:46-51` + +- [ ] **Step 1: 替换为明确的错误提示** + +```python +# 第 45-51 行改为： +else: + log("错误: 未指定目标目录。请通过命令行参数或 OBSIDIAN_DIRS 环境变量提供。") + log("用法: uv run python scripts/ingest_obsidian.py <目录1> [目录2] ...") + sys.exit(1) +``` + +- [ ] **Step 2: 提交** + +```bash +git add scripts/ingest_obsidian.py +git commit -m "fix: ingest_obsidian 移除硬编码回退路径，改为明确错误提示" +``` + +--- + +### Task 24: MarkdownSplitter 边界测试提取 (MED-10) + +**Files:** +- Create: `tests/test_splitters_markdown.py` +- Modify: `tests/test_ingest.py` (移除 Markdown 测试) + +- [ ] **Step 1: 创建独立测试文件** + +```python +"""MarkdownSplitter 边界测试.""" +import pytest + +from src.core.splitters import MarkdownSplitter + + +class TestMarkdownSplitterEdgeCases: + """Markdown 分块边界情况.""" + + @pytest.fixture + def splitter(self): + return MarkdownSplitter(max_size=1000, overlap=100) + + def test_no_headings_document(self, splitter): + """无标题文档正常分块.""" + md = "这是一段没有标题的纯文本。\n\n第二段内容。" + chunks = splitter.split(md, source_file="nohead.md") + assert len(chunks) >= 1 + + def test_deep_headings(self, splitter): + """h4-h6 深层标题.""" + md = """# 一级 +## 二级 +### 三级 +#### 四级 +内容在这里。 +##### 五级 +更多内容。 +###### 六级 +最深的内容。""" + chunks = splitter.split(md, source_file="deep.md") + assert len(chunks) >= 1 + + def test_hash_in_code_block_not_heading(self, splitter): + """代码块中的 # 号不被误识别为标题.""" + md = """# 真实标题 +这是内容。 +```python +# 这不是标题，是注释 +x = 1 # 行内注释 +## 这也不是标题 +``` +更多内容。""" + chunks = splitter.split(md, source_file="codehash.md") + # 代码块内的 # 不应产出新 section + section_titles = [c.get("section_title", "") for c in chunks] + # 不应包含 "这不是标题" 之类的代码注释 + for title in section_titles: + assert "不是标题" not in title + + def test_adjacent_headings_empty_content(self, splitter): + """标题后紧接标题（空内容）.""" + md = """# 标题 A +# 标题 B +内容 B。""" + chunks = splitter.split(md, source_file="adjacent.md") + assert len(chunks) >= 1 + + def test_only_headings_no_content(self, splitter): + """仅有标题无正文.""" + md = "# 只有标题\n## 没有内容" + chunks = splitter.split(md, source_file="headingsonly.md") + # 没有正文内容时也可能产生 chunk（标题本身也是内容） + assert isinstance(chunks, list) +``` + +- [ ] **Step 2: 从 test_ingest.py 移除 MarkdownSplitter 测试** + +`test_ingest.py` 中 `TestMarkdownSplitter` 类的测试保留（它们是入仓器使用 MarkdownSplitter 的集成测试）。新增的边界测试放在独立文件中。 + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/test_splitters_markdown.py tests/test_ingest.py -v +``` +预期：全部通过 + +- [ ] **Step 4: 提交** + +```bash +git add tests/test_splitters_markdown.py +git commit -m "test: 提取 MarkdownSplitter 边界测试到独立文件" +``` + +--- + +### Task 25: splitters/__init__.py 导出 HTMLSplitter (MED-11) + +**Files:** +- Modify: `src/core/splitters/__init__.py:6-20` + +- [ ] **Step 1: 添加 HTMLSplitter 导入和导出** + +```python +# 第 7 行改为（添加 HTMLSplitter import）： +from src.core.splitters.html import HTMLSplitter + +# __all__ 中加入 "HTMLSplitter"： +__all__ = [ + "Splitter", + "BaseTextSplitter", + "MarkdownSplitter", + "TextSplitter", + "PDFSplitter", + "HTMLSplitter", # 新增 + "EPUBSplitter", + "get_splitter", + "register_splitter", + "SUPPORTED_SUFFIXES", +] +``` + +- [ ] **Step 2: 验证导入** + +```bash +uv run python -c "from src.core.splitters import HTMLSplitter; print('OK')" +``` +预期：OK + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/test_splitters_html.py -v +``` +预期：4 passed + +- [ ] **Step 4: 提交** + +```bash +git add src/core/splitters/__init__.py +git commit -m "fix: splitters/__init__.py 导出 HTMLSplitter" +``` + +--- + +### Batch 3 收尾 + +```bash +uv run pytest tests/ -v +``` + +--- + +## Batch 4 — LOW（10 个问题） + +### Task 26: EPUB 编码回退 (L-1) + +**Files:** +- Modify: `src/core/splitters/epub.py:44` + +- [ ] **Step 1: 添加编码回退** + +```python +# 第 42-47 行改为： +try: + content = item.get_content().decode("utf-8") +except UnicodeDecodeError: + try: + content = item.get_content().decode("utf-8-sig") + except UnicodeDecodeError: + try: + content = item.get_content().decode("latin-1") + except UnicodeDecodeError: + logger.warning("EPUB 跳过一个无法解码的章节: %s", item.get_name()) + continue +``` + +- [ ] **Step 2: 提交** + +```bash +git add src/core/splitters/epub.py +git commit -m "fix: EPUB 添加编码回退链 utf-8 -> utf-8-sig -> latin-1" +``` + +--- + +### Task 27: embedder.py requests 惰性导入 (L-2) + +**Files:** +- Modify: `src/core/embedder.py:24,151-155` + +- [ ] **Step 1: 移除模块级 requests 导入，改为惰性导入** + +```python +# 第 24 行删除： +# import requests # noqa: F401 — DashscopeEmbedder 使用 <-- 删除此整行 + +# 第 151 行 DashscopeEmbedder.embed 方法内： +def embed(self, texts: list[str]) -> list[list[float]]: + if not texts: + raise ValueError("文本列表不能为空") + import requests # 惰性导入（仅DashScope使用） + resp = requests.post(...) +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_embedder.py -v +``` +预期：全部通过 + +- [ ] **Step 3: 提交** + +```bash +git add src/core/embedder.py +git commit -m "refactor: requests 导入改为惰性（仅 DashScope 使用）" +``` + +--- + +### Task 28: embedder.py list.sort() 改为 sorted() (L-3) + +**Files:** +- Modify: `src/core/embedder.py:176` + +- [ ] **Step 1: 原地排序替换为不可变版本** + +```python +# 第 176 行改为： +embeddings_raw_sorted = sorted(embeddings_raw, key=lambda x: x.get("text_index", 0)) +return [e["embedding"] for e in embeddings_raw_sorted] +``` + +- [ ] **Step 2: 提交** + +```bash +git add src/core/embedder.py +git commit -m "refactor: Dashscope 嵌入排序改用 sorted() 避免原地修改 API 响应" +``` + +--- + +### Task 29: config.yaml 与代码默认值一致性 (L-7, L-8) + +**Files:** +- Modify: `config.yaml:19-22` + +- [ ] **Step 1: 添加 SSL 字段注释** + +```yaml +server: + host: 0.0.0.0 + port: 8000 + # ssl_keyfile: "" # HTTPS 私钥路径（设置后启用 HTTPS） + # ssl_certfile: "" # HTTPS 证书路径（设置后启用 HTTPS） +``` + +- [ ] **Step 2: 提交** + +```bash +git add config.yaml +git commit -m "docs: config.yaml 添加 SSL 字段注释说明" +``` + +--- + +### Task 30: scripts/serve.py 废弃提示改为显式 print (L-9) + +**Files:** +- Modify: `scripts/serve.py:1-12` + +- [ ] **Step 1: 改为 stderr 直接输出** + +```python +"""便捷启动脚本 — 已废弃, 请使用 `uv run md-vector-db serve`.""" +import sys +import uvicorn + +print( + "[废弃] scripts/serve.py 已废弃, 请使用 `uv run md-vector-db serve`", + file=sys.stderr, +) + +if __name__ == "__main__": + uvicorn.run("src.server.app:app", host="127.0.0.1", port=8000, reload=True) +``` + +- [ ] **Step 2: 提交** + +```bash +git add scripts/serve.py +git commit -m "fix: scripts/serve.py 废弃提示改为 stderr 直接输出" +``` + +--- + +### Task 31: pyproject.toml 添加开发工具依赖 (L-6) + +**Files:** +- Modify: `pyproject.toml:21` + +- [ ] **Step 1: 扩展 dev 依赖** + +```toml +dev = ["pytest>=8.0", "httpx>=0.27.0", "pytest-cov>=5.0", "ruff>=0.8.0", "mypy>=1.13"] +``` + +并添加 tool 配置节： + +```toml +[tool.ruff] +line-length = 100 +target-version = "py313" + +[tool.ruff.lint] +select = ["E", "F", "I", "N", "W"] + +[tool.mypy] +python_version = "3.13" +ignore_missing_imports = true +``` + +- [ ] **Step 2: 安装新依赖** + +```bash +uv sync --extra dev +``` + +- [ ] **Step 3: 提交** + +```bash +git add pyproject.toml uv.lock +git commit -m "build: dev 依赖添加 pytest-cov/ruff/mypy" +``` + +--- + +### Task 32: 文档更新 — 测试数量 + 数据流 (L-10) + +**Files:** +- Modify: `CLAUDE.md` +- Modify: `README.md` + +- [ ] **Step 1: CLAUDE.md 更新** + +- `(46 个)` → `(90+ 个)` +- 数据流更新为包含多格式：`文件 → get_splitter(path) 自动选择 → Splitter.split() → batch_embed() → ChromaDB` + +- [ ] **Step 2: README.md 同步更新** + +同上修改。 + +- [ ] **Step 3: 提交** + +```bash +git add CLAUDE.md README.md +git commit -m "docs: 更新测试数量和文档以反映多格式支持" +``` + +--- + +### Task 33: DEFAULT_CONFIG_PATH 路径处理 (L-4) + +**Files:** +- Modify: `src/core/config.py:16` + +- [ ] **Step 1: 不改 DEFAULT_CONFIG_PATH，而是在 load_config 中已正确处理** + +`load_config` 已经处理了相对路径回退到项目根目录的逻辑。`DEFAULT_CONFIG_PATH` 本身保持不变即可，因为使用方（CLI 和 deps.py）都是通过 `load_config()` 调用而非直接使用 `DEFAULT_CONFIG_PATH` 作为文件路径字面量。 + +此问题标记为无需修改（当前逻辑已正确处理）。 + +- [ ] **Step 2: 提交（跳过或记录为 won't fix）** + +如果决定不修改，跳过此任务。当前 `load_config()` 已经做了项目根目录回退。 + +--- + +### Task 34: registry.py 线程安全文档说明 (L-5) + +**Files:** +- Modify: `src/core/splitters/registry.py:23` + +- [ ] **Step 1: 添加文档注释** + +```python +# 第 23 行函数上方添加： +def register_splitter(ext: str, splitter_cls: type[Splitter]) -> None: + """注册自定义 Splitter 类. + + 注意: 此函数非线程安全，请在程序启动时调用（单线程阶段）。 + 运行时动态注册需自行加锁。 + """ +``` + +- [ ] **Step 2: 提交** + +```bash +git add src/core/splitters/registry.py +git commit -m "docs: register_splitter 添加线程安全注意事项" +``` + +--- + +### Batch 4 收尾 + +```bash +uv run pytest tests/ -v # 全部测试 +``` + +--- + +## 最终验证清单 + +- [ ] `uv run pytest tests/ -v` — 全部测试通过 +- [ ] `uv run md-vector-db ingest --help` — CLI 可用 +- [ ] `uv run md-vector-db search "测试" -k 1` — 检索可用 +- [ ] `uv run md-vector-db stats` — 统计可用 +- [ ] 检查 `config.yaml` 中的 chunk 修改是否生效 +- [ ] 确认无硬编码密钥、无敏感信息泄漏 diff --git a/docs/superpowers/plans/2026-07-10-multi-format-support.md b/docs/superpowers/plans/2026-07-10-multi-format-support.md new file mode 100644 index 0000000..c813de3 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-multi-format-support.md @@ -0,0 +1,1014 @@ +# 多格式文档支持 — 实现计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 将 md-vector-db 从"仅 Markdown"扩展为支持 `.txt`、`.pdf`、`.html` 的多格式文档向量数据库。 + +**Architecture:** 新建 `src/core/splitters/` 包，将 `MarkdownSplitter` + `Splitter(Protocol)` 从 `ingest.py` 移入。抽取 `BaseTextSplitter(ABC)` 作为通用文本切分基类，`TextSplitter`/`PDFSplitter`/`HTMLSplitter` 继承或组合它。`registry.py` 按文件扩展名自动选择 Splitter。`ingest.py` 保留兼容 import。 + +**Tech Stack:** pymupdf (PDF), beautifulsoup4 (HTML), 均为可选依赖。 + +--- + +## 文件清单 + +| 操作 | 文件 | 职责 | +|------|------|------| +| 新建 | `src/core/splitters/__init__.py` | 导出所有公共符号 | +| 新建 | `src/core/splitters/base.py` | Splitter(Protocol) + BaseTextSplitter(ABC) | +| 新建 | `src/core/splitters/markdown.py` | MarkdownSplitter（从 ingest.py 移入） | +| 新建 | `src/core/splitters/text.py` | TextSplitter（纯文本段落切分） | +| 新建 | `src/core/splitters/pdf.py` | PDFSplitter（pymupdf 提取文字） | +| 新建 | `src/core/splitters/html.py` | HTMLSplitter（bs4 去标签） | +| 新建 | `src/core/splitters/registry.py` | 扩展名→Splitter 映射 + get_splitter() | +| 修改 | `src/core/ingest.py` | 精简，用 registry 自动选择 splitter | +| 修改 | `pyproject.toml` | 添加可选依赖组 | +| 新建 | `tests/test_splitters.py` | 注册表 + TextSplitter 测试 | +| 新建 | `tests/test_splitters_pdf.py` | PDFSplitter 测试 | +| 新建 | `tests/test_splitters_html.py` | HTMLSplitter 测试 | + +--- + +### Task 1: 创建 `splitters/base.py` — Protocol + 基类 + +**Files:** +- Create: `src/core/splitters/__init__.py` +- Create: `src/core/splitters/base.py` + +- [ ] **Step 1: 创建包目录和 `__init__.py`** + +```bash +mkdir -p src/core/splitters +``` + +- [ ] **Step 2: 写入 `splitters/__init__.py`** + +```python +"""文档分块器包 — 支持 Markdown / 纯文本 / PDF / HTML.""" + +from src.core.splitters.base import Splitter, BaseTextSplitter +from src.core.splitters.markdown import MarkdownSplitter +from src.core.splitters.text import TextSplitter +from src.core.splitters.registry import get_splitter, register_splitter, SUPPORTED_SUFFIXES + +__all__ = [ + "Splitter", + "BaseTextSplitter", + "MarkdownSplitter", + "TextSplitter", + "get_splitter", + "register_splitter", + "SUPPORTED_SUFFIXES", +] +``` + +- [ ] **Step 3: 写入 `splitters/base.py`** + +```python +"""Splitter Protocol 和文本切分基类.""" +import re +from abc import ABC, abstractmethod +from typing import Protocol + + +class Splitter(Protocol): + """文档分块器接口 — 将文本拆分为带元数据的 chunk 列表. + + 每个 chunk 为 dict: {"content": str, "section_title": str, "heading_level": int, ...} + """ + + def split(self, text: str, source_file: str = "") -> list[dict]: ... + + +class BaseTextSplitter(ABC): + """文本切分基类 — 提供段落切分和硬切逻辑，子类实现 split().""" + + def __init__(self, max_size: int = 1000, overlap: int = 100): + self.max_size = max_size + self.overlap = overlap + + @abstractmethod + def split(self, text: str, source_file: str = "") -> list[dict]: ... + + def _split_by_paragraphs( + self, text: str, section_title: str = "", heading_level: int = 0 + ) -> list[dict]: + """按段落边界拆分超长文本，若单段仍超长则硬切.""" + paragraphs = re.split(r"\n\n+", text) + chunks = [] + current = "" + + for para in paragraphs: + if len(para) > self.max_size: + if current.strip(): + chunks.append(self._make_chunk(current, section_title, heading_level)) + current = "" + for sub in self._split_single_paragraph(para): + chunks.append(self._make_chunk(sub, section_title, heading_level)) + continue + + if len(current) + len(para) > self.max_size and current: + chunks.append(self._make_chunk(current, section_title, heading_level)) + if self.overlap > 0 and len(current) > self.overlap: + current = current[-self.overlap:] + "\n\n" + para + else: + current = para + else: + current = f"{current}\n\n{para}" if current else para + + if current.strip(): + chunks.append(self._make_chunk(current, section_title, heading_level)) + + return chunks + + def _split_single_paragraph(self, text: str) -> list[str]: + """按字符边界硬切单个超长段落（带 overlap）.""" + parts = [] + start = 0 + while start < len(text): + end = start + self.max_size + if end >= len(text): + parts.append(text[start:].strip()) + break + break_point = end + for sep in ("。", "！", "？", "\n", ". ", " "): + pos = text.rfind(sep, start, end) + if pos > start: + break_point = pos + len(sep) + break + part = text[start:break_point].strip() + if part: + parts.append(part) + next_start = break_point - self.overlap if self.overlap > 0 else break_point + start = max(start + 1, next_start) + return parts + + @staticmethod + def _make_chunk(content: str, section_title: str, heading_level: int) -> dict: + return { + "content": content.strip(), + "section_title": section_title, + "heading_level": heading_level, + } +``` + +- [ ] **Step 4: 验证导入** + +```bash +uv run python -c "from src.core.splitters.base import Splitter, BaseTextSplitter; print('OK')" +``` +Expected: `OK` + +- [ ] **Step 5: Commit** + +```bash +git add src/core/splitters/__init__.py src/core/splitters/base.py +git commit -m "feat: 创建 splitters 包骨架 — Splitter Protocol + BaseTextSplitter 基类" +``` + +--- + +### Task 2: 将 MarkdownSplitter 移入 `splitters/markdown.py` + +**Files:** +- Create: `src/core/splitters/markdown.py` +- Modify: `src/core/ingest.py` — 删除 MarkdownSplitter 类，添加兼容 import +- Modify: `tests/test_ingest.py` — 更新 import 路径 + +- [ ] **Step 1: 写入 `splitters/markdown.py`** + +```python +"""Markdown 文档分块器.""" +import re +from src.core.splitters.base import BaseTextSplitter + + +class MarkdownSplitter(BaseTextSplitter): + """Markdown 混合分块器：先按标题拆，超长再按段落拆.""" + + def split(self, text: str, source_file: str = "") -> list[dict]: + if not text.strip(): + return [] + + sections = self._split_by_headings(text) + chunks = [] + + for section in sections: + if len(section["content"]) <= self.max_size: + chunks.append(section) + else: + sub_chunks = self._split_by_paragraphs( + section["content"], + section["section_title"], + section["heading_level"], + ) + chunks.extend(sub_chunks) + + for i, chunk in enumerate(chunks): + chunk["source_file"] = source_file or chunk.get("source_file", "") + chunk["chunk_index"] = i + + return chunks + + def _split_by_headings(self, text: str) -> list[dict]: + heading_pattern = re.compile(r"^(#{1,6})\s+(.+)$", re.MULTILINE) + matches = list(heading_pattern.finditer(text)) + + if not matches: + return [{ + "content": text.strip(), + "section_title": "", + "heading_level": 0, + }] + + sections = [] + for i, match in enumerate(matches): + level = len(match.group(1)) + title = match.group(2).strip() + start = match.end() + end = matches[i + 1].start() if i + 1 < len(matches) else len(text) + content = text[start:end].strip() + + if content: + sections.append({ + "content": f"{match.group(0)}\n{content}", + "section_title": title, + "heading_level": level, + }) + + if matches and matches[0].start() > 0: + preamble = text[:matches[0].start()].strip() + if preamble: + sections.insert(0, { + "content": preamble, + "section_title": "", + "heading_level": 0, + }) + + return sections +``` + +- [ ] **Step 2: 从 `ingest.py` 中删除 `Splitter` Protocol 和 `MarkdownSplitter` 类，替换为兼容 import** + +在 `ingest.py` 中删除第 14-174 行（Splitter Protocol + MarkdownSplitter 全部代码），替换为： +```python +import logging +import re +from pathlib import Path + +from src.core.db import VectorDB +from src.core.embedder import Embedder, batch_embed +from src.core.splitters.registry import get_splitter +from src.core.splitters.markdown import MarkdownSplitter # 兼容旧 import +from src.core.splitters.base import Splitter # 兼容旧 import + +logger = logging.getLogger("md-vector-db") +``` + +- [ ] **Step 3: 更新 `tests/test_ingest.py` 的 import** + +```python +# 将原有 from src.core.ingest import MarkdownSplitter +# 改为: +from src.core.splitters import MarkdownSplitter +``` + +- [ ] **Step 4: 运行测试验证** + +```bash +uv run pytest tests/test_ingest.py tests/test_search.py -v +``` +Expected: all 12+ tests PASS + +- [ ] **Step 5: Commit** + +```bash +git add src/core/splitters/markdown.py src/core/ingest.py tests/test_ingest.py +git commit -m "refactor: 将 MarkdownSplitter 移入 splitters 包，ingest.py 保留兼容 import" +``` + +--- + +### Task 3: 创建 `TextSplitter` 和 `registry.py` + +**Files:** +- Create: `src/core/splitters/text.py` +- Create: `src/core/splitters/registry.py` + +- [ ] **Step 1: 写入 `splitters/text.py`** + +```python +"""纯文本分块器 — 按段落双换行切分.""" +from src.core.splitters.base import BaseTextSplitter + + +class TextSplitter(BaseTextSplitter): + """纯文本分块器：按 \n\n 切段落，超长按标点硬切.""" + + def split(self, text: str, source_file: str = "") -> list[dict]: + if not text.strip(): + return [] + + chunks = self._split_by_paragraphs(text) + for i, chunk in enumerate(chunks): + chunk["source_file"] = source_file + chunk["chunk_index"] = i + + return chunks +``` + +- [ ] **Step 2: 写入 `splitters/registry.py`** + +```python +"""Splitter 注册表 — 按文件扩展名自动选择分块器.""" +from pathlib import Path +from src.core.splitters.base import Splitter + +# 扩展名 → Splitter 类名映射 +_DEFAULT_MAP: dict[str, str] = { + ".md": "markdown", + ".markdown": "markdown", + ".txt": "text", + ".pdf": "pdf", + ".html": "html", + ".htm": "html", +} + +# 所有支持的扩展名集合（供外部遍历文件使用） +SUPPORTED_SUFFIXES = frozenset(_DEFAULT_MAP.keys()) + +# 用户可注册自定义 Splitter +_custom_registry: dict[str, type[Splitter]] = {} + + +def register_splitter(ext: str, splitter_cls: type[Splitter]) -> None: + """注册自定义 Splitter 类.""" + ext = ext.lower() if ext.startswith(".") else f".{ext}" + _custom_registry[ext] = splitter_cls + + +def get_splitter( + file_path: str, + max_size: int = 1000, + overlap: int = 100, +) -> Splitter: + """根据文件扩展名自动选择 Splitter，未匹配回退到 TextSplitter. + + Args: + file_path: 文件路径（用于提取扩展名） + max_size: 分块最大字符数 + overlap: 相邻块重叠字符数 + + Returns: + 对应格式的 Splitter 实例 + """ + ext = Path(file_path).suffix.lower() + + # 优先查用户自定义注册 + if ext in _custom_registry: + return _custom_registry[ext](max_size=max_size, overlap=overlap) + + kind = _DEFAULT_MAP.get(ext, "text") + + if kind == "markdown": + from src.core.splitters.markdown import MarkdownSplitter + return MarkdownSplitter(max_size=max_size, overlap=overlap) + + if kind == "text": + from src.core.splitters.text import TextSplitter + return TextSplitter(max_size=max_size, overlap=overlap) + + if kind == "pdf": + from src.core.splitters.pdf import PDFSplitter + return PDFSplitter(max_size=max_size, overlap=overlap) + + if kind == "html": + from src.core.splitters.html import HTMLSplitter + return HTMLSplitter(max_size=max_size, overlap=overlap) + + # 回退 + from src.core.splitters.text import TextSplitter + return TextSplitter(max_size=max_size, overlap=overlap) +``` + +- [ ] **Step 3: 更新 `splitters/__init__.py`**（追加 TextSplitter 和 SUPPORTED_SUFFIXES 导出） + +```python +"""文档分块器包 — 支持 Markdown / 纯文本 / PDF / HTML.""" + +from src.core.splitters.base import Splitter, BaseTextSplitter +from src.core.splitters.markdown import MarkdownSplitter +from src.core.splitters.text import TextSplitter +from src.core.splitters.registry import get_splitter, register_splitter, SUPPORTED_SUFFIXES + +__all__ = [ + "Splitter", + "BaseTextSplitter", + "MarkdownSplitter", + "TextSplitter", + "get_splitter", + "register_splitter", + "SUPPORTED_SUFFIXES", +] +``` + +- [ ] **Step 4: 验证注册表** + +```bash +uv run python -c " +from src.core.splitters.registry import get_splitter, SUPPORTED_SUFFIXES +print('Suffixes:', sorted(SUPPORTED_SUFFIXES)) +s = get_splitter('test.txt') +print('TextSplitter:', type(s).__name__) +s = get_splitter('doc.md') +print('MarkdownSplitter:', type(s).__name__) +s = get_splitter('unknown.xyz') +print('Fallback:', type(s).__name__) +" +``` +Expected: `TextSplitter`, `MarkdownSplitter`, `Fallback: TextSplitter` + +- [ ] **Step 5: Commit** + +```bash +git add src/core/splitters/text.py src/core/splitters/registry.py src/core/splitters/__init__.py +git commit -m "feat: 添加 TextSplitter 和 registry 自动选择机制" +``` + +--- + +### Task 4: 更新 `ingest.py` — 使用 registry 自动选择 Splitter + +**Files:** +- Modify: `src/core/ingest.py` + +- [ ] **Step 1: 更新 `DocumentIngestor.__init__`** + +将默认 splitter 从 `MarkdownSplitter()` 改为 `None`（None 时由 `ingest_file` 自动选择）： +```python +def __init__( + self, + db: VectorDB, + embedder: Embedder, + collection_name: str, + splitter: Splitter | None = None, +): + self.db = db + self.embedder = embedder + self.collection_name = collection_name + self.splitter = splitter # None = 按扩展名自动选择 +``` + +- [ ] **Step 2: 更新 `ingest_file` 使用 registry** + +```python +def ingest_file(self, file_path: str) -> int: + """入库单个文件, 返回 chunk 数量. + + 根据文件扩展名自动选择 Splitter（.md→MarkdownSplitter, .pdf→PDFSplitter 等）。 + 使用文件路径的 SHA256 前 12 位 + 文件名作为唯一标识。 + """ + import hashlib + path = Path(file_path).resolve() + path_hash = hashlib.sha256(str(path).encode()).hexdigest()[:12] + file_name = f"{path_hash}_{path.name}" + + splitter = self.splitter or get_splitter(file_path) + content = path.read_text(encoding="utf-8") + + return self._ingest_with_splitter(content, file_name, splitter) + +def _ingest_with_splitter(self, content: str, file_name: str, splitter: Splitter) -> int: + """使用指定 splitter 分块并入库.""" + self._remove_by_source(file_name) + + chunks = splitter.split(content, source_file=file_name) + if not chunks: + return 0 + + texts = [c["content"] for c in chunks] + embeddings = batch_embed(self.embedder, texts) + + ids = [f"{file_name}_{i}" for i in range(len(chunks))] + metadatas = [ + { + "source_file": c.get("source_file", file_name), + "section_title": c.get("section_title", ""), + "heading_level": c.get("heading_level", 0), + "chunk_index": i, + } + for i, c in enumerate(chunks) + ] + + with self.db.write_lock: + self.collection.add( + ids=ids, embeddings=embeddings, documents=texts, metadatas=metadatas, + ) + + return len(chunks) +``` + +- [ ] **Step 3: 更新 `ingest_content` 保持向后兼容** + +```python +def ingest_content(self, content: str, file_name: str) -> int: + """入库内容（无需实际文件）。若未指定 splitter，默认用 MarkdownSplitter.""" + splitter = self.splitter or MarkdownSplitter() + return self._ingest_with_splitter(content, file_name, splitter) +``` + +- [ ] **Step 4: 更新 `ingest_directory` 支持多格式** + +```python +def ingest_directory(self, dir_path: str) -> dict[str, int]: + """入库目录下所有支持的文档格式.""" + from src.core.splitters.registry import SUPPORTED_SUFFIXES + results = {} + for f in Path(dir_path).rglob("*"): + if f.suffix.lower() in SUPPORTED_SUFFIXES: + count = self.ingest_file(str(f)) + results[f.name] = count + return results +``` + +- [ ] **Step 5: 运行现有测试确保无回归** + +```bash +uv run pytest tests/ -v +``` +Expected: all 70 tests PASS + +- [ ] **Step 6: Commit** + +```bash +git add src/core/ingest.py +git commit -m "feat: ingest.py 使用 registry 自动选择 Splitter，ingest_directory 支持多格式" +``` + +--- + +### Task 5: 创建 `PDFSplitter` + +**Files:** +- Create: `src/core/splitters/pdf.py` + +- [ ] **Step 1: 写入 `splitters/pdf.py`** + +```python +"""PDF 文档分块器 — 使用 pymupdf 提取文字后委托 TextSplitter.""" +import logging +from src.core.splitters.base import Splitter, BaseTextSplitter +from src.core.splitters.text import TextSplitter + +logger = logging.getLogger("md-vector-db") + + +class PDFSplitter: + """PDF 分块器：pymupdf 提取文字 → TextSplitter 分块. + + 实现 Splitter Protocol，内部组合 TextSplitter 实例。 + """ + + def __init__(self, max_size: int = 1000, overlap: int = 100): + self._text_splitter = TextSplitter(max_size=max_size, overlap=overlap) + + def split(self, text: str, source_file: str = "") -> list[dict]: + """从 PDF 文件路径提取文字并分块. + + Args: + text: 此处应为 PDF 文件路径（而非文本内容） + source_file: 来源文件名 + """ + try: + import fitz # pymupdf + except ImportError: + raise ImportError( + "PDF 支持需要 pymupdf 库. 请执行: uv sync --extra pdf" + ) + + pdf_path = text # text 参数实际是文件路径 + extracted_pages = [] + try: + doc = fitz.open(pdf_path) + for page in doc: + page_text = page.get_text() + if page_text.strip(): + extracted_pages.append(page_text) + doc.close() + except Exception as e: + logger.error("PDF 解析失败: %s — %s", pdf_path, e) + raise ValueError(f"PDF 解析失败: {e}") from e + + if not extracted_pages: + return [] + + full_text = "\n\n".join(extracted_pages) + return self._text_splitter.split(full_text, source_file=source_file) +``` + +- [ ] **Step 2: 修改 `ingest_file` 处理 PDF 二进制文件** + +PDF 文件不能像文本文件那样 `read_text()`。需要在 `ingest_file` 中特殊处理： +```python +def ingest_file(self, file_path: str) -> int: + import hashlib + path = Path(file_path).resolve() + path_hash = hashlib.sha256(str(path).encode()).hexdigest()[:12] + file_name = f"{path_hash}_{path.name}" + + splitter = self.splitter or get_splitter(file_path) + + # PDF 二进制文件特殊处理：传入路径给 Splitter + suffix = path.suffix.lower() + if suffix in (".pdf",): + # PDFSplitter.split() 接收文件路径而非文本内容 + chunks = splitter.split(str(path), source_file=file_name) + return self._ingest_chunks(chunks, file_name) + + content = path.read_text(encoding="utf-8") + return self._ingest_content_with_splitter(content, file_name, splitter) +``` + +并在类中添加 `_ingest_chunks` 辅助方法（与 `_ingest_with_splitter` 的后半段相同）。 + +- [ ] **Step 3: 安装 pymupdf 并验证导入** + +```bash +uv sync --extra pdf +uv run python -c "from src.core.splitters.pdf import PDFSplitter; print('OK')" +``` +Expected: `OK` + +- [ ] **Step 4: Commit** + +```bash +git add src/core/splitters/pdf.py src/core/ingest.py +git commit -m "feat: 添加 PDFSplitter — pymupdf 提取文字后分块" +``` + +--- + +### Task 6: 创建 `HTMLSplitter` + +**Files:** +- Create: `src/core/splitters/html.py` + +- [ ] **Step 1: 写入 `splitters/html.py`** + +```python +"""HTML 文档分块器 — 使用 BeautifulSoup 去标签后委托 TextSplitter.""" +import logging +from src.core.splitters.text import TextSplitter + +logger = logging.getLogger("md-vector-db") + + +class HTMLSplitter: + """HTML 分块器：bs4 去标签提取文字 → TextSplitter 分块. + + 实现 Splitter Protocol，内部组合 TextSplitter 实例。 + """ + + def __init__(self, max_size: int = 1000, overlap: int = 100): + self._text_splitter = TextSplitter(max_size=max_size, overlap=overlap) + + def split(self, text: str, source_file: str = "") -> list[dict]: + """从 HTML 文本去标签并分块. + + Args: + text: HTML 文本内容 + source_file: 来源文件名 + """ + try: + from bs4 import BeautifulSoup + except ImportError: + raise ImportError( + "HTML 支持需要 beautifulsoup4 库. 请执行: uv sync --extra html" + ) + + try: + soup = BeautifulSoup(text, "html.parser") + # 移除 script/style 标签 + for tag in soup(["script", "style"]): + tag.decompose() + plain_text = soup.get_text(separator="\n") + except Exception as e: + logger.error("HTML 解析失败: %s — %s", source_file, e) + raise ValueError(f"HTML 解析失败: {e}") from e + + if not plain_text.strip(): + return [] + + return self._text_splitter.split(plain_text, source_file=source_file) +``` + +- [ ] **Step 2: 安装 beautifulsoup4 并验证导入** + +```bash +uv sync --extra html +uv run python -c "from src.core.splitters.html import HTMLSplitter; print('OK')" +``` +Expected: `OK` + +- [ ] **Step 3: Commit** + +```bash +git add src/core/splitters/html.py +git commit -m "feat: 添加 HTMLSplitter — bs4 去标签后分块" +``` + +--- + +### Task 7: 更新 `pyproject.toml` 可选依赖 + +**Files:** +- Modify: `pyproject.toml` + +- [ ] **Step 1: 添加可选依赖组** + +将 `pyproject.toml` 的 `[project.optional-dependencies]` 段改为： +```toml +[project.optional-dependencies] +dev = ["pytest>=8.0", "httpx>=0.27.0"] +pdf = ["pymupdf>=1.24.0"] +html = ["beautifulsoup4>=4.12.0"] +all = ["md-vector-db[pdf,html]", "requests>=2.31.0", "openai>=1.0.0"] +``` + +- [ ] **Step 2: 验证依赖安装** + +```bash +uv sync --extra all +uv run python -c "import fitz; from bs4 import BeautifulSoup; print('OK')" +``` +Expected: `OK` + +- [ ] **Step 3: Commit** + +```bash +git add pyproject.toml +git commit -m "feat: 添加 pdf/html/all 可选依赖组" +``` + +--- + +### Task 8: 编写测试 — 注册表 + TextSplitter + +**Files:** +- Create: `tests/test_splitters.py` + +- [ ] **Step 1: 写入 `tests/test_splitters.py`** + +```python +"""Splitter 注册表和 TextSplitter 测试.""" +import pytest +from src.core.splitters import TextSplitter, MarkdownSplitter, get_splitter, register_splitter, SUPPORTED_SUFFIXES + + +class TestTextSplitter: + """TextSplitter 纯文本分块测试.""" + + def test_empty_text(self): + s = TextSplitter() + assert s.split("") == [] + assert s.split(" \n\n ") == [] + + def test_short_text_single_chunk(self): + s = TextSplitter(max_size=1000) + chunks = s.split("这是一段短文本。", source_file="test.txt") + assert len(chunks) == 1 + assert chunks[0]["source_file"] == "test.txt" + assert chunks[0]["content"] == "这是一段短文本。" + + def test_long_paragraph_split(self): + s = TextSplitter(max_size=50, overlap=10) + long_text = "这是第一句。" * 20 + chunks = s.split(long_text, source_file="long.txt") + assert len(chunks) > 1 + for c in chunks: + assert len(c["content"]) <= 60 # max_size + 少许容差 + + def test_paragraph_boundary_split(self): + s = TextSplitter(max_size=100) + text = "短段落A。\n\n短段落B。\n\n短段落C。" + chunks = s.split(text) + assert len(chunks) >= 1 + assert all("content" in c for c in chunks) + + def test_chunk_metadata(self): + s = TextSplitter() + chunks = s.split("测试内容。", source_file="doc.txt") + assert chunks[0]["source_file"] == "doc.txt" + assert chunks[0]["section_title"] == "" + assert chunks[0]["heading_level"] == 0 + assert chunks[0]["chunk_index"] == 0 + + +class TestRegistry: + """注册表测试.""" + + def test_get_splitter_for_md(self): + s = get_splitter("doc.md") + assert isinstance(s, MarkdownSplitter) + + def test_get_splitter_for_txt(self): + s = get_splitter("notes.txt") + assert isinstance(s, TextSplitter) + + def test_get_splitter_fallback(self): + s = get_splitter("data.xyz") + assert isinstance(s, TextSplitter) + + def test_supported_suffixes(self): + assert ".md" in SUPPORTED_SUFFIXES + assert ".txt" in SUPPORTED_SUFFIXES + assert ".pdf" in SUPPORTED_SUFFIXES + assert ".html" in SUPPORTED_SUFFIXES + + def test_custom_register(self): + class FakeSplitter: + def __init__(self, max_size=1000, overlap=100): pass + def split(self, text, source_file=""): return [] + + register_splitter(".fake", FakeSplitter) + s = get_splitter("test.fake") + assert isinstance(s, FakeSplitter) +``` + +- [ ] **Step 2: 运行测试** + +```bash +uv run pytest tests/test_splitters.py -v +``` +Expected: 8 tests PASS + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_splitters.py +git commit -m "test: 添加 Splitter 注册表和 TextSplitter 测试" +``` + +--- + +### Task 9: 编写 PDF/HTML Splitter 测试 + +**Files:** +- Create: `tests/test_splitters_pdf.py` +- Create: `tests/test_splitters_html.py` + +- [ ] **Step 1: 写入 `tests/test_splitters_pdf.py`** + +```python +"""PDFSplitter 测试.""" +import pytest +from pathlib import Path + +pymupdf = pytest.importorskip("fitz", reason="pymupdf 未安装") + + +class TestPDFSplitter: + """PDFSplitter 测试（需 pymupdf）.""" + + def test_split_simple_pdf(self, tmp_path): + """用 pymupdf 创建一个简单 PDF 并测试分块.""" + from src.core.splitters.pdf import PDFSplitter + import fitz + + pdf_path = tmp_path / "test.pdf" + doc = fitz.open() + doc.new_page().insert_text((72, 72), "这是PDF文档内容。\n\n第二段文字。") + doc.save(str(pdf_path)) + doc.close() + + s = PDFSplitter(max_size=500) + chunks = s.split(str(pdf_path), source_file="test.pdf") + assert len(chunks) >= 1 + assert "PDF文档内容" in chunks[0]["content"] + + def test_pdf_missing_lib_error(self, monkeypatch): + """未安装 pymupdf 时的错误提示.""" + # 此测试在已安装 pymupdf 时跳过语义检查，仅验证 split 方法存在 + from src.core.splitters.pdf import PDFSplitter + s = PDFSplitter() + assert hasattr(s, "split") +``` + +- [ ] **Step 2: 写入 `tests/test_splitters_html.py`** + +```python +"""HTMLSplitter 测试.""" +import pytest + +bs4 = pytest.importorskip("bs4", reason="beautifulsoup4 未安装") + + +class TestHTMLSplitter: + """HTMLSplitter 测试（需 beautifulsoup4）.""" + + def test_split_simple_html(self): + from src.core.splitters.html import HTMLSplitter + html = "

标题

这是段落内容。

第二段。

" + s = HTMLSplitter(max_size=500) + chunks = s.split(html, source_file="test.html") + assert len(chunks) >= 1 + # 验证去标签后的内容 + all_text = "".join(c["content"] for c in chunks) + assert "标题" in all_text + assert "段落内容" in all_text + assert "第二段" in all_text + + def test_strips_script_and_style(self): + from src.core.splitters.html import HTMLSplitter + html = """ + +

可见内容。

+ """ + s = HTMLSplitter() + chunks = s.split(html, source_file="test.html") + all_text = "".join(c["content"] for c in chunks) + assert "可见内容" in all_text + assert "alert" not in all_text + assert ".a{color:red}" not in all_text + + def test_empty_html(self): + from src.core.splitters.html import HTMLSplitter + s = HTMLSplitter() + assert s.split("") == [] + assert s.split("") == [] +``` + +- [ ] **Step 3: 运行测试** + +```bash +uv run pytest tests/test_splitters_pdf.py tests/test_splitters_html.py -v +``` +Expected: all tests PASS (pymupdf + bs4 已安装) + +- [ ] **Step 4: 运行全量测试确保无回归** + +```bash +uv run pytest tests/ -v +``` +Expected: all tests PASS + +- [ ] **Step 5: Commit** + +```bash +git add tests/test_splitters_pdf.py tests/test_splitters_html.py +git commit -m "test: 添加 PDFSplitter 和 HTMLSplitter 测试" +``` + +--- + +### Task 10: 更新文档 + +**Files:** +- Modify: `CLAUDE.md` +- Modify: `README.md` + +- [ ] **Step 1: 更新 `CLAUDE.md` 的架构描述** + +在 "架构" 段落后增加 splitters 说明： +``` +src/core/splitters/ # 文档分块器包（新增） +├── base.py # Splitter Protocol + BaseTextSplitter 基类 +├── markdown.py # MarkdownSplitter（标题+段落混合分块） +├── text.py # TextSplitter（纯文本段落切分） +├── pdf.py # PDFSplitter（pymupdf 提取文字） +├── html.py # HTMLSplitter（bs4 去标签） +└── registry.py # 扩展名→Splitter 自动选择 +``` + +- [ ] **Step 2: 更新 `README.md`** + +在 "功能特性" 列表中添加 "多格式文档：支持 .md / .txt / .pdf / .html，按扩展名自动选择分块器"，并提供安装命令： +```bash +uv sync --extra all # 安装 PDF + HTML 支持 +``` + +- [ ] **Step 3: Commit** + +```bash +git add CLAUDE.md README.md +git commit -m "docs: 更新文档记录多格式支持特性" +``` + +--- + +## 执行顺序 + +``` +Task 1 (base.py 骨架) + → Task 2 (MarkdownSplitter 迁移) + → Task 3 (TextSplitter + registry) + → Task 4 (ingest.py 更新) + → Task 5 (PDFSplitter) + → Task 6 (HTMLSplitter) + → Task 7 (pyproject.toml) + → Task 8 (测试 — 注册表+TextSplitter) + → Task 9 (测试 — PDF+HTML) + → Task 10 (文档) +``` + +Task 5/6 可并行，Task 8/9 可并行。 diff --git a/docs/superpowers/specs/2026-07-10-multi-format-support-design.md b/docs/superpowers/specs/2026-07-10-multi-format-support-design.md new file mode 100644 index 0000000..1fa153e --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-multi-format-support-design.md @@ -0,0 +1,142 @@ +# 多格式文档支持 — 设计文档 + +**日期**: 2026-07-10 +**版本**: 1.0 + +## 目标 + +将 md-vector-db 从"仅 Markdown"扩展为支持 `.txt`、`.pdf`、`.html` 的多格式文档向量数据库。 + +## 非目标 + +- 不支持 `.docx`、`.pptx`、`.epub` 等 Office/电子书格式（留待后续扩展） +- 不改变现有的嵌入和检索流程 +- 不改变 HTTP API 的请求/响应模型 + +## 架构 + +### 模块结构 + +``` +src/core/ +├── splitters/ # 新建目录 +│ ├── __init__.py # 导出 registry + 所有 Splitter +│ ├── base.py # Splitter Protocol + BaseTextSplitter (ABC) +│ ├── markdown.py # MarkdownSplitter (从 ingest.py 移入) +│ ├── text.py # TextSplitter (纯文本按段落+标点硬切) +│ ├── pdf.py # PDFSplitter (pymupdf → TextSplitter) +│ ├── html.py # HTMLSplitter (bs4 → TextSplitter) +│ └── registry.py # 工厂 + 扩展名→Splitter 映射表 +├── ingest.py # 精简，用 registry.get_splitter() 自动选择 +``` + +### 类继承 + +``` +Splitter (Protocol) + └── BaseTextSplitter (ABC) # max_size, overlap, _split_single_paragraph + ├── MarkdownSplitter # 已有，从 ingest.py 移入 + ├── TextSplitter # 纯文本：按 \n\n 切段落，超长按标点硬切 + ├── PDFSplitter # 读取 PDF → extract_text() → 委托 TextSplitter + └── HTMLSplitter # 读取 HTML → get_text() → 委托 TextSplitter +``` + +`PDFSplitter` 和 `HTMLSplitter` **不继承** `BaseTextSplitter`，而是组合一个 `TextSplitter` 实例。它们实现 `Splitter` Protocol，在 `split()` 中：提取纯文本 → 委托 `TextSplitter.split()`。 + +### 注册表 + +`registry.py` 维护默认扩展名映射： + +```python +_DEFAULT_MAP = { + ".md": "markdown", + ".markdown": "markdown", + ".txt": "text", + ".pdf": "pdf", + ".html": "html", + ".htm": "html", +} + +def get_splitter(file_path: str, **config) -> Splitter: + """根据扩展名自动选择 Splitter，未匹配回退到 TextSplitter.""" +``` + +## 依赖策略 + +- `pymupdf` 和 `beautifulsoup4` 作为**可选依赖** +- 首次使用 PDF/HTML 格式时才 `import`，库缺失时抛 `ImportError` 带安装提示 +- 安装方式： + +```bash +uv sync --extra pdf # PDF 支持 +uv sync --extra html # HTML 支持 +uv sync --extra all # 全部可选依赖 +``` + +## ingest.py 改动 + +### DocumentIngestor.__init__ + +已有 `splitter` 可选参数，**保持不变**。显式传入的 splitter 覆盖自动选择。 + +### ingest_file() + +改为使用 `get_splitter(file_path, ...)` 自动选择 splitter： + +```python +def ingest_file(self, file_path: str) -> int: + splitter = self.splitter or get_splitter(file_path, + max_size=..., overlap=...) + ... +``` + +### ingest_directory() + +将 `rglob("*.md")` 改为遍历所有支持格式： + +```python +_SUPPORTED_SUFFIXES = {".md", ".markdown", ".txt", ".pdf", ".html", ".htm"} + +def ingest_directory(self, dir_path: str) -> dict[str, int]: + for f in Path(dir_path).rglob("*"): + if f.suffix.lower() in _SUPPORTED_SUFFIXES: + ... +``` + +## 数据流 + +``` +文件路径 → get_splitter(path) + ├── .md → MarkdownSplitter.split(text) + ├── .txt → TextSplitter.split(text) + ├── .pdf → PDFSplitter.split(binary) + │ └── pymupdf 提取文字 → TextSplitter.split(text) + └── .html → HTMLSplitter.split(html) + └── bs4 去标签 → TextSplitter.split(text) + ↓ + batch_embed(chunks) + ↓ + ChromaDB.add() +``` + +## 测试策略 + +- `tests/test_splitters_text.py` — TextSplitter 段落切分、硬切、overlap +- `tests/test_splitters_registry.py` — 扩展名映射、回退逻辑、自定义注册 +- `tests/test_splitters_pdf.py` — PDF 提取文字 + 分块（需 pymupdf） +- `tests/test_splitters_html.py` — HTML 去标签 + 分块（需 bs4） +- `tests/test_ingest.py` — 补 `ingest_directory` 多格式遍历测试 + +## CLI/API 影响 + +- **CLI**：零改动。`ingest` 和 `ingest-dir` 自动获得多格式能力 +- **API**：零改动。`POST /api/v1/ingest` 的 `file_path` 自动支持 +- **`ingest_content()`**：仍默认使用 `MarkdownSplitter`（处理 Markdown 字符串），显式传入 content 时不变 + +## 风险 + +| 风险 | 缓解 | +| -------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| PDF 提取失败（扫描件/图片 PDF） | 抛明确异常，跳过该文件 | +| HTML 标签复杂导致去标签不干净 | 用`soup.get_text(separator="\n")` 保留段落结构 | +| `ingest.py` 迁移 MarkdownSplitter 后旧 import 路径失效 | 在`ingest.py` 保留兼容 import：`from src.core.splitters.markdown import MarkdownSplitter` | diff --git a/docs/代码审计/全面安全与质量审计报告-2026-07-10.md b/docs/代码审计/全面安全与质量审计报告-2026-07-10.md new file mode 100644 index 0000000..75d0191 --- /dev/null +++ b/docs/代码审计/全面安全与质量审计报告-2026-07-10.md @@ -0,0 +1,369 @@ +# md-vector-db 全面安全与质量审计报告 + +> **审计日期**: 2026-07-10 +> **审计范围**: 全部源代码（22 个源文件）、13 个测试文件、配置、文档 +> **测试基线**: 90 个测试全部通过 +> **项目版本**: commit `3b8b585` + +--- + +## 一、审计概览 + +本次审计由三个并行代理分别审查 **核心模块**（config/db/embedder/ingest/search/security/splitters）、**服务层与 CLI**（app/auth/deps/main.py/scripts）、**测试与配置**（tests/pyproject.toml/config.yaml/文档），最终整合为本报告。 + +### 问题统计 + +| 严重级别 | 数量 | 含义 | +|----------|------|------| +| **CRITICAL** | **1** | 阻断 — 数据重复处理浪费 GPU 资源 | +| **HIGH** | **13** | 警告 — 合并前应修复 | +| **MEDIUM** | **11** | 信息 — 应考虑修复 | +| **LOW** | **10** | 注意 — 可选修复 | +| **总计** | **35** | | + +### 覆盖评估 + +| 模块 | 覆盖程度 | +|------|----------| +| `core/security.py` | ✅ 优秀 — 所有路径穿越场景已覆盖 | +| `core/config.py` | ✅ 良好 — YAML 加载/默认值/环境变量 | +| `core/embedder.py` | ✅ 良好 — 所有 provider 初始化和 batch_embed | +| `core/splitters/*` (除 markdown) | ✅ 良好 | +| `core/db.py` | ⚠️ 基本 — 缺少 list_collections、write_guard、并发测试 | +| `core/ingest.py` | ❌ 不足 — 缺少 ingest_file、ingest_directory | +| `core/search.py` | ❌ 不足 — 缺少 3/4 方法测试 | +| `server/deps.py` / `app.py` | ⚠️ 基本 — 缺少认证/速率限制/CORS 配置测试 | +| `server/auth.py` | ❌ 零覆盖 | +| `cli/main.py` | ❌ 零覆盖 | +| **整体估算** | **~55-60%**（低于 80% 目标） | + +--- + +## 二、CRITICAL 级别（1 个） + +### [CRIT-1] `ingest_obsidian.py` 顶层 .md 文件被重复入库 + +- **文件**: `scripts/ingest_obsidian.py`，第 54-77 行 +- **根因**: 第一轮用 `rglob("*.md")` 递归扫描所有文件，第二轮用 `glob("*.md")` 再次收集顶层文件。由于 `rglob` 已经包含顶层文件，每个顶层 `.md` 被处理两次。 +- **影响**: 嵌入计算浪费（GPU 资源），入库时间翻倍于顶层文件数量。第二次入库虽然会覆盖第一次（file_name 相同，先删后加），但嵌入已完成，白白消耗了 GPU。 +- **修复**: 删除第二轮循环（第 67-77 行），`rglob` 已覆盖所有文件。 + +--- + +## 三、HIGH 级别（13 个） + +### 安全 + +#### [HIGH-1] CORS `allow_credentials=True` 与 `allow_origins="*"` 冲突 + +- **文件**: `src/server/app.py`，第 66-72 行 +- **根因**: CORS 规范禁止 `Access-Control-Allow-Origin: *` 与 `Access-Control-Allow-Credentials: true` 同时使用。浏览器会拒绝此配置。当前默认 `CORS_ORIGINS="*"` 且 `allow_credentials=True` 直接违反规范。 +- **此外**: `*` 允许任意来源跨域访问，安全上过于宽松。 +- **修复**: + ```python + app.add_middleware( + CORSMiddleware, + allow_origins=os.environ.get("CORS_ORIGINS", "http://localhost:3000").split(","), + allow_credentials=False, # 默认关闭 + ... + ) + ``` + +#### [HIGH-2] `search_documents` 端点缺少异常处理 + +- **文件**: `src/server/app.py`，第 147-155 行 +- **根因**: `ingest_document` 有完整的 `try/except` 包裹，但 `search_documents` 完全没有。如果嵌入模型加载失败或 ChromaDB 查询异常，会返回 FastAPI 默认的原始 traceback，泄露内部路径和调用栈。 +- **修复**: 添加异常捕获： + ```python + try: + searcher = state.get_searcher(req.collection) + results = searcher.search(req.query, top_k=req.top_k) + return {"results": results, "collection": searcher.collection_name} + except Exception: + logger.exception("检索失败") + raise HTTPException(status_code=500, detail="服务器内部错误") + ``` + +#### [HIGH-3] `delete_document` 的 `collection` 查询参数缺少输入校验 + +- **文件**: `src/server/app.py`，第 163 行 +- **根因**: `IngestRequest.collection` 和 `SearchRequest.collection` 都使用了 `pattern=r"^[a-zA-Z0-9_-]+$"` 校验，但 `delete_document` 的 `collection` 查询参数没有任何正则校验。ChromaDB collection 名称允许的字符集有限，传入非法字符可能导致未预期的 ChromaDB 内部错误。 +- **修复**: 添加 `Query(default=None, max_length=128, pattern=r"^[a-zA-Z0-9_-]+$")`。 + +#### [HIGH-4] CLI 与 API 的路径安全检查逻辑不一致 + +- **文件**: `src/cli/main.py` 第 85 行 vs `src/server/app.py` 第 125-130 行 +- **根因**: CLI 仅用 `is_safe_path()` 做基础检查（拒绝 `..` + 绝对路径），而 API 还额外做了 `os.path.commonpath` 绑定当前工作目录的检查。两处策略不同增加了安全漏洞风险。 +- **修复**: 提取统一的 `is_path_within_workspace()` 函数到 `security.py`，两处共用： + ```python + def is_path_within_workspace(path_str: str) -> bool: + """检查路径是否在当前工作目录内（防路径穿越）。""" + if not is_safe_path(path_str): + return False + path = Path(path_str).resolve() + cwd = Path.cwd().resolve() + try: + return Path(os.path.commonpath([str(path), str(cwd)])) == cwd + except ValueError: + return False + ``` + +### 功能 + +#### [HIGH-5] chunk 配置项被硬编码覆盖 — YAML 中的 chunk 设置完全无效 + +- **文件**: `src/core/ingest.py`，第 46 行 +- **根因**: `get_splitter(file_path, max_size=1000, overlap=100)` 写死了值。用户无论怎么修改 `config.yaml` 里的 `chunk.max_size` 和 `chunk.overlap`，都完全不会生效。 +- **影响**: 所有通过 API/CLI 正常流程的入库都使用硬编码值。`AppConfig.chunk` 数据类从未在正常入库链路中被读取。 +- **修复**: `DocumentIngestor` 构造函数接受 `ChunkConfig` 参数，`deps.py` 的 `AppState.get_ingestor` 传入 `self.config.chunk`。 + +#### [HIGH-6] CLI `serve` 命令双重加载嵌入模型 + +- **文件**: `src/cli/main.py` 第 174 行 + `src/server/deps.py` +- **根因**: `serve` 命令调用 `get_state().config`，触发 `AppState.__init__` → `create_embedder()` 加载模型（第一次）。uvicorn 子进程启动后，`src.server.app:app` 生命周期内再次调用 `get_state()` → 加载模型（第二次）。对于 `serve` 命令，只需要服务配置（host, port, SSL），不需要加载 ML 模型。 +- **影响**: 模型被加载两次，浪费几 GB 内存和数秒启动时间。 +- **修复**: 将配置加载与模型加载解耦——提供一个不初始化模型的轻量配置读取函数给 CLI `serve` 命令使用。 + +### 代码质量 + +#### [HIGH-7] PDF 文件句柄泄漏 — fitz.Document 在异常路径下未关闭 + +- **文件**: `src/core/splitters/pdf.py`，第 34-43 行 +- **根因**: `doc.close()` 在 `try` 块内，如果在 `for page in doc` 或 `page.get_text()` 期间抛出异常，`close()` 永远不会被调用。 +- **修复**: 使用 `with fitz.open(pdf_path) as doc:`（pymupdf 支持上下文管理器协议）。 + +#### [HIGH-8] embedder.py 中环境变量操作不是线程安全的 + +- **文件**: `src/core/embedder.py`，第 80-90 行 +- **根因**: `os.environ["HF_ENDPOINT"]` 的读写没有锁保护。如果多个线程同时创建 `LocalEmbedder` 实例，可能产生竞态——线程 A 设置的值被 B 覆盖，A 的 `finally` 恢复时读到 B 的值。 +- **修复**: 使用线程锁保护该区域，或通过 `huggingface_hub` 的 API 在进程启动时统一设置而非按实例设置。 + +#### [HIGH-9] `ingest_obsidian.py` 初始化阶段无异常处理 + +- **文件**: `scripts/ingest_obsidian.py`，第 32-35 行 +- **根因**: `load_config()` / `VectorDB()` / `create_embedder()` / `DocumentIngestor()` 任一失败都会直接崩溃，没有友好的错误信息或恢复路径。 +- **修复**: 添加顶层 `try/except` 包裹，捕获异常后打印 traceback 并以 `sys.exit(1)` 退出。 + +### 测试缺口 + +#### [HIGH-10] CLI 模块零测试覆盖 + +- **文件**: `src/cli/main.py`（216 行，5 个命令） +- **影响**: 通配符展开、stdin 输入、JSON 格式化输出、路径安全检查等逻辑无自动化验证。 +- **修复**: 创建 `tests/test_cli.py`，使用 `typer.testing.CliRunner` 覆盖每个命令。 + +#### [HIGH-11] 认证与速率限制零单元测试 + +- **文件**: `src/server/auth.py`（58 行） +- **影响**: `verify_api_key()` 和 `RateLimiter` 两个安全关键组件完全没有独立测试。所有 API 测试因未设置 `MD_VECTOR_API_KEY` 而跳过认证验证。 +- **修复**: 创建 `tests/test_auth.py`，覆盖密钥匹配/不匹配/未提供、速率限制正常/超限/过期恢复、并发线程安全。 + +#### [HIGH-12] `ingest_file` / `ingest_directory` 零测试 + +- **文件**: `src/core/ingest.py`（121 行） +- **影响**: 包含扩展名检测、路径哈希、PDF/EPUB 二进制特殊处理、目录递归扫描和过滤逻辑，全部未测试。 +- **修复**: 在 `tests/test_ingest.py` 中补充文件入库和多格式目录入库的集成测试。 + +#### [HIGH-13] `Searcher` 三个方法零测试 + +- **文件**: `src/core/search.py`（97 行） +- **影响**: `list_sources()`、`delete_by_source()`、`get_collection_info()` 以及 `search(source_file=...)` 过滤均无测试。 +- **修复**: 在 `tests/test_search.py` 中补充。 + +--- + +## 四、MEDIUM 级别（11 个） + +### [MED-1] `content` 模式默认 file_name 为 `"untitled.md"` 导致数据竞态覆盖 + +- **文件**: `src/server/app.py`，第 137 行 +- **根因**: 多个并发请求不提供 `file_name` 时都写入 `"untitled.md"`。`ingest_content` 先 `_remove_by_source(file_name)` 再 `_add_chunks()`，且两次操作之间锁释放，存在竞态——后完成的请求覆盖先完成的请求数据。 +- **修复**: 自动生成 UUID 唯一名称：`file_name = req.file_name or f"untitled_{uuid.uuid4().hex[:8]}.md"` + +### [MED-2] `X-XSS-Protection` 头已过时且可能引入安全问题 + +- **文件**: `src/server/app.py`，第 81 行 +- **根因**: 此响应头已被所有现代浏览器废弃。旧版 IE 的 XSS Auditor 自身存在安全漏洞。OWASP 等安全组织明确建议不要设置此头。 +- **修复**: 替换为 `Content-Security-Policy: default-src 'self'` + +### [MED-3] `EXPECTED_API_KEY` 在模块导入时求值，存在加载顺序隐患 + +- **文件**: `src/server/auth.py`，第 14 行 +- **根因**: 该常量在模块被 import 时求值。虽然目前 `config.py` 先调用 `load_dotenv()` 再触发导入链，但这是一个隐式依赖——如果有人调整导入顺序或在测试中直接 import `auth.py`，`EXPECTED_API_KEY` 可能为空。 +- **修复**: 改为惰性求值函数：`def _get_api_key() -> str: return os.environ.get("MD_VECTOR_API_KEY", "")` + +### [MED-4] ingest.py 与 search.py 存在去重删除逻辑重复 + +- **文件**: `ingest.py:108-120` 和 `search.py:83-97` +- **根因**: 两处 `_remove_by_source` / `delete_by_source` 逻辑几乎一模一样：获取匹配文档、删除 IDs、捕获 `ValueError`。 +- **修复**: 统一放到 `VectorDB.delete_by_metadata(where)` 方法中。 + +### [MED-5] `list_collections_with_stats` 静默吞掉所有异常 + +- **文件**: `src/server/deps.py`，第 54-59 行 +- **根因**: 当 ChromaDB 不可用时，返回空列表而不是错误。调用方向用户报告"没有任何集合"，而非"数据库连接失败"。错误语义被扭曲。 +- **修复**: 让异常向上传播，由 API 层统一处理并返回适当的 HTTP 错误。 + +### [MED-6] EPUB 缺失依赖测试设计有误 + +- **文件**: `tests/test_splitters_epub.py`，第 51-56 行 +- **根因**: `test_epub_missing_dependency_message` 声称测试"未安装 ebooklib 时应给出明确提示"，但模块顶部有 `pytest.importorskip("ebooklib", ...)`，意味着当 ebooklib 未安装时整个文件被 skip，该测试永远不会在目标场景下执行。 +- **修复**: 将此测试移到独立文件中（不使用 `importorskip`），或使用 `monkeypatch` 模拟 `ImportError`。 + +### [MED-7] 环境变量测试使用 `importlib.reload` 过于脆弱 + +- **文件**: `tests/test_config.py`，第 89-106 行 +- **根因**: `importlib.reload(config)` 有全局副作用，可能影响后续测试。`finally` 块中的清理逻辑也不完整。 +- **修复**: 使用 `monkeypatch.setenv` + 重新实例化代替 `reload`。 + +### [MED-8] 速率限制器的配置值硬编码 + +- **文件**: `src/server/auth.py`，第 58 行 +- **根因**: `RateLimiter(max_requests=30, window_seconds=60)` 无法通过环境变量或配置文件调整。 +- **修复**: 从环境变量读取：`max_requests=int(os.environ.get("RATE_LIMIT_MAX", "30"))` + +### [MED-9] `ingest_obsidian.py` 存在硬编码的绝对回退路径 + +- **文件**: `scripts/ingest_obsidian.py`，第 47-51 行 +- **根因**: `targets` 中的三个路径仅在本机有效，在其他开发者的机器上毫无意义。如果恰好存在同名目录，会意外入库其他数据。 +- **修复**: 移除硬编码回退路径，直接报错提示用户提供参数。 + +### [MED-10] MarkdownSplitter 测试组织不清晰 + +- **文件**: 分散在 `tests/test_ingest.py` 中 +- **根因**: Markdown 分块测试与入仓器测试混在一起，缺少 Markdown 特有的边界测试：无标题文档、深层标题、代码块内含 `#` 号被误识别为标题等。 +- **修复**: 提取到 `tests/test_splitters_markdown.py`，补充边界测试。 + +### [MED-11] `splitters/__init__.py` 未导出 HTMLSplitter + +- **文件**: `src/core/splitters/__init__.py`，第 10-20 行 +- **根因**: `HTMLSplitter` 在第 7 行被 import 但未加入 `__all__` 列表，而 `PDFSplitter` 和 `EPUBSplitter` 都在列表中。 +- **修复**: 在 `__all__` 中加入 `"HTMLSplitter"`。 + +--- + +## 五、LOW 级别（10 个） + +| # | 文件 | 问题 | 建议 | +|---|------|------|------| +| L-1 | `src/core/splitters/epub.py:44` | EPUB 硬编码 UTF-8 解码，非 UTF-8 文件会丢失章节 | 添加编码检测回退（先 UTF-8，失败后 chardet） | +| L-2 | `src/core/embedder.py:24` | 模块级 `import requests` 依赖传递性依赖项 | 改为惰性导入（在 DashscopeEmbedder.embed 内部） | +| L-3 | `src/core/embedder.py:176` | `list.sort()` 原地修改 API 响应 | 改为 `sorted(embeddings_raw, key=...)` | +| L-4 | `src/core/config.py:16` | `DEFAULT_CONFIG_PATH` 使用相对路径 | 统一使用 `load_config` 的路径解析逻辑 | +| L-5 | `src/core/splitters/registry.py:20` | `_custom_registry` 模块级字典无线程锁保护 | 添加 `threading.Lock` 或文档说明仅限启动时调用 | +| L-6 | `pyproject.toml:21` | `dev` 依赖缺少 `pytest-cov` 和 `ruff`/`mypy` | 添加 `pytest-cov>=5.0`、`ruff>=0.8.0`、`mypy>=1.13` 及相关 tool 配置节 | +| L-7 | `config.yaml` | `host: 0.0.0.0` 与代码默认值 `127.0.0.1` 不一致 | 统一默认值或在配置文件中添加注释说明差异 | +| L-8 | `config.yaml` | SSL 字段（ssl_keyfile/ssl_certfile）未在配置文件中暴露 | 添加注释说明可用配置 | +| L-9 | `scripts/serve.py` | `DeprecationWarning` 被 Python 默认过滤，多数用户看不到 | 改为 `print(..., file=sys.stderr)` 或 `FutureWarning` | +| L-10 | `CLAUDE.md` / `README.md` | 文档中测试数量写"46 个"，实际 90 个；数据流未反映多格式支持 | 更新为实际数量，数据流加入 `get_splitter()` 描述 | + +--- + +## 六、架构与设计评估 + +### 优点 + +1. **策略模式设计良好** — `Embedder(Protocol)` 和 `Splitter(Protocol)` 接口清晰，新增 provider/格式只需实现接口并注册 +2. **扩展名自动路由** — `registry.py` 的 `get_splitter()` 工厂 + 回退到 TextSplitter 的设计优雅 +3. **线程安全基础扎实** — `VectorDB._write_lock` + `deps.py` 的 `_state_lock`/`_cache_lock` 构成多层保护 +4. **安全防御分层** — API Key 认证 → 速率限制 → 路径穿越防护，防御纵深合理 +5. **配置热加载友好** — `load_config()` 支持文件变更检测，适合长期运行的服务 + +### 需要改进的架构问题 + +1. **配置与模型加载未解耦** — `AppState.__init__` 做了太多事（配置+模型+数据库），导致 CLI `serve` 双重加载。建议拆分为 `load_config_only()` 和 `init_full_state()` +2. **chunk 配置未贯传整个链路** — `config.yaml → AppConfig.chunk` 存在但被 `ingest.py` 硬编码覆盖。应该在 `DocumentIngestor` 构造函数中接受 `ChunkConfig` +3. **路径安全检查逻辑分散** — CLI 和 API 各有一套检查，应该提取为统一的 `is_path_within_workspace()` +4. **去重删除逻辑重复** — `ingest.py` 和 `search.py` 中几乎相同的代码应合并到 `VectorDB` + +--- + +## 七、测试策略建议 + +### 当前测试分布 + +``` +tests/ +├── test_api.py # 14 个 — API 端点基本功能 +├── test_config.py # 7 个 — 配置加载 +├── test_db.py # 4 个 — 数据库 CRUD +├── test_deps.py # 11 个 — AppState 生命周期 +├── test_embedder.py # 8 个 — 嵌入 provider +├── test_ingest.py # 11 个 — 入仓（偏 MarkdownSplitter） +├── test_search.py # 8 个 — 基本搜索 +├── test_security.py # 11 个 — 路径安全 +├── test_splitters.py # 10 个 — TextSplitter + 注册表 +├── test_splitters_html.py # 4 个 +├── test_splitters_pdf.py # 3 个 +├── test_splitters_epub.py # 3 个 +└── (缺失) + ├── test_auth.py ❌ 认证与速率限制 + ├── test_cli.py ❌ CLI 命令 + └── test_splitters_markdown.py ❌ Markdown 边界 +``` + +### 建议新增测试 + +1. **`tests/test_cli.py`** — CLI 5 个命令（使用 `CliRunner`） +2. **`tests/test_auth.py`** — `verify_api_key` + `RateLimiter` +3. **`tests/test_splitters_markdown.py`** — 标题边界、代码块内 `#` 号、空文档 +4. **补充 `tests/test_ingest.py`** — `ingest_file` / `ingest_directory` +5. **补充 `tests/test_search.py`** — `list_sources` / `delete_by_source` / `get_collection_info` + +--- + +## 八、修复优先级路线图 + +### 第一阶段（立即修复 — 1 个 CRITICAL） + +| 问题 | 文件 | 工作量 | +|------|------|--------| +| CRIT-1: 顶层 .md 文件重复入库 | `scripts/ingest_obsidian.py` | 5 分钟 | + +### 第二阶段（尽快修复 — 13 个 HIGH） + +| 问题 | 领域 | 工作量 | +|------|------|--------| +| HIGH-5: chunk 配置被子覆盖 | 功能 | 30 分钟 | +| HIGH-7: PDF 文件句柄泄漏 | 资源 | 5 分钟 | +| HIGH-1: CORS 配置冲突 | 安全 | 5 分钟 | +| HIGH-2: search 端点无异常处理 | 安全 | 5 分钟 | +| HIGH-3: delete collection 缺少校验 | 安全 | 5 分钟 | +| HIGH-4: 路径安全检查不一致 | 安全 | 20 分钟 | +| HIGH-6: serve 双重加载模型 | 性能 | 30 分钟 | +| HIGH-8: HF_ENDPOINT 非线程安全 | 稳定性 | 15 分钟 | +| HIGH-9: ingest_obsidian 无异常处理 | 稳定性 | 10 分钟 | +| HIGH-10: CLI 零测试 | 测试 | 1-2 小时 | +| HIGH-11: auth 零测试 | 测试 | 1 小时 | +| HIGH-12: ingest_file/dir 零测试 | 测试 | 1 小时 | +| HIGH-13: Searcher 方法零测试 | 测试 | 30 分钟 | + +### 第三阶段（计划修复 — 11 个 MEDIUM） + +涉及竞态覆盖、过时安全头、重复代码、错误语义、测试设计等。 + +### 第四阶段（择机修复 — 10 个 LOW） + +涉及文档过时、编码回退、线程锁、工具链配置等。 + +--- + +## 九、合规性检查 + +对照用户全局安全规则（`~/.claude/rules/ecc/zh/security.md`）进行逐项检查： + +| 检查项 | 状态 | 说明 | +|--------|------|------| +| 无硬编码密钥 | ✅ 通过 | API Key 均从环境变量读取 | +| 所有用户输入已验证 | ⚠️ 部分 | `delete_document` 的 collection 参数缺少正则校验 | +| SQL 注入防护 | N/A | 无 SQL 数据库 | +| XSS 防护 | ⚠️ 部分 | `X-XSS-Protection` 已过时，应换为 CSP | +| CSRF 保护 | ⚠️ 部分 | CORS 配置存在规范冲突 | +| 认证/授权已验证 | ✅ 通过 | API Key 认证 + hmac.compare_digest | +| 所有端点启用速率限制 | ✅ 通过 | RateLimiter 中间件 | +| 错误消息不泄露敏感数据 | ⚠️ 部分 | `search_documents` 缺少异常处理会泄露 traceback | + +--- + +*审计人: Claude Code（多代理协作审查）* +*下次审计建议: 在修复 CRITICAL 和 HIGH 问题后进行复验* diff --git a/src/server/app.py b/src/server/app.py index ad1eefb..dc69e1d 100644 --- a/src/server/app.py +++ b/src/server/app.py @@ -65,10 +65,10 @@ app = FastAPI(title="md-vector-db", version="0.1.0", lifespan=lifespan) # CORS 中间件 app.add_middleware( CORSMiddleware, - allow_origins=os.environ.get("CORS_ORIGINS", "*").split(","), - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], + allow_origins=os.environ.get("CORS_ORIGINS", "http://localhost:3000").split(","), + allow_credentials=False, + allow_methods=["GET", "POST", "DELETE", "OPTIONS"], + allow_headers=["Content-Type", "Authorization", "X-API-Key"], ) diff --git a/uv.lock b/uv.lock index 4bb1da5..6968fdf 100644 --- a/uv.lock +++ b/uv.lock @@ -214,6 +214,19 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/27/44/d2ef5e87509158ad2187f4dd0852df80695bb1ee0cfe0a684727b01a69e0/bcrypt-5.0.0-cp39-abi3-win_arm64.whl", hash = "sha256:f2347d3534e76bf50bca5500989d6c1d05ed64b440408057a37673282c654927", size = 144953, upload-time = "2025-09-25T19:50:37.32Z" }, ] +[[package]] +name = "beautifulsoup4" +version = "4.15.0" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +dependencies = [ + { name = "soupsieve" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/43/65/318323f98dbee45d42dff61d8f047181bc6f2268a9068cfad035a46be5af/beautifulsoup4-4.15.0.tar.gz", hash = "sha256:288e3ca7d54b06f2ac191970bc275c1939cb46d450b255bf6718b04aa37ab4f7", size = 632571, upload-time = "2026-06-07T16:44:20.453Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/88/c6/92fcd42f1ba33e1184263f25bfabf3d27c383410470f169e4b8163bf9c17/beautifulsoup4-4.15.0-py3-none-any.whl", hash = "sha256:d6f88de62e1d4e38ecb1077eb9724cd0eff29d2a08ca16a401e9b9e93f117cf9", size = 109924, upload-time = "2026-06-07T16:44:21.566Z" }, +] + [[package]] name = "build" version = "1.5.0" @@ -357,6 +370,15 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] +[[package]] +name = "distro" +version = "1.9.0" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed", size = 60722, upload-time = "2023-12-24T09:54:32.31Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" }, +] + [[package]] name = "durationpy" version = "0.10" @@ -366,6 +388,19 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b0/0d/9feae160378a3553fa9a339b0e9c1a048e147a4127210e286ef18b730f03/durationpy-0.10-py3-none-any.whl", hash = "sha256:3b41e1b601234296b4fb368338fdcd3e13e0b4fb5b67345948f4f2bf9868b286", size = 3922, upload-time = "2025-05-17T13:52:36.463Z" }, ] +[[package]] +name = "ebooklib" +version = "0.20" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +dependencies = [ + { name = "lxml" }, + { name = "six" }, +] +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/77/85/322e8882a582d4b707220d1929cfb74c125f2ba513991edbce40dbc462de/ebooklib-0.20.tar.gz", hash = "sha256:35e2f9d7d39907be8d39ae2deb261b19848945903ae3dbb6577b187ead69e985", size = 127066, upload-time = "2025-10-26T20:56:20.968Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/bf/ee/aa015c5de8b0dc42a8e507eae8c2de5d1c0e068c896858fec6d502402ed6/ebooklib-0.20-py3-none-any.whl", hash = "sha256:fff5322517a37e31c972d27be7d982cc3928c16b3dcc5fd7e8f7c0f5d7bcf42b", size = 40995, upload-time = "2025-10-26T20:56:19.104Z" }, +] + [[package]] name = "fastapi" version = "0.139.0" @@ -681,6 +716,56 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "jiter" +version = "0.16.0" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/1d/1f/10936e16d8860c70698a1aa939a46aa0224813b782bce4e000e637da0b2d/jiter-0.16.0.tar.gz", hash = "sha256:7b24c3492c5f4f84a37946ad9cf504910cf6a782d6a4e0689b6673c5894b4a1c", size = 176431, upload-time = "2026-06-29T13:05:13.657Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/91/c0/555fc60473d30d66894ba825e63615e3be7524fac23858356afa7a38906c/jiter-0.16.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:41977aa5654023948c2dae2a81cbf9c43343954bef1cd59a154dd15a4d84c195", size = 306203, upload-time = "2026-06-29T13:03:36.243Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d0/2b/c3eaf16f5d7c9bad66ea32f40a95bd169b29a91217fcc7f081375157e99c/jiter-0.16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d28bb3c26762358dadf3e5bf0bccd29ae987d65e6988d2e6f49829c76b003c09", size = 306489, upload-time = "2026-06-29T13:03:37.846Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/96/3f/02fdfc6705cad96127d883af5c34e4867f554f29ec7705ec1a46156400a9/jiter-0.16.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0542a7189c26920778658fc8fcf2af8bae05bae9924577f71804acef37996536", size = 335453, upload-time = "2026-06-29T13:03:39.221Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b2/a6/e4bda5920d4b0d7c5dfb7174ce4a6b2e4d3e11c9162c452ef0eab4cdbdbd/jiter-0.16.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8fb8de1e23a0cb2a7f53c335049c7b72b6db41aa6227cdcc0972a1de5cb39450", size = 361625, upload-time = "2026-06-29T13:03:40.597Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b7/97/4e6b59b2c6e55cbb3e183595f81ad65dcfb21c915fee5e19e335df21bc55/jiter-0.16.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b72d0b2990ca754a9102779ac98d8597b7cb31678958562214a007f909eab78e", size = 456958, upload-time = "2026-06-29T13:03:42.074Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/15/e0/97e9557686d2f94f4b93786eccb7eed28e9228ad132ea8237f44727314a7/jiter-0.16.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d5f91b1c27fc22a57993d5a5cb8a627cb8ed4b10502716fac1ffbfe1d19d84e8", size = 372017, upload-time = "2026-06-29T13:03:43.658Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/0f/94/db768b6938e0df35c86beeba3dfbbb025c9ee5c19e1aa271f2396e50864d/jiter-0.16.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c682bea068a90b764577bdb78a60a4c1d1606daf9cd4c893832a37c7cc9d9026", size = 343320, upload-time = "2026-06-29T13:03:45.226Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/c1/d6/5a59d938244a30735fe62d9433fd325f9021ea29d89780ea4596ea93bc89/jiter-0.16.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:8d031aabecc4f1b6276adfb42e3aabb77c89d468bf616600e8d3a11328929053", size = 350520, upload-time = "2026-06-29T13:03:46.671Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/67/f8/c4a857f49c9af125f6bbcac7e3eee7f7978ed89682833062e2dbf62576b1/jiter-0.16.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:eab2cd170150e70153de16896a1774e3a1dca80154c56b54d7a812c479a7165e", size = 387550, upload-time = "2026-06-29T13:03:48.361Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/8b/d6/5fbc2f7d6b67b754caa61a993a2e626e815dec47ffc2f9e35f01adfebec7/jiter-0.16.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:6edb63a46e65a82c26800a868e49b2cac30dd5a4218b88d74bc2c848c8ad60bb", size = 515424, upload-time = "2026-06-29T13:03:49.881Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/ed/54/284f0164b64a5fed915fea6ba7e9ba9b3d8d37c67d59cf2e3bb99d45cdfe/jiter-0.16.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:659039cc50b5addcc35fcc87ae2c1833b7c0a8e5326ef631a75e4478447bcf84", size = 546981, upload-time = "2026-06-29T13:03:51.363Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/13/c5/2a467585a576594384e1d2c43e1224deaafc085f24e243529cf98beef8e1/jiter-0.16.0-cp313-cp313-win32.whl", hash = "sha256:c9c53be232c2e206ef9cdbad81a48bfa74c3d3f08bcf8124630a8a748aad993e", size = 202853, upload-time = "2026-06-29T13:03:53.015Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/88/6a/de61d04b9eec69c71719968d2f716532a3bc121170c44a39e14979c6be81/jiter-0.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:baad945ed47f163ad833314f8e3288c396118934f94e7bbb9e243ce4b341a4fd", size = 196160, upload-time = "2026-06-29T13:03:54.447Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/19/4b/b390ed59bafb3f31d008d1218578f10327714484b334439947f7e5b11e7f/jiter-0.16.0-cp313-cp313-win_arm64.whl", hash = "sha256:3c1fd2dbe1b0af19e987f03fe66c5f5bd105a2229c1aff4ab14890b24f41d21a", size = 189862, upload-time = "2026-06-29T13:03:55.754Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a7/89/bc4f1b57d5da938fd344a466396541e586d161320d70bffd929aaafcd8f4/jiter-0.16.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:b2c61484666ad42726029af0c00ef4541f0f3b5cdc550221f56c2343208018ee", size = 308239, upload-time = "2026-06-29T13:03:57.205Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/65/7a/c415453e5213001bf3b411ff65dec3d303b0e76a4a2cfea9768cd4960994/jiter-0.16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:63efadc657488f45db1c676d81e704cac2abf3fdb892def1faea61db053127e2", size = 308928, upload-time = "2026-06-29T13:03:58.643Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/11/fc/1f4fb7ebf9a724c7741994f4aae18fba1e2f3133df14521a79194952c34a/jiter-0.16.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cf0d73f50e7b6935677854f6e8e31d499ca7064dd24734f703e060f5b237d883", size = 336998, upload-time = "2026-06-29T13:04:00.071Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a0/8d/72cadaac05ccfa7cc3a0a2232862e6c72443ca40cf300ba8b57f9f18b69b/jiter-0.16.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bf3ea07d9bc8e7d03a9fbc051295462e6dbc295b894fd72457c3136e3e43d898", size = 362112, upload-time = "2026-06-29T13:04:01.52Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/58/4a/c4b0d5f651fda90a24ffce9f8d56cde462a2e09d31ae3de3c68cef34c04e/jiter-0.16.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:26798522707abb47d767db536e4148ceac1b14446bf028ee85e579a2e043cfe5", size = 459807, upload-time = "2026-06-29T13:04:03.214Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/80/58/ef77879ea9aa56b50824edc5a445e226422c7a8d211f3fd2a56bcb9493cf/jiter-0.16.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bc837c1b9631be10abfe0191537fe8009838204cec7e44827401ace390ddb567", size = 373181, upload-time = "2026-06-29T13:04:04.629Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/49/2e/ffbc3f254e4d8a66da3062c624a7df4b7c2b2cf9e1fe43cf394b3e104041/jiter-0.16.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49060fd70737fad59d33ba9dcc0d83247dc9e77187de26053a19c16c9f32bd69", size = 344927, upload-time = "2026-06-29T13:04:06.067Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/9a/f6/0be5dc6d64a89f80aa8fec984f94dedb2973e251edcae55841d60786d578/jiter-0.16.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:adbb8edeadd431bc4477879d5d371ece7cb1334486584e0f252656dd7ffada29", size = 352754, upload-time = "2026-06-29T13:04:07.477Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/da/6e/7d31243b3b91cd261dd19e9d3557fc3251a80883d3d8049c86174e7ab7af/jiter-0.16.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:31aaee5b80f672c1dc21272bcfb9cbdcfc1ea04ff50f00ed5af500b80c44fa93", size = 390553, upload-time = "2026-06-29T13:04:08.92Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/25/33/51ae371fde3c88897520f62b4d5f8b27ad7103e2bb10812ff52195609853/jiter-0.16.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:6722bcef4ffc86c835574b1b2fac6b33b9fb4a889c781e67950e891591f3c55a", size = 516900, upload-time = "2026-06-29T13:04:10.407Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a0/45/6449b3d123ea439ba79507c657288f461d55049e7bcbdc2cf8eb8210f491/jiter-0.16.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:5ab4f50ff971b611d656554ea10b75f80097392c827bc32923c6eeb6386c8b00", size = 548754, upload-time = "2026-06-29T13:04:12.046Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/9b/e7/fd2fb11ae3e2649333da3aa170d04d7b3000bbdc3b270f6513382fdf4e04/jiter-0.16.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:710cc51d4ebdcd3c1f70b232c1db1ea1344a075770422bbd4bede5708335acbe", size = 122381, upload-time = "2026-06-29T13:04:13.413Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/26/80/f0b147a62c315a164ed2168908286ca302310824c218d3aae52b06c0c9a9/jiter-0.16.0-cp314-cp314-win32.whl", hash = "sha256:57b37fc887a32d44798e4d8ebfa7c9683ff3da1d5bf38f08d1bb3573ccb39106", size = 204578, upload-time = "2026-06-29T13:04:14.813Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/5e/e6/4758a14304b4523a6f5adb2419340086aa3593bd4327c2b25b5948a90548/jiter-0.16.0-cp314-cp314-win_amd64.whl", hash = "sha256:cbd18dd5e2df96b580487b5745adf57ef64ad89ba2d9662fc3c19386acce7db8", size = 198154, upload-time = "2026-06-29T13:04:16.272Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/26/be/41fa54a2e7ea41d6c99f1dc5b1f0fd4cb474680304b5d268dd518e81da3a/jiter-0.16.0-cp314-cp314-win_arm64.whl", hash = "sha256:a32d2027a9fa67f109ff245a3252ece3ccc32cc56703e1deab6cc846a59e0585", size = 191458, upload-time = "2026-06-29T13:04:17.707Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/81/6b/59127338b86d9fe4d99418f5a15118bea778103ee0fe9d9dd7e0af174e95/jiter-0.16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2577196f4474ef3fc4779a088a23b0897bbf86f9ea3679c372d45b8383b43207", size = 316739, upload-time = "2026-06-29T13:04:19.663Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/2d/95/49461034d5388196d3dabf98748935f017b7785d8f3f5349f834bcc4ed0d/jiter-0.16.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:616e89e008a93c01104161c75b4988e58716b01d62307ebfe161e52a56d2a818", size = 340911, upload-time = "2026-06-29T13:04:21.257Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/cd/97/a4369f2fb82cb3dda13b98622f31249b2e014b223fe64ee534413ad72294/jiter-0.16.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0e2e9efbe042210df657bade597f66d6d75723e3d8f45a12ea6d8167ff8bbce3", size = 361747, upload-time = "2026-06-29T13:04:22.677Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/28/51/49b6ed456261646e1906016a6760367a28aacd3c24805e4e5fe64116c1db/jiter-0.16.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3f4d9e473a5ce7d27fef8b848df4dc16e283893d3f53b4a585e72c9595f3c284", size = 460225, upload-time = "2026-06-29T13:04:24.441Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/33/b5/5689aff4f66c5b60be63106e591dbfcba2190df97d2c9c7cf052361ddb98/jiter-0.16.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8d30a4a1c87713060c8d1cc59a7b6c8fb6b8ef0a6900368014c76c87922a2929", size = 373169, upload-time = "2026-06-29T13:04:25.884Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a2/96/3ae1b85ee0d6d6cab254fb7f8da018272b932bbf2d69b07e98aa2a96c746/jiter-0.16.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bae96332410f866e5900d809298b1ed82735932986c672495f9701daacd80620", size = 350332, upload-time = "2026-06-29T13:04:27.302Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/15/32/c99d7bafd78986556c95bf60ce84c6cc98786eac56066c12d7f828bb6747/jiter-0.16.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:da3d7ec75dc83bb18bca888b5edfae0656a26849056c59e05a7728badd17e7af", size = 353377, upload-time = "2026-06-29T13:04:28.731Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/0e/4b/f99a8e571287c3dec766bcc18528bbe8e8fb5365522ab5e6d64c93e87066/jiter-0.16.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ee6162b77d49a9939229df666dfa8af3e656b6701b54c4c84966d740e189264e", size = 387746, upload-time = "2026-06-29T13:04:30.319Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/75/69/c78a5b3f71040e34eb5917df26fb7ae9a2174cad1ccbf277512507c53a6e/jiter-0.16.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:63ffdbdae7d4499f4cda14eadc12ddcabef0fc0c081191bdc2247489cb698077", size = 517292, upload-time = "2026-06-29T13:04:31.709Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/c2/f7/095b38eda4c70d03651c403f29a5590f16d12ddc5d544aac9f9cddf72277/jiter-0.16.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:a111256a7193bea0759267b10385e5870949c239ed7b6ddbaaf57573edb38734", size = 549259, upload-time = "2026-06-29T13:04:33.721Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/2e/c5/6a0207d90e5f656d95af98ebd0934f382d37674416f215aeda2ff8063e51/jiter-0.16.0-cp314-cp314t-win32.whl", hash = "sha256:de5ba8763e56b793561f43bed197c9ea55776daa5e9a6b91eed68a909bc9cdbf", size = 206523, upload-time = "2026-06-29T13:04:35.068Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a5/31/c757d5f30a8980fd945ce7b98be10be9e4ff59c7c42f5fd86804c2e87db8/jiter-0.16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b8a3f9a6008048fe9def7bf465180564a6e458047d2ce499149cfbe73c3ae9db", size = 200366, upload-time = "2026-06-29T13:04:36.61Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/7c/a2/d88de6d313d734a544a7901353ad5db67cb38dcfcd91713b7979dafc345d/jiter-0.16.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0fa25b09b13075c46f5bc174f2690525a925a4fc2f7c82969a2bbabff22386ce", size = 190516, upload-time = "2026-06-29T13:04:38.004Z" }, +] + [[package]] name = "joblib" version = "1.5.3" @@ -738,6 +823,68 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/46/2c/5c160dbdef7123f8cc97fd8ece7e0198627a426a2a49614845e9086feb8d/kubernetes-36.0.2-py2.py3-none-any.whl", hash = "sha256:faf9b5241b58de0c4a5069f2a0ffc8ac06fece7215156cd3d3ba081a78a858b6", size = 4617568, upload-time = "2026-06-01T18:20:28.737Z" }, ] +[[package]] +name = "lxml" +version = "6.1.1" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/05/3b/aab6728cae887456f409b4d75e8a01856e4f04bd510de38052a47768b680/lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40", size = 4197430, upload-time = "2026-05-18T19:19:06.424Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a5/eb/7e6f37c5584ccbb2ff267f56fd0339016938c1c8684cfefab9b33ffc2f36/lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736", size = 8559780, upload-time = "2026-05-18T19:17:57.661Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a1/36/587c2521cf23a2cd6c9c22108aa7528f683a1f195ed7ccd23a4b1786ad36/lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9", size = 4618006, upload-time = "2026-05-18T19:18:04.452Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/6e/ca/ab7bfe2bf4c972af5e7878262845ead3a24a929a9b04bc11c7c1ece6c82a/lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354", size = 4924139, upload-time = "2026-05-18T19:19:04.873Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/6b/55/a0c72851dfee5ecc689f949723a73dea457758912542cb955b108eaf0d8f/lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca", size = 5082329, upload-time = "2026-05-18T19:19:09.728Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/f0/b6/0608f7d61a3b96cc67e5648a3d906e31a5082093e10e7be65b3886289938/lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099", size = 4993564, upload-time = "2026-05-18T19:19:13.608Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/4c/66/ae227524b066d29d55bf0b453d93d2d793c40218657d643dcbbca13b8faf/lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6", size = 5613467, upload-time = "2026-05-18T19:19:16.228Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a6/76/dbe4a00b50385e40194231dcfe5a12c059de7cf90e89c83407d2b085b719/lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085", size = 5228304, upload-time = "2026-05-18T19:19:19.354Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/1c/01/00b1b8442ed2041793336868ba0b9ea4b13d7da7c085c6404c207a63bf79/lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e", size = 5341607, upload-time = "2026-05-18T19:19:22.297Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/63/36/1ad29931e9a4638bb707869f01d423a6c815f82152138d1a40dfcfde2b95/lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f", size = 4700168, upload-time = "2026-05-18T19:19:25.133Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/3c/d1/a9536cecf9be18a0dc72d32bead283a2332d1ffebd2dd3ac70ce444686e5/lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c", size = 5232487, upload-time = "2026-05-18T19:19:28.603Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/0e/77/b4fb1e03bf5d130e879214d3100092e386418807fb74dd0adc4b0a48f351/lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b", size = 5044231, upload-time = "2026-05-18T19:18:42.246Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/26/4c/d00daeeb0a5530c4028a9232aa1b93db3ef4ed2158c116ea73c79a9765b3/lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2", size = 4769450, upload-time = "2026-05-18T19:18:48.013Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/ed/6a/715a3a8d156ce42f29cf014706f5410c2ff3b02267774110fc23266409fe/lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5", size = 5635874, upload-time = "2026-05-18T19:18:51.914Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/45/37/0544bc21dde2a88f3a17b504e6fc79c0e01d25a33c2f6079724e9e72b9c7/lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785", size = 5223987, upload-time = "2026-05-18T19:18:59.715Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/4d/f8/f6a5e8185bcb28c2befae3d31f8e3df3b811cb0f47746517a81279fcafe1/lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947", size = 5250276, upload-time = "2026-05-18T19:19:03.834Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/c7/f2/1a2b9f1b7a49d45495369be7ef9ad05b262930f2eab3e3145706fca8083f/lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca", size = 3596903, upload-time = "2026-05-18T19:17:29.863Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/e6/99/f4ffb024f238eec2131aaa09f3278fb6129cf892741bf68e1fc1afb8c100/lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660", size = 3995869, upload-time = "2026-05-18T19:18:02.596Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d1/53/70eb8c5c6037f27448f1e3c54ebede9545a801ae63f0a7254afca4fe8e45/lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc", size = 3658490, upload-time = "2026-05-19T19:22:53.846Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/13/e2/2e325795566de01d0d7c3bb57d3c370616b2d07b01214e84eec5d3b10963/lxml-6.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:19b7ab10b210b0b3ad7985d9ac4eb66ab09a90b20fe6e2f7ba55d01a234345d0", size = 8577146, upload-time = "2026-05-18T19:18:17.765Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/93/cf/5630b5e4be7d2e6bee8efe83865c925221103cf0221303b104ce134b01e2/lxml-6.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c08e5c694306507275f2290073350c4f32e383db15213b2c69e7ff39c1193840", size = 4623866, upload-time = "2026-05-18T19:18:30.669Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d2/51/3904907c063451cf8d4a5c9fe0cad95fa1f4ec57f4e3884fa0731bd7a305/lxml-6.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:74a9717fd0d82effef5c2854f0d917231d5324b5a3eb7275c43ac9fa32f97a14", size = 4950022, upload-time = "2026-05-18T19:19:31.958Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/94/cd/9c7611a51c37a2830928405817cc5d56a97f64fab83cc3f628748b135749/lxml-6.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:efe0374196335f93b53269acd811b944f2e6bdc88e8894f214bd636455484909", size = 5086695, upload-time = "2026-05-18T19:19:34.764Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/da/d6/24e3b5906abb0b674ff2ae195bc3ce59708df2bcd17cf17703b2d7dd643a/lxml-6.1.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac931cdc9442c1763b8a8f6cd62c0c938737eafc5be75eff88df55fc73bc0d00", size = 5031642, upload-time = "2026-05-18T19:19:37.771Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/2d/db/6ec54f99019838bff54785c51da07f189eb4676861c5f2730962b0d8d665/lxml-6.1.1-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:aee395f5d0927f947758b4ec119fd5fc8ec71f07a1c5c52077b30b04c0fa6955", size = 5647338, upload-time = "2026-05-18T19:19:40.553Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/42/3d/ef4dcfffd22d27a61805d8ed9f7fb888495bc6aa88648fa07c1eaa5586b6/lxml-6.1.1-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9395002973c827b3ed67db77e6ec09f092919a587022174554096a269378fb13", size = 5239528, upload-time = "2026-05-18T19:19:43.657Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/62/bb/37fb3f0dff146bdcfa78eec47879273820b2a0bf350ec236ce14bd0b1c26/lxml-6.1.1-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:73bc2086f141224ebddb7fc5c6a36ca58b31b94b561e1dfe8e073e3270fad1e7", size = 5350730, upload-time = "2026-05-18T19:19:46.307Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/90/42/43253f168388df4fae1f38c01df36ddb9bee39e2048167b54cdcbae85ea3/lxml-6.1.1-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:3779def59032b81e44a5f70096ef6bf2082f8d901937dca354474ba09782e245", size = 4697530, upload-time = "2026-05-18T19:19:49.889Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/eb/a8/c5a8504f81bbdfc8e7094c2c850cdb4ed6777fc4d5ddd9e5ab819f3b0d54/lxml-6.1.1-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:86c89b9d55ebf820ad7c90bc533410f0d098054f293351f10603c0c46ff598f5", size = 5250670, upload-time = "2026-05-18T19:19:53.199Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/77/b7/c7e76ab18744d75e21f320ebf9ff9d1ceae2b54dd431ea5a64caf26c9672/lxml-6.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19607c6bbff2a44cf3fe8250abccd20942d3462473e0a721d01d379ed017e462", size = 5084485, upload-time = "2026-05-18T19:19:08.422Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/31/31/b35c53f8ef7b7c31cacd23d3638652fff7bcd1deb6eedb709ab43b685908/lxml-6.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:c6ed5141a5c7507cf3ee76bd363b0d6f801e3321adc35b5d825a23115faa5465", size = 4737635, upload-time = "2026-05-18T19:19:12.321Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d9/06/31f23c813a7fe8e0cb1b175e915b08c9bf4e86d225b210feadbdbe519667/lxml-6.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:62aeb7e85b5d60320b9d77eef2e773994e2c0ce10121b277e0a19804e1654a5a", size = 5670681, upload-time = "2026-05-18T19:19:15.001Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/1a/bc/ce619bccc89b1fd9ad8a8e1330ee3f3beff9f2ff95b712d7bbcdd6e22fc3/lxml-6.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b1b963fd8f5caa68e99dfae060d54de1fe9cba899b8718b44a00cdca53c3e590", size = 5238229, upload-time = "2026-05-18T19:19:18.131Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/2f/5d/b329acbbedc0b619ebc2be6cf7ee9ed07e80892c88d4dfd612c33805789a/lxml-6.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:63876be28efefa04a1df615b46770e82042cce445cfdce55160522f57b231ccb", size = 5264191, upload-time = "2026-05-18T19:19:21.118Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d6/85/be36fb1425b30db3c3f9df75fe86343ebffb79e6320bd7f588e25bfeac39/lxml-6.1.1-cp314-cp314-win32.whl", hash = "sha256:7f7a92e8583f06b1fd49d01158143b8461cfcd135dcb10ec807270a3051bd603", size = 3657202, upload-time = "2026-05-18T19:17:39.509Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b8/ce/3cf9a827342269f54d405a6202397de63f07c69cbd6ce7d183a3f0cba1e9/lxml-6.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:b2d444f2e66624d68e9c6b211e28a76e22fff5fcabcfff4deac18b529b7d4137", size = 4064497, upload-time = "2026-05-18T19:18:14.662Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/d9/3e/1a957bde8f0760039e627f94699f82caa782c9d838d86c3d28245ee67212/lxml-6.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:3fd9728a2735fda14f4e8235830c86b539e9661e849665bf926d3f867943b4bf", size = 3741991, upload-time = "2026-05-19T19:22:59.111Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/78/b2/00ed55b3a2efa4658fb795c38d1090ec9b3e8a6c3683d4441fa517f09c3b/lxml-6.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:787b2496d0dbe8cd180984e8d29e3a6f76e7ea34db781cb3bd55e4ba1ef8b4ee", size = 8827545, upload-time = "2026-05-18T19:18:41.193Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/c0/73/74573db19baa618d5f266f2407898b087ff6927115b00b71e5fc1b700847/lxml-6.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2c8daa471358dc2d6fcf02165e80ec68f77871a286df95bc5cc3816153b0fd2c", size = 4735736, upload-time = "2026-05-18T19:18:46.761Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/16/02/6f7061f4f95f51e545d48e87647c54791d204a4e881be4156e7a26ba5338/lxml-6.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acd7d70b64c0aae0c7922cca83d288a16f5f6da523637697872253415269baef", size = 4970291, upload-time = "2026-05-18T19:19:56.215Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b0/02/55fc057d8283427dea7d6edb102e7a840239c77a64a983d92f62a304c0e9/lxml-6.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4f0dd2f01f9f8a89f565d000e03abcf0a13d692a346c8d22f628d49af098777a", size = 5102822, upload-time = "2026-05-18T19:19:59.223Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/e4/48/8e1cf78d89d66850121d9255a2a24414c98f775da93b90cf976956c24b14/lxml-6.1.1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b7e8a14c8634bf6f7a568634cb395305a6d964aeb5b7ee32248094bed3a7e2c", size = 5027923, upload-time = "2026-05-18T19:20:01.549Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/ed/00/0632a0647612c8af24d26997b3b961397daa9d5b2581444805933629a4cb/lxml-6.1.1-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86281fbdd6a8162756f8d603f37e3435bfa38043adb79c6dc6a2dfee065e7525", size = 5595843, upload-time = "2026-05-18T19:20:03.93Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/bc/86/ab008a7dc360711b66858d61c80a5979a70a09f2aa2b05d9698df80b803d/lxml-6.1.1-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5d7152ec39ca7c402d8fb9bad86140a15b9503bd0c54484e3f1bbe3dd37ceca", size = 5224515, upload-time = "2026-05-18T19:20:06.381Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/75/c6/2702ff375e728e34f56d9a45339a9cf7e4427e917f542225242d63a05afa/lxml-6.1.1-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:88d8cb75b9d82858497a5393e3c63cfbf03035225e4b35a49ed7ccb151e4dc0e", size = 5312511, upload-time = "2026-05-18T19:20:09.308Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b7/57/a5807c98f87a86f10ef9ffab35516df7c0f0c4b6d5d33e9f608ab9c04a31/lxml-6.1.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f64ec5397ea6a41fc1b4af0380d79b44a755b5531dcaccd9940fb260dca93038", size = 4639206, upload-time = "2026-05-18T19:20:11.704Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/1f/e1/8a0a2c35734812395f4da4eaf33748a7e5705bfb2a58b128da764339d5ec/lxml-6.1.1-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d34bbf07dbc7ca5970671b1512e928991fb5e9d95365636c9b2d8b4f53af405e", size = 5232404, upload-time = "2026-05-18T19:20:14.064Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/c2/e2/0e6a4dd5ad84d01d99aa7bae7cfefd4a760a0e0f8176818241de17d9b6c0/lxml-6.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:17e0e18d4ad8adbd0399291bc44845b69d9dd68439a3cdebdf35ff902ec05072", size = 5083769, upload-time = "2026-05-18T19:19:23.758Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a0/7e/161f33d463f6ffc1c7679104b65086dea120080d49dde4d238f015aaee2f/lxml-6.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:3ab541146f1f6968c462d6c2ac495148e8cdba2f8347700b2141b6ec5a75bf52", size = 4758936, upload-time = "2026-05-18T19:19:27.256Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/f1/fb/2369825e3f6ca99305bf9f7b7085fda91c8b0922a89e54d900974aa3ef85/lxml-6.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2a0217714657e023ef4293500f65aa20fce6164c8fd6b08fa5bd4a859fb14b9b", size = 5620296, upload-time = "2026-05-18T19:19:29.993Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/30/90/d61e383146f74c5ab683947ea14dc7b82778838ab9b95ea73a23b60d0191/lxml-6.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:05a82eb6e1530a64f26225b55cbd178113bd0b5af1c2b625f25e5296742c26d2", size = 5228598, upload-time = "2026-05-18T19:19:33.523Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/76/2d/2dafd8149e94b05bb070690efd5bb2680720681e03ff03fc57d2b70a1105/lxml-6.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9e36f163528fc50cbef305f02a5fd66d404edf7049cdaff211dbc2cba5a7013e", size = 5247845, upload-time = "2026-05-18T19:19:36.649Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/ce/68/b30e913340c380ddac9580c6e6230991fc37240ec4f64704833e4f3e2769/lxml-6.1.1-cp314-cp314t-win32.whl", hash = "sha256:649dda677cf3bd6ac9ae14007ba0c824ded8ce5808b53fc7431d9140399118c1", size = 3897345, upload-time = "2026-05-18T19:17:33.562Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/3c/4e/9eb2af5335545f9fbcd7af57bcf87c6025d31eaa31b14ec184a6c8675328/lxml-6.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:793033d6c5cdf33a573f910d9bea14ef8f5771820411d118da8e1182edb53d5e", size = 4393350, upload-time = "2026-05-18T19:18:10.076Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/7f/2c/0f1e93c636720e8a3eb59af2bfda99d98b55891e1c53bc30c2e0e865f01b/lxml-6.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:58bb955caba94e467d2a96da17660d2d704e0675894cba21ab8a775b8621fd1c", size = 3817223, upload-time = "2026-05-19T19:22:56.823Z" }, +] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -818,25 +965,47 @@ dependencies = [ ] [package.optional-dependencies] +all = [ + { name = "beautifulsoup4" }, + { name = "ebooklib" }, + { name = "openai" }, + { name = "pymupdf" }, + { name = "requests" }, +] dev = [ { name = "httpx" }, { name = "pytest" }, ] +epub = [ + { name = "ebooklib" }, +] +html = [ + { name = "beautifulsoup4" }, +] +pdf = [ + { name = "pymupdf" }, +] [package.metadata] requires-dist = [ + { name = "beautifulsoup4", marker = "extra == 'html'", specifier = ">=4.12.0" }, { name = "chromadb", specifier = ">=0.5.0" }, + { name = "ebooklib", marker = "extra == 'epub'", specifier = ">=0.18" }, { name = "fastapi", specifier = ">=0.115.0" }, { name = "httpx", marker = "extra == 'dev'", specifier = ">=0.27.0" }, { name = "markdown-it-py", specifier = ">=3.0.0" }, + { name = "md-vector-db", extras = ["pdf", "html", "epub"], marker = "extra == 'all'" }, + { name = "openai", marker = "extra == 'all'", specifier = ">=1.0.0" }, + { name = "pymupdf", marker = "extra == 'pdf'", specifier = ">=1.24.0" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0" }, { name = "python-dotenv", specifier = ">=1.2.2" }, { name = "pyyaml", specifier = ">=6.0" }, + { name = "requests", marker = "extra == 'all'", specifier = ">=2.31.0" }, { name = "sentence-transformers", specifier = ">=3.0.0" }, { name = "typer", specifier = ">=0.12.0" }, { name = "uvicorn", extras = ["standard"], specifier = ">=0.30.0" }, ] -provides-extras = ["dev"] +provides-extras = ["dev", "pdf", "html", "epub", "all"] [[package]] name = "mdurl" @@ -1215,6 +1384,25 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b7/f6/2bac21f722aa45d876d4a51f26bd0ef30e704068a3cd5021a5a7cd784271/onnxruntime-1.27.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:370d211e1ceeac4cd5f45301655463ac59e27cdc74d9f7aeb2d19ff4b7a76715", size = 18670781, upload-time = "2026-06-15T22:43:17.151Z" }, ] +[[package]] +name = "openai" +version = "2.45.0" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +dependencies = [ + { name = "anyio" }, + { name = "distro" }, + { name = "httpx" }, + { name = "jiter" }, + { name = "pydantic" }, + { name = "sniffio" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/78/60/d4219875289b11d2c2f7da93c36283da224a2e55865ed865ab64e0ce9217/openai-2.45.0.tar.gz", hash = "sha256:10d34ca9c5643bce775852fddbfc172505cb1d4de1ccd101696c3ecff358765d", size = 1109653, upload-time = "2026-07-09T18:02:44.091Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/f1/b0/2291689e3ec4723fbf5bbf3b54afcd7b160f9ddc98ca7aedfd0132af5677/openai-2.45.0-py3-none-any.whl", hash = "sha256:5df105f5f8c9b711fcb9d06d2d3888cebc82506db216484c14a4e53cdf651777", size = 1629470, upload-time = "2026-07-09T18:02:42.21Z" }, +] + [[package]] name = "opentelemetry-api" version = "1.43.0" @@ -1641,6 +1829,23 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, ] +[[package]] +name = "pymupdf" +version = "1.28.0" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/8e/e9/6d6c5d6c0a3551bffd47681a6240caf941727f195b45593cf20ab36f018f/pymupdf-1.28.0.tar.gz", hash = "sha256:e53f3567403a92da15caa9e7ae0164327fff48817e9f40175367fb9de524258d", size = 87637751, upload-time = "2026-06-29T09:08:47.547Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/c8/b7/88043e38cc7529de070f0c9bd267fa258035cca0b4ad5260536b994594a7/pymupdf-1.28.0-cp310-abi3-macosx_10_15_x86_64.whl", hash = "sha256:892b89ba88e8f98b53133b62877a9dc9b5e7dc6a4aeb837b612db56a8d2e03ac", size = 24597385, upload-time = "2026-06-29T09:03:30.608Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/33/f4/23775bbda0781b61fc398cc75079a2b0e64696d8fcf93271748883e9627e/pymupdf-1.28.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:4d692dcf44d3566ae96bc6f6346c6ad432274a29ba617bf7a9fe18009e24adb4", size = 23828292, upload-time = "2026-06-29T09:03:46.129Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/1c/f5/bf75fc7a415722f8b33662054f82d88520c0cbfd4c36d0e08aeaec605e49/pymupdf-1.28.0-cp310-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:47a5c29ed4eb0744de9c4e37bb49b1259b18d4d75fcc8a7c130f7c9fa15956f6", size = 25045507, upload-time = "2026-06-29T09:04:03.86Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/58/69/5d12c9f1f2d76f28383d6110a069c79fbfced5a4f97bb1ee6e8354f52bb7/pymupdf-1.28.0-cp310-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:44f0973f5e5edbaec95bc34b64e71d1959d4ee90b1328de1b4f4f5b4fa78673f", size = 25716599, upload-time = "2026-06-29T09:04:19.367Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/4d/b4/ec0e017bc42857cc86bd651441dbc41cc18be48d4698ecd27aac491e0c9a/pymupdf-1.28.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4d61ec323a706e153a12e262e51febfb43eeaa20977785ace135d18d48bcdc83", size = 25940489, upload-time = "2026-06-29T09:04:36.624Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/06/86/f831fef09013f33b3c9c09fb3923f2ff53e1e437f6ace14b8ae46392f558/pymupdf-1.28.0-cp310-abi3-win32.whl", hash = "sha256:caea2b3b67347fd79e5d15ed7929b0e886aac594ea228073b6d39de0078189da", size = 18489703, upload-time = "2026-06-29T20:50:30.599Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/2e/5d/1a03f53eb0449900469335fcfc742ca28e3ba159b7d650e0921d50b8b308/pymupdf-1.28.0-cp310-abi3-win_amd64.whl", hash = "sha256:e01e90fd86abfeb37ceb921eddb951f988a11d45ff6ce6b7664f2039849068ec", size = 19773102, upload-time = "2026-06-29T09:04:49.773Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/72/f6/1e52ce243ca792254f6223b4017c5667194c146ce9b88baf37bc5eb3d1c9/pymupdf-1.28.0-cp313-abi3-pyemscripten_2025_0_wasm32.whl", hash = "sha256:74c6d00ba2a9aad3a635db73b07c15db462b480741d831a34a75a56535ebc22b", size = 18357011, upload-time = "2026-06-29T20:50:50.353Z" }, + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/62/b1/46b5b3d8ef3cc71114667cf10c4d8b33f39af97253af32e9a0986775b638/pymupdf-1.28.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:b3e1399c7a64c6914239116a369efcdaac4cfb9e838bde2656d7accc4a85c72d", size = 25753599, upload-time = "2026-06-29T09:05:09.398Z" }, +] + [[package]] name = "pypika" version = "0.51.1" @@ -2084,6 +2289,24 @@ wheels = [ { url = "https://pypi.tuna.tsinghua.edu.cn/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] +[[package]] +name = "sniffio" +version = "1.3.1" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, +] + +[[package]] +name = "soupsieve" +version = "2.8.4" +source = { registry = "https://pypi.tuna.tsinghua.edu.cn/simple/" } +sdist = { url = "https://pypi.tuna.tsinghua.edu.cn/packages/47/2c/0a5f6f8ee0d5589e48c7640213ed5175d52cf540a06725b628cc1a45d6ce/soupsieve-2.8.4.tar.gz", hash = "sha256:e121fd02e975c695e4e9e8774a5ee35d74714b59307868dcc5319ad2d9e3328e", size = 121110, upload-time = "2026-05-24T13:55:57.154Z" } +wheels = [ + { url = "https://pypi.tuna.tsinghua.edu.cn/packages/5e/f5/0c41cb68dcae6b7de4fac4188a3a9589e21fb31df21ea3a2e888db95e6c9/soupsieve-2.8.4-py3-none-any.whl", hash = "sha256:e7e6b0769c8f51ed59acab6e994b00621096cfb1c640a7509295987388fbaf65", size = 37304, upload-time = "2026-05-24T13:55:55.406Z" }, +] + [[package]] name = "starlette" version = "1.3.1"