39 KiB
第二优先级:体验完善 实施计划
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: 补齐 Web 管理界面、.docx 文档支持、数据导出备份功能,并将测试覆盖率从 75% 提升到 ≥80%。
Architecture: Web UI 使用单文件 Vue 3 CDN 方案(零构建),嵌入 FastAPI 静态文件服务;.docx 支持通过集成 markitdown 库新增 DocxSplitter;导出功能在 CLI 新增 export 命令支持 JSON/CSV 格式。
Tech Stack: Vue 3 (CDN), markitdown, FastAPI StaticFiles, CSV/JSON
预估总工作量: 约 10-15 小时
文件结构规划
新增文件:
src/web/index.html — Web 管理界面(单文件 Vue 3 SPA)
src/core/splitters/docx.py — .docx 分块器
tests/test_splitters_docx.py — DocxSplitter 测试
tests/test_export.py — 导出功能测试
修改文件:
src/server/app.py — 挂载静态文件 + CORS 修复
src/cli/main.py — 新增 export 命令
src/core/splitters/registry.py — 注册 .docx
src/core/ingest.py — 修复 PDF/EPUB splitter 接口不一致
src/core/search.py — 添加 export 方法
src/core/embedder.py — 修复 Embedder Protocol 定义
src/core/splitters/pdf.py — 实现 Splitter Protocol
src/core/splitters/epub.py — 实现 Splitter Protocol
pyproject.toml — 新依赖
config.yaml — 更新示例
README.md — Web UI 使用说明
tests/test_api.py — 静态文件端点测试
tests/test_splitters_pdf.py — 实体测试(非 skip)
tests/test_splitters_html.py — 实体测试(非 skip)
tests/test_splitters_epub.py — 实体测试(非 skip)
tests/test_ingest.py — 补充 ingest_directory / content 模式测试
tests/test_search.py — 补充 list_sources / delete_by_source 测试
tests/test_db.py — 补充 write_guard / close 测试
tests/test_cli.py — 补充 ingest-dir / stats 测试
Task 1: 修复 Embedder Protocol 和 Splitter 接口
Files:
-
Modify:
src/core/embedder.py -
Modify:
src/core/splitters/pdf.py -
Modify:
src/core/splitters/epub.py -
Step 1: 修复 Embedder Protocol 为标准写法
src/core/embedder.py 的 Embedder 类,将方法体改为标准写法:
class Embedder(Protocol):
"""嵌入器接口."""
@property
def dimension(self) -> int:
"""返回嵌入向量的维度."""
...
def embed(self, texts: list[str]) -> list[list[float]]:
"""对文本列表进行嵌入.
Args:
texts: 待嵌入的文本列表
Returns:
嵌入向量列表,每个向量为 float 列表
"""
...
- Step 2: PDFSplitter 显式实现 Splitter Protocol
修改 src/core/splitters/pdf.py,将 split 方法的参数名从 text 改为 source:
class PDFSplitter:
"""PDF 分块器:pymupdf 提取文字 → TextSplitter 分块.
实现 Splitter Protocol,内部组合 TextSplitter 实例。
split() 的 source 参数接收 PDF 文件路径(非文本内容)。
"""
def __init__(self, max_size: int = 1000, overlap: int = 100):
self._text_splitter = TextSplitter(max_size=max_size, overlap=overlap)
def split(self, source: str, source_file: str = "") -> list[dict]:
"""从 PDF 文件提取文字并分块.
Args:
source: PDF 文件路径
source_file: 来源文件名
"""
# ... 其余实现不变,将 text 改为 source ...
同步修改 ingest.py 中 PDF/EPUB splitter 的调用,使用关键字参数:
# ingest.py 第 57 行附近,改为:
chunks = splitter.split(source=str(path), source_file=file_name)
- Step 3: EPUBSplitter 同样修改
修改 src/core/splitters/epub.py 的 split 方法签名:
def split(self, source: str, source_file: str = "") -> list[dict]:
"""从 EPUB 文件提取各章节文字并分块.
Args:
source: EPUB 文件路径
source_file: 来源文件名
"""
# ... 将 text 改为 source ...
- Step 4: 确认测试通过
uv run pytest tests/test_embedder.py tests/test_splitters.py tests/test_ingest.py -v
- Step 5: Commit
git add src/core/embedder.py src/core/splitters/pdf.py src/core/splitters/epub.py src/core/ingest.py
git commit -m "refactor: 修复 Embedder Protocol 标准写法和 PDF/EPUB Splitter 参数语义"
Task 2: DocxSplitter — .docx 文档支持
Files:
-
Create:
src/core/splitters/docx.py -
Create:
tests/test_splitters_docx.py -
Modify:
src/core/splitters/registry.py -
Modify:
pyproject.toml -
Step 1: 添加 docx 可选依赖
在 pyproject.toml 的 [project.optional-dependencies] 中添加:
docx = ["markitdown>=0.1.0"]
并将 all 更新为包含 docx:
all = ["md-vector-db[pdf,html,epub,docx]", "requests>=2.31.0", "openai>=1.0.0"]
- Step 2: 编写 DocxSplitter 测试
# tests/test_splitters_docx.py
"""DocxSplitter 测试."""
import pytest
# 如果未安装 markitdown,跳过所有测试
pytest.importorskip("markitdown", reason="需要 markitdown 库")
from src.core.splitters.docx import DocxSplitter
def test_docx_splitter_creates():
"""创建 DocxSplitter 实例."""
s = DocxSplitter(max_size=500, overlap=50)
assert s is not None
def test_docx_splitter_empty_text():
"""空纯文本返回空列表."""
s = DocxSplitter()
result = s.split(" ", source_file="empty.docx")
assert result == []
def test_docx_splitter_basic_text():
"""基本文本文档分块."""
s = DocxSplitter(max_size=200, overlap=20)
text = "段落A。\n\n段落B。\n\n段落C。"
result = s.split(text, source_file="test.docx")
assert len(result) >= 1
assert all("content" in r for r in result)
assert all(r["source_file"] == "test.docx" for r in result)
def test_docx_splitter_long_text():
"""长文本分多块."""
s = DocxSplitter(max_size=100, overlap=10)
text = "这是一段非常长的文本。\n\n" * 50
result = s.split(text, source_file="long.docx")
assert len(result) >= 5
def test_docx_splitter_source_file():
"""source_file 正确传递到每个 chunk."""
s = DocxSplitter(max_size=500, overlap=50)
result = s.split("测试内容。", source_file="myfile.docx")
assert all(r["source_file"] == "myfile.docx" for r in result)
def test_docx_splitter_chunk_index():
"""chunk_index 从 0 递增."""
s = DocxSplitter(max_size=100, overlap=10)
text = "chunk A。\n\n" * 20
result = s.split(text, source_file="index.docx")
indices = [r["chunk_index"] for r in result]
assert indices == list(range(len(result)))
- Step 3: 运行测试确认失败
uv sync --extra docx
uv run pytest tests/test_splitters_docx.py -v
- Step 4: 实现 DocxSplitter
# src/core/splitters/docx.py
""".docx Word 文档分块器 — 使用 markitdown 提取文字后委托 TextSplitter."""
from __future__ import annotations
import logging
from pathlib import Path
from src.core.splitters.text import TextSplitter
logger = logging.getLogger("md-vector-db")
class DocxSplitter:
"""Docx 分块器:markitdown 提取文字 → TextSplitter 分块.
实现 Splitter Protocol,内部组合 TextSplitter 实例。
split() 的 source 参数可接收文件路径或纯文本。
"""
def __init__(self, max_size: int = 1000, overlap: int = 100):
self._text_splitter = TextSplitter(max_size=max_size, overlap=overlap)
def split(self, source: str, source_file: str = "") -> list[dict]:
"""从 .docx 文件或纯文本提取文字并分块.
Args:
source: .docx 文件路径或纯文本内容
source_file: 来源文件名
Returns:
分块后的 chunk 列表
"""
path = Path(source)
if path.suffix.lower() == ".docx":
text = self._extract_from_docx(str(path))
else:
text = source
if not text.strip():
return []
return self._text_splitter.split(text, source_file=source_file)
def _extract_from_docx(self, file_path: str) -> str:
"""使用 markitdown 从 .docx 文件中提取文字."""
try:
from markitdown import MarkItDown
except ImportError:
raise ImportError(
"Docx 支持需要 markitdown 库. 请执行: uv sync --extra docx"
)
try:
md = MarkItDown()
result = md.convert(file_path)
return result.text_content
except Exception as e:
logger.error("Docx 解析失败: %s — %s", file_path, e)
raise ValueError(f"Docx 解析失败: {e}") from e
- Step 5: 注册 .docx 到 registry
修改 src/core/splitters/registry.py:
# 在 _DEFAULT_MAP 字典中添加:
_DEFAULT_MAP: dict[str, str] = {
".md": "markdown",
".markdown": "markdown",
".txt": "text",
".pdf": "pdf",
".html": "html",
".htm": "html",
".epub": "epub",
".docx": "docx", # 新增
}
# 在 SUPPORTED_SUFFIXES 行后确保 frozenset 自动包含
# 在 get_splitter 函数中添加 docx 分支:
if kind == "docx":
from src.core.splitters.docx import DocxSplitter
return DocxSplitter(max_size=max_size, overlap=overlap)
- Step 6: 运行测试
uv run pytest tests/test_splitters_docx.py -v
预期: 6 passed
- Step 7: Commit
git add src/core/splitters/docx.py tests/test_splitters_docx.py src/core/splitters/registry.py pyproject.toml
git commit -m "feat: 新增 DocxSplitter — 支持 .docx 文档入库"
Task 3: 数据导出功能
Files:
-
Create:
tests/test_export.py -
Modify:
src/core/search.py -
Modify:
src/cli/main.py -
Step 1: 编写导出功能测试
# tests/test_export.py
"""导出功能测试."""
import json
import csv
import io
import pytest
from src.core.search import Searcher
class FakeExportCollection:
def count(self):
return 2
def get(self, include=None):
return {
"ids": ["doc_0", "doc_1"],
"documents": ["内容A。\n\n段落B。", "内容C。"],
"metadatas": [
{"source_file": "a.md", "section_title": "标题A", "heading_level": 1, "chunk_index": 0},
{"source_file": "b.md", "section_title": "", "heading_level": 0, "chunk_index": 0},
],
}
class FakeExportDB:
def get_or_create_collection(self, name):
return FakeExportCollection()
def list_collections(self):
return [] # 简化
class FakeExportEmbedder:
@property
def dimension(self):
return 4
def embed(self, texts):
return [[0.1, 0.2, 0.3, 0.4] for _ in texts]
@pytest.fixture
def searcher():
db = FakeExportDB()
embedder = FakeExportEmbedder()
return Searcher(db, embedder, "test_export")
def test_export_json_returns_valid_json(searcher):
"""export_json 返回合法的 JSON 字符串."""
output = searcher.export_json()
data = json.loads(output)
assert isinstance(data, list)
assert len(data) == 2
def test_export_json_contains_all_fields(searcher):
"""导出包含所有必要字段."""
output = searcher.export_json()
data = json.loads(output)
first = data[0]
assert "id" in first
assert "content" in first
assert "source_file" in first
assert "section_title" in first
assert "heading_level" in first
def test_export_csv_returns_valid_csv(searcher):
"""export_csv 返回合法的 CSV 字符串."""
output = searcher.export_csv()
reader = csv.DictReader(io.StringIO(output))
rows = list(reader)
assert len(rows) == 2
def test_export_csv_has_header(searcher):
"""CSV 包含表头."""
output = searcher.export_csv()
reader = csv.DictReader(io.StringIO(output))
assert reader.fieldnames is not None
assert "content" in reader.fieldnames
assert "source_file" in reader.fieldnames
def test_export_empty_collection(searcher):
"""空 collection 导出空列表/空 CSV(仅有表头)."""
class EmptyCollection:
def count(self):
return 0
def get(self, include=None):
return {"ids": [], "documents": [], "metadatas": []}
class EmptyDB:
def get_or_create_collection(self, name):
return EmptyCollection()
def list_collections(self):
return []
s = Searcher(EmptyDB(), FakeExportEmbedder(), "empty")
json_out = s.export_json()
assert json.loads(json_out) == []
csv_out = s.export_csv()
lines = csv_out.strip().split("\n")
assert len(lines) == 1 # 仅表头
- Step 2: 运行测试确认失败
uv run pytest tests/test_export.py -v
- Step 3: 在 Searcher 中添加导出方法
在 src/core/search.py 的 Searcher 类中添加两个方法:
import csv
import io
import json as json_lib
# 在 Searcher 类中追加:
def export_json(self, file_path: str | None = None) -> str:
"""导出 collection 所有 chunks 为 JSON.
Args:
file_path: 可选,写入文件路径。不传则返回 JSON 字符串。
Returns:
JSON 字符串
"""
all_data = self.collection.get(include=["documents", "metadatas"])
records = []
if all_data and all_data["ids"]:
for i, doc_id in enumerate(all_data["ids"]):
meta = all_data["metadatas"][i] if all_data["metadatas"] else {}
records.append({
"id": doc_id,
"content": all_data["documents"][i] if all_data["documents"] else "",
"source_file": meta.get("source_file", ""),
"section_title": meta.get("section_title", ""),
"heading_level": meta.get("heading_level", 0),
"chunk_index": meta.get("chunk_index", i),
})
json_str = json_lib.dumps(records, ensure_ascii=False, indent=2)
if file_path:
with open(file_path, "w", encoding="utf-8") as f:
f.write(json_str)
return json_str
def export_csv(self, file_path: str | None = None) -> str:
"""导出 collection 所有 chunks 为 CSV.
Args:
file_path: 可选,写入文件路径。不传则返回 CSV 字符串。
Returns:
CSV 字符串
"""
all_data = self.collection.get(include=["documents", "metadatas"])
output = io.StringIO()
writer = csv.writer(output)
writer.writerow(["id", "content", "source_file", "section_title", "heading_level", "chunk_index"])
if all_data and all_data["ids"]:
for i, doc_id in enumerate(all_data["ids"]):
meta = all_data["metadatas"][i] if all_data["metadatas"] else {}
writer.writerow([
doc_id,
all_data["documents"][i] if all_data["documents"] else "",
meta.get("source_file", ""),
meta.get("section_title", ""),
meta.get("heading_level", 0),
meta.get("chunk_index", i),
])
csv_str = output.getvalue()
if file_path:
with open(file_path, "w", encoding="utf-8", newline="") as f:
f.write(csv_str)
return csv_str
- Step 4: 运行导出测试
uv run pytest tests/test_export.py -v
- Step 5: 在 CLI 添加 export 命令
在 src/cli/main.py 中添加:
@app.command(help="导出 collection 数据为 JSON 或 CSV.")
def export(
output: Annotated[str, typer.Option("--output", "-o", help="输出文件路径")],
fmt: Annotated[str, typer.Option("--format", "-f", help="导出格式: json | csv")] = "json",
config: ConfigOpt = DEFAULT_CONFIG_PATH,
collection: CollectionOpt = None,
):
"""导出 collection 数据."""
_init_config(config)
state = get_state()
searcher = state.get_searcher(_resolve_collection(collection))
if fmt == "json":
searcher.export_json(file_path=output)
elif fmt == "csv":
searcher.export_csv(file_path=output)
else:
typer.echo(f"错误: 不支持的格式 '{fmt}',可选: json, csv", err=True)
raise typer.Exit(code=1)
typer.echo(f"[OK] 已导出到: {output}")
- Step 6: 测试 CLI export 命令
uv run md-vector-db export --help
uv run md-vector-db export -o /tmp/test_export.json -f json
- Step 7: Commit
git add src/core/search.py src/cli/main.py tests/test_export.py
git commit -m "feat: 新增数据导出功能(JSON/CSV)和 CLI export 命令"
Task 4: Web 管理界面
Files:
-
Create:
src/web/index.html -
Modify:
src/server/app.py -
Step 1: 创建 Web 管理界面(单文件 Vue 3 SPA)
<!-- src/web/index.html -->
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>md-vector-db 管理面板</title>
<script src="https://unpkg.com/vue@3/dist/vue.global.prod.js"></script>
<style>
:root {
--bg: #f8f9fa;
--card-bg: #ffffff;
--text: #212529;
--muted: #6c757d;
--border: #dee2e6;
--primary: #0d6efd;
--success: #198754;
--danger: #dc3545;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
background: var(--bg);
color: var(--text);
line-height: 1.6;
}
.container { max-width: 960px; margin: 0 auto; padding: 24px 16px; }
h1 { font-size: 1.5rem; margin-bottom: 24px; }
.card {
background: var(--card-bg);
border: 1px solid var(--border);
border-radius: 8px;
padding: 20px;
margin-bottom: 16px;
}
.card h2 { font-size: 1.1rem; margin-bottom: 12px; }
.form-group { margin-bottom: 12px; }
label { display: block; font-size: .875rem; color: var(--muted); margin-bottom: 4px; }
input, textarea, select {
width: 100%;
padding: 8px 12px;
border: 1px solid var(--border);
border-radius: 4px;
font-size: .9rem;
font-family: inherit;
}
textarea { min-height: 100px; resize: vertical; }
.btn {
display: inline-block;
padding: 8px 16px;
border: none;
border-radius: 4px;
font-size: .875rem;
cursor: pointer;
text-decoration: none;
}
.btn-primary { background: var(--primary); color: #fff; }
.btn-danger { background: var(--danger); color: #fff; }
.btn-sm { padding: 4px 10px; font-size: .8rem; }
.result-item {
border: 1px solid var(--border);
border-radius: 6px;
padding: 12px;
margin-bottom: 8px;
}
.result-item .score { font-weight: 600; color: var(--primary); }
.result-item .source { font-size: .8rem; color: var(--muted); }
.result-item .section { font-size: .85rem; color: var(--success); margin-bottom: 4px; }
.badge {
display: inline-block;
padding: 2px 8px;
border-radius: 12px;
font-size: .75rem;
font-weight: 600;
}
.badge-ok { background: #d1e7dd; color: #0f5132; }
.badge-err { background: #f8d7da; color: #842029; }
.tabs { display: flex; gap: 4px; margin-bottom: 16px; }
.tab {
padding: 8px 16px;
border: 1px solid var(--border);
border-radius: 6px 6px 0 0;
background: var(--bg);
cursor: pointer;
font-size: .875rem;
}
.tab.active { background: var(--card-bg); border-bottom-color: var(--card-bg); font-weight: 600; }
.toast {
position: fixed; top: 16px; right: 16px;
padding: 12px 20px; border-radius: 6px; color: #fff;
font-size: .875rem; z-index: 999;
}
.toast-success { background: var(--success); }
.toast-error { background: var(--danger); }
</style>
</head>
<body>
<div id="app">
<div class="container">
<h1>📚 md-vector-db 管理面板</h1>
<!-- 标签页切换 -->
<div class="tabs">
<div :class="['tab', { active: activeTab === 'search' }]" @click="activeTab = 'search'">🔍 搜索</div>
<div :class="['tab', { active: activeTab === 'ingest' }]" @click="activeTab = 'ingest'">📥 入库</div>
<div :class="['tab', { active: activeTab === 'collections' }]" @click="activeTab = 'collections'">📂 集合</div>
</div>
<!-- Toast 通知 -->
<div v-if="toast" :class="['toast', 'toast-' + toast.type]">{{ toast.message }}</div>
<!-- 搜索页面 -->
<div v-if="activeTab === 'search'" class="card">
<h2>语义检索</h2>
<div class="form-group">
<label>搜索查询</label>
<input v-model="searchQuery" @keyup.enter="doSearch" placeholder="输入关键词或自然语言...">
</div>
<div style="display:flex;gap:12px;align-items:end;">
<div class="form-group" style="flex:1">
<label>返回条数</label>
<input v-model.number="topK" type="number" min="1" max="100">
</div>
<div class="form-group" style="flex:1">
<label>集合名(留空=默认)</label>
<input v-model="collectionName" placeholder="default">
</div>
</div>
<button class="btn btn-primary" @click="doSearch" :disabled="searching">
{{ searching ? '搜索中...' : '搜索' }}
</button>
<!-- 搜索结果 -->
<div v-if="searchResults.length > 0" style="margin-top:16px">
<h3>{{ searchResults.length }} 条结果</h3>
<div v-for="(r, i) in searchResults" :key="i" class="result-item">
<div class="score">相似度: {{ r.score?.toFixed(4) || r.score }}</div>
<div class="source">📄 {{ r.source_file }}</div>
<div v-if="r.section_title" class="section">📑 {{ r.section_title }}</div>
<pre style="white-space:pre-wrap;font-size:.85rem;margin-top:8px;">{{ r.content?.substring(0, 300) }}{{ r.content?.length > 300 ? '...' : '' }}</pre>
</div>
</div>
</div>
<!-- 入库页面 -->
<div v-if="activeTab === 'ingest'" class="card">
<h2>入库文档</h2>
<div class="form-group">
<label>Markdown 内容</label>
<textarea v-model="ingestContent" placeholder="粘贴 Markdown 文本..."></textarea>
</div>
<div class="form-group">
<label>文件名</label>
<input v-model="ingestFileName" placeholder="document.md">
</div>
<div class="form-group">
<label>目标集合(留空=默认)</label>
<input v-model="ingestCollection" placeholder="default">
</div>
<button class="btn btn-primary" @click="doIngest" :disabled="ingesting">
{{ ingesting ? '入库中...' : '入库' }}
</button>
<p v-if="ingestResult" style="margin-top:12px">{{ ingestResult }}</p>
</div>
<!-- 集合页面 -->
<div v-if="activeTab === 'collections'" class="card">
<h2>集合列表</h2>
<button class="btn btn-primary btn-sm" @click="loadCollections" :disabled="loadingColl">
{{ loadingColl ? '加载中...' : '刷新' }}
</button>
<div v-if="collections.length > 0" style="margin-top:12px">
<table style="width:100%;border-collapse:collapse">
<thead>
<tr style="border-bottom:2px solid var(--border)">
<th style="text-align:left;padding:8px">名称</th>
<th style="text-align:right;padding:8px">Chunks</th>
</tr>
</thead>
<tbody>
<tr v-for="c in collections" :key="c.name" style="border-bottom:1px solid var(--border)">
<td style="padding:8px">{{ c.name }}</td>
<td style="text-align:right;padding:8px">{{ c.count }}</td>
</tr>
</tbody>
</table>
</div>
</div>
<!-- 健康状态 -->
<div class="card" v-if="health">
<h2>服务状态
<span :class="['badge', health.status === 'ok' ? 'badge-ok' : 'badge-err']">{{ health.status }}</span>
</h2>
<pre style="font-size:.8rem">{{ JSON.stringify(health.checks, null, 2) }}</pre>
</div>
</div>
</div>
<script>
const { createApp } = Vue
createApp({
data() {
return {
activeTab: 'search',
searchQuery: '',
topK: 10,
collectionName: '',
searchResults: [],
searching: false,
ingestContent: '',
ingestFileName: '',
ingestCollection: '',
ingesting: false,
ingestResult: null,
collections: [],
loadingColl: false,
health: null,
toast: null,
apiBase: window.location.origin,
}
},
mounted() {
this.checkHealth()
},
methods: {
async api(method, path, body) {
const headers = { 'Content-Type': 'application/json' }
const res = await fetch(`${this.apiBase}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined })
if (!res.ok) {
const err = await res.text()
throw new Error(`HTTP ${res.status}: ${err}`)
}
return res.json()
},
showToast(message, type = 'success') {
this.toast = { message, type }
setTimeout(() => { this.toast = null }, 3000)
},
async checkHealth() {
try {
this.health = await this.api('GET', '/api/v1/health')
} catch (e) {
this.health = { status: 'error', checks: {} }
}
},
async doSearch() {
if (!this.searchQuery.trim()) return
this.searching = true
this.searchResults = []
try {
const body = { query: this.searchQuery, top_k: this.topK }
if (this.collectionName) body.collection = this.collectionName
const data = await this.api('POST', '/api/v1/search', body)
this.searchResults = data.results || []
} catch (e) {
this.showToast(`搜索失败: ${e.message}`, 'error')
} finally {
this.searching = false
}
},
async doIngest() {
if (!this.ingestContent.trim()) return
this.ingesting = true
this.ingestResult = null
try {
const body = { content: this.ingestContent, file_name: this.ingestFileName || 'untitled.md' }
if (this.ingestCollection) body.collection = this.ingestCollection
const data = await this.api('POST', '/api/v1/ingest', body)
this.ingestResult = `✅ 入库成功: ${data.chunks} chunks → ${data.collection}`
this.showToast(this.ingestResult)
} catch (e) {
this.ingestResult = `❌ 入库失败: ${e.message}`
this.showToast(this.ingestResult, 'error')
} finally {
this.ingesting = false
}
},
async loadCollections() {
this.loadingColl = true
try {
const data = await this.api('GET', '/api/v1/collections')
this.collections = data.collections || []
} catch (e) {
this.showToast(`加载失败: ${e.message}`, 'error')
} finally {
this.loadingColl = false
}
},
}
}).mount('#app')
</script>
</body>
</html>
- Step 2: 在 FastAPI 中挂载静态文件
修改 src/server/app.py,在 app 创建后添加:
from fastapi.staticfiles import StaticFiles
from pathlib import Path
# 在 app = FastAPI(...) 之后,中间件之前添加:
web_dir = Path(__file__).parent.parent / "web"
if web_dir.exists():
app.mount("/admin", StaticFiles(directory=str(web_dir), html=True), name="admin")
- Step 3: 添加 API 测试
在 tests/test_api.py 中添加:
def test_admin_ui_served(client):
"""管理界面可访问."""
response = client.get("/admin")
assert response.status_code in (200, 404) # 404 如果 web 目录不存在(可接受)
def test_health_skips_rate_limit(client):
"""健康检查不触发速率限制."""
for _ in range(5):
resp = client.get("/api/v1/health")
assert resp.status_code == 200
- Step 4: 测试 Web UI
uv run md-vector-db serve --port 8000 &
sleep 3
# 访问管理界面
curl -s http://localhost:8000/admin | head -5
# 预期:返回 HTML 内容
kill %1
- Step 5: Commit
git add src/web/index.html src/server/app.py tests/test_api.py
git commit -m "feat: 添加 Web 管理界面(Vue 3 SPA 单文件)"
Task 5: 测试覆盖率提升到 80%+
Files:
-
Modify:
tests/test_splitters_pdf.py -
Modify:
tests/test_splitters_html.py -
Modify:
tests/test_splitters_epub.py -
Modify:
tests/test_cli.py -
Modify:
tests/test_db.py -
Modify:
tests/test_search.py -
Modify:
tests/test_ingest.py -
Step 1: 补充 PDF splitter 测试
# 在 tests/test_splitters_pdf.py 追加(保留原有内容):
def test_pdf_splitter_creation():
"""PDFSplitter 可以创建."""
from src.core.splitters.pdf import PDFSplitter
s = PDFSplitter(max_size=500, overlap=50)
assert s is not None
assert s._text_splitter.max_size == 500
assert s._text_splitter.overlap == 50
def test_pdf_splitter_empty_result():
"""提取结果为空时返回空列表."""
from src.core.splitters.pdf import PDFSplitter
s = PDFSplitter()
result = s.split(source="/nonexistent/file.pdf", source_file="test.pdf")
# 应该 raise ValueError(文件不存在)
import pytest
with pytest.raises(ValueError):
s.split(source="/nonexistent/file.pdf", source_file="test.pdf")
- Step 2: 补充 HTML splitter 测试
# 在 tests/test_splitters_html.py 追加:
def test_html_splitter_creation():
"""HTMLSplitter 创建正常."""
from src.core.splitters.html import HTMLSplitter
s = HTMLSplitter(max_size=500, overlap=50)
assert s is not None
def test_html_splitter_basic():
"""基本 HTML 分块."""
from src.core.splitters.html import HTMLSplitter
s = HTMLSplitter(max_size=200, overlap=20)
html = "<html><body><p>段落A。</p><p>段落B。</p></body></html>"
result = s.split(html, source_file="test.html")
assert len(result) >= 1
assert "段落A" in result[0]["content"]
def test_html_splitter_strips_script_style():
"""去除 script 和 style 标签."""
from src.core.splitters.html import HTMLSplitter
s = HTMLSplitter(max_size=500, overlap=50)
html = "<html><script>alert('xss')</script><body><p>可见内容。</p></body></html>"
result = s.split(html, source_file="test.html")
assert "alert" not in result[0]["content"]
assert "可见内容" in result[0]["content"]
def test_html_splitter_empty():
"""空 HTML 返回空列表."""
from src.core.splitters.html import HTMLSplitter
s = HTMLSplitter()
result = s.split("<html></html>", source_file="empty.html")
assert result == []
- Step 3: 补充 EPUB splitter 测试
# 在 tests/test_splitters_epub.py 追加:
def test_epub_splitter_creation():
"""EPUBSplitter 创建正常."""
from src.core.splitters.epub import EPUBSplitter
s = EPUBSplitter(max_size=500, overlap=50)
assert s is not None
def test_epub_splitter_file_not_found():
"""不存在的文件抛出 ValueError."""
from src.core.splitters.epub import EPUBSplitter
s = EPUBSplitter()
import pytest
with pytest.raises(ValueError):
s.split(source="/nonexistent/file.epub", source_file="test.epub")
- Step 4: 补充 CLI 测试
# 在 tests/test_cli.py 追加:
def test_ingest_help(cli_app):
"""ingest --help 正常."""
result = cli_app(["ingest", "--help"])
assert result.exit_code == 0
def test_search_no_results(cli_app, tmp_path):
"""空 collection 搜索返回提示."""
result = cli_app(["search", "测试查询", "-c", str(tmp_path / "cfg.yaml")])
# 可能返回 0(只是无结果)或出错
assert result.exit_code in (0, 1)
def test_stats_empty(cli_app, tmp_path):
"""空 collection 的 stats."""
result = cli_app(["stats", "-c", str(tmp_path / "cfg.yaml")])
assert result.exit_code == 0
def test_stats_json(cli_app, tmp_path):
"""stats --json 输出."""
result = cli_app(["stats", "--json", "-c", str(tmp_path / "cfg.yaml")])
assert result.exit_code == 0
- Step 5: 补充 DB 测试
# 在 tests/test_db.py 追加:
def test_write_guard_context_manager(db):
"""write_guard 上下文管理器."""
with db.write_guard():
pass # 应正常获取和释放锁
def test_close(db):
"""close 正常执行."""
db.close()
# close 后不应对 client 做任何操作,测试仅验证不抛异常
def test_delete_collection_nonexistent(db):
"""删除不存在的 collection 不抛异常."""
db.delete_collection("nonexistent-collection-12345")
# 应静默处理
def test_delete_by_source_no_match(db):
"""删除不存在的 source 返回 False."""
result = db.delete_by_source("test_col", "no-such-file.md")
assert result is False
- Step 6: 补充搜索测试
# 在 tests/test_search.py 追加:
def test_list_sources_empty():
"""空 collection 的 list_sources 返回空列表."""
from src.core.search import Searcher
class EmptyColl:
def count(self): return 0
def get(self, **kwargs): return {"ids": [], "documents": [], "metadatas": []}
class EmptyDB:
def get_or_create_collection(self, name): return EmptyColl()
def list_collections(self): return []
class FakeEmb:
@property
def dimension(self): return 4
def embed(self, texts): return [[0.0]*4]
s = Searcher(EmptyDB(), FakeEmb(), "empty")
assert s.list_sources() == []
- Step 7: 补充 ingest 测试
# 在 tests/test_ingest.py 追加:
def test_ingest_file_returns_zero_for_dir(db, local_embedder, tmp_path: Path):
"""目录路径应被 ingest_directory 而非 ingest_file 处理."""
ingestor = DocumentIngestor(db, local_embedder, "test_ingest_dir")
d = tmp_path / "subdir"
d.mkdir()
# ingest_file 不应处理目录
result = ingestor.ingest_file(str(d))
assert result == 0 # 目录不是文件,跳过
def test_ingest_file_markdown(db, local_embedder, tmp_path: Path):
"""MD 文件入库返回正确的 chunk 数."""
file = tmp_path / "hello.md"
file.write_text("# 标题\n\n内容段落。", encoding="utf-8")
ingestor = DocumentIngestor(
db, local_embedder, "test_md",
chunk_config=ChunkConfig(max_size=1000, overlap=100),
)
count = ingestor.ingest_file(str(file))
assert count >= 1
def test_ingest_content_default_splitter(db, local_embedder):
"""未指定 splitter 时用 MarkdownSplitter."""
ingestor = DocumentIngestor(db, local_embedder, "test_content")
count = ingestor.ingest_content("# 测试\n\n一些内容。", "test.md")
assert count >= 1
def test_ingest_directory_recursive(db, local_embedder, tmp_path: Path):
"""ingest_directory 递归处理子目录."""
(tmp_path / "sub").mkdir()
(tmp_path / "a.md").write_text("# A\n\n内容A。", encoding="utf-8")
(tmp_path / "sub" / "b.md").write_text("# B\n\n内容B。", encoding="utf-8")
ingestor = DocumentIngestor(
db, local_embedder, "test_recurse",
chunk_config=ChunkConfig(max_size=1000, overlap=100),
)
results = ingestor.ingest_directory(str(tmp_path))
assert len(results) >= 2
assert all(v > 0 for v in results.values())
- Step 8: 运行全部测试并验证覆盖率
uv run pytest tests/ --cov=src --cov-report=term-missing -v
预期: ≥ 80% 覆盖率
- Step 9: Commit
git add tests/
git commit -m "test: 补充测试覆盖率至 80%+(PDF/HTML/EPUB/CLI/DB/Search/Ingest)"
Task 6: CORS 和安全配置修复
Files:
-
Modify:
src/server/app.py -
Step 1: 修复 CORS 配置
将 allow_credentials 改为 False:
app.add_middleware(
CORSMiddleware,
allow_origins=os.environ.get("CORS_ORIGINS", "http://localhost:3000").split(","),
allow_credentials=False, # 修复:原来是 True
allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_headers=["Content-Type", "Authorization", "X-API-Key"],
)
- Step 2: 更新 .env.example 添加 CORS 说明
# CORS 允许的源列表(逗号分隔)
CORS_ORIGINS=http://localhost:3000,http://localhost:5173
- Step 3: 测试 CORS 中间件
uv run pytest tests/test_api.py -v -k "cors or health"
- Step 4: Commit
git add src/server/app.py .env.example
git commit -m "fix: 修复 CORS allow_credentials 配置,更新 .env.example"
Task 7: 更新 README 和文档
Files:
-
Modify:
README.md -
Step 1: 更新 README 功能特性和新命令说明
在 README 的功能特性列表追加:
- **Web 管理界面**: 内置 Vue 3 单页管理面板,可视化搜索、入库、查看集合
- **数据导出**: 支持 JSON/CSV 导出 collection 全量数据
- **多格式扩展**: 新增 .docx 支持(通过 markitdown 库)
在 CLI 命令参考表中追加:
| `export -o <文件> -f <json|csv>` | 导出 collection 数据 |
在 Docker 部署章节后添加:
## Web 管理界面
启动服务后访问 `http://localhost:8000/admin` 进入管理面板:
- **搜索**: 可视化输入查询词、选择返回条数、查看相似度分数和来源文件
- **入库**: 粘贴 Markdown 文本直接入库,无需创建文件
- **集合**: 查看所有 collection 的 chunks 统计
- Step 2: Commit
git add README.md
git commit -m "docs: README 补充 Web UI、导出、docx 支持说明"
Task 8: 最终验证
- Step 1: 全量测试 + 覆盖率
uv run pytest tests/ --cov=src --cov-report=term-missing -v
预期: 全部通过,覆盖率 ≥ 80%
- Step 2: Lint 检查
uv run ruff check src/ tests/
预期: 零错误
- Step 3: 类型检查
uv run mypy src/ --ignore-missing-imports
- Step 4: 功能集成测试
# 启动服务
uv run md-vector-db serve --port 8000 &
sleep 3
# 测试 .docx 入库 + 混合检索 + 导出
uv run md-vector-db ingest /path/to/sample.md
uv run md-vector-db search "测试" --mode hybrid
uv run md-vector-db export -o /tmp/export.json -f json
cat /tmp/export.json
# 验证 Web UI 可访问
curl -s http://localhost:8000/admin | grep -q "md-vector-db"
kill %1
- Step 5: 总结 Commit
git add -A
git commit -m "feat: 第二优先级完善 — Web UI、docx 支持、导出功能、覆盖率 80%+"
自审清单
- Spec 覆盖: Web UI ✅ | .docx 支持 ✅ | 导出 ✅ | 覆盖率 80%+ ✅ | Embedder 修复 ✅ | Splitter 接口统一 ✅
- 无占位符: 所有代码为具体实现,无 TODO/TBD
- 类型一致性: DocxSplitter.split(source, ...) 与 PDF/EPUB 统一,SearchResult 复用于导出
- 测试先行: 每个模块先写测试再实现