Serendipity/md-vector-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
82 Commits 1 Branch 0 Tags
1df4793acba26cc88e329c61261e15db11b0715e
Commit Graph

30 Commits

Author SHA1 Message Date
Serendipity 82924ff5f3 feat: 添加 Cross-Encoder Reranker + 集成到 Searcher 2026-07-11 19:44:56 +08:00
Serendipity cb470e516b feat: 添加 HybridRetriever — BM25+向量混合检索 2026-07-11 19:42:20 +08:00
Serendipity 7b3c7d5323 feat: DocumentIngestor 支持增量入库（FileTracker） 2026-07-11 19:41:21 +08:00
Serendipity 3e2c61402a feat: 添加 FileTracker — 基于 SHA256 的文件变更追踪 2026-07-11 19:39:12 +08:00
Serendipity ec3898ecb7 fix: auth 测试改用 monkeypatch.setattr，MarkdownSplitter 跳过代码块内标题识别 2026-07-10 15:30:58 +08:00
Serendipity f8ec682474 test: 提取 MarkdownSplitter 边界测试到独立文件 2026-07-10 15:29:10 +08:00
Serendipity 76ae81c772 test: test_config 用 monkeypatch 替代 importlib.reload 2026-07-10 15:28:22 +08:00
Serendipity 788f610d3f test: 修复 EPUB 缺失依赖测试设计问题 2026-07-10 15:28:13 +08:00
Serendipity b0319f366a test: 补充 Searcher list_sources/delete_by_source/get_collection_info 测试 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 15:25:54 +08:00
Serendipity 925499b05b test: 补充 ingest_file 和 ingest_directory 的测试覆盖 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 15:25:50 +08:00
Serendipity e0190e8d2d test: 添加 verify_api_key 和 RateLimiter 完整单元测试 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 15:25:47 +08:00
Serendipity a368e640c1 test: 添加 CLI 模块基础测试 (ingest/search/stats/json) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 15:25:43 +08:00
Serendipity 3b8b585f31 feat: 添加 EPUBSplitter — ebooklib 提取章节文字后分块 
- 新增 splitters/epub.py: ebooklib 读取 EPUB → BeautifulSoup 去标签 → TextSplitter 分块
- epub 作为可选依赖: uv sync --extra epub
- registry 新增 .epub 映射, ingest_file 处理 EPUB 二进制文件
- 测试: 3 个 EPUBSplitter 测试 (含空 EPUB/Protocol 合规)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-10 14:32:41 +08:00
Serendipity b8afc9043d test: 添加 PDFSplitter 和 HTMLSplitter 测试 2026-07-10 14:25:40 +08:00
Serendipity 9db0c15dee test: 添加 Splitter 注册表和 TextSplitter 测试 2026-07-10 14:23:34 +08:00
Serendipity 4dc51dccd2 refactor: 将 MarkdownSplitter 移入 splitters 包，ingest.py 保留兼容 import 2026-07-10 14:15:59 +08:00
Serendipity 405303e82c fix: 修复 44 个代码审查问题 (CRITICAL/HIGH/MEDIUM/LOW) 
Batch 1 — CRITICAL (1):
- 提取 is_safe_path() 到 src/core/security.py 公共模块
- CLI 和 ingest_obsidian.py 统一添加路径遍历防护

Batch 2 — HIGH (13) + 架构重构:
- CLI 复用 deps.py AppState, 消除 30 行重复代码
- AppState/get_state 添加线程安全锁
- serve 命令传递 --config 到 uvicorn (H1)
- OpenAIEmbedder 懒创建+复用 HTTP 客户端 (H2)
- DashscopeEmbedder import 移到模块顶部 (H3)
- 路径检查改用 os.path.commonpath (H4)
- embedder.embed() 返回值长度检查 (H5)
- 健康检查不泄露内部错误详情 (H7)
- /api/v1/collections 添加 API Key 认证 (H8)
- API Key 使用 hmac.compare_digest 恒定时间比较 (H9)
- 添加 CORS 中间件 (H10)
- ServerConfig 支持 SSL 配置 (H11)
- HF_ENDPOINT 修改添加详细注释 (H12)

Batch 3 — MEDIUM (20) + Splitter Protocol:
- 定义 Splitter(Protocol) 接口, DocumentIngestor 接受可选 splitter
- DashScope 响应添加结构验证 (M2)
- ingest_obsidian.py 支持 CLI 参数和 OBSIDIAN_DIRS 环境变量 (M6)
- scripts/serve.py 添加废弃警告 (M7)
- content 限制 500KB, collection 正则限制字符集 (M12-M14)
- 默认监听地址 127.0.0.1 (M16)
- 添加安全响应头中间件 (M17)
- verify_api_key 认证失败记录日志 (M19)

Batch 4 — LOW (10):
- CLI emoji 清理为纯文本标记 (L5)
- logging.basicConfig 移到 FastAPI lifespan (L1)
- VectorDB 添加 write_guard() 上下文管理器 (L3)
- IngestRequest file_path/content 互斥校验 (L10)
- ingest_obsidian.py 注释修正 (L6)

测试: 46 → 70 (+24)
- tests/test_security.py: 11 个路径安全测试
- tests/test_deps.py: 11 个依赖注入测试

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 16:56:38 +08:00
Serendipity 832201186d fix: 修复 11 个代码架构审计问题 
H1: AppConfig 自定义 __init__ 改用 from_dict() 类方法
H2: list_collections_with_stats 改为从 ChromaDB 直接查询
H3: RateLimiter 过期 key 自动清理, 防止内存泄漏
M1: delete_by_source 区分 ValueError 与真实异常, 记日志
M2: VectorDB 新增 list_collections() 封装方法
M3: _remove_by_source 异常记日志, 不再静默吞掉
M4: CLI 集合回退支持 MD_VECTOR_DB_COLLECTION 环境变量
L1: DEFAULT_CONFIG_PATH 自动从项目根目录解析
L2: ingest 命令内重复 import 移至模块顶部
L3: 新增死循环回归测试 + 密集分隔符分块测试
L4: 统一 logger 名称为 md-vector-db
删除旧版审计文档

测试: 48 passed
 2026-07-06 13:16:26 +08:00
Serendipity b89c6076e1 feat: multi-provider embedder — support OpenAI/DashScope with provider-specific defaults 2026-07-05 01:46:59 +08:00
Serendipity 441ba8ab86 test: add DELETE endpoint tests and ingest integration tests 2026-07-05 01:46:59 +08:00
Serendipity ad9e5ce449 refactor: strategy-pattern embedder (Local/API), remove global env vars, add batch embedding 2026-07-05 01:46:59 +08:00
Serendipity 2ef0cf4a4e fix: add path traversal protection to ingest endpoint 2026-07-05 01:46:58 +08:00
Serendipity 5dcf1eed52 fix: remove api_key from config.yaml, read from EMBED_API_KEY env var; unify config path constant 2026-07-05 01:46:58 +08:00
Serendipity d021390fd3 feat: add FastAPI HTTP API layer 
- Implement FastAPI server with health, collections, ingest, search, and delete endpoints
- Add Pydantic request models for ingest and search
- Add lazy singleton initialization for DB, embedder, and services
- Support environment variable override for data dir and collection name
- Add integration tests with TestClient and temp directory
- Set TRANSFORMERS_OFFLINE/HF_HUB_OFFLINE for offline model loading in tests
 2026-07-05 01:46:44 +08:00
Serendipity 7309b0a437 feat: add semantic search module 
实现语义检索模块 Searcher 类，支持：
- search(): 向量语义检索，返回格式化结果（含 content、source_file、section_title、heading_level、chunk_index、score 字段）
- get_collection_info(): 获取 collection 统计信息
- list_sources(): 列出已入库源文件
- delete_by_source(): 按源文件删除文档

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-05 01:46:44 +08:00
Serendipity 757bd8b336 feat: add Markdown splitter and document ingestor 
- MarkdownSplitter: hybrid splitting by headings then paragraphs
- DocumentIngestor: read MD -> split -> embed -> store with dedup
- All 6 tests passing
 2026-07-05 01:46:44 +08:00
Serendipity f9e5c37d3e feat: add embedder with local/API dual mode 2026-07-05 01:46:44 +08:00
Serendipity b756c53e25 feat: add ChromaDB database layer 2026-07-05 01:46:44 +08:00
Serendipity 138c5c1881 feat: add config loading module with YAML support 
实现应用配置加载模块，支持从 YAML 文件或默认值加载配置。
覆盖 ChromaDB、嵌入模型、文档分块和 HTTP 服务配置。

- src/core/config.py: 配置数据类 (AppConfig/ChromaConfig/EmbedConfig/ChunkConfig/ServerConfig) + load_config 函数
- tests/test_config.py: 5 个单元测试覆盖默认值、API 模式和文件加载

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-05 01:46:44 +08:00
Serendipity 3eb4b30858 chore: scaffold md-vector-db project structure 2026-07-05 01:46:44 +08:00