Serendipity/md-vector-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
44 Commits 1 Branch 0 Tags
30716f10dce35b545a4b57989bc0f40ea40d78c4
Commit Graph

44 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Serendipity 30716f10dc feat: 添加 HTMLSplitter — bs4 去标签后分块 2026-07-10 14:22:28 +08:00
Serendipity 69fb8cac9f feat: 添加 PDFSplitter — pymupdf 提取文字后分块 2026-07-10 14:20:53 +08:00
Serendipity cde4ae7ddc feat: ingest.py 使用 registry 自动选择 Splitter，ingest_directory 支持多格式 2026-07-10 14:19:03 +08:00
Serendipity 13d63ba6ff feat: 添加 TextSplitter 和 registry 自动选择机制 2026-07-10 14:17:17 +08:00
Serendipity 4dc51dccd2 refactor: 将 MarkdownSplitter 移入 splitters 包，ingest.py 保留兼容 import 2026-07-10 14:15:59 +08:00
Serendipity 30c4d65cf0 feat: 创建 splitters 包骨架 — Splitter Protocol + BaseTextSplitter 基类 2026-07-10 14:12:38 +08:00
Serendipity 11b6171e21 docs(test-guide): 重写为完整的项目使用与测试文档 
替换原有的通用向量数据库入门内容，新增项目简介、核心概念、快速上手流程、CLI与HTTP API使用方法、配置说明、嵌入提供商选择、安全配置、架构概览、GPU加速、测试命令和常见问题等完整文档内容，适配作为项目的使用指南与测试手册。
 2026-07-07 15:04:45 +08:00
Serendipity 405303e82c fix: 修复 44 个代码审查问题 (CRITICAL/HIGH/MEDIUM/LOW) 
Batch 1 — CRITICAL (1):
- 提取 is_safe_path() 到 src/core/security.py 公共模块
- CLI 和 ingest_obsidian.py 统一添加路径遍历防护

Batch 2 — HIGH (13) + 架构重构:
- CLI 复用 deps.py AppState, 消除 30 行重复代码
- AppState/get_state 添加线程安全锁
- serve 命令传递 --config 到 uvicorn (H1)
- OpenAIEmbedder 懒创建+复用 HTTP 客户端 (H2)
- DashscopeEmbedder import 移到模块顶部 (H3)
- 路径检查改用 os.path.commonpath (H4)
- embedder.embed() 返回值长度检查 (H5)
- 健康检查不泄露内部错误详情 (H7)
- /api/v1/collections 添加 API Key 认证 (H8)
- API Key 使用 hmac.compare_digest 恒定时间比较 (H9)
- 添加 CORS 中间件 (H10)
- ServerConfig 支持 SSL 配置 (H11)
- HF_ENDPOINT 修改添加详细注释 (H12)

Batch 3 — MEDIUM (20) + Splitter Protocol:
- 定义 Splitter(Protocol) 接口, DocumentIngestor 接受可选 splitter
- DashScope 响应添加结构验证 (M2)
- ingest_obsidian.py 支持 CLI 参数和 OBSIDIAN_DIRS 环境变量 (M6)
- scripts/serve.py 添加废弃警告 (M7)
- content 限制 500KB, collection 正则限制字符集 (M12-M14)
- 默认监听地址 127.0.0.1 (M16)
- 添加安全响应头中间件 (M17)
- verify_api_key 认证失败记录日志 (M19)

Batch 4 — LOW (10):
- CLI emoji 清理为纯文本标记 (L5)
- logging.basicConfig 移到 FastAPI lifespan (L1)
- VectorDB 添加 write_guard() 上下文管理器 (L3)
- IngestRequest file_path/content 互斥校验 (L10)
- ingest_obsidian.py 注释修正 (L6)

测试: 46 → 70 (+24)
- tests/test_security.py: 11 个路径安全测试
- tests/test_deps.py: 11 个依赖注入测试

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 16:56:38 +08:00
Serendipity 832201186d fix: 修复 11 个代码架构审计问题 
H1: AppConfig 自定义 __init__ 改用 from_dict() 类方法
H2: list_collections_with_stats 改为从 ChromaDB 直接查询
H3: RateLimiter 过期 key 自动清理, 防止内存泄漏
M1: delete_by_source 区分 ValueError 与真实异常, 记日志
M2: VectorDB 新增 list_collections() 封装方法
M3: _remove_by_source 异常记日志, 不再静默吞掉
M4: CLI 集合回退支持 MD_VECTOR_DB_COLLECTION 环境变量
L1: DEFAULT_CONFIG_PATH 自动从项目根目录解析
L2: ingest 命令内重复 import 移至模块顶部
L3: 新增死循环回归测试 + 密集分隔符分块测试
L4: 统一 logger 名称为 md-vector-db
删除旧版审计文档

测试: 48 passed
 2026-07-06 13:16:26 +08:00
Serendipity 7e77c31e16 docs: 更新 README/CLAUDE 文档 + 配置 uv CUDA 兼容 
- README: 补充 GPU 加速章节、多集合、CLI 新选项 (-C/--json)
- CLAUDE: 修正 uv run 可用说明、补充 [tool.uv] 配置文档
- pyproject.toml: 添加 [tool.uv] 配置, 通过 find-links + unsafe-best-match
  让 uv 从本地 CUDA wheel 解析 torch, 不再覆盖 GPU 环境
- uv.lock: 更新 lockfile, Windows 平台锁定 torch 2.6.0+cu124
 2026-07-06 12:44:06 +08:00
Serendipity eebdef739c fix: 修复 MarkdownSplitter 死循环 + 添加 GPU 自动检测 
- ingest.py: _split_single_paragraph 中分隔符距 start 小于 overlap 时
  start 会回退为负数，Python rfind 负索引导致无限循环
  修复: start = max(start + 1, next_start) 确保始终前进
- embedder.py: LocalEmbedder 自动检测 CUDA，优先使用 GPU
- ingest_obsidian.py: MAX_SIZE 调整为 200KB 适配 GPU 嵌入
- 清理临时日志文件
 2026-07-06 00:04:04 +08:00
Serendipity d141baf964 fix: use path hash prefix for source_file names to prevent same-name collisions; add size filter to ingest script 2026-07-05 04:46:47 +08:00
Serendipity d3cb88525c feat: multi-file, stdin, and glob support for ingest; fix stderr UTF-8 2026-07-05 02:14:17 +08:00
Serendipity 4b2fd3d33a feat: add --json output and --collection to CLI for MCP scripting 2026-07-05 02:10:23 +08:00
Serendipity 0aee167085 feat: multi-collection support — each project uses its own isolated collection 2026-07-05 02:07:02 +08:00
Serendipity 8042cf288b docs: comprehensive README with quickstart, API reference, provider guide, and architecture 2026-07-05 01:50:05 +08:00
Serendipity f61f631d71 docs: update CLAUDE.md to reflect current architecture 2026-07-05 01:47:38 +08:00
Serendipity 88c109d3dd docs: enhance CLI help with argument descriptions, --config option, and epilog examples 2026-07-05 01:46:59 +08:00
Serendipity b89c6076e1 feat: multi-provider embedder — support OpenAI/DashScope with provider-specific defaults 2026-07-05 01:46:59 +08:00
Serendipity 5672472030 feat: load .env file via python-dotenv for API key management 2026-07-05 01:46:59 +08:00
Serendipity 441ba8ab86 test: add DELETE endpoint tests and ingest integration tests 2026-07-05 01:46:59 +08:00
Serendipity 4114a168b4 refactor: FastAPI Depends injection replacing globals, real health check, logging config 2026-07-05 01:46:59 +08:00
Serendipity ad9e5ce449 refactor: strategy-pattern embedder (Local/API), remove global env vars, add batch embedding 2026-07-05 01:46:59 +08:00
Serendipity 81a8173a38 fix: add thread-safe write lock to ChromaDB operations 2026-07-05 01:46:59 +08:00
Serendipity 10eda446ea fix: add field validation to API request models (top_k limit, file_name pattern) 2026-07-05 01:46:58 +08:00
Serendipity 157155c0d2 fix: add API key auth, rate limiting, and safe error messages 2026-07-05 01:46:58 +08:00
Serendipity 2ef0cf4a4e fix: add path traversal protection to ingest endpoint 2026-07-05 01:46:58 +08:00
Serendipity 5dcf1eed52 fix: remove api_key from config.yaml, read from EMBED_API_KEY env var; unify config path constant 2026-07-05 01:46:58 +08:00
Serendipity 10f9aaede4 docs: add architecture audit report with 23 findings 2026-07-05 01:46:58 +08:00
Serendipity d82d728818 docs: add CLAUDE.md for AI agent guidance 2026-07-05 01:46:58 +08:00
Serendipity 2f0aa38fbf fix: add root redirect to /docs and show localhost in serve output 2026-07-05 01:46:58 +08:00
Serendipity 30a1c1af57 fix: SSL certificate issue and Windows terminal encoding 2026-07-05 01:46:58 +08:00
Serendipity 8a646be06c docs: add README and test data 2026-07-05 01:46:57 +08:00
Serendipity 2c1dcf22cf feat: add serve script and console entry point 2026-07-05 01:46:45 +08:00
Serendipity 9ae676d9cc feat: add CLI management tool 2026-07-05 01:46:45 +08:00
Serendipity d021390fd3 feat: add FastAPI HTTP API layer 
- Implement FastAPI server with health, collections, ingest, search, and delete endpoints
- Add Pydantic request models for ingest and search
- Add lazy singleton initialization for DB, embedder, and services
- Support environment variable override for data dir and collection name
- Add integration tests with TestClient and temp directory
- Set TRANSFORMERS_OFFLINE/HF_HUB_OFFLINE for offline model loading in tests
 2026-07-05 01:46:44 +08:00
Serendipity 7309b0a437 feat: add semantic search module 
实现语义检索模块 Searcher 类，支持：
- search(): 向量语义检索，返回格式化结果（含 content、source_file、section_title、heading_level、chunk_index、score 字段）
- get_collection_info(): 获取 collection 统计信息
- list_sources(): 列出已入库源文件
- delete_by_source(): 按源文件删除文档

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-05 01:46:44 +08:00
Serendipity 757bd8b336 feat: add Markdown splitter and document ingestor 
- MarkdownSplitter: hybrid splitting by headings then paragraphs
- DocumentIngestor: read MD -> split -> embed -> store with dedup
- All 6 tests passing
 2026-07-05 01:46:44 +08:00
Serendipity f9e5c37d3e feat: add embedder with local/API dual mode 2026-07-05 01:46:44 +08:00
Serendipity b756c53e25 feat: add ChromaDB database layer 2026-07-05 01:46:44 +08:00
Serendipity 138c5c1881 feat: add config loading module with YAML support 
实现应用配置加载模块，支持从 YAML 文件或默认值加载配置。
覆盖 ChromaDB、嵌入模型、文档分块和 HTTP 服务配置。

- src/core/config.py: 配置数据类 (AppConfig/ChromaConfig/EmbedConfig/ChunkConfig/ServerConfig) + load_config 函数
- tests/test_config.py: 5 个单元测试覆盖默认值、API 模式和文件加载

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-05 01:46:44 +08:00
Serendipity cfc41f8af5 chore: improve .gitignore with build artifacts and venv patterns 2026-07-05 01:46:44 +08:00
Serendipity 3eb4b30858 chore: scaffold md-vector-db project structure 2026-07-05 01:46:44 +08:00
Serendipity 77d9ec6570 Initial commit 2026-07-05 01:25:37 +08:00