Skip to content

CODEBASE.md

Latest commit

 

History

History
165 lines (140 loc) · 8.45 KB

CODEBASE.md

File metadata and controls

165 lines (140 loc) · 8.45 KB
Raw

Codebase Overview

Project Overview

Arag CLI is a high-performance Agent RAG search tool written in Rust, providing keyword search, semantic search, chunk reading, and more. Minimum Rust version 1.75, MIT license.

Tech Stack

  • Language: Rust (edition 2021)
  • CLI Framework: Clap 4.4 (derive + env features)
  • Serialization: serde / serde_json
  • Concurrency: rayon
  • HTTP Client: ureq
  • Progress Bar: indicatif
  • CSV Parsing: csv
  • Environment Variables: dotenv

Directory Structure

arag-cli/
├── src/
│   ├── main.rs           # CLI entry, Clap command routing (19 subcommands)
│   ├── lib.rs            # Library root: declares layers + pub use glob re-export
│   ├── utils/            # 基础设施层（无内部依赖）
│   │   ├── mod.rs
│   │   ├── tokenizer.rs  # Token counting and truncation
│   │   ├── error.rs      # Error code definitions
│   │   └── output.rs     # JSON output
│   ├── core/             # 核心引擎层（依赖 utils）
│   │   ├── mod.rs
│   │   ├── bm25.rs       # BM25 search engine + snippet extraction
│   │   ├── chunker.rs    # Structure-aware chunking (Markdown parsing / chunk aggregation / markitdown integration)
│   │   ├── markitdown.rs # markitdown subprocess wrapper (file format conversion)
│   │   ├── index.rs      # Index loading (JSON + binary embedding) + model_name validation
│   │   ├── manifest.rs   # KB file manifest (mtime/size diff, append/update incremental support)
│   │   ├── embedding.rs      # Embedding entry (calls HTTP API)
│   │   ├── embedding_api.rs  # HTTP Embedding client (ureq, exponential backoff retry)
│   │   └── config.rs     # Embedding configuration (base_url / model / api_key via environment variables)
│   ├── business/         # 业务层（依赖 core + utils）
│   │   ├── mod.rs
│   │   ├── kb.rs         # Multi-KB lifecycle management (directory / activation / migration)
│   │   ├── cache.rs      # Query cache (JSONL append write + in-memory index)
│   │   ├── history.rs    # Query history (JSONL append write + followup inference)
│   │   ├── neighbors.rs  # Snippet neighborhood expansion (adjacent chunks in the same file)
│   │   ├── notes.rs      # Chunk annotation storage (Markdown files)
│   │   ├── profile.rs    # User profile aggregation (hot/cold chunks, clustering, blind spots, adoption rate)
│   │   └── session.rs    # AgentContext persistence
│   ├── search/           # Hybrid search module（依赖 core + business）
│   │   ├── mod.rs
│   │   ├── hybrid.rs      # BM25 + semantic RRF fusion
│   │   └── adaptive.rs    # Personalized configuration and adaptive scoring
│   └── commands/         # 19 subcommand implementations（依赖所有层）
│       ├── mod.rs
│       ├── common.rs      # Shared utilities (download / extract / version comparison)
│       ├── keyword_search.rs
│       ├── semantic_search.rs
│       ├── read_chunk.rs
│       ├── build_index.rs
│       ├── inspect_index.rs
│       ├── health.rs
│       ├── schema.rs
│       ├── update.rs
│       ├── kb.rs
│       ├── daemon.rs
│       ├── cache.rs
│       ├── history.rs
│       ├── profile.rs
│       ├── note.rs
│       ├── pin.rs
│       ├── search.rs
│       ├── lint.rs
│       └── config.rs
├── api/                 # FastAPI HTTP service
│   ├── app/main.py      # Application entry, middleware, health, and exception handling
│   ├── app/api/v1/      # REST route definitions
│   ├── app/core/        # Settings, logging, and request context
│   ├── app/schemas/     # Pydantic request/response models
│   ├── app/tools/       # Async arag-cli subprocess adapter
│   └── README.md        # API operator and developer guide
├── benches/             # Performance benchmarks
├── tests/               # Integration tests + test data
├── skill/               # Agent skill definitions
├── docs/                # Documentation
├── install.sh           # Linux/macOS install script
└── install.bat          # Windows install script

Core Module Responsibilities

API Service

Module Responsibility
api/app/main.py Configure FastAPI, request IDs, CORS, health checks, and global errors
api/app/api/v1/arag.py Validate HTTP operations, invoke the CLI adapter, and map errors to responses
api/app/schemas/arag.py Define request constraints and the common /arag response envelope
api/app/tools/arag_cli.py Execute arag-cli asynchronously, retry timeouts, parse JSON, and format text output
api/app/core/ Load environment settings and configure request-aware rotating logs

Detailed setup and usage instructions are maintained in api/README.md.

Search Module

Module Responsibility
bm25.rs BM25 keyword search, supports multi-keyword joint queries, snippet extraction
embedding.rs / embedding_api.rs Semantic search entry, obtains embedding vectors via HTTP API
search/hybrid.rs BM25 + semantic RRF (Reciprocal Rank Fusion) hybrid search
search/adaptive.rs Personalized configuration and adaptive scoring

Index Module

Module Responsibility
chunker.rs Structure-aware chunking: Markdown parsing, chunk aggregation, markitdown integration
markitdown.rs markitdown subprocess wrapper, supports any file format conversion
index.rs Index loading (JSON + binary embedding) + model_name validation
manifest.rs KB file manifest, supports mtime/size diff detection and incremental updates

Knowledge Base Management

Module Responsibility
kb.rs Multi-KB lifecycle management: directory creation, activation switching, legacy migration
cache.rs Query cache: JSONL append write + in-memory index
history.rs Query history: JSONL append write + followup inference
notes.rs Chunk annotation storage (Markdown files)
profile.rs User profile aggregation: hot/cold chunks, clustering, blind spots, adoption rate

Infrastructure

Module Responsibility
config.rs Embedding configuration (base_url / model / api_key via environment variables)
error.rs Error code definitions
output.rs JSON output formatting
session.rs AgentContext persistence
tokenizer.rs Token counting and truncation
neighbors.rs Snippet neighborhood expansion (adjacent chunks in the same file)

Dependency Hierarchy

utils (tokenizer, error, output)
  -> core (bm25, chunker, embedding, embedding_api, index, manifest, markitdown, config)
    -> business (search, kb, cache, history, notes, profile, session, neighbors)
      -> commands (19 subcommands)
        -> main (CLI entry)

Circular dependencies are prohibited; upper layers may depend on lower layers, but not vice versa.

物理归档与重导出约定

源码按上述四层归档到 src/utils/、src/core/、src/business/、src/commands/、src/search/ 目录。lib.rs / main.rs 通过 pub use <layer>::*;（glob 重导出）将各层子模块名暴露到 crate 根，因此所有下游 use crate::<module>::... 路径保持不变——归档前后 92 处引用零修改，运行时逻辑零影响。新增模块时应归入对应层目录并在该层 mod.rs 声明。

Design Decisions

  1. Rust Rewrite: The original Python version took 7s+ to start; the Rust version reduced it to 160ms, with memory dropping from 1.2GB to 110MB
  2. Structure-Aware Chunking: Tables and code blocks are treated as atomic units and never truncated; each chunk is injected with heading chain context
  3. Multi-KB Support: Different knowledge bases are isolated via ~/.arag/kb/<name>/ directories, supporting creation, switching, and deletion
  4. Incremental Updates: Change detection based on file mtime/size differences, rebuilding only the changed parts
  5. Hybrid Search: BM25 + semantic RRF fusion, combined with personalized scoring
  6. Agent Integration: JSON schema output, supports --format text conversion to LLM-readable format