RAG Practice

This repo is a source-agnostic retrieval backend for local knowledge corpora. It is designed so any LLM or chatbot can use the repo tools to retrieve fast, source-grounded context without needing a custom chat interface.

The included articles/ PDFs are the demo corpus. For production-style evaluation, put new files in corpus/ and rebuild from a clean database.

Quick Start

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
rag init
rag rebuild articles --yes
rag eval
rag context "Which sources discuss immobilized antibody orientation and ToF-SIMS?"

If rag is not on PATH, use .venv/bin/rag ... from the repo root after setup.

Fresh Production Corpus

1. Add files to corpus/.
2. Run:

rag rebuild corpus --yes

This deletes generated data, creates a clean SQLite database, ingests the current corpus, chunks/enriches it, and builds the vector index. Source files are not deleted. If corpus/ contains no real source files, the command exits with an error instead of creating a misleading empty production index.

For an isolated trial run:

rag --data-dir data-production-test rebuild corpus --yes
rag --data-dir data-production-test status
rag --data-dir data-production-test eval --file eval/your-production-questions.yml
rag --data-dir data-production-test context "your question"

3. Update or create a production eval YAML file.
4. Run against the same data directory you rebuilt:

rag --data-dir data-production-test eval --file eval/your-production-questions.yml
rag --data-dir data-production-test context "your question"

Supported Sources

• PDFs
• Markdown
• plain text
• HTML snapshots
• .urls files with one webpage URL per line
• .jsonl post exports, including X/thread-style exports

Main Commands

rag status
rag ingest articles
rag build-index
rag search "query"
rag context "query" --format json
rag inspect-doc <document_id>
rag inspect-chunk <chunk_id>
rag eval
rag reset --yes
rag rebuild corpus --yes

Documentation

• Agent instructions
• Architecture
• Retrieval contract
• Ingestion
• Evaluation
• Operations
• AWS document intelligence platform spec
• Agent review notes

Deploy to AWS

cd infra
set -a
source .env
set +a

npx cdk bootstrap aws://$AWS_ACCOUNT_ID/$AWS_REGION
npx cdk deploy RagDocIntelDev

• Once deployed compy the CDK credentials into the frontend .env file

set -a
source .env
set +a

npm run build

then deploy it

set -a
source .env
set +a

aws s3 sync dist/ "s3://$FrontendBucketName/" --delete

aws cloudfront create-invalidation \
  --distribution-id "$CloudFrontDistributionId" \
  --paths "/*"