Verified Scholar Agent

A self-correcting RAG research assistant built with LangGraph.

Ingest documents (URLs or PDFs), ask questions, and get answers that are graded for groundedness before delivery. If the answer fails verification, the agent rewrites the query and retries retrieval — reducing confident hallucinations in a transparent, traceable loop.

Portfolio v1 — config-driven, multi-project vector stores, Streamlit UI, and LangSmith evaluation.

---

Highlights

• LangGraph pipeline — retrieve → generate → grade → conditional retry
• Structured grading — Pydantic GradeAnswer (yes / no + rationale) via with_structured_output
• RAG — Chroma + OpenAI embeddings, recursive text splitting
• Multi-project corpora — isolated collections per notebook (scholar_{project_id})
• Streamlit UI — ingest sources, chat, verification badge, chunk inspection
• LangSmith eval — dataset + LLM-as-a-judge (faithfulness & correctness)

---

Screenshots

Streamlit chat — grounded answers with verification status, grader rationale, and retrieved chunks.

LangSmith — experiment runs, trace tree, and evaluation feedback.

---

Architecture

flowchart TD
    UI[Streamlit / CLI] --> Ingest[ingest.py]
    UI --> Agent[agent.py]
    Ingest --> Chroma[(Chroma DB)]
    Agent --> Graph[LangGraph app]
    Graph --> R[retrieve]
    R --> G[generate]
    G --> GR[grade_generation_node]
    GR -->|verified| END([END])
    GR -->|not verified| R
    R --> Chroma
    Graph --> LangSmith[LangSmith traces]
    Eval[eval_suite.py] --> LangSmith

Loading

State channels (AgentState):

Field	Role
messages	Chat history (accumulates with operator.add)
documents	Retrieved chunks for the current turn
is_verified	Grader score: yes or no
grade_rationale	Why the answer passed or failed
retry_count	Loop guard (max configurable)

---

Project structure

├── app/
│   ├── config.py      # YAML + env settings
│   ├── ingest.py      # Load, chunk, embed → Chroma
│   ├── chains.py      # LCEL prompts & LLM chains
│   ├── graph.py       # LangGraph nodes & routing
│   ├── agent.py       # build_agent(), run_chat()
│   └── __main__.py    # CLI entrypoint
├── config.yaml        # Models, retrieval, sources
├── ui.py              # Streamlit app
├── eval_suite.py      # LangSmith dataset & evaluation
├── main_demo.py       # Retry-loop demos (--demo force-bad)
├── main.py            # Minimal legacy runner
└── requirements.txt

---

Quick start

1. Clone and install

git clone https://github.com/gaurika05/verified-scholar-agent.git
cd verified-scholar-agent
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

2. Environment

cp .env.example .env
# Edit .env — add OPENAI_API_KEY and LANGCHAIN_API_KEY

3. Configure sources (optional)

Edit config.yaml to point at your documents:

default_project: default
sources:
  - type: url
    path: https://blog.langchain.dev/langgraph/
  - type: pdf
    path: ./docs/paper.pdf

4. Ingest documents

python -m app ingest
# Or a specific project / source:
python -m app ingest -p my_research -s https://example.com/article

5. Chat (CLI)

python -m app chat "What is LangGraph used for?"
python -m app chat -p my_research "Your question here"

6. Streamlit UI

streamlit run ui.py

• Ingest tab — add URLs, upload PDFs, or load config.yaml sources
• Chat tab — ask questions; see verification status, grader rationale, and retrieved chunks

---

Self-correction demo

Watch the graph retry when an answer is not grounded:

python main_demo.py --demo force-bad --trace --skip-ingest

This injects a hallucinated answer on the first pass, triggers grade → no → rewrite query → retrieve again, then generates a grounded answer.

---

LangSmith evaluation

Creates dataset Scholar_Agent_Test_Set (on-topic + off-topic questions) and runs an experiment with faithfulness and correctness judges.

python eval_suite.py --skip-ingest

Requires LANGCHAIN_API_KEY and LANGCHAIN_TRACING_V2=true in .env.
View results under your LangSmith project (e.g. scholar_agent).

---

Configuration reference

File / variable	Purpose
config.yaml	LLM model, chunk size, top_k, sources, max_retries
OPENAI_API_KEY	Embeddings + chat + judges
LANGCHAIN_API_KEY	LangSmith tracing & eval
LANGCHAIN_PROJECT	Trace grouping in LangSmith UI
default_project	Default Chroma collection suffix

Each project id maps to collection name scholar_{project_id} (e.g. default → scholar_default).

---

Tech stack

Python · LangChain · LangGraph · OpenAI · Chroma · LangSmith · Streamlit · Pydantic · PyYAML

---

Roadmap

• Source citations in answers (chunk metadata → UI)
• FastAPI backend + session auth
• CI job running eval_suite.py on prompt/graph changes
• Ingest deduplication (skip unchanged documents)

---

License

MIT — see LICENSE.

---

Author

Gaurika Gupta — built as a self-directed portfolio project exploring agentic RAG and evaluation-driven LLM workflows.

If this project helped you or you have feedback, feel free to open an issue or star the repo.