feat(mcp): Add documentation browsing and search tools by bisilaj · Pull Request #72 · cisagov/thorium

bisilaj · 2026-02-18T01:47:06Z

feat(mcp): Add documentation browsing and search tools

🗣 Description

Adds three new MCP tools that allow AI agents to browse and search Thorium's documentation through the existing MCP server at /mcp:

get_docs_toc — Returns the structured table of contents (titles, paths, nesting depth) parsed from SUMMARY.md. An agent calls this first to orient itself.
get_doc_page — Fetches a specific documentation page's raw markdown by relative path (e.g. concepts/files.md). Includes path traversal protection via canonicalization.
search_docs — Case-insensitive full-text search across all 90+ markdown files. Returns the top 10 matching pages ranked by match count, with page titles and context snippets.

Unlike the existing MCP tools which use the loopback HTTP client pattern to call Thorium's REST API, the docs tools read markdown files directly from the container filesystem. This is simpler and
appropriate since docs are static content that don't require database access or complex auth beyond validating the MCP session token.

Files changed:

api/src/routes/mcp/docs.rs (new) — Three MCP tool handlers with helper functions and 24 unit tests
api/src/routes/mcp.rs — Added docs_path: PathBuf to McpConfig, registered docs module and router, removed Copy derive (PathBuf isn't Copy), added .clone() in mount closure
api/src/conf.rs — Added docs_src field to Assets struct with default path "docs/src"
api/Cargo.toml — Added tempfile dev-dependency for unit tests
Cargo.lock — Updated to reflect new dev-dependency
Dockerfile — Added ADD ./api/docs/src docs/src to include raw markdown source in the container image

💭 Motivation and context

Currently, an AI agent connected to Thorium's MCP server has no way to learn how Thorium works — it can interact with files, images, and pipelines but can't access any documentation. This means agents
require external context or human guidance to understand Thorium's concepts, workflows, and configuration.

These tools give agents a self-service path to learn Thorium through a progressive disclosure pattern: get the table of contents for orientation, search for specific topics, then fetch individual pages as
needed. This keeps agent context windows lean while providing full access to all documentation.

🧪 Testing

Unit tests (24 tests, all passing):

Run with: cargo +nightly test -p thorium-api --lib -- mcp::docs::tests

Function	Tests	Coverage
`parse_summary`	5	Basic parsing, depth levels, `./` stripping, non-link lines, empty input, real SUMMARY.md
`validate_doc_path`	4	Traversal rejection (`..`), nested traversal, valid path acceptance, nonexistent file
`collect_md_files`	3	Finds .md only, ignores other extensions, recurses subdirs, empty dir, real docs tree
`extract_title`	5	Finds H1, returns first H1, ignores H2+, empty content, leading whitespace
`extract_snippets`	7	Basic context, case insensitivity, max limit, adjacent dedup, first/last line edges, no matches

Three tests also validate against the real documentation source files on disk.

Integration testing (recommended for reviewers):

Connect an MCP client to /mcp and verify get_docs_toc returns structured TOC entries
Verify get_doc_page with a valid path (e.g. intro.md) returns markdown content
Verify get_doc_page with a traversal path (e.g. ../../../etc/passwd) returns an error
Verify search_docs with "redis" returns relevant results with snippets from deployment docs
Verify existing MCP tools (get_sample, list_images, etc.) still function after McpConfig changes

✅ Pre-approval checklist

✅ Pre-merge checklist

Revert dependencies to default branches.
Finalize version.

✅ Post-merge checklist

Create a release (necessary if and only if the version was bumped).

Add three new MCP tools (get_docs_toc, get_doc_page, search_docs) that allow AI agents to browse and search Thorium's documentation through the existing MCP server. Tools read raw markdown directly from the container filesystem using a progressive disclosure pattern: TOC for orientation, search for discovery, and page fetch for full content. Extends McpConfig with a docs_path field, adds docs_src to the Assets config, and includes the raw markdown source in the Docker image. Includes 24 unit tests covering SUMMARY.md parsing, path traversal prevention, markdown file collection, title extraction, and snippet generation — including tests that validate against the real docs.

The rmcp #[tool] macro generates an empty JSON schema automatically when no Parameters<T> argument is present, so the Empty struct was unnecessary.

Replace the manual for loop with filter_map and the ? operator for a more idiomatic Rust style. Added inline comments explaining each step of the markdown link parsing.

Adopt a split_once chain approach for parsing SUMMARY.md markdown links, replacing index arithmetic. This eliminates manual offset tracking and off-by-one risk while being marginally faster. Improved variable names and comments for readability.

Refactor helper functions to use idiomatic Rust iterator patterns: collect_md_files returns a Vec using map/flatten/filter/collect, extract_title uses find_map, and the search_docs for loop becomes a filter_map chain. Make snippet context (context_lines param) and search truncation (MAX_SEARCH_RESULTS, MAX_SNIPPETS_PER_PAGE, SNIPPET_CONTEXT_LINES constants) configurable instead of hardcoded. Log file read errors with tracing::warn instead of silently discarding them. Add rustdoc # Errors sections to all fallible functions. Document the content vs structured_content distinction in the module doc. Derive PartialEq on TocEntry so tests can assert against reference vectors. Rewrite all tests as property tests using proptest, and turn the silent skip on missing docs into a hard assertion.

mjcarson · 2026-02-26T04:34:33Z

+/// Maximum number of search results to return from `search_docs`.
+const MAX_SEARCH_RESULTS: usize = 10;
+
+/// Maximum number of context snippets per search result.
+const MAX_SNIPPETS_PER_PAGE: usize = 3;
+
+/// Number of context lines to include above and below each snippet match.
+const SNIPPET_CONTEXT_LINES: usize = 1;


Should we make these query params so the caller can adjust them as needed? Will that be too complex or confusing for LLMs to call reliably?

I think the main issue is that these calls are intended to be stateless, and thus they would have to request an amount of context every single time rather than set it once. @bisilaj is that correct?

mjcarson · 2026-02-26T04:42:35Z

+fn collect_md_files(dir: &Path) -> Result<Vec<std::path::PathBuf>, std::io::Error> {
+    Ok(std::fs::read_dir(dir)?
+        .map(|entry| {
+            let path = entry?.path();
+            if path.is_dir() {
+                collect_md_files(&path)
+            } else {
+                Ok(vec![path])
+            }
+        })
+        .collect::<Result<Vec<Vec<_>>, _>>()?
+        .into_iter()
+        .flatten()
+        .filter(|p| p.extension().and_then(|e| e.to_str()) == Some("md"))
+        .collect())
+}


We should use AsyncWalkDir here instead of the blocking calls to prevent blocking the thread if we have large amounts of files to crawl. That would also mean this doesn't have to be a recursive function.

mjcarson · 2026-02-26T04:43:55Z

+    content
+        .lines()
+        .find_map(|line| line.trim().strip_prefix("# ").map(String::from))
+        .unwrap_or_else(|| "Untitled".to_string())


Should we default to the file name (sans extension?) if no title can be found?

That probably makes more sense

mjcarson · 2026-02-26T04:45:50Z

+    /// Context snippets around the matches.
+    pub snippets: Vec<String>,


If we were to make snippets a Vec of structs with the snippet text and the offset into the file we could allow people to jump to docs if this exposed in the UI.

mjcarson · 2026-02-26T04:50:36Z

+fn extract_snippets(
+    content: &str,
+    query_lower: &str,
+    max_snippets: usize,
+    context_lines: usize,
+) -> Vec<String> {
+    let lines: Vec<&str> = content.lines().collect();
+    let mut snippets = Vec::new();
+    let mut last_snippet_line: Option<usize> = None;
+
+    for (i, line) in lines.iter().enumerate() {
+        if snippets.len() >= max_snippets {
+            break;
+        }
+        if line.to_lowercase().contains(query_lower) {
+            // skip if too close to the previous snippet to avoid overlap
+            if let Some(last) = last_snippet_line {
+                if i <= last + context_lines + 1 {
+                    continue;
+                }
+            }
+            // grab the matching line with context on each side
+            let start = i.saturating_sub(context_lines);
+            let end = (i + context_lines + 1).min(lines.len());
+            let snippet: String = lines[start..end].join("\n");
+            snippets.push(snippet);
+            last_snippet_line = Some(i);
+        }
+    }
+    snippets
+}


What are you thoughts on using this library to perform the search?

https://docs.rs/grep-searcher/latest/grep_searcher/

I think that's the right tool if we scale out but probably not yet unless you want to do that next. Right now we're doing case-insensitive substring matching across ~90 small markdown files, and the current implementation is simple and fast enough for that.

Where grep-searcher becomes compelling is if we add regex support to search queries (which feels like a natural next step, agents will want patterns like file.*upload). At that point we'd need a regex engine with context windowing and line tracking, and reimplementing that ourselves would just be rebuilding what grep-searcher already does. It would also give us byte/line offsets for free, which pairs nicely with the new Snippet.line_offset field.

A second trigger would be reuse beyond docs, if we ever want MCP tools that search source code or analysis results, having grep-searcher as shared infrastructure makes the second tool nearly free.

mjcarson · 2026-02-26T04:51:37Z

+        let summary_path = self.conf.docs_path.join("SUMMARY.md");
+        let content =
+            tokio::fs::read_to_string(&summary_path)
+                .await
+                .map_err(|e| ErrorData {
+                    code: rmcp::model::ErrorCode::INTERNAL_ERROR,
+                    message: format!("Failed to read documentation index: {e}").into(),
+                    data: None,
+                })?;
+
+        let entries = parse_summary(&content);


Should we move the summary path and the content reading into the parse_summary function?

mjcarson · 2026-02-26T04:54:32Z

+        let mut results: Vec<SearchResult> = md_files
+            .iter()
+            .filter_map(|file_path| {
+                let content = match std::fs::read_to_string(file_path) {


we don't want to use the blocking fs calls in the API.

mjcarson · 2026-02-26T04:54:56Z

+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::fs;


We should use the tokio fs module instead.

Replace all blocking filesystem calls with async equivalents: collect_md_files now uses async-walkdir (already a dependency in thorctl, thoradm, and agent) instead of recursive std::fs::read_dir, and search_docs uses tokio::fs::read_to_string instead of std::fs. Add optional max_results, max_snippets, and context_lines parameters to SearchDocs so callers can tune search behavior without changing defaults. Introduce a Snippet struct with a 1-based line_offset field alongside the snippet text to support jump-to-docs in UIs. Make extract_title fall back to the file stem (sans extension) when no H1 heading is found, instead of returning Untitled. Move SUMMARY.md file reading into parse_summary so the handler doesn't need to know about the file convention; the pure parsing logic remains available as parse_toc_entries for direct testing.

bisilaj added 2 commits February 17, 2026 18:33

refactor(mcp): Remove Empty struct from get_docs_toc handler

7c4b527

The rmcp #[tool] macro generates an empty JSON schema automatically when no Parameters<T> argument is present, so the Empty struct was unnecessary.

bisilaj closed this Feb 18, 2026

bisilaj reopened this Feb 18, 2026

bisilaj added 3 commits February 18, 2026 08:48

refactor(mcp): Use iterator pattern for parse_summary

33e05eb

Replace the manual for loop with filter_map and the ? operator for a more idiomatic Rust style. Added inline comments explaining each step of the markdown link parsing.

mjcarson reviewed Feb 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(mcp): Add documentation browsing and search tools#72

feat(mcp): Add documentation browsing and search tools#72
bisilaj wants to merge 6 commits into
cisagov:mainfrom
bisilaj:feat/mcp-docs-tools

bisilaj commented Feb 18, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

jbteves-sandia Mar 20, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

jbteves-sandia Mar 20, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

bisilaj Mar 4, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

mjcarson Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		/// Context snippets around the matches.
		pub snippets: Vec<String>,

Uh oh!

Conversation

bisilaj commented Feb 18, 2026

feat(mcp): Add documentation browsing and search tools

🗣 Description

💭 Motivation and context

🧪 Testing

✅ Pre-approval checklist

✅ Pre-merge checklist

✅ Post-merge checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants