🔍 URL Intelligence API

A production-quality FastAPI project that performs deep analysis of any URL — built to show off what FastAPI does best.

What it does

Paste any URL and get back an instant intelligence report:

Signal	Details
Performance	Total time, TTFB, content size, status code, redirect chain
SEO	Title, description, OG tags, canonical, H1s, word count, reading time
Tech Stack	Frameworks, analytics tools, CDN/infra (50+ fingerprints)
Link Graph	Internal vs external links, external domain breakdown
Keywords	Top 10 content keywords by frequency (stopwords filtered)
Security	HTTPS, HSTS, CSP, X-Frame, X-Content-Type, Referrer-Policy audit

FastAPI features demonstrated

• async/await throughout — non-blocking HTTP fetching via httpx
• WebSockets — real-time step-by-step progress streaming to the browser
• Pydantic v2 models — full request validation and typed response schemas
• @field_validator — auto-prepends https:// to bare hostnames
• Auto-generated docs — Swagger UI at /docs, ReDoc at /redoc
• Background-friendly design — easily extendable with BackgroundTasks
• CORS middleware — configured for API use from any origin
• Static file serving — built-in UI served from /static
• Dual endpoints — both POST /analyze (JSON body) and GET /analyze?url=…

Project structure

url-intel/
├── app/
│   └── main.py          # All FastAPI routes, models, and analysis logic
├── static/
│   └── index.html       # Dark-mode UI with live WebSocket progress
├── requirements.txt
├── run.sh
└── README.md

Quickstart

# 1. Clone / unzip the project
cd url-intel

# 2. (Optional) create a virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install and run
./run.sh

Then open http://localhost:8000 in your browser.

API usage

REST — POST

curl -X POST http://localhost:8000/analyze \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com"}'

REST — GET (handy for quick tests)

curl "http://localhost:8000/analyze?url=https://stripe.com"

WebSocket (real-time progress)

const ws = new WebSocket("ws://localhost:8000/ws/analyze");
ws.onopen = () => ws.send(JSON.stringify({ url: "https://vercel.com" }));
ws.onmessage = e => console.log(JSON.parse(e.data));

Each message includes step, message, progress (1–8), and the final message includes the full report object.

Health check

curl http://localhost:8000/health

Example response shape

{
  "url": "https://github.com",
  "analyzed_at": "2025-08-14T10:23:11Z",
  "performance": {
    "total_time_ms": 312.4,
    "ttfb_ms": 124.9,
    "content_size_bytes": 248310,
    "status_code": 200,
    "redirect_count": 0
  },
  "seo": {
    "title": "GitHub · Build and ship software on a single, collaborative platform",
    "word_count": 1842,
    "reading_time_minutes": 9.2
  },
  "tech_stack": {
    "frameworks": ["React"],
    "analytics": ["Google Analytics"],
    "cdn": ["Fastly"]
  },
  "security": {
    "https": true,
    "hsts": true,
    "csp": false,
    "score": "4/6"
  }
}

Extending it

Some ideas to take this further:

• Lighthouse integration — spawn a headless Chrome audit
• Screenshot endpoint — playwright for full-page captures
• Diff mode — compare two URLs side-by-side
• History — store reports in SQLite via databases + aiosqlite
• Rate limiting — add slowapi middleware
• Auth — protect with fastapi-users or API key header dependency

Requirements

• Python 3.11+
• Internet access (the API fetches external URLs)