ForumGetter

A Laravel web application for crawling, archiving, and browsing Dutch forum sites. Built around a queue-based pipeline that scrapes pages, extracts reader-friendly content, and caches assets for offline viewing.

Features

• Multi-site management — add any number of seed URLs; each site is crawled independently
• Crawl pipeline — shallow one-level scan or recursive full spider
• Page archive — fetches HTML, stores compressed (gzip + base64) in the database
• Reader extraction — strips nav, ads, and sidebars to produce a clean reading view
• Asset snapshots — inlines CSS and images as data URIs for fully offline pages (CSS-only, popular-images, or full modes)
• Thread reconstruction — detects paginated forum threads; CSS selector wizard with live iframe preview
• Link graph — extracts page-to-page links; vis.js force-directed graph + popular pages ranking
• Exports — single-page HTML download; EPUB 3 export for threads
• Setup wizard — guided scan → plan → archive flow when adding a new site
• Light/dark mode — toggle persisted in localStorage
• Compression stats — DB and disk usage per site card
• Real-time queue status — Livewire polling shows pending/failed jobs; dedicated Queue page

Requirements

• PHP 8.4+
• Composer
• A database (SQLite works for development, MySQL for production)
• Queue worker (php artisan queue:work)

Setup

composer install
cp .env.example .env
php artisan key:generate
php artisan migrate
php artisan queue:work
php artisan serve

Usage

Crawl → Import → Enrich pipeline

1. Add a site — paste a seed URL; redirected to the setup wizard which auto-starts a shallow scan
2. Wizard — watch URLs being discovered, then choose archive mode (Reader / CSS snapshot / Full) and optionally filter by path prefix, then start importing
3. Enrich (optional, from the advanced panel on the Sites page):
   • Reader view — extract main content, strip chrome
   • Extract links — build the page link graph
   • Snapshot — inline CSS (and optionally images) for offline rendering
4. Detect threads — groups paginated URLs into multi-page threads

Thread wizard

Open Thread wizard from a site's advanced panel to configure CSS selectors for post extraction. Type a selector in the left panel and the right iframe updates live with highlighted matches.

Browsing

• Browse — full-text search across all archived pages; availability dots indicate reader/snapshot status per row; relative timestamps with absolute tooltip
• Tree — navigate the URL hierarchy with per-node import actions
• Popular — pages ranked by inbound link count
• Graph — vis.js force-directed link network
• Threads — stitched multi-page thread view with per-post cards
• Queue — live view of running, waiting, and failed jobs (also reachable via the header badge)

Architecture

Job pipeline

Job	Purpose
ScanDomainJob	Seeds recursive crawl
ScanUrlJob	Recursive spider — dispatches itself for each new URL
ShallowScanJob	One-level link extraction (no recursion)
ResumeScanJob	Re-dispatches ScanUrlJob for all unimported URLs
ImportPagesJob	Fetches HTML, compresses, saves to pages; auto-runs reader extraction
ExtractReaderContentJob	Readability.php-based reader extraction (Mozilla Readability port); falls back to plain text
InlineAssetsJob	Fetches CSS/images, saves to disk, builds asset cache
ExtractLinksJob	Parses hrefs from all pages, populates page_links
DetectThreadsJob	Groups paginated URLs into threads records
ScanPdfLinksJob	Finds external .pdf links, exports CSV

Services

Class	Purpose
App\Services\Parser	cURL scraper — extracts links, images, title, meta tags
App\Services\UrlNormaliser	Strips query strings, normalises www., strips pagination suffixes
App\Services\CssToXPath	Converts CSS selectors to XPath for DOM extraction

Controllers

Controller	Purpose
SiteController	Site CRUD, job dispatch, tree browser, link graph data, popular pages
SiteSetupController	Setup wizard — show, status polling, job dispatch, favicon serving
DataPageController	Archive browser, page detail, content serving, internal link rewriting
ThreadController	Thread list/show, selector wizard, live preview, DetectThreadsJob dispatch
ExportController	HTML download (single page or thread), EPUB 3 export
QueueController	Queue page and JSON data endpoint

Database schema

Table	Key columns
sites	domain, seed_url, name, thread_config (JSON)
urls	site_id, url, imported
pages	url_id, content (gzip+base64), reader_content, title
site_assets	site_id, url, type, mime, disk_path, size
page_links	from_page_id, to_url, to_page_id
threads	site_id, title, slug, base_url, page_count
thread_pages	thread_id, page_id, page_number
jobs / failed_jobs	Laravel queue tables

Page content is stored compressed. To read it back:

gzuncompress(base64_decode($page->content))

Running tests

php artisan test
# or
./vendor/bin/phpunit