LinkedIn Workforce Report Scraper

Scrapes LinkedIn's monthly Workforce Reports into a local SQLite database and serves them through a web UI.

Each report publishes hiring rate data by industry — an indexed rate normalized to a 2016 baseline of 1.0, with month-over-month and year-over-year change columns. The scraper discovers reports from the LinkedIn workforce-data list page, parses the HTML tables, and stores results incrementally so re-runs only fetch new reports.

---

Screenshots

Highlights — KPI cards, full hiring rate strip chart, Top 5/Bottom 5, market landscape scatter, and biggest MoM movers.

Trends — Sector hiring rate over time with stitched historical data across LinkedIn's taxonomy changes, seasonality heatmap, and compare overlay.

Snapshot — Sortable table of all sectors for any report with color-coded MoM% and YoY% changes.

Rankings — Horizontal bar chart ranking all sectors by hiring rate, MoM%, or YoY% for any report.

Matrix — Scatter plot of sectors by any two axes (Rate, MoM%, YoY%) with quadrant shading and timeline scrubber.

---

Requirements

• Bun 1.x
• Internet access (scrapes linkedin.com)

---

Setup

bun install
bunx playwright install chromium   # one-time browser download (~92 MB)

---

Usage

1. Scrape reports

bun run scrape

Launches a headless browser to discover report links from the LinkedIn workforce-data page, then fetches and parses each one. Already-scraped reports are skipped automatically.

# Options
bun run scrape --db ./data/my.db        # custom database path (default: data/linkedin_hiring.db)
bun run scrape --limit 10               # stop after N reports
bun run scrape --force                  # re-scrape and overwrite existing records
bun run scrape --include-international  # also scrape UK, Canada, India reports

On a full run, the scraper processes ~90 US reports. Errors on individual reports (e.g. pre-2018 reports with no data table) are logged and skipped without interrupting the run.

2. Choose a serving mode

The frontend can run against a live Bun backend or from pre-baked static JSON files (for hosting without a server). The mode is set in public/config.js:

const APP_CONFIG = {
  USE_STATIC_API: true   // true = static files, false = live Bun server
};

Option A — Live server (default for local development)

Set USE_STATIC_API: false in public/config.js, then:

bun run server

Opens a local server at http://localhost:3000, reading directly from SQLite on each request.

bun run server --db ./data/my.db    # match the DB path used by the scraper
bun run server --port 8080

Option B — Static files (GitHub Pages / Vercel / any static host)

Set USE_STATIC_API: true in public/config.js (already the default), then bake the data:

bun run build

This writes pre-computed JSON files into public/data/ (one file per report, per sector, per canonical trend). Serve the public/ directory with any static file server:

bunx serve public
# or
python3 -m http.server -d public 3000

Re-run bun run build whenever the database is updated.

---

Web UI

Five tabs:

Highlights — Select any report and a metric (YoY%, MoM%, or Latest Rate). Shows the top 3 and bottom 3 sectors for that metric at a glance.

Snapshot — Select any report from the dropdown. Shows a sortable table of all sectors with their latest hiring rate, MoM% change, and YoY% change. Changes are color-coded green/red. Reports from 2018–2019 show — for YoY (those reports predated the two-column change format).

Trends — Select a sector to see a Chart.js line chart of its hiring rate over time. Includes:

• Compare overlay: add a second (or more) sectors to the same chart
• Baseline overlay toggle: show the all-sector average as a reference line
• Seasonality heatmap: MoM% by month across years
• Sector alias signals: renamed sectors show a dashed amber vertical line at the transition date and a "Previously known as" note; merged sectors show individual predecessor charts plus a full-range stitched chart with a predecessor selector dropdown

Matrix — Scatter plot of sectors for a selected report. X/Y axes are configurable (Hiring Rate, MoM%, YoY%). A scrubber lets you move through the report timeline. Supports multi-report comparison via compare tags.

Rankings — Horizontal bar chart ranking all sectors by a chosen metric (Hiring Rate, MoM%, YoY%) for a selected report.

---

Sector taxonomy

LinkedIn changed its sector taxonomy around 2021–2022: 24 old sector names consolidated into 20 new names. The scraper stores original names as-is. The UI surfaces the mapping explicitly:

• Renamed (1:1): e.g. Finance → Financial Services. The Trends chart stitches old and new data into a single continuous line with an amber marker at the transition.
• Merged (many-to-1): e.g. Software & IT Services + Hardware & Networking + Media & Communications → Technology, Information and Media. The Trends page shows each predecessor as its own chart plus a full-range stitched chart where you pick which predecessor to use as the pre-2022 baseline.
• Legacy: six sectors discontinued after 2021 with no modern equivalent (Arts, Design, Nonprofit, Public Safety, Recreation & Travel, Wellness & Fitness). Shown with a "Legacy sector" note.

The mapping lives in src/database/aliases.ts and can be updated as LinkedIn's taxonomy evolves.

---

Database schema

Three tables in SQLite (default path: data/linkedin_hiring.db):

reports — one row per scraped report

Column	Type	Description
id	INTEGER	Primary key
url	TEXT	Source URL
slug	TEXT	e.g. linkedin-workforce-report-march-2026
label	TEXT	Human-readable label, e.g. March 2026
country	TEXT	us, uk, canada, or india
scraped_at	TEXT	ISO 8601 timestamp

sectors — one row per unique industry name seen across all reports

hiring_metrics — one row per (report, sector) pair

Column	Type	Description
anchor_month / anchor_rate	TEXT / REAL	Year-ago reference month and its rate
m1_label–m4_label	TEXT	Month labels for the 4-month window
m1_rate–m4_rate	REAL	Hiring rates for those months
mom_change	REAL	Month-over-month % change
yoy_change	REAL	Year-over-year % change (null for pre-2020 reports)

---

Project structure

src/
  cli/
    scraper.ts          entry point — CLI args, orchestration loop
    discover.ts         Playwright-based link discovery from the list page
    parse.ts            HTML table parsing, column detection, number extraction
    build_static.ts     bakes DB data into public/data/ JSON files
  database/
    db.ts               SQLite schema, insert/delete helpers
    aliases.ts          sector rename/merge/legacy mapping
  server/
    server.ts           Bun HTTP server — API endpoints + static file serving
  shared/
    constants.ts        shared Chrome request headers
    types.ts            shared TypeScript types
data/
  linkedin_hiring.db    SQLite database (created by scraper)
public/
  index.html            page shell (5 tabs)
  style.css             styles
  app.js                tab switching, all chart/table logic
  config.js             USE_STATIC_API toggle
  data/                 pre-baked JSON files (created by bun run build)

---

Notes

• Column format changes: reports from 2018–2019 include an extra "Hiring Rate" summary column and a single % Change column instead of separate MoM/YoY columns. The parser handles both formats automatically.
• Pre-2018 reports appear on the list page but use a different CMS format with no data table. They are skipped with a logged error and do not affect the run.
• Sector names vary slightly across years (42 unique names across all reports vs. 20 in any given modern report). The Trends view uses canonical names with explicit alias signals.