Leadgen Orchestrator

A parallel, multi-ICP lead-generation pipeline built on Apify. Define one or more Ideal Customer Profiles (ICPs) in config, and the orchestrator runs a self-improving scrape -> validate -> tune -> retry loop for each one in parallel, with cost tracking, quality scoring, and local storage.

Each ICP pipeline is autonomous: it scrapes leads, scores their quality (0.0-1.0), and if the batch falls below the quality threshold it tunes its own search parameters and retries - warm-starting future runs from the best parameters it has seen so far.

What it does

• Two scrape strategies, config-driven:
  • linkedin - Google Search (site:linkedin.com/in queries) to discover profile URLs, then the LinkedIn Profile Scraper to enrich name / title / company / email.
  • maps - Google Maps (Google Places) to find local businesses with address, phone, website, and email.
• Self-improving loop - low-relevance or low-contact batches trigger automatic query narrowing, lead enrichment, and broader pagination on retry.
• Quality scoring - every lead is scored on completeness, relevance, and contact info, fully driven by the ICP's quality_criteria.
• Cost tracking - per-run and per-session USD estimates based on Apify actor pricing (see config/actors.json).
• Local storage - raw leads, quality reports, iteration logs (the "memory"), and an export-ready CSV, all written under output/<icp_id>/.

Architecture

orchestrator.py
   |
   |  (ThreadPoolExecutor: one thread per ICP)
   v
ICPPipeline  x N
   |
   |-- strategy "linkedin": GoogleSearchScraper -> LinkedInProfileScraper
   |-- strategy "maps":     GoogleMapsScraper
   |
   v
QualityValidator  (score 0.0-1.0, suggestions for tuning)
   |
   v
LeadStorage  ->  output/<icp_id>/   (JSON + CSV + iterations.json)

• src/orchestrator.py - loads ICP configs, runs every pipeline in parallel, prints a results table, writes a master report.
• src/pipeline.py - the self-improving loop for a single ICP.
• src/scrapers/ - thin wrappers over the Apify actors (apify_client.py, google_search.py, linkedin.py, google_maps.py).
• src/validators/quality.py - config-driven quality scoring and dedup.
• src/memory/storage.py - local JSON/CSV persistence and warm-start memory.

Apify actors used

Strategy	Actor	Purpose	Cost
both	apify/google-search-scraper	SERP + LinkedIn URL discovery + lead enrichment	~$1.80 / 1000 pages
linkedin	apify/linkedin-profile-scraper	Full profile enrichment (no cookies)	~$10 / 1000 profiles
maps	apify/crawler-google-places	Local business data from Google Maps	~$5 / 1000 places

Actor IDs, default parameters, and cost metadata live in config/actors.json.

Setup

pip install -r requirements.txt

cp .env.example .env
# edit .env and set your Apify token (from https://console.apify.com/settings/integrations)
#   APIFY_TOKEN=your_apify_token_here

cp config/icps.example.json config/icps.json
# edit config/icps.json to describe your own ICPs (see schema below)

Sanity-check the install:

python test_smoke.py          # config + import check, no Apify calls
python test_smoke.py --live   # + one real Google Search call (~$0.002)

Usage

# Preview the plan without hitting Apify
python -m src.orchestrator --dry-run

# Run a specific ICP (or several)
python -m src.orchestrator --icp saas_marketing_leads
python -m src.orchestrator --icp saas_marketing_leads local_law_firms

# Run every ICP defined in config/icps.json, in parallel
python -m src.orchestrator

Outputs land in output/<icp_id>/:

• leads_final_<timestamp>.csv - export-ready for your email / outreach tool
• leads_iter<N>_<timestamp>.json - raw leads per iteration
• quality_iter<N>_<timestamp>.json - quality report per iteration
• iterations.json - cumulative log used to warm-start future runs

Bring your own ICP config

The engine is fully config-driven - no code changes are needed to target a new audience. Add entries to config/icps.json. config/icps.example.json ships with two neutral examples (saas_marketing_leads, local_law_firms) you can copy and edit.

ICP config schema

Each entry in the icps array supports the following fields:

Field	Required	Applies to	Description
id	yes	both	Unique slug. Used for --icp filtering and as the output/<id>/ directory.
name	yes	both	Human-readable label shown in the results table.
strategy	yes	both	"linkedin" or "maps" - selects the scrape pipeline.
priority	no	both	Integer for your own ordering/notes (informational).
target_count	no	both	Target number of leads (informational, shown in the plan).
geography	no	both	Free-text region appended to enrichment/expansion queries (e.g. "United States").
target_roles	linkedin	linkedin	List of role titles; the first is used to narrow queries when relevance is low.
linkedin_keywords	linkedin	linkedin	Keywords used to broaden the search when too few leads are found.
google_search_queries	linkedin	linkedin	Initial Google Search queries (typically site:linkedin.com/in ...).
google_maps_queries	maps	maps	Search terms for Google Maps (e.g. "law firm").
google_maps_cities	maps	maps	Cities to search; omit for a single nationwide query per term.
quality_criteria.required_fields	yes	both	Fields that must be present for a complete lead.
quality_criteria.preferred_fields	no	both	Nice-to-have fields that boost the completeness score.
quality_criteria.relevance_keywords	no	both	Keywords matched against title/company/description to score relevance.

Quality scoring

Each lead is scored from 0.0 to 1.0:

• 40% completeness - share of required_fields (weighted 0.7) and preferred_fields (weighted 0.3) that are present.
• 40% relevance - fraction of relevance_keywords matched in the lead text.
• 20% contact info - presence of LinkedIn URL, a valid email, and a phone number.

A batch scoring below the quality threshold triggers the self-improvement loop: the pipeline narrows queries, enables lead enrichment, and/or widens pagination, then retries (up to a fixed iteration cap).

Cost model

Costs are estimated from the per-actor rates in config/actors.json and tracked per session. Typical guardrails in src/pipeline.py:

• LinkedIn profile scrapes are capped per run (default 100 profiles ~ $1).
• Google Search costs ~$1.80 / 1000 result pages.
• Google Maps costs ~$5 / 1000 places.

The orchestrator prints a session cost summary when it finishes.

License

MIT - see LICENSE.