🐦 TweetScraper

A cost-effective Twitter/X data scraper with anti-bot detection, location extraction, engagement metrics, and resume-capable scraping — built as a free alternative to expensive paid tweet scraping APIs.

🎯 Problem Statement

Commercial tweet scraping APIs charge $100-500/month for basic data access, and Twitter's official API (now X API) has aggressive rate limits and costly tiers. For research projects requiring 10K+ tweets with rich metadata, these costs are prohibitive.

TweetScraper provides the same capabilities at zero cost — and adds features that paid APIs don't offer, such as profile location extraction and anti-bot detection for uninterrupted long-running scrapes.

✨ Features

Feature	Description
🛡️ Anti-Bot Detection	Randomized scroll distances, typing delays, mouse simulation, viewport rotation, and rate-limiting cooldowns
📍 Dual Location Extraction	Extracts both tweet geo-tags AND user profile locations with a caching layer
📊 Engagement Metrics	Replies, retweets, likes — handles K/M shorthand and comma-separated numbers
💬 Reply Scraping	Extracts up to 50 replies per tweet with spam filtering
🔄 Resume-Capable	Automatically resumes from where it stopped — no duplicate processing
📁 Dual Export	JSON + CSV output with proper escaping and flattened structure
🍪 Session Persistence	Cookie-based auth with auto-refresh — login once, scrape endlessly
⚡ CLI Interface	Fully configurable via command-line args (tags, dates, limits, features)
🧪 Tested	29 unit tests across metrics parsing, data export, and config validation

🏗️ Architecture

server/src/
├── config/
│   ├── env.js                 # Zod-validated environment config
│   └── scraping-options.js    # Configurable scraping parameters
├── lib/
│   ├── anti-detect.js         # Human-like behavior simulation
│   ├── auth.js                # Cookie-based authentication flow
│   ├── browser.js             # Stealth Puppeteer setup
│   ├── export.js              # JSON + CSV data export
│   ├── location.js            # Tweet & profile location extraction
│   ├── metrics.js             # Engagement metric parsing
│   └── scraper.js             # Core scraping engine
├── utils/
│   ├── delay.js               # Timing utilities with jitter
│   ├── logger.js              # Winston structured logging
│   └── retry.js               # Exponential backoff retry
├── tests/
│   ├── config.test.js         # Config validation tests
│   ├── export.test.js         # Export pipeline tests
│   └── metrics.test.js        # Metric parsing tests
├── main.js                    # CLI entry point & orchestrator
└── package.json

🚀 Quick Start

1. Clone & Install

git clone https://github.com/aneebnaqvi15/TweetScraper.git
cd TweetScraper/server/src
npm install

2. Configure

cp .env.example .env
# Edit .env with your Twitter/X credentials

3. Run

# Default run
node main.js

# Custom search with CSV export
node main.js --tags "#AI" "#MachineLearning" --max 500 --csv

# Fast mode (no location/reply extraction)
node main.js --no-location --no-replies --max 1000

# Show all options
node main.js --help

4. Test

npm test           # Run all tests
npm run test:coverage  # With coverage report

🤖 AI-Assisted Development

This project was built with an AI-first engineering approach. Here's how AI tools were integrated throughout the development workflow:

Debugging Twitter Selectors

Twitter/X frequently updates their DOM structure, breaking selectors like [data-testid="tweetText"]. AI was used to analyze page structures and suggest robust fallback selector strategies, including the aria-label fallback pattern used in lib/metrics.js.

Anti-Bot Detection Strategy

The stealth techniques in lib/anti-detect.js were iteratively refined with AI assistance — analyzing what patterns get flagged by Twitter's bot detection and designing randomization strategies (scroll jitter, viewport rotation, mouse simulation) to mimic human behavior.

Code Architecture Refactoring

AI helped transform a 444-line monolithic script into a clean modular architecture with proper separation of concerns — extracting 7 focused modules from a single main.js file while preserving all original functionality.

Test Generation

AI assisted in generating comprehensive test cases including edge cases for metric parsing (K/M notation, commas, null/undefined inputs) and CSV escaping (special characters, newlines, quotes).

What I Validated Manually

• Scraped 10,000+ tweets for a health research project (chiropractic care data 2020-2024)
• Verified anti-bot detection by running continuous 4-hour scraping sessions without blocks
• Confirmed profile location cache prevents redundant page loads (50%+ speed improvement)
• Cross-checked exported CSV data with original JSON to verify flattening accuracy

🌍 Real-World Impact

Successfully scraped 10,000+ health-related tweets (chiropractic care hashtags, 2020–2024) for a health research project, including:

• Tweet text, dates, and engagement metrics
• User profile locations for geographic analysis
• Reply threads for sentiment analysis
• All delivered at zero API cost vs. $200+ for equivalent paid API usage

🛠️ Tech Stack

• Runtime: Node.js 18+
• Browser Automation: Puppeteer + Stealth Plugin
• Validation: Zod (env + config schemas)
• CLI: Commander.js
• Logging: Winston (console + file transports)
• Testing: Jest
• Anti-Detection: Custom stealth techniques (viewport rotation, scroll jitter, timing randomization)

📝 License

MIT — See LICENSE for details.

⚠️ Disclaimer

This tool is for educational and research purposes. Ensure compliance with Twitter/X Terms of Service and applicable data privacy regulations when using this scraper.