Godspeed Crawaler

[hardik4tiwari]

Godspeed Crawler: Structured Content Sync from GitHub & Web URLs to Local Files

Description

The Godspeed Crawler is a file-based content ingestion system built using the Godspeed Framework. It extracts and continuously syncs content from GitHub repositories (both public and private) and various web URLs (blogs, news sites, Twitter feeds, etc.) and saves the extracted content locally in structured folders.

Key features include:

• Detect changes using GitHub commit hashes, RSS feeds, or sitemaps
• Extract only new or modified content to avoid redundant processing
• Store content in a local folder that mirrors the structure of:
  • the GitHub repo (for GitHub content), or
  • the website's sitemap hierarchy (for URLs)
• Save content as .md or .txt files, with metadata headers (YAML frontmatter or JSON)
• Track and update only changed files without re-downloading everything

The crawler is ideal for offline indexing, static content generation, or RAG pipelines that load files from local disk.

Categories

• Backend
• Automation
• Data Ingestion
• Developer Tools

Type

Large

Requirements

• Godspeed Framework (for project scaffolding and event-driven flow)
• GitHub API (to access private/public repos and detect commit diffs)
• Cheerio / Playwright / Puppeteer (for scraping HTML content)
• RSS Parser (to detect web updates)
• cron / godspeed-cron (for scheduled syncs)
• Filesystem module or fs-extra (to write, diff, and update content files)
• dotenv (for managing API keys and tokens)
• Optional: content hashing to detect duplicates or changes (e.g., SHA-256)

Acceptance Criteria

• Godspeed project structured using standard conventions
• GitHub content ingested via repo URL (supports private repos with token)
• Web URLs ingested via sitemap or direct crawl (RSS preferred if available)
• Local folder structure mimics:
  • GitHub repo structure (for repos)
  • Sitemap hierarchy or domain path (for websites)
• Content saved in plain text or markdown format
• YAML or JSON metadata saved at top of each file:
  • source_url
  • last_updated
  • commit_hash or content_hash
• Only new/modified content is written to disk; no unnecessary rewrites
• Configurable output root path (e.g., ./data/github/, ./data/web/)
• Logging added for all sync events (created, updated, skipped, failed)
• Easy to extend for new sources like PDF, Twitter feeds, Notion, etc.
• CLI or API to trigger manual sync (in addition to scheduled cron jobs)
• README included with usage instructions and environment setup