KLScrapper

KLScrapper is a multi-threaded batch media downloader and web crawler. Built with Python and PyQt6, it provides a desktop interface for managing parallel downloads from direct URLs, standard HTML pages, and modern video platforms.

The application unifies several extraction methods, supporting headless browser rendering for dynamic content and stream multiplexing for high-resolution video formats.

Core Capabilities

• Unified Extraction Engine: Automatically determines the best extraction method per URL. Supports direct file links, HTML parsing (<img>, <video>, <a>), and hundreds of platforms via native yt-dlp integration.
• Headless JavaScript Rendering: Integrates Playwright to evaluate JavaScript-heavy sites and Single-Page Applications (SPAs) before executing DOM extraction.
• Segmented Downloading: Splits large media files into byte-range chunks and downloads them concurrently, maximizing available network bandwidth.
• Stream Multiplexing: Automatically detects when platforms host video and audio tracks separately. Downloads the highest quality available streams and merges them using FFmpeg without re-encoding.
• State Persistence: Maintains a continuous extraction and download queue. Session state is written to ~/.klscrapper_queue.json, allowing the application to be closed and resumed without losing progress.
• Bandwidth Management: Includes a built-in rate limiter to cap maximum throughput during background operation.
• Rule-based Organization: Sorts downloaded media dynamically based on domain source, file extension, or the current date.

Prerequisites

• Python: Version 3.10 or higher.
• FFmpeg: Required for multiplexing split video and audio streams. The application will search for the ffmpeg executable in the following locations:
  1. The application's root directory (same folder as main.py or the compiled executable).
  2. The system PATH environment variable.
  3. The WinGet package directory (%LOCALAPPDATA%\Microsoft\WinGet\Packages).
  4. Standard Windows installation paths (C:\ffmpeg\bin\ffmpeg.exe or C:\Program Files\ffmpeg\bin\ffmpeg.exe).

If FFmpeg is not found, KLScrapper will continue to function but may fall back to lower-quality pre-merged streams or fail to process high-resolution videos from certain platforms.

Installation

1. Clone the repository to your local machine:

   git clone https://github.com/yourusername/KLScrapper.git
   cd KLScrapper

2. Install the required Python dependencies:

   pip install -r requirements.txt

3. (Optional) If you intend to extract media from JavaScript-rendered sites, install the necessary Playwright browser binaries:

   playwright install

Usage

To start the application, run the main entry point:

python main.py

Interface Overview

• Input Field: Paste individual URLs or a batch of links.
• Queue Manager: View the status of extractions, file sizes, download progress, and processing speeds in real time.
• Configuration Panel: Set your output directory, toggle headless extraction, adjust the concurrent worker count, and configure file organization rules.

Architecture Pipeline

The system operates on a four-stage pipeline:

1. Extraction: URLs are added to a concurrent worker pool. The system resolves redirects, identifies the content source, and extracts the direct media URLs using either standard HTTP requests, Playwright DOM evaluation, or yt-dlp metadata extraction.
2. Verification: Before allocating disk space or beginning a transfer, the manager performs a HEAD request to verify the file's MIME type and size, ensuring it matches the user's filtering rules.
3. Transfer: Validated files are segmented. Each chunk is downloaded in parallel to temporary .part files. Once all segments are acquired, they are sequentially joined.
4. Post-Processing: For split-stream media, the application invokes FFmpeg subprocesses to map and mux the tracks into a final container format. Temporary stream files are subsequently purged.

License

This project is open-source and available under the standard MIT License.