A simple Python-based web crawler that extracts and filters URLs from a given website while avoiding unwanted paths and file types. The crawler follows links recursively within the same domain and provides a clean list of URLs found across the website.
A fast, fully-typed Python crawler that collects every hook documented at developer.wordpress.org/reference/hooks and exports the data in five formats: JSON, YAML, Markdown, HTML, and plain text.
Add a description, image, and links to the py-crawler topic page so that developers can more easily learn about it.
To associate your repository with the py-crawler topic, visit your repo's landing page and select "manage topics."