Lost and Found

Overview

Lost and Found is an OSINT multi-tool for uncovering forgotten hosts and URLs, retired services, archived webpages and content missing from existing web estate registries.

It can help answer questions such as:

• What URLs exist under a domain?
• What content has been captured historically?
• What exists outside our current registry?
• Which forgotten services are still online?
• Where are the gaps in web archive coverage?

Originally developed for web archiving and web estate mapping at Heritage Collections, The University of Edinburgh.

Hack the planet!

Installation

1. Clone the repository:

   git clone https://github.com/UoEMainLibrary/lost-and-found.git
   cd lost-and-found

2. Install Python 3.10 or newer (download Python)

3. Install required packages:

   pip install -r requirements.txt

4. Create a uniquely named folder inside the input/ folder, then place your web estate registry file in that folder. For example:

   input/
   └── my_registry/
       └── urls.csv
   

Tip

Your registry should be a CSV file containing URLs that Lost and Found can extract and process. It may originate from any source, including existing inventories, CMS exports, previous crawl outputs, web archive seed lists, or manually curated URL collections.

Quick Start

1. Discover hosts and URLs under a domain using one of the available Discovery Tools:

   python tools/internet_archive.py ed.ac.uk

2. Compare discovered hosts and URLs against your existing registry:

   python tools/compare.py input/my_registry/urls.csv output/ed.ac.uk/internet_archive/hosts.csv

3. Check whether discovered hosts and URLs are still active:

   python tools/validate.py output/ed.ac.uk/__comparisons/my_registry__internet_archive.csv

Toolkit

Tools in the Lost and Found toolkit are grouped by purpose:

• 🧲 Discovery Tools: discover hosts and URLs from external archives and datasets
• 🔎 Analysis Tools: process discovered hosts and URLs, compare results and validate findings

Tool	Purpose	File
🧲 Internet Archive Extractor	Extract hosts and URLs from Internet Archive's Wayback CDX Server API	internet_archive.py
🧲 Common Crawl Extractor	Extract hosts and URLs from the Common Crawl CDX URL Index	common_crawl.py
🧲 UKWA Extractor	Extract hosts and URLs from UKWA seed lists	ukwa.py
🧲 CRT.sh Extractor	Extract hosts from CRT.sh certificate transparency logs	crt_sh.py
🧲 archive.today Extractor	Extract hosts and URLs from search results of archive.today	archive_today.user.js
🔎 Registry Comparator	Compare two registries to identify new hosts and URLs	compare.py
🔎 Activity Validator	Check whether discovered hosts and URLs are still active	validate.py

Workflow

The recommended Lost and Found discovery workflow is:

flowchart TD
   A{Select Discovery Tool}
   A --> B[🧲 ⠀Internet Archive Extractor]
   A --> C[🧲 ⠀Common Crawl Extractor]
   A --> D[🧲 ⠀UKWA Extractor]
   A --> E[🧲 ⠀CRT.sh Extractor]
   A --> F[🧲 ⠀archive.today Extractor]
   B --> G[Extract hosts and URLs]
   C --> G
   D --> G
   E --> G
   F --> G
   G --> H{Compare Against Registry?<br/>🔎 ⠀Registry Comparator}
   H -->|Yes| I{Already Known?}
   H -->|No| K[New Discovery]
   I -->|Yes| J[Ignore]
   I -->|No| K
   K --> L[Check Status<br/>🔎 ⠀Activity Validator]
   L --> M{HTTP Response<br/>2xx / 3xx?}
   M -->|Yes| N[(Save Active Discovery)]
   M -->|No| O[Ignore]
   N --> P[(Add to Main Registry)]

Loading

Instructions

🧲 Internet Archive Extractor

Click to expand

⠀

Run internet_archive.py to extract hosts and URLs from Internet Archive's Wayback CDX Server API.

Functions

1. Search the Internet Archive for hosts and URLs under a domain:

   python tools/internet_archive.py ed.ac.uk

Results

• Results are written to:

  output/
  └── ed.ac.uk/
      └── internet_archive/
          ├── hosts.csv
          └── urls.csv
  

• urls.csv contains discovered URLs and hosts.csv contains unique hosts.

Back up to the toolkit ↑

🧲 Common Crawl Extractor

Click to expand

⠀

Run common_crawl.py to extract hosts and URLs from the Common Crawl CDX URL Index.

Functions

1. Search all Common Crawl indexes for hosts and URLs under a domain:

   python tools/common_crawl.py ed.ac.uk

2. Search latest Common Crawl index for hosts and URLs under a domain:

   python tools/common_crawl.py ed.ac.uk --latest

Results

• Results are written to:

  output/
  └── ed.ac.uk/
      └── common_crawl/
          ├── hosts.csv
          └── urls.csv
  

• urls.csv contains discovered URLs and hosts.csv contains unique hosts.

Back up to the toolkit ↑

🧲 UKWA Extractor

Click to expand

⠀

Run ukwa.py to extract hosts and URLs from UKWA seed lists.

Functions

1. Search a local UKWA seed list (JSON/CSV) for hosts and URLs under a domain:

   python tools/ukwa.py input/ukwa/urls.csv ed.ac.uk

2. Search a remote UKWA seed list (JSON/CSV) for hosts and URLs under a domain:

   python tools/ukwa.py https://librarylabs.ed.ac.uk/Annotation_Curation_Tool_Metadata-Collection_Seed_List_JSON.json ed.ac.uk

Results

• Results are written to:

  output/
  └── ed.ac.uk/
      └── ukwa/
          ├── hosts.csv
          └── urls.csv
  

• urls.csv contains discovered URLs and hosts.csv contains unique hosts.

Back up to the toolkit ↑

🧲 CRT.sh Extractor

Click to expand

⠀

Run crt_sh.py to extract hosts from CRT.sh certificate transparency logs.

Functions

1. Search CRT.sh certificate transparency logs for hosts under a domain:

   python tools/crt_sh.py ed.ac.uk

Results

• Results are written to:

  output/
  └── ed.ac.uk/
      └── crt_sh/
          └── hosts.csv
  

Back up to the toolkit ↑

🧲 archive.today Extractor

Click to expand

⠀

Use archive_today.user.js to extract hosts and URLs from search results of archive.today.

⚠️ archive.today can be a useful research tool, but users should be aware of concerns surrounding the service. See Wikipedia's archive.today guidance for more details.

Installation

1. Install a userscript manager such as Tampermonkey (install Tampermonkey)

2. Open the userscript in your browser (open userscript)

3. Wait for Tampermonkey to detect the userscript, then click Install

Functions

1. Open the archive.today 'wildcard' search page (https://archive.ph/search/?q=*)

2. Append your target domain after the existing *. in the search query and click Search (pre-filled search).

3. Click Start in the userscript interface and wait for it to process all available results pages

4. Wait until the extraction process is complete, then click Export to download the results or Copy to copy them to your clipboard

5. Click Reset to clear stored results before starting a new search

Results

• Exported results will be saved to your Downloads folder. We recommend adding the results to the repository under:

  output/
  └── ed.ac.uk/
      └── archive_today/
          └── urls.csv
  

Back up to the toolkit ↑

🔎 Registry Comparator

Click to expand

⠀

Run compare.py to compare two registries to identify new hosts and URLs.

Functions

1. Compare discovered hosts and URLs against your existing registry:

   python tools/compare.py input/my_registry/urls.csv output/ed.ac.uk/internet_archive/hosts.csv

Results

• Output files are named using the folder names of the compared sources:

  my_registry__internet_archive.csv

• Results are written to:

  output/
  └── ed.ac.uk/
      └── __comparisons/
          ├── my_registry__internet_archive.csv 
          └── ...
  

Back up to the toolkit ↑

🔎 Activity Validator

Click to expand

⠀

Run validate.py to check whether discovered hosts and URLs are still active. Treats HTTP 2xx and 3xx responses as active.

Functions

1. Check whether discovered hosts and URLs are still active:

   python tools/validate.py output/ed.ac.uk/__comparisons/my_registry__internet_archive.csv

Results

• Output files are named automatically using the source file name:

  my_registry__internet_archive__live.csv

• Results are written to:

  output/
  └── ed.ac.uk/
      └── __live/
          ├── my_registry__internet_archive__live.csv
          └── ...
          
  

Back up to the toolkit ↑

Credits

Developed by David Mahoney at Heritage Collections, The University of Edinburgh.

Citing

If you use, implement, or reference this project, please cite it as 'Lost and Found' and include clear attribution in publications, software, or documentation where appropriate.

Licenses

Lost and Found is licensed under Apache 2.0. For full licensing details, see the LICENSE file.