Wayback URLs

Overview

waybackurls is a Node.js tool for pulling every historical URL the Wayback Machine has archived for a domain. It queries the Internet Archive's CDX API directly and returns a deduplicated list of every path, parameter, and subdomain that has ever been crawled — including pages long since removed from the live site.

It ships as both a global CLI and an importable library, so it fits equally well into a one-off terminal recon session or an automated Node.js pipeline. It's published on npm under the MIT license.

The primary use case is security reconnaissance: bug bounty hunters and pentesters use it to surface forgotten endpoints, old API routes, exposed parameters, and stale admin panels that no longer show up in a live crawl but were never actually decommissioned. It's equally useful for SEO audits (finding orphaned or redirect-broken URLs), content archaeology, and general OSINT on a target domain.

Key Features

• CDX API access — talks directly to the Wayback Machine's Capture Index, no scraping or HTML parsing involved
• Dual interface — works as a standalone CLI and as an importable library (require('waybackurls'))
• Subdomain matching — optionally expands the query to *.domain.com to pull URLs across every subdomain, not just the apex
• Deduplication — collapses near-duplicate captures using the CDX urlkey strategy by default
• File output — write results straight to a file for downstream tooling (ffuf, gf, httpx, etc.)
• Zero configuration — no API key required; a single dependency (axios)

Installation

# Global install — for CLI usage
npm install -g waybackurls
 
# Local install — for use as a library
npm install waybackurls

Usage

CLI

# Print archived URLs to stdout
waybackurls -d example.com
 
# Save results to a file
waybackurls -d example.com -o results.txt
 
# Legacy shorthand (still supported)
waybackurls example.com

As a library

const { fetchWaybackUrls } = require('waybackurls');
 
(async () => {
  const urls = await fetchWaybackUrls('example.com', {
    matchPrefix: true,  // include *.example.com subdomains
    collapse: 'urlkey', // CDX deduplication strategy
  });
 
  console.log(`Found ${urls.length} archived URLs`);
 
  const uniqueSubdomains = new Set(urls.map((u) => new URL(u).hostname));
  console.log('Unique subdomains:', [...uniqueSubdomains]);
})();

fetchWaybackUrls(domain, options) returns a Promise<string[]> and throws if the domain is invalid or the CDX request fails — safe to drop into an existing async recon or crawling script.

How It Works

The Wayback Machine keeps a searchable index of every snapshot it has ever taken, called the CDX (Capture Index). Instead of crawling archive.org's UI, waybackurls queries that index's HTTP API directly, requesting every unique URL captured under a domain (and its subdomains, if matchPrefix is enabled), then deduplicates the results before returning them. Because it reads from the archive rather than the live site, it surfaces URLs even if the underlying pages have since been deleted, moved, or blocked by robots.txt.

Links

View on GitHub ↗ · View on npm ↗