How to Download an Entire Website from the Wayback Machine

An entire website should mean a documented set of hosts and paths. Decide whether subdomains, language folders, media hosts, and downloadable documents belong in scope. Without that boundary, a crawler can either miss required assets or follow links far beyond the owned site. Use the Archive.org CDX API recovery guide to build a candidate URL inventory without treating every replay record as a distinct page.

Use a normalized URL key that handles fragments, tracking parameters, trailing slashes, and default index files. Preserve meaningful query parameters only when they produce distinct content. Then apply the Wayback Machine URL format guide so timestamps, modifiers, original URLs, queries, and fragments are normalized consistently.

A target date is a preference, not a guarantee. For each URL, record the nearest valid capture and how far it is from the target. Avoid silently combining years of unrelated redesigns. If the closest asset is structurally incompatible, flag it for review rather than forcing it into the package. After the inventory is bounded, submit one exact capture to the Wayback Machine downloader to estimate and configure the authorized recovery.

Count discovered pages, downloaded pages, missing assets, redirects, and external dependencies. Open the recovery through a local web server rather than directly from the file system because relative paths and browser security rules differ. Test with archive.org blocked to confirm the package stands alone.

An archive index can contain many addresses that do not represent distinct pages. Tracking parameters, session identifiers, print views, calendar loops, case variants, trailing slash variants, and repeated captures can inflate the count. Normalize only with written rules. Keep query parameters that change meaningful content, and retain a sample of discarded variants so the reduction can be reviewed later.

Count final original URLs after normalization, not replay URLs. Separate HTML pages from redirects, assets, error responses, and files. Then group pages by template family and priority. This produces a more useful estimate than a raw capture count because it shows both how much public content may be recoverable and how many distinct layouts require verification.

Use two inventories. The crawl inventory contains URLs reached through recovered navigation, body links, feeds, styles, scripts, and media references. The archive-index inventory contains in-scope captures discovered from metadata even when no recovered page links to them. Compare the sets rather than assuming either one is complete.

A URL found only in the index is a candidate orphan, not automatically a page worth publishing. Check its status, MIME type, content, canonical purpose, date, and relationship to the approved site. A URL found only through a page may have no valid capture and belongs in the missing-file report. The final reconciliation should label every candidate as recovered, redirected, duplicated, excluded, missing, or pending review.

Completion should be measured against the approved boundary and manifest, not against every URL Archive.org has ever observed for the domain. Agree on required hosts, path prefixes, language versions, file types, capture dates, and exclusions. Set acceptance rules for representative templates, navigation, documents, images, and offline loading before the full crawl begins.

The handoff should include the portable files, normalized URL inventory, source manifest, missing-file report, redirect plan, broken-link results, and a summary of capture substitutions. If the output uses a framework or CMS, also include the route and content mapping. These records allow the owner to see what was recovered without mistaking the package for a server backup.