Build Your Own Wayback Machine: Automated Website Archiving

The Transient Web

Did you know that the average lifespan of a webpage is just 100 days? Sites go down, companies pivot, and designs change. Often, valuable information is lost forever.

While the Internet Archive (Wayback Machine) does an amazing job, it can't capture everything, and you can't control its schedule. By building your own, you ensure that the content you care about is preserved.

Why Build an Archive?

• Proof of Existence: Keep records of your own work or portfolio.
• Compliance: Archive your company's TOCs and Privacy Policies as they change.
• Design Inspiration: Build a mood board of evolving UI trends over time.
• Sentimental Value: Keep a copy of your favorite blogs or communities.

The Architecture

Our goal is to build a "set and forget" script.

1. Input: A simple text file (urls.txt) containing the sites to track.
2. Engine: Python script using SnapAPI to render pages just like a user sees them.
3. Output: A structured folder system organized by Year/Month/Day.

The Code

We'll create a script called archiver.py. It reads URLs and saves full-page screenshots.

import requests
import os
from datetime import datetime

# Setup
API_KEY = 'YOUR_SNAPAPI_KEY'
SOURCE_FILE = 'urls.txt'
ARCHIVE_ROOT = 'my_archive'

def get_todays_folder():
    """Creates a folder path like my_archive/2026/01/27/"""
    now = datetime.now()
    path = os.path.join(ARCHIVE_ROOT, str(now.year), f"{now.month:02}", f"{now.day:02}")
    if not os.path.exists(path):
        os.makedirs(path)
    return path

def archive_url(url, folder):
    print(f"Archiving {url}...")
    
    # Clean filename (convert https://google.com -> google.com)
    filename = url.replace('https://', '').replace('http://', '').replace('/', '_') + '.png'
    save_path = os.path.join(folder, filename)
    
    # Call SnapAPI
    # full_page=true ensures we get the entire scrollable area
    endpoint = f"https://snapapi.alterra-solutions.com/v1/screenshot"
    params = {
        "url": url,
        "api_key": API_KEY,
        "full_page": "true",
        "response": "binary"
    }
    
    try:
        r = requests.get(endpoint, params=params)
        if r.status_code == 200:
            with open(save_path, 'wb') as f:
                f.write(r.content)
            print(f"✅ Saved to {save_path}")
        else:
            print(f"❌ Error: {r.status_code}")
    except Exception as e:
        print(f"❌ Failed: {e}")

# Main execution
if __name__ == "__main__":
    if not os.path.exists(SOURCE_FILE):
        print(f"Create a {SOURCE_FILE} file with one URL per line first!")
        exit()
        
    todays_folder = get_todays_folder()
    
    with open(SOURCE_FILE, 'r') as f:
        urls = [line.strip() for line in f if line.strip()]
        
    for url in urls:
        archive_url(url, todays_folder)

Smart Storage Tips

To keep your archive efficient:

• Compress Images: Use a tool like pngquant to reduce file size by 70% without visible loss.
• Deduplicate: Before saving, check if the file is identical to yesterday's snapshot. If it is, use a symlink instead of duplicating the file.
• Cloud Sync: Point your ARCHIVE_ROOT to a Dropbox or Google Drive folder for automatic cloud backup.

Preserve History

With this simple tool, you're not just taking screenshots; you're preserving a slice of digital history. Start your archive today before the pages you love change forever.