Web Scraping Without Getting Blocked: Best Practices in 2026

Why Do Websites Block Scrapers?

Websites employ anti-bot measures to protect their infrastructure, prevent data theft, and enforce their terms of service. Common detection methods include:

• IP Rate Limiting: Too many requests from the same IP address in a short period.
• User-Agent Analysis: Default bot user-agents (like python-requests/2.x) are instantly flagged.
• Browser Fingerprinting: JavaScript checks that detect headless browsers (missing plugins, WebGL, etc.).
• CAPTCHAs: Challenge-response tests triggered by suspicious behavior.
• Honeypot Traps: Invisible links that only bots follow.

Technique 1: User-Agent Rotation

The simplest improvement is to rotate your User-Agent header. Never use the default library user-agent.

import requests
import random

USER_AGENTS = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2_1) AppleWebKit/605.1.15 Safari/605.1.15',
    'Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Edg/120.0.0.0',
]

headers = {
    'User-Agent': random.choice(USER_AGENTS),
    'Accept': 'text/html,application/xhtml+xml',
    'Accept-Language': 'en-US,en;q=0.9',
}

response = requests.get('https://target.com', headers=headers)

Technique 2: Proxy Rotation

If you're making hundreds of requests, you need to distribute them across multiple IP addresses. Residential proxies are ideal because they look like real home connections.

# Using a rotating proxy service
proxies = {
    'http': 'http://user:[email protected]:8080',
    'https': 'http://user:[email protected]:8080',
}

response = requests.get('https://target.com', proxies=proxies)

# Or rotate from a list of proxies
PROXY_LIST = [
    'http://proxy1.example.com:8080',
    'http://proxy2.example.com:8080',
    'http://proxy3.example.com:8080',
]

proxy = {'http': random.choice(PROXY_LIST), 'https': random.choice(PROXY_LIST)}
response = requests.get('https://target.com', proxies=proxy)

Technique 3: Stealth Headless Browsers

Modern anti-bot systems can detect Puppeteer and Playwright. The puppeteer-extra-plugin-stealth plugin patches many of these detection vectors.

import puppeteer from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';

// Apply stealth plugin
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({ headless: 'new' });
const page = await browser.newPage();

// Set realistic viewport
await page.setViewport({ width: 1920, height: 1080 });

// Navigate like a human
await page.goto('https://target.com', { waitUntil: 'networkidle2' });

// Add random delays between actions
await page.waitForTimeout(Math.random() * 2000 + 1000);

const data = await page.evaluate(() => {
    return document.querySelector('.price').textContent;
});

await browser.close();

Common detection vectors that stealth plugins patch:

• navigator.webdriver being true
• Missing Chrome runtime objects
• Incorrect plugin/mime-type counts
• WebGL renderer strings revealing headless mode

Technique 4: Respectful Rate Limiting

The most overlooked technique is simply being polite. Many blocks happen because scrapers are too aggressive.

import time
import random

def polite_request(url, session):
    # Random delay between 2-5 seconds
    delay = random.uniform(2.0, 5.0)
    time.sleep(delay)
    
    response = session.get(url)
    
    # Back off on errors
    if response.status_code == 429:
        retry_after = int(response.headers.get('Retry-After', 60))
        print(f"Rate limited. Waiting {retry_after}s...")
        time.sleep(retry_after)
        return session.get(url)
    
    return response

Technique 5: Screenshot-Based Scraping

Sometimes the smartest approach is to skip HTML parsing entirely. Instead, take a screenshot and use OCR or visual AI to extract the data you need.

This approach is nearly undetectable because:

• You make a single request per page (minimal footprint).
• You don't interact with the DOM in suspicious ways.
• The rendered output is identical to what a real user sees.

# Using SnapAPI for screenshot-based scraping
import requests

api_url = "https://snapapi.alterra-solutions.com/v1/screenshot"
params = {
    "url": "https://target.com/product/123",
    "api_key": "YOUR_KEY",
    "width": 1280,
    "response": "binary",
    "wait_until": "networkidle0"
}

response = requests.get(api_url, params=params)

with open("product_page.png", "wb") as f:
    f.write(response.content)

# Now use OCR (Tesseract, Google Vision, etc.) to extract prices

Ethical Scraping Guidelines

With great power comes great responsibility. Follow these guidelines:

1. Check robots.txt: Always respect the website's crawling rules.
2. Read the ToS: Some sites explicitly prohibit scraping.
3. Don't overload servers: Add delays between requests.
4. Cache aggressively: Don't re-scrape data you already have.
5. Identify yourself: Consider adding contact info to your User-Agent.

Conclusion

Web scraping in 2026 requires a layered approach. Combine user-agent rotation, proxy management, stealth browsers, and respectful rate limiting for the best results. And when traditional scraping is too fragile, consider screenshot-based approaches for rock-solid reliability.