Tutorials 5 min read

How to Avoid Getting Blocked When Web Scraping (2026 Guide)

Most scraping blocks are self-inflicted: too fast, too uniform, too obviously a script. Here's how to scrape responsibly and stay unblocked.

If you want to avoid getting blocked web scraping, start by accepting that sites block scrapers for understandable reasons: excessive load, abusive bots and data theft. Most blocks you'll hit aren't sophisticated; they're reactions to traffic that is too fast, too uniform or too obviously a script. This guide covers the practices that keep a legitimate scraper healthy: pacing, realistic requests, session handling, the right proxy type, sensible retries, and respecting the site's rules.

One boundary up front: this is about collecting public data responsibly. It doesn't cover defeating CAPTCHAs, getting around logins you don't own, or ignoring a site that has clearly told you no. If a site's terms prohibit scraping, or the data is behind authentication, get permission or use an official API. Our guide on whether proxies are legal explains the basics.

1. Respect robots.txt and Rate Limits

Read robots.txt before you start. It tells you which paths the site doesn't want crawled and sometimes a crawl delay. Python's standard library can check it for you:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("MyScraper/1.0", "https://example.com/products/"))
print(rp.crawl_delay("MyScraper/1.0"))  # None if not set

Then pace yourself. Request rate is the single biggest factor in getting blocked. A human doesn't load 20 pages per second, and neither should your scraper on a small site. Start slow, such as one request every second or two per site, and only speed up while your success rate stays high. Add random jitter so requests don't arrive at perfectly regular intervals. Scrape during the target's off-peak hours when you can.

2. Send Realistic, Consistent Headers

Default library headers like python-requests/2.x are an instant giveaway. Send what a real browser sends, and keep it consistent:

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}

Consistency matters more than variety. Rotating random user agents on every request while keeping the same cookies looks less human, not more. Pick a plausible browser profile per session, and make Accept-Language match the proxy's country: a German IP with en-US only isn't impossible, but a French IP claiming Japanese is unusual.

3. Use Sessions and Cookies Like a Browser

Real visitors accept cookies and keep them. Many sites set a cookie on the first visit and treat requests without it as suspicious. Use a session object that persists cookies, and pair each cookie jar with a sticky IP so the session's IP doesn't jump around mid-visit:

import uuid, requests

sid = uuid.uuid4().hex[:8]
proxy = f"http://USERNAME-country-us-session-{sid}-time-15:[email protected]:8080"

s = requests.Session()
s.headers.update(HEADERS)
s.proxies = {"http": proxy, "https": proxy}

s.get("https://example.com/", timeout=30)              # picks up cookies
r = s.get("https://example.com/category/1", timeout=30)  # sends them back

For independent pages with no state, rotating IPs spread load and are fine. For anything multi-step, use sticky. Our rotating vs sticky guide goes deeper.

4. Choose the Right Proxy Type

Many sites check whether an IP belongs to a hosting provider. Datacenter IPs are fast and cheap, and work well on sites that don't filter them. Residential IPs come from consumer ISPs and are treated like normal visitors. Mobile IPs are shared by many real users on carrier networks, so sites are cautious about blocking them.

  • Start with datacenter for tolerant targets and APIs. It's the cheapest way to get volume.
  • Switch to residential when you see blocks that follow the IP type rather than your behavior.
  • Use mobile for mobile-first platforms where residential still struggles.

Our datacenter vs residential comparison covers the tradeoffs, and residential proxies for scraping shows typical setups. Remember: a better IP doesn't fix bad behavior. Residential IPs sending 50 requests per second still get blocked.

5. Retry With Exponential Backoff

When a site answers 429 or 503, it's asking you to slow down. Back off exponentially, honor Retry-After if present, and give up after a few attempts:

import random, time, requests

def fetch(session, url, max_tries=4):
    for attempt in range(max_tries):
        try:
            r = session.get(url, timeout=30)
        except requests.RequestException:
            r = None
        if r is not None and r.status_code not in (429, 500, 502, 503, 504):
            return r
        retry_after = r.headers.get("Retry-After") if r is not None else None
        if retry_after and retry_after.isdigit():
            wait = int(retry_after)
        else:
            wait = (2 ** attempt) + random.uniform(0, 1)
        time.sleep(wait)
    return r

Don't retry 403s in a tight loop. A 403 usually means the site has decided you're a bot, and hammering it makes things worse. Stop, look at what changed, and adjust. If the proxy itself returns 407 or 402, that's a credentials or bandwidth issue, not a block; see how to fix proxy error 407.

6. Watch Your Numbers

Log the status code, response size and exit IP for every request. Watch for success rate falling, pages that return 200 but contain a block message, and response sizes that suddenly shrink. These are early warnings that it's time to slow down. When blocks do happen, why proxies get blocked walks through the diagnosis.

Putting It Together With NodeProxies

NodeProxies gives you datacenter, residential and mobile IPs behind the same gateway, so you can test which type a target needs by changing credentials, not code. Pay-as-you-go pricing starts at $0.42/GB for datacenter and $0.65/GB for residential, with no contract. Check pricing or sign up to start with a small plan.

FAQ

Why do websites block web scrapers?

To protect server load, stop abusive bots and control access to their data. Scrapers that send too many requests, use default library headers or come from flagged IP ranges are the easiest to spot.

Do proxies stop you from getting blocked?

They spread requests across IPs and let you use IP types sites trust more, but they don't fix aggressive request rates or inconsistent headers. Behavior matters as much as the IP.

How many requests per second is safe when scraping?

There's no universal number. Start around one request per second per site, honor any crawl-delay in robots.txt, and only increase while your success rate stays high.

Is it legal to scrape a website using proxies?

Scraping public data is often lawful, but it depends on the jurisdiction, the site's terms and the data involved. Personal data and content behind logins carry more risk; get advice for your specific case.

NodeProxies

Ready to get started?

90M+ residential, mobile & datacenter proxies. Datacenter from $0.30/GB, residential from $0.42/GB. No contracts.

Create free account