A Scrapy proxy setup is simpler than most tutorials make it look. Scrapy's built-in HttpProxyMiddleware already reads a proxy URL from each request, extracts the username and password, and sends the Proxy-Authorization header for you. You don't need a third-party rotation package for a gateway-style proxy. This guide shows the one-line version, a small downloader middleware for rotating and sticky sessions, and the retry and throttling settings that keep your crawl healthy.
The Quick Version: meta['proxy']
Set the proxy on each request through meta. Credentials go in the URL:
import scrapy
PROXY = "http://USERNAME-country-us:[email protected]:8080"
class IpSpider(scrapy.Spider):
name = "ip"
def start_requests(self):
for i in range(3):
yield scrapy.Request(
"https://api.ipify.org?format=json",
meta={"proxy": PROXY},
dont_filter=True,
cb_kwargs={"i": i},
)
def parse(self, response, i):
self.logger.info("request %s exit ip %s", i, response.json()["ip"])HttpProxyMiddleware is enabled by default. It strips the credentials from the URL, base64-encodes them into a Proxy-Authorization header and uses CONNECT for HTTPS targets. With no session option in the username, the gateway assigns a new IP per request. If your password has special characters, URL-encode it with urllib.parse.quote(password, safe=""), or you'll hit a 407 error.
A Downloader Middleware for Rotation and Sticky Sessions
Setting meta on every request gets repetitive, and most crawls need both modes: rotating IPs for independent pages and a sticky IP for flows that carry state, like paginated sessions behind a login to your own account. A small middleware handles both. Put this in middlewares.py:
from urllib.parse import quote
GATEWAY = "global.nodeproxies.xyz:8080"
class NodeProxiesMiddleware:
def __init__(self, user, password, country, session_minutes):
self.user = user
self.password = quote(password, safe="")
self.country = country
self.session_minutes = session_minutes
@classmethod
def from_crawler(cls, crawler):
s = crawler.settings
return cls(
s.get("NP_USER"),
s.get("NP_PASS"),
s.get("NP_COUNTRY"),
s.getint("NP_SESSION_MINUTES", 10),
)
def process_request(self, request, spider):
if "proxy" in request.meta:
return None # respect a proxy set explicitly on the request
username = self.user
if self.country:
username += f"-country-{self.country}"
session_id = request.meta.get("proxy_session")
if session_id:
username += f"-session-{session_id}-time-{self.session_minutes}"
request.meta["proxy"] = f"http://{username}:{self.password}@{GATEWAY}"
return NoneEnable it in settings.py with a priority lower than 750, so it runs before the built-in HttpProxyMiddleware, which then handles the authentication:
import os
NP_USER = os.environ["NP_USER"]
NP_PASS = os.environ["NP_PASS"]
NP_COUNTRY = "us" # or None for any country
NP_SESSION_MINUTES = 10
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.NodeProxiesMiddleware": 350,
}Now every request rotates by default. When a sequence of requests must share an IP, give them the same proxy_session value:
def parse_listing(self, response):
sid = response.meta.get("proxy_session") or "crawl42"
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(
next_page,
callback=self.parse_listing,
meta={"proxy_session": sid},
)Same session ID means the same exit IP until the time runs out; a new ID gives a new IP. For a deeper explanation of when to use each mode, see rotating vs sticky proxies.
Retry Settings That Fit Proxies
Scrapy's RetryMiddleware retries failed requests. With a rotating proxy, a retry naturally goes out from a different IP, which is often all it takes to get past a one-off failure. Tune it in settings.py:
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
DOWNLOAD_TIMEOUT = 30Don't add 407 or 402 to the retry list. Those come from the proxy gateway and mean bad credentials, an inactive plan, or no bandwidth left; retrying just burns time. Be careful with 403 too: if a site is blocking you, retrying harder makes it worse. Fix the cause instead, covered in why proxies get blocked.
AutoThrottle and Concurrency
Proxies don't make it OK to hammer a site. AutoThrottle adjusts the delay between requests based on how fast the server responds, which keeps your crawl polite and also reduces blocks:
ROBOTSTXT_OBEY = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8AUTOTHROTTLE_TARGET_CONCURRENCY is the average number of parallel requests Scrapy aims for per remote site; start low and raise it only while your success rate holds. CONCURRENT_REQUESTS_PER_DOMAIN is a hard cap. Keep ROBOTSTXT_OBEY on and check the site's terms; our guide to proxy legality covers the basics.
Verifying It Works
Run the IP spider above with the middleware enabled and no meta proxy. Three requests should log three different IPs. Keep dont_filter=True for this test, since Scrapy's duplicate filter would otherwise drop the repeated URL, and watch the log for HttpProxyMiddleware errors, which usually point to a malformed proxy URL. Then add the same proxy_session to all three and confirm they log one IP. If you get 407s, check the credentials with curl first, as described in how to test if a proxy is working.
Running Scrapy on NodeProxies
Scrapy pairs well with pay-per-GB pricing because you only pay for the pages you download. Start crawls on datacenter proxies from $0.42/GB and switch the same middleware to residential from $0.65/GB for sites that block datacenter ranges; only the credentials change. For very large crawls, unlimited residential is billed by the hour instead of per GB. See pricing for volume tiers.
FAQ
How do I add a proxy with username and password in Scrapy?
Set request.meta["proxy"] to http://user:pass@host:port. The built-in HttpProxyMiddleware turns the credentials into a Proxy-Authorization header automatically.
Do I need scrapy-rotating-proxies with a rotating gateway?
No. Those packages rotate through a list of individual proxies. A gateway already assigns a new IP per request, so a single proxy URL is enough.
Why does my Scrapy middleware proxy get ignored?
Usually the priority is wrong. Your middleware must run before HttpProxyMiddleware (750), so give it a lower number such as 350.
Does Scrapy support SOCKS5 proxies?
Not out of the box. Use the HTTP endpoint on port 8080, which works with Scrapy's built-in middleware.
