Getting blocked while scraping is frustrating - but entirely avoidable. In this guide, we'll walk through exactly how to set up and use rotating residential proxies for web scraping, from authentication to advanced rotation strategies.
Why You Need Proxies for Web Scraping
Without proxies, every request you make comes from the same IP address. Even modest scraping volumes will trigger rate limiting or IP bans on most modern websites. Rotating residential proxies solve this by distributing your requests across thousands of different IPs.
Step 1: Get Your Proxy Credentials
Create a NodeProxies account, add balance, and select a residential or datacenter plan. From your dashboard, you'll get:
- Proxy host: eu-1.nodeproxies.xyz (or eu-2, na, global)
- Port: 8080
- Username: your generated username
- Password: your generated password
Step 2: Configure Rotation
NodeProxies controls rotation through your username. To rotate on every request, use your username as-is. To use a sticky session (same IP for multiple requests), append a session ID:
- Rotating: username:password
- Sticky session: username-session-abc123:password
- Country targeting: username-country-us:password
- City targeting: username-country-us-city-newyork:password
Step 3: Python Example (Requests + Scrapy)
Here's a basic Python setup with the requests library:
- import requests
- proxies = {'http': 'http://user:[email protected]:8080', 'https': 'http://user:[email protected]:8080'}
- r = requests.get('https://target.com', proxies=proxies)
Step 4: Handle Retries and Errors
Even with residential proxies, some requests will fail. Implement a retry loop with exponential backoff, and rotate to a new session on 403 or 429 responses. Libraries like tenacity (Python) or axios-retry (Node.js) make this easy.
Advanced Tips
- Match your User-Agent to a real browser for maximum stealth
- Add realistic delays between requests (0.5-3 seconds)
- Use headless browsers (Playwright/Puppeteer) for JavaScript-heavy sites
- Combine city targeting with sticky sessions for localized scraping
- Monitor your bandwidth usage in the NodeProxies dashboard to optimize costs