Sellers, brands and analysts scrape Amazon for product titles, prices, ratings, Buy Box status and stock levels. Amazon serves enormous traffic and protects itself accordingly: request too fast from one IP and you get a CAPTCHA page or an error instead of a product. The right proxies for Amazon scraping, combined with sensible request habits, are what separate a scraper that works for a day from one that works every day.
Before building anything, check whether an official route covers your needs. Sellers have access to Amazon's own seller and advertising APIs, and those are more stable than scraping. If you do scrape public product pages, keep rates reasonable, avoid anything behind a login and read are proxies legal for the general rules.
Why Residential Proxies Work Best for Amazon
Amazon sees a lot of automated traffic from cloud and datacenter ranges, so requests from those ranges are much more likely to be challenged. Residential IPs come from home ISPs, the same networks real shoppers use, so they are treated like ordinary visitors as long as each IP behaves like one.
- Residential proxies: the default for product and search pages. From $0.65/GB, falling to $0.35/GB at 1,000 GB and above.
- Datacenter proxies: cheaper, but expect higher block rates on Amazon; test them before committing.
- Unlimited residential: billed at $3.90 per hour with no GB cap, worth comparing if you run large crawls continuously.
Rotation: A New IP per Request
For product page crawling, rotate on every request. With NodeProxies that is the default when you leave the session out of the username. Each request goes out from a different residential IP, so no single address requests hundreds of pages.
Target the marketplace's country. Amazon.com prices and availability depend on the delivery location, so use -country-us for amazon.com, -country-gb for amazon.co.uk, -country-de for amazon.de and so on. Use a sticky session (for example -session-zip1-time-10) only when you need several requests from the same visitor, such as setting a delivery postcode and then reading prices.
Headers That Look Like a Real Browser
The default Python requests user agent is an instant giveaway. At minimum, send a current desktop browser User-Agent, an Accept header and an Accept-Language that matches the marketplace (en-US for .com, de-DE for .de). Keep the set consistent with the browser you claim to be, and rotate between a few realistic user agents rather than random strings.
Python Example: Amazon Scraping With Retries and Backoff
This example rotates IPs through the NodeProxies gateway, detects CAPTCHA pages, backs off exponentially and stops immediately on proxy account errors instead of retrying them forever:
import random
import time
import requests
PROXY = 'http://USERNAME-country-us:[email protected]:8080'
PROXIES = {'http': PROXY, 'https': PROXY}
USER_AGENTS = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
]
def headers():
return {
'User-Agent': random.choice(USER_AGENTS),
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9',
}
def fetch(url, max_tries=5):
for attempt in range(max_tries):
try:
r = requests.get(url, headers=headers(), proxies=PROXIES, timeout=30)
if r.status_code == 200 and 'captcha' not in r.text.lower():
return r.text
print(f'attempt {attempt + 1}: status {r.status_code} or captcha page')
except requests.exceptions.ProxyError as e:
if '407' in str(e) or '402' in str(e):
raise # credentials wrong, plan inactive or out of bandwidth
print(f'attempt {attempt + 1}: proxy error {e}')
except requests.RequestException as e:
print(f'attempt {attempt + 1}: {e}')
time.sleep(2 ** attempt + random.random())
return None
html = fetch('https://www.amazon.com/dp/ASIN_HERE')
print('ok' if html else 'gave up')Because there is no session in the username, every retry automatically uses a new IP. The CAPTCHA check is a simple heuristic; tighten it once you see what the challenge pages look like for your marketplace. A 407 means the username or password is wrong or the plan is not active, and a 402 means the plan is out of bandwidth - our guide to fixing proxy error 407 covers both.
Parsing and Keeping the Scraper Stable
- Parse with an HTML parser such as BeautifulSoup or lxml and keep selectors in one place; Amazon changes layouts and runs A/B tests, so expect to update them.
- Validate each record: a page without a title or price is a failed fetch, even if it returned 200.
- Limit concurrency. A few parallel workers with jitter between requests usually beats a hundred threads that trigger challenges.
- Cache pages you have already fetched today, and only re-crawl products whose data you actually need refreshed.
- If you need JavaScript-rendered content, use a headless browser but block images and media to save bandwidth.
For a broader look at why scrapers get blocked and how to fix it, read avoiding blocks when web scraping.
Estimating Bandwidth
Amazon product pages are heavy. Measure len(r.content) on a sample of your targets, multiply by your monthly page count and add a margin for retries. That figure, divided into GB, is what you buy. Since NodeProxies is pay as you go with no expiry on balance, you can start with a small test plan and scale once you know the real number.
Getting Started With NodeProxies
- Create an account with no ID verification and top up from $3 by card or crypto.
- Buy a residential plan and copy your username and password from the dashboard.
- Set the country for your marketplace and run the script above against a handful of product pages.
- Scale up and pick a volume tier on the pricing page.
FAQ
What are the best proxies for Amazon scraping?
Rotating residential proxies targeted to the marketplace's country are the most reliable choice. Datacenter IPs are cheaper but are challenged much more often on Amazon.
Why does Amazon show me a CAPTCHA when scraping?
Usually because too many requests come from one IP, the headers look automated, or the IP belongs to a datacenter range. Rotate IPs, send realistic headers and slow down.
How many requests per IP can I send to Amazon?
There is no published number and it varies. The safe pattern is one request per IP with rotation, plus delays and backoff when you see challenge pages.
Is scraping Amazon legal?
Collecting public product information is common, but Amazon's terms restrict automated access and laws vary by country. Use official APIs where they fit, avoid logged-in areas and personal data, and get legal advice for commercial projects.
