Competitive price intelligence sounds simple: visit competitor product pages, record the price, repeat. In practice, the price you see depends on where you appear to be, retailers rate-limit repeated visits, and a crawl that worked last week starts returning error pages. Proxies for price monitoring fix the first two problems and make the third manageable. This guide covers geo-specific pricing, rotation, scheduling and a realistic way to estimate cost.
Why Price Monitoring Needs Proxies
- Prices, currencies, taxes, shipping and stock are often localised by country, and sometimes by region or city. A crawler running from one server sees one version.
- Retailers rate-limit or block IPs that request thousands of product pages. Spreading requests over many IPs keeps the load per IP low.
- Datacenter IP ranges are often served different content or challenged on protected stores, so the data you collect may not be what real shoppers see.
Before you start, check each site's robots.txt and terms, and keep request rates modest. Collecting publicly displayed prices is common practice, but how you do it matters; see are proxies legal for the broader picture.
Collecting Geo-Specific Prices With Country and City Targeting
To see what a shopper in Germany sees, your request has to come from a German IP. With NodeProxies the target goes in the username, so one script can loop over markets:
import requests
USER, PASS = 'USERNAME', 'PASSWORD'
def proxies_for(country):
url = f'http://{USER}-country-{country}:{PASS}@global.nodeproxies.xyz:8080'
return {'http': url, 'https': url}
PRODUCT = 'https://shop.example.com/product/123'
for country in ['us', 'gb', 'de', 'fr']:
r = requests.get(PRODUCT, proxies=proxies_for(country), timeout=30)
print(country, r.status_code, len(r.content), 'bytes')For retailers that price by region (grocery chains, delivery services, local electronics stores), residential plans also support state and city targeting, for example USERNAME-country-us-state-NY1-city-NYC1. Remember that many sites also choose currency and region from cookies or a store selector, so set those explicitly where possible and do not rely on the IP alone.
Rotating vs Sticky Sessions for Price Checks
For most catalogue crawls, rotate: with no session in the username, every request goes out on a new IP. That spreads load and keeps any single IP from looking like a bot.
Use a short sticky session when a price only appears after several steps on the same visitor, such as choosing a store location, adding to cart or entering a postcode. Something like USERNAME-country-gb-session-basket42-time-10 keeps one IP for ten minutes, long enough to complete the flow. Our post on rotating vs sticky proxies shows both patterns in code.
Scheduling: How Often to Check Prices
Crawling everything every hour is the fastest way to burn bandwidth and get blocked. Match frequency to how often prices actually move:
- Fast-moving categories (marketplaces, electronics, travel): every few hours for your key SKUs.
- Typical retail catalogues: once or twice a day.
- Long-tail products: daily or weekly, or only when a cheaper signal (a sitemap lastmod date, a category page) shows a change.
- Spread jobs across the day with random jitter instead of firing every request at the top of the hour.
Fetch the lightest page that contains the price. A category listing with 48 products in one request is far cheaper than 48 product pages, and some sites embed prices in structured data you can parse without loading the whole page.
Estimating Price Monitoring Proxy Costs per 10,000 Pages
Proxy bandwidth is billed per GB, so cost is roughly bytes per page times pages. The numbers below are illustrative estimates, not measurements of any particular site; measure your own targets with the len(r.content) line in the script above, and remember that compression and TLS overhead mean the billed figure will differ somewhat from what your code sees.
- HTML only, about 150 KB per page: 10,000 pages is about 1.5 GB, roughly $0.98 at the $0.65/GB residential rate.
- HTML only, about 500 KB per page (heavy modern store pages): about 5 GB, roughly $3.25 at $0.65/GB.
- Full headless browser load with images and scripts, about 2 MB per page: about 20 GB, roughly $12 at the $0.60/GB rate for plans of 10-50 GB.
The biggest saving is not the per-GB rate but the bytes. Fetching raw HTML instead of rendering, and blocking images, fonts and video in headless browsers, often cuts usage several times over. Larger plans lower the rate further: residential drops to $0.50/GB at 200-500 GB and $0.35/GB above 1,000 GB. If your crawl runs around the clock, unlimited residential proxies at $3.90 per hour can work out cheaper than paying per GB.
Keeping Your Price Data Clean
- Validate every response: check for the price element, not just HTTP 200. Block pages and CAPTCHAs often return 200.
- Retry failures with exponential backoff on a new IP, and cap retries so one broken page does not eat your budget.
- Store the country, timestamp and raw price string alongside the parsed number, so you can audit odd values later.
- Alert when a site's success rate drops; it usually means the page layout changed. Our guide to avoiding blocks when web scraping covers the most common causes.
Running Price Monitoring on NodeProxies
- Sign up and add balance from $3; there is no subscription and the balance does not expire.
- Buy a residential proxy plan sized to your estimate, starting from 100 MB for a test run.
- Plug the gateway into your crawler with a country per market, and add city targeting where prices are local.
- Check your real GB per 10,000 pages after the first run, then pick the right tier on the pricing page.
FAQ
What type of proxy is best for price monitoring?
Rotating residential proxies are the safest default because they show the same prices real shoppers see and are less likely to be blocked. Datacenter proxies can work for sites with little bot protection and are cheaper per GB.
How do I see prices for a different country?
Route the request through an IP in that country, for example by adding -country-de to your NodeProxies username, and set the site's own region or currency selector where it has one.
How much bandwidth does price scraping use?
It depends almost entirely on page weight. Raw HTML pages are often tens to hundreds of kilobytes, while full browser loads can be several megabytes. Measure a sample of your targets and multiply by your page count.
Is competitor price monitoring legal?
Collecting publicly displayed prices is widely practised, but laws and site terms vary by country. Respect robots.txt and rate limits, avoid personal data and get legal advice for your specific case.
