Researchers in computer science, communication, economics and law use proxies for academic research when the thing they study looks different depending on where you are. Do search results differ by country? Are prices personalised by location? Which news sources does a platform show in one market but not another? Answering these requires observing the web from many vantage points, and a residential proxy network provides them without travelling or recruiting volunteers in every country.
Typical Research Designs That Need Proxies
- Algorithm audits: comparing search, recommendation or ad results across countries with otherwise identical queries.
- Price discrimination studies: checking whether the same product is priced differently by visitor country.
- Censorship and availability measurement: recording which pages load, are blocked or are altered in different countries.
- Localization studies: how global brands and platforms change messaging by market.
Ethics Review and Terms of Service
Talk to your institution's ethics board (IRB or equivalent) early, even if you only collect public pages. Points they usually ask about:
- Whether you collect personal data, and how you minimise, secure and delete it. Data protection law such as GDPR applies to public personal data too.
- Whether the collection could harm the sites studied (load) or the people whose content is collected.
- How you handle the platforms' terms of service, which often restrict automated access. Courts and institutions treat this differently by country; get advice from your university's legal office.
- Whether you use sock-puppet accounts, which need a stronger justification than logged-out observation.
Keep request rates low, honour robots.txt unless your ethics approval addresses it explicitly, and identify your crawler with a contact address when the design allows. Our overview of whether proxies are legal is a starting point, not legal advice.
Controlling Your Variables
In an audit, the only thing that should differ between observations is the variable you are testing - usually country. Hold everything else fixed: a fresh browser context with no cookies for every observation, the same browser version and window size, the same time window, and a language setting chosen deliberately (either matched to each country, or fixed across all, depending on your design). Write the choice down; reviewers will ask.
Log the Vantage Point with Every Observation
Reproducibility depends on recording where each observation came from. Record the exit IP, the requested country, the session ID and the timestamp alongside the data, in a format you can publish with your replication package (with IPs truncated if your ethics plan requires it):
import json, uuid, datetime, requests
LOG = open('observations.jsonl', 'a', encoding='utf-8')
def observe(url, country, condition):
sid = uuid.uuid4().hex[:12]
proxy = f'http://USERNAME-country-{country}-session-{sid}-time-5:[email protected]:8080'
px = {'http': proxy, 'https': proxy}
ip = requests.get('https://api.ipify.org', proxies=px, timeout=20).text
r = requests.get(url, proxies=px, timeout=30,
headers={'User-Agent': 'UniStudy/1.0 (contact: [email protected])'})
record = {
'utc': datetime.datetime.now(datetime.timezone.utc).isoformat(),
'condition': condition, 'country': country, 'session': sid,
'exit_ip': ip, 'url': url, 'status': r.status_code,
'bytes': len(r.content), 'body': r.text,
}
LOG.write(json.dumps(record) + '\n')
LOG.flush()
return record
for cc in ['us', 'gb', 'de', 'fr']:
rec = observe('https://example.org/', cc, 'baseline')
print(cc, rec['exit_ip'], rec['status'])A new session ID per observation gives each one its own IP, held for the IP check and the page fetch together. A JSON Lines file is easy to append to, robust to crashes and simple to load in pandas or R. For browser-based measurements, the same idea works with Playwright; rotating vs sticky proxies in Python and Playwright shows the session mechanics.
Validate Your Vantage Points
Before the main collection, run a pilot: for each country, make a few dozen requests and confirm with an independent IP geolocation lookup that exits are in the intended country. Report the validation in your methods section. Also run a few repeated observations from the same country to estimate the natural noise in your measure, so you can tell real cross-country differences from random variation. If something looks off, how to test if a proxy is working lists quick checks.
Budgeting a Grant
Research budgets are fixed in advance and spent over months or years, which suits pay-as-you-go pricing. Estimate bytes per observation from the pilot, multiply by observations per country, countries and repeats, then add a margin for re-runs. Residential proxies start at $0.65/GB and drop with volume. A NodeProxies balance never expires, so funds deposited at the start of a grant period are still there for the second wave of collection. Institutions that need invoices or a specific payment method should check with their finance office; deposits are by card or crypto.
Getting Started with NodeProxies
Register, deposit from $3, and buy a small plan for the pilot - 100 MB is enough to validate vantage points. There is no subscription to cancel when the project ends. See pricing for volume tiers and the API documentation if you manage usage programmatically.
FAQ
Do I need ethics approval to scrape public websites for research?
Many institutions require review whenever personal data may be collected, even from public pages. Ask your ethics board before collecting.
How do I make proxy-based measurements reproducible?
Log the country, exit IP, session ID, timestamp and browser settings with every observation, and publish your collection code.
Which proxy type is best for algorithm audits?
Residential proxies, because platforms treat them as ordinary users, which is what most audits aim to measure.
