Labour market analysts, recruiting startups and HR teams use proxies for job listings data to follow hiring across many companies: who is hiring, for which roles, in which countries, at what salary ranges where published. The most reliable source is usually not a giant job board but the employers' own career pages, which are hosted on a handful of common applicant tracking systems and often expose clean JSON behind the page.
Career Pages Beat Job Boards
- They are the original source, so postings are fresher and removed when filled.
- They are spread across many hosts, so your load on any single site is small.
- Many publish structured job data for search engines, which is easy to parse.
- Large job boards usually have strict terms against scraping and strong bot protection; career pages are meant to be found.
Find the JSON Behind the Page
Open a company's careers page, open your browser's developer tools on the Network tab, filter by Fetch/XHR and reload. Career widgets usually load the job list from a public JSON endpoint on the applicant tracking system's domain. That endpoint is what you collect - one small request per company instead of rendering the page. Many career pages also embed schema.org JobPosting markup (a script tag of type application/ld+json) on each job page, which gives you title, location, employment type, date posted and sometimes salary in a standard format.
import json, requests
from bs4 import BeautifulSoup
PROXY = 'http://USERNAME-country-us:[email protected]:8080'
def job_postings(url):
r = requests.get(url, proxies={'http': PROXY, 'https': PROXY},
headers={'User-Agent': 'JobsResearch/1.0 (contact: [email protected])'},
timeout=30)
soup = BeautifulSoup(r.text, 'html.parser')
found = []
for tag in soup.find_all('script', type='application/ld+json'):
try:
data = json.loads(tag.string or '')
except json.JSONDecodeError:
continue
items = data if isinstance(data, list) else [data]
found += [d for d in items if isinstance(d, dict) and d.get('@type') == 'JobPosting']
return found
for job in job_postings('https://careers.example.com/jobs/12345'):
print(job.get('title'), '|', job.get('datePosted'), '|', job.get('employmentType'))Deduplicate Before You Count
The same job appears on the career page, two job boards and a recruiter's site, often with slightly different titles. If you are measuring hiring volume, duplicates inflate every number. A practical key is a hash of normalised company name, normalised title and location:
import hashlib, re
ALIASES = {'acme inc': 'acme', 'acme gmbh': 'acme'}
def norm(s):
return re.sub('[^a-z0-9]+', ' ', (s or '').lower()).strip()
def job_key(company, title, location):
c = norm(company)
c = ALIASES.get(c, c)
raw = '|'.join([c, norm(title), norm(location)])
return hashlib.sha1(raw.encode()).hexdigest()
seen = set()
for job in [('Acme Inc.', 'Senior Data Engineer', 'Berlin'),
('ACME', 'Senior Data Engineer ', 'Berlin')]:
k = job_key(*job)
print('dup' if k in seen else 'new', job)
seen.add(k)Company name normalisation (Acme Inc., ACME, Acme GmbH) is the hardest part; keep a small alias table for the employers you track most.
Track Freshness, Not Just Counts
Record first seen and last seen dates for every job key. A posting that disappears was probably filled or cancelled; time-to-fill by role is one of the most useful signals in hiring data. Revisit each company daily or every few days - more often adds load without adding information.
Why Proxies, and Which Country
Some career sites show different jobs or redirect to a country-specific page based on the visitor's IP, for example a global company that shows only UK roles to UK visitors. To see every market's listings, request through each country: -country-gb, -country-de, -country-us. Proxies also spread a crawl of thousands of career pages so your own server's IP is not the one making all the requests, and they keep crawling going when a particular host throttles one IP. For general patterns, see how to use proxies for web scraping.
Proxy type
Career page JSON endpoints are usually unprotected and small, so datacenter proxies from $0.42/GB handle most of the crawl. Use residential proxies for the minority of sites that block server IPs. Our datacenter vs residential comparison explains the split.
Be a Good Crawler
Respect robots.txt, identify your crawler with a contact address, keep one or two requests per host at a time, and do not collect applicants' or recruiters' personal contact details. Job postings are published to be read, but terms still apply - see are proxies legal. If a large board offers a feed or partnership, use it.
Setting It Up with NodeProxies
Register, deposit from $3 by card or crypto, and start with a datacenter plan for the bulk of the crawl plus a small residential plan for protected hosts. Change the country code per market. There is no subscription and no KYC, and balance never expires. See pricing for tiers.
FAQ
Where can I get job listings data?
Company career pages, which often load jobs from public JSON endpoints or embed JobPosting structured data. Large boards may offer feeds or partner access.
Why do I see different jobs from different countries?
Some career sites filter or redirect by the visitor's country. Use a proxy in each market to see its listings.
Do I need residential proxies to collect job postings?
Mostly not. Datacenter proxies handle career page endpoints well; keep residential for sites that block server IPs.
