About our crawler

Last updated November 15, 2026

CareersFresh lists internships and co-ops so students can find them soon after they open. To do that, a small crawler reads the public job listings that employers publish on their own career sites and job-board APIs, and every listing links straight back to the employer’s posting.

How it identifies itself

Requests carry this User-Agent (InternRadar is the project’s code name):

InternRadar/0.1 (+https://careersfresh.com/crawler)

For the rare site that only serves ordinary browsers, we load public pages in a normal Chrome and never alter what it reports about itself. Those requests add an X-Crawler-Contact header pointing here, so you can still tell who is visiting.

What it does and doesn’t do

• It reads only public listings: titles, locations, dates, apply links and the description, which we use to recognise internships. We don’t republish descriptions; we link to your page instead.
• It never logs in, never solves captchas, and never works around bot protection or other access controls. A site that refuses us is left alone.
• It follows robots.txt: we check it before adding a site, and pages loaded in a browser are checked against it on every visit.
• It is gentle. Most boards are read every few minutes, using conditional requests where the site supports them; large boards are read in full only a few times a day. It slows down when told to (HTTP 429 or Retry-After) and backs off after a refusal.

Contact us or opt out

If our crawler causes you any trouble, or you’d rather your roles weren’t listed, email hello@careersfresh.com with your careers site’s address. We’ll stop checking it and take your listings off CareersFresh.

You can also disallow us in robots.txt:

User-agent: InternRadar
Disallow: /

Most of our requests go to job-board APIs we review when adding your company, so please also email us when you do.

Students: see how fresh each employer’s listings are on the source status page.