Our crawler.

What it is for

Competitor Intelligence shows car dealerships the new vehicles and advertised prices at the dealerships around them, so nobody has to check rival websites by hand. Our crawler gathers that public information from dealership websites. Today it reads new 2026 Chevrolet, GMC, Buick and Ford vehicles at dealerships in Western Canada.

How it identifies itself

Every request carries the user agent AddedHours-CompetitorIntel/2.0. It never presents itself as a web browser or a person.

Some dealership sites show their inventory only after JavaScript runs. Those pages are loaded through Firecrawl, a page-rendering service, which sends the same user agent on our behalf. Our requests come from Vercel’s and Firecrawl’s cloud infrastructure, so our IP addresses are not fixed.

What it reads

It reads robots.txt, sitemaps, inventory listing pages and vehicle detail pages. From those pages it keeps the facts about each vehicle for sale: year, make, model, trim, VIN or stock number where shown, and the advertised price.

It never logs in, never submits a form and does not collect personal information.

How it behaves

• It reads a site’s robots.txt before any other page and reads it again at least every hour. It never fetches a path robots.txt disallows.
• It honours Crawl-delay up to 10 seconds and Retry-After up to an hour.
• It sends at most two requests at a time to a site, at least half a second apart.
• It visits a dealership about once a week, and at most once a day when a customer follows that dealership closely.
• It slows down when a site asks it to, and backs off from a site whose pages keep failing.
• It never tries to solve or get around a CAPTCHA, a bot check or a block. When a site refuses it, it stops.

How to limit or block it

To keep our crawler off your whole site, add these two lines to your robots.txt:

User-agent: AddedHours-CompetitorIntel
Disallow: /

To keep it out of part of your site, or to space out its visits, use the same group with your own rules:

User-agent: AddedHours-CompetitorIntel
Disallow: /service/
Crawl-delay: 5

The name is matched without regard to case, version number or punctuation, so AddedHours-CompetitorIntel/2.0 and addedhourscompetitorintel work too. A change takes effect within an hour.

If you would rather we stopped reading your dealership altogether, or need it handled another way, write to system@addedhours.ai and we will take care of it.

Contact

If our crawler causes you any trouble, or you would like to let it through a firewall or bot check, write to system@addedhours.ai. We are a small team in Calgary, and a person reads every message.