Skip to main content
Our crawler

ComparePVBot

Reads public product pages on solar shops so ComparePV can show what hardware costs. One request at a time, identified in every header, robots.txt obeyed.

User-Agent ComparePVBot/1.0 (+https://comparepv.com/bot; respects robots.txt; opt-out there)
Rate
1 request, 2s+ apart
Reads
Name, price, stock, image
Block takes effect
within 24h
Contact
[email protected]
01

Block it in robots.txt

Paste one of these at your-shop.example/robots.txt. Our fetcher reads it before every page. Nothing to confirm here.

Stop us 
User-agent: ComparePVBot
Disallow: /
Nothing on the host gets fetched again.
Keep us out of a section 
User-agent: ComparePVBot
Disallow: /clearance/
Disallow: /b2b/
One Disallow per path prefix.
Slow us down 
User-agent: ComparePVBot
Crawl-delay: 30
Honoured whenever it asks for more than our 2s default.
Three ways a block silently fails
A comment is not a rule

# ComparePVBot: Deny reads as a note to a human and as nothing at all to a crawler. Only a User-agent group with a Disallow line under it blocks us.

Groups do not stack

If you have a User-agent: * group, we read our own group instead of it, not on top of it. Repeat any line that should still apply to us.

One file per host and scheme

A file on the apex domain does not cover shop.your-shop.example. Each host needs its own reachable /robots.txt.

02

What it reads

Takes
  • Product name
  • Price and currency
  • Availability
  • Product image
  • Sitemaps and category listings that lead to products
Never touches
  • Forms and logins
  • Baskets and checkout
  • Account pages
  • Anything behind a paywall or a price gate
  • Data about your visitors

No personal data, so nothing for your consent banner. A Crawl-delay widens our gap between requests; a 429 or 503 with Retry-After pauses us for as long as it asks, up to a minute.

03

Or just email us

Can't edit robots.txt

Send us the domain and we will stop crawling it. Write from an address on that domain if you can, so we can tell an owner from a competitor.

Email opt-out
Delete what we already hold

A block stops new prices; it does not remove published ones. Ask in the same email and your domain's offers go with it. Specifications stay, because those come from manufacturer datasheets, not from your shop.

See what we publish

Wanted the opposite? A shop that publishes panel prices on its own site can ask to be read rather than blocked. We add them by hand, so leave an address we can answer.

04

Questions we get

How fast does a robots.txt block take effect?

Within a day. The crawler re-reads robots.txt for each host at most once every 24 hours, and the answer it gets is the one it obeys until the next read. There is nothing to confirm on our side.

Does a comment like "# ComparePVBot: Deny" work?

No. Everything after a # is a comment and is ignored by every crawler that reads the file, including ours. A block has to be a User-agent group with a Disallow line under it.

I do not want to block you, just slow you down.

Add a Crawl-delay line to the ComparePVBot group. We honour it whenever it asks for more than our own default of two seconds between requests, and we honour Retry-After on a 429 or 503 up to one minute.

Do you collect personal data from my shop?

No. The crawler reads public product pages: product name, price, currency, availability and the product image. It does not submit forms, does not log in, does not touch checkout or account pages, and stores nothing about your visitors.

Will blocking you remove my prices from ComparePV?

New prices stop arriving from the next crawl. To have the offers we already hold deleted as well, email [email protected] from an address on the domain and say so.

How do I verify a request really came from ComparePVBot?

Our requests carry the exact User-agent string shown on this page and a From header with our contact address. Anyone can copy a User-agent, so treat an unusual request pattern as unrelated traffic wearing our name and tell us about it.