lowerSpendingsBot
lowerSpendingsBot collects public product prices for lowerSpendings, a price comparison site for
Australian shoppers. If you saw lowerSpendingsBot/1.0 in your logs, this page explains
what it does.
What it fetches
- Public product listings: name, price, availability, product identifiers (such as GTINs and
manufacturer part numbers), the main image link and the page link.
- Where a store has a catalogue endpoint or sitemap, it uses that instead of crawling pages. It
never logs in, adds to carts or submits forms.
- It keeps a short excerpt of a product description at most, and every listing on lowerSpendings
links back to the store.
How it behaves
- It reads your
robots.txt and follows it, including Crawl-delay. - It makes one request at a time to a site, at least 5 seconds apart, usually once a day, and
sends conditional requests (ETag / If-Modified-Since) so unchanged pages cost little.
- If a site answers with 403, 429 or a challenge page, it stops for the rest of the day. It
always uses this name and the same addresses: it never changes its identity, IP address or
headers to get past a block.
Opting out or asking us to slow down
To stop it, add this to your robots.txt:
User-agent: lowerSpendingsBot
Disallow: /
To ask for a different rate, offer a product feed or anything else, email michael@lowerspendings.com.