About our crawler

We run a small crawler that reads publicly available RSS, Atom and JSON feeds to build a news index for AI tools. If you landed here from your server logs, this page tells you exactly what it does and how to stop it.

How to recognise it

AIScannerBot/0.1 (+https://ainews.webscanreport.com/bot)

What it does

• It reads feeds only — the URLs publishers provide for exactly this purpose. It does not spider your site, follow links, or fetch pages, images or assets.
• It sends a conditional request with If-None-Match and If-Modified-Since. If nothing changed, your server answers 304 and sends no body at all.
• It waits at least 2 seconds between requests to the same host, and never runs two requests against one host at the same time.
• It checks a feed at most once every 60 minutes, and backs off exponentially when something fails.
• It gives up after 15 seconds and stops reading after 5 MB.

What it stores

The headline, at most 300 characters of the summary you put in the feed, the publication date and the link. Nothing else. We never store or republish a full article, and every headline on the site links straight back to you.

robots.txt

We follow RFC 9309, and we fail safe: if your robots.txt is unreachable, returns a server error or rate-limits us, we do not crawl at all. Several sources are switched off for exactly this reason and we list them publicly on the sources page.

To keep us off entirely, this is enough:

User-agent: AIScannerBot
Disallow: /

Changes take effect within twelve hours, which is how long we cache a robots.txt. A failed fetch is only remembered for ten minutes, so a hiccup on your side never locks us out for long.

Getting in touch

If the crawler is causing trouble, or you would rather we did not index your feed at all, say so and we will remove it — no robots.txt required. Mail mr.tomstraathof@gmail.com.