IOCCheckBot

IOCCheckBot is the crawler behind IOC Site Check — a free tool that tells website owners whether AI systems can access and read their pages.

If you found this page because you saw IOCCheckBot in your server logs, this page explains what it does and how to control it.

What it does

When someone enters a URL into IOC Site Check, IOCCheckBot fetches a small number of pages from that site and reports on technical accessibility and machine readability.

It does not collect personal data, does not submit forms, does not log in, and does not follow links beyond the pages it needs.

How to identify it

User-Agent: IOCCheckBot/1.0 (+https://ioclabsco.com/bot)

All requests come from a fixed IP address with matching forward and reverse DNS:

169.58.231.36 → checkbot.ioclabsco.com → 169.58.231.36

You can verify any request claiming to be IOCCheckBot by checking the reverse DNS of the source IP.

A machine-readable list of our IP addresses is available at /bot/ips.json.

Verifying the crawler

IP addresses can change and user-agent strings can be forged. The reliable check is forward-confirmed reverse DNS: our crawler’s IP resolves to a hostname under ioclabsco.com, and that hostname resolves back to the same IP.

host 169.58.231.36
# → checkbot.ioclabsco.com
host checkbot.ioclabsco.com
# → 169.58.231.36

Signed requests (Web Bot Auth)

IOCCheckBot also signs its requests using Web Bot Auth, so a server can verify our identity cryptographically rather than by IP address or user-agent alone.

Standard: RFC 9421 HTTP Message Signatures, using Ed25519 keys. The Signature-Agent value is https://ioclabsco.com.

Our public key directory (JWKS) is published at https://ioclabsco.com/.well-known/http-message-signatures-directory.

How it behaves

• Respects robots.txt, including rules for IOCCheckBot
• Honors crawl-delay from robots.txt (up to 30 seconds); otherwise at most one request per second per domain
• At most 10 pages per check
• Respects Retry-After and backs off on 429
• Stops on 403 rather than retrying
• Does not impersonate other crawlers

How to block it

Add this to your robots.txt:

User-agent: IOCCheckBot
Disallow: /

We check robots.txt before every crawl and stop immediately if disallowed. No further action is needed.

If instead you want the tool to reach your site, see how to allow IOCCheckBot.

Contact

If IOCCheckBot is causing problems for your site, or you have questions about its behaviour, write to abuse@ioclabsco.com. We answer requests to slow down or stop crawling.

Please include the timestamp and the affected URLs — it lets us find the requests in our logs.