Daily snapshot 2026-10-28
2026-10-28 UTC
The numbers come from one website. They describe what these crawlers did here, not across the web.
Period: 2026-10-28T00:00:00Z to 2026-10-29T00:00:00Z (00:00Z to 24:00Z). Generated 2026-10-29T03:18:29.230Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method
Facts
- F-261028-01 Bingbot made 31 requests on 2026-10-28, its highest daily count since the site launched on 2026-09-13; its previous high was 24, on 2026-10-27. Source
- F-261028-02 PerplexityBot has made no requests since 2026-10-21, 7 days. Source
- F-261028-03 Bingbot first fetched the 2026-10-27 snapshot 3 h 54 min after it was published. Source
- F-261028-04 Googlebot first fetched the 2026-10-27 snapshot 8 h 46 min after it was published. Source
- F-261028-05 GoogleOther first fetched the 2026-10-27 snapshot 8 h 46 min after it was published. Source
- F-261028-06 ClaudeBot first fetched the 2026-10-27 snapshot 8 h 48 min after it was published. Source
- F-261028-07 Every request on 2026-10-28 from ClaudeBot, Bingbot, OAI-SearchBot, GPTBot and GoogleOther was verified against the operator's published method. Source
Records selected at random for 2026-10-29
Ten records from the field guide are chosen at random each day. The selection and how it was drawn are listed in /snapshot.json.
- https://rattlesnakesbymail.com/fields/what_it_controls
- https://rattlesnakesbymail.com/claims/57
- https://rattlesnakesbymail.com/claims/33
- https://rattlesnakesbymail.com/claims/120
- https://rattlesnakesbymail.com/claims/79
- https://rattlesnakesbymail.com/claims/71
- https://rattlesnakesbymail.com/compare/google-extended-vs-perplexitybot
- https://rattlesnakesbymail.com/observed/applebot-extended
- https://rattlesnakesbymail.com/observed/gptbot
- https://rattlesnakesbymail.com/claims/147
Crawlers
| Crawler | Requests | Verified share | Distinct paths | 28-day median | Sweep day | First path | robots.txt | sitemap |
|---|---|---|---|---|---|---|---|---|
| ClaudeBot | 42 | 100% | 6 | 48 | no | /sitemap-index.xml | 9 | 28 |
| Googlebot | 35 | 94.3% | 21 | 15 | no | /robots.txt | 12 | 0 |
| Bingbot | 31 | 100% | 31 | 12 | no | /compare/bingbot-vs-googleother | 1 | 1 |
| OAI-SearchBot | 6 | 100% | 3 | 5 | no | /robots.txt | 4 | 0 |
| GPTBot | 3 | 100% | 3 | 3 | no | /sitemap-index.xml | 0 | 3 |
| ChatGPT-User | 2 | 0% | 2 | 0 | no | /.env.backup | 0 | 0 |
| GoogleOther | 2 | 100% | 2 | 0 | no | /snapshots | 0 | 0 |
MCP calls
| Client | Tool | Calls |
|---|---|---|
| BrickBlueBot | get_snapshot | 1 |
| BrickBlueBot | identify_user_agent | 1 |
| BrickBlueBot | list_changes | 1 |
| BrickBlueBot | lookup_crawler | 1 |
| rokmcp-probe | list_changes | 1 |
| verifymcp-probe | __verifymcp_auth_probe_5887e10ecf28e8ce__ | 1 |
Snapshot pickup
For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.
| Snapshot | Crawler | Minutes after publication | Tag |
|---|---|---|---|
| 2026-10-27 | Bingbot | 234 | none |
| 2026-10-27 | Googlebot | 526 | none |
| 2026-10-27 | GoogleOther | 526 | h |
| 2026-10-27 | ClaudeBot | 528 | h |
| 2026-10-26 | Bingbot | 234 | none |
| 2026-10-26 | Googlebot | 842 | none |
| 2026-10-26 | GoogleOther | 1,544 | none |
| 2026-10-26 | ClaudeBot | 2,031 | none |
| 2026-10-25 | Bingbot | 239 | none |
| 2026-10-25 | Googlebot | 631 | none |
| 2026-10-25 | ClaudeBot | 3,532 | none |
| 2026-10-24 | Bingbot | 233 | none |
| 2026-10-24 | Googlebot | 2,945 | none |
| 2026-10-24 | GoogleOther | 3,124 | none |
| 2026-10-24 | ClaudeBot | 5,044 | none |
| 2026-10-23 | Bingbot | 234 | none |
| 2026-10-23 | Googlebot | 423 | none |
| 2026-10-23 | GoogleOther | 584 | none |
| 2026-10-23 | ClaudeBot | 6,548 | none |
| 2026-10-22 | ClaudeBot | 214 | none |
| 2026-10-22 | Googlebot | 225 | none |
| 2026-10-22 | GoogleOther | 225 | h |
| 2026-10-22 | Bingbot | 235 | none |
Open questions
- Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? (id 26, asked 5 times)
- How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations? (id 34, asked 5 times)
- Was a specific document used in a model's training, fine-tuning, or evaluation? (id 38, asked 4 times)
- Why wasn't a specific page cited by an AI answer instead of a competitor's page? (id 7, asked 3 times)
- How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited? (id 11, asked 3 times)
If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.
Ask for something this snapshot lacks
What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261028.
To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261028 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261028. The optional author_claim field says who is writing: a person, a model or an agent.