Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
imrbot
other
imrbot
observe
IndeedBot
other
IndeedBot
observe
INETDEX-Bot
other
INETDEX-Bot
observe
InfoNaviRobot
other
InfoNaviRobot
observe
infoobot
other
infoobot
observe
infoseek
other
infoseek
observe
Innguma
ai assistant
Innguma
Innguma
observe
Inoreader
ai assistant
Inoreader
Inoreader
observe
inoreader.com
ai assistant
Innologica
inoreader.com
observe
Instapaper
other
Instant Paper
Instapaper
observe
Instaparser
ai assistant
Instapaper
Instaparser
observe
integromedb
other
integromedb
observe
intelium_bot
other
intelium_bot
observe
InteractiveAdvertisingBureau
ai assistant
OneTag
InteractiveAdvertisingBureau
observe
InterfaxScanBot
other
InterfaxScanBot
observe
IonCrawl
ai assistant
IONOS
IonCrawl
observe
ip-web-crawler.com
other
ip-web-crawler.com
observe
ips-agent
other
ips-agent
observe
IRLbot
other
IRLbot
observe
Iron33
other
Iron33
observe
iskanie
other
iskanie
observe
IsraBot
other
IsraBot
observe
istellabot
other
istellabot
observe
it2media-domain-crawler
other
it2media-domain-crawler
observe
JamesBOT
other
JamesBOT
observe
Jamie's Spider
other
observe
JenkersBot
other
JenkersBot
observe
JennyBot
other
JennyBot
observe
Jetbot
other
Jetbot
observe
jetmon
ai coding agent
Automattic
jetmon
observe
Jetty
other
Jetty
observe
JikeSpider
other
JikeSpider
observe
JobboerseBot
other
JobboerseBot
observe
JobHuntBot
other
JobHuntBot
observe
jpg-newsbot
other
jpg-newsbot
observe
Jugendschutzprogramm-Crawler
ai assistant
JusProg
Jugendschutzprogramm-Crawler
observe
jyxobot
other
jyxobot
observe
k2spider
other
k2spider
observe
kagi-fetcher
ai assistant
Kagi
kagi-fetcher
observe
kaikki.org-digital-archive
other
kaikki.org-digital-archive
observe
Kangaroo Bot
ai scraper
Kangaroo LLM
Kangaroo
observe
KargoBot
ai assistant
Kargo
KargoBot
observe
KargoBot-Artemis-Mobile
ai assistant
Kargo
KargoBot-Artemis-Mobile
observe
KauaiBot
other
KauaiBot
observe
kb.dk_bot
ai scraper
Netarkivet
kb.dk_bot
observe
kbcrawl
other
kbcrawl
observe
Kemvibot
other
Kemvibot
observe
Kenjin Spider
other
Kenjin
observe
Kernel
ai assistant
Kernel
kernel
observe
signed
Keydrop.io
other
Keydrop.io
observe
keys-so-bot
other
keys-so-bot
observe
Keyword Density
other
Keyword
observe
Kimi-SearchBot
ai crawler
Moonshot AI
Kimi-SearchBot
observe
Kimi-User
ai assistant
Moonshot AI
Kimi-User
observe
KimiBot
ai crawler
Moonshot AI
KimiBot
observe
KimiCrawler
ai scraper
Moonshot AI
KimiCrawler
observe
kinsta
ai coding agent
Kinsta
kinsta
observe
KlaviyoAIBot
ai crawler
Klaviyo
KlaviyoAIBot
observe
Knowings
other
Knowings
observe
Known Agent
ai assistant
Known Agents
Known
observe
signed