Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Innguma Fetcher
social media
Innguma
Innguma Fetcher
allow
Innologica
social media
Innologica
inoreader.com
allow
Inoreader
ai assistant
Inoreader
Inoreader
observe
inoreader.com
ai assistant
Innologica
inoreader.com
observe
Instapaper
other
Instant Paper
Instapaper
observe
Instaparser
ai assistant
Instapaper
Instaparser
observe
Instinct
ai agent
Instinct
observe
Integromat
other
Make
Integromat
allow
integromedb
other
integromedb
observe
intelium_bot
other
intelium_bot
observe
intelx.io_bot
search engine
Intelligence X
intelx.io_bot
allow
IntentBeamBot
ai assistant
IntentBeamBot
observe
InteractiveAdvertisingBureau
ai assistant
OneTag
InteractiveAdvertisingBureau
observe
InterfaxScanBot
other
InterfaxScanBot
observe
Internet Archive
archiver
Internet Archive
archive.org_bot
allow
Internet Archive - Archive-It
archiver
Archive-It
Archive-It
allow
InternetArchiveBot
monitoring
Internet Archive
InternetArchiveBot
allow
InternetMeasurement
security
Driftnet
InternetMeasurement
allow
InterviewOS
ai assistant
Subspace
InterviewOS
observe
IntricleBot
ai assistant
IntricleBot
observe
IonCrawl
ai assistant
IONOS
IonCrawl
observe
ip-web-crawler.com
other
ip-web-crawler.com
observe
ips-agent
other
ips-agent
observe
IRLbot
other
IRLbot
observe
Iron33
other
Iron33
observe
IronFountain-Leads
other
Iron Fountain
IronFountain-Leads
observe
IsDownBot
monitoring
IsDownBot
IsDownBot
allow
iskanie
other
iskanie
observe
IsonSearchBot
ai crawler
IsonAI
IsonSearchBot
observe
IsraBot
other
IsraBot
observe
ISSCyberRiskCrawler
security
ISS
ISSCyberRiskCrawler
allow
istellabot
other
istellabot
observe
it2media-domain-crawler
other
it2media-domain-crawler
observe
Jagged Pixel UptimeBot
monitoring
Jagged Pixel Inc.
www.getuptime.co
allow
Jambot
search engine
JamBot
Jambot
allow
James BOT
seo crawler
cognitiveSEO
James
challenge
JamesBOT
other
JamesBOT
observe
Jamie's Spider
other
Jamie&
observe
JenkersBot
other
JenkersBot
observe
JennyBot
other
JennyBot
observe
Jetbot
other
Jetbot
observe
jetmon
ai coding agent
Automattic
jetmon
observe
Jetpack
monitoring
Automattic
Jetpack
allow
Jetty
other
Jetty
observe
JikeSpider
other
JikeSpider
observe
JobboerseBot
other
JobboerseBot
observe
JobHuntBot
other
JobHuntBot
observe
Jobiglo-Bot
search engine
Jobiglo
Jobiglo-Bot
allow
jobs-crawler
other
jobs-crawler
observe
jobswithgptcom-bot
search engine
JobsWithGPT
jobswithgptcom-bot
allow
Jooblebot
search engine
Jooble
Jooblebot
allow
jpg-newsbot
other
jpg-newsbot
observe
Jugendschutzprogramm-Crawler
ai assistant
JusProg
Jugendschutzprogramm-Crawler
observe
Jumio
security
Coinbase
Jumio
allow
jyxobot
other
jyxobot
observe
k2spider
other
k2spider
observe
K7MLWCBot
security
K7 Computing
K7MLWCBot
allow
Kagi Bot
search engine
Kagi
allow
kagi-fetcher
ai assistant
Kagi
kagi-fetcher
observe
Kagibot
search engine
Kagi
Kagibot
allow
rDNS