Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
ichiro
other
ichiro
observe
Idealo
seo crawler
Idealo internet GmbH
challenge
IDG/IT
ai assistant
SpazioDati
IDG/IT
observe
Iframely
social media
Iframely
Iframely
allow
IFTTT
ai assistant
IFTTT
IFTTT
observe
IFTTT RSS Feed Service
social media
IFTTT RSS Feed Servuce
allow
ImagesiftBot
ai assistant
Hive
ImagesiftBot
observe
imageSpider
ai scraper
ByteDance
imageSpider
observe
IMJ-CompanyPage-Scraper
other
IMJ-CompanyPage-Scraper
observe
Impact.com
other
Impact.com
observe
imrbot
other
imrbot
observe
IndeedBot
other
IndeedBot
observe
IndeedJobBot
seo crawler
Indeed
IndeedJobBot
challenge
INETDEX-Bot
other
INETDEX-Bot
observe
InfoNaviRobot
other
InfoNaviRobot
observe
infoobot
other
infoobot
observe
infoseek
other
infoseek
observe
Infrawatch
security
Infrawatch
Infrawatch
allow
Innguma Fetcher
social media
Innguma
Innguma
allow
Innologica
social media
Innologica
Innologica
allow
Inoreader
ai assistant
Inoreader
Inoreader
observe
inoreader.com
ai assistant
Innologica
inoreader.com
observe
Instapaper
other
Instant Paper
Instapaper
observe
Instaparser
ai assistant
Instapaper
Instaparser
observe
Integromat
other
Make
allow
integromedb
other
integromedb
observe
intelium_bot
other
intelium_bot
observe
intelx.io_bot
search engine
Intelligence X
intelx.io_bot
allow
InteractiveAdvertisingBureau
ai assistant
OneTag
InteractiveAdvertisingBureau
observe
InterfaxScanBot
other
InterfaxScanBot
observe
Internet
archiver
Internet Archive
Internet
allow
Internet Archive
archiver
Internet Archive
archive.org_bot
allow
Internet Archive - Archive-It
archiver
Archive-It
allow
InternetArchiveBot
monitoring
Internet Archive
InternetArchiveBot
allow
InternetMeasurement
security
Driftnet
InternetMeasurement
allow
IonCrawl
ai assistant
IONOS
IonCrawl
observe
ip-web-crawler.com
other
ip-web-crawler.com
observe
ips-agent
other
ips-agent
observe
IRLbot
other
IRLbot
observe
Iron33
other
Iron33
observe
IsDownBot
monitoring
IsDownBot
IsDownBot
allow
iskanie
other
iskanie
observe
IsraBot
other
IsraBot
observe
ISSCyberRiskCrawler
security
ISS
ISSCyberRiskCrawler
allow
istellabot
other
istellabot
observe
it2media-domain-crawler
other
it2media-domain-crawler
observe
Jagged
ai coding agent
Jagged
observe
Jagged Pixel UptimeBot
monitoring
Jagged Pixel Inc.
allow
Jambot
search engine
JamBot
Jambot
allow
James
seo crawler
cognitiveSEO
James
challenge
JamesBOT
other
JamesBOT
observe
Jamie's
other
Jamie's
observe
JenkersBot
other
JenkersBot
observe
JennyBot
other
JennyBot
observe
Jetbot
other
Jetbot
observe
jetmon
ai coding agent
Automattic
jetmon
observe
Jetpack
monitoring
Automattic
Jetpack
allow
Jetty
other
Jetty
observe
JikeSpider
other
JikeSpider
observe
JobboerseBot
other
JobboerseBot
observe