Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
908 results · Other
GenomeCrawlerd
other
Nokia
GenomeCrawlerd
observe
Gensparkbot
other
Gensparkbot
observe
GermCrawler
other
GermCrawler
observe
GetRight
other
GetRight
observe
gigablast
other
gigablast
observe
GingerCrawler
other
GingerCrawler
observe
Gluten Free Crawler
other
Gluten
observe
gnam
other
gnam
observe
GnowitNewsbot
other
Gnowit
GnowitNewsbot
observe
GolgulBot
other
Golgul
GolgulBot
observe
Google API
other
APIs-Google
allow
rDNS
Google Read Aloud
other
Google-Read-Aloud
allow
Google-Ads-Creatives-Assistant
other
Google-Ads-Creatives-Assistant
observe
Google-Certificates-Bridge
other
Google-Certificates-Bridge
observe
Google-PhysicalWeb
other
Google-PhysicalWeb
observe
GoParserBot
other
GoParserBot
observe
GoPay
other
GoPay.cz
GoPay
allow
Gowikibot
other
Gowikibot
observe
GraphEmbedScraper
other
GraphEmbedScraper
observe
Grobbot
other
Grobbot
observe
GroupHigh
other
GroupHigh
observe
grub-client
other
grub-client
observe
grub.org
other
grub.org
observe
gsa-crawler
other
gsa-crawler
observe
gslfbot
other
gslfbot
observe
Gwene
other
Gwene
observe
HaloBot
other
Haloscan
HaloBot
observe
HanaleiBot
other
HanaleiBot
observe
Harvest
other
Harvest
observe
HawaiiBot
other
HawaiiBot
observe
HeadlessChrome
other
HeadlessChrome
observe
HeadOnlyScraper
other
HeadOnlyScraper
observe
HeadOnlyScraperV2
other
HeadOnlyScraperV2
observe
HelloWork
other
HelloWork Group
HelloWork
observe
HetrixTools Uptime Monitoring Bot
other
observe
hgfAlphaXCrawl
other
hgfAlphaXCrawl
observe
HIFIBot
other
HIFI
HIFIBot
observe
HSTS preload bot
other
Chromium
hstspreload-bot
observe
HTTP Banner Detection
other
observe
HubSeedsBot
other
HubSeedsBot
observe
HubSpot Crawler
other
HubSpot
HubSpot Connect 2.0 (http:
observe
humanlinks
other
humanlinks
observe
HUNT-Bot
other
HUNT-Bot
observe
HyScore
other
HyScore
observe
hyscore.io
other
hyscore.io
observe
ias-va
other
ias-va
observe
ICBot
other
ICBot
observe
ichiro
other
ichiro
observe
ImageBot
other
ImageBot
observe
IMJ-CompanyPage-Scraper
other
IMJ-CompanyPage-Scraper
observe
Impact.com Agent
other
Impact.com
observe
imrbot
other
imrbot
observe
IndeedBot
other
IndeedBot
observe
INETDEX-Bot
other
INETDEX-Bot
observe
InfoNaviRobot
other
InfoNaviRobot
observe
infoobot
other
infoobot
observe
infoseek
other
infoseek
observe
Instapaper
other
Instant Paper
Instapaper
observe
Integromat
other
Make
Integromat
allow
integromedb
other
integromedb
observe