Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

114 results · Search Engine

360Spider search engine

Qihoo 360

Alexandria.org

AlexandriaOrgBot search engine

Alexandria.org

AlexandriaOrgBot

allow

Algolia search engine

Algolia

Algolia

allow

All Africa Crawler search engine

AllAfrica Global Media

RSSMicro

Baidu ADS Server Proxy search engine

Baidu

Baidu-ADS

allow

Baiduspider search engine

Baidu

Baiduspider

allow

Baiduspider-render search engine

Baidu

Baiduspider-render

allow

BingBot search engine

Microsoft

bingbot

allow

Cloudflare Web Scanner search engine

Cloudflare

allow

cludo.com search engine

Cludo

cludo.com

allow

Cốc Cốc search engine

Coccoc

coccocbot

allow

coccocbot-image search engine

Coc Coc

coccocbot-image

allow

coccocbot-web search engine

Coc Coc

coccocbot-web

allow

Coveo Bot search engine

Coveo

Coveobot

allow

crawler.freespoke.com search engine

Freespoke

crawler.freespoke.com

allow

Crawlson search engine

Crawlson

Crawlson

allow

Dataprovider search engine

Dataprovider.com

Dataprovider

allow

Dataprovider.com search engine

Dataprovider.com

Daum

DocSpotBot search engine

DocSpot

DocSpotBot

allow

DuckDuckBot search engine

DuckDuckGo

DuckDuckBot

allow

DuckDuckGo-Favicons-Bot search engine

DuckDuckGo

DuckDuckGo-Favicons-Bot

allow

Feedfetcher-Google search engine

Google

Feedfetcher-Google

allow

FindFiles.net search engine

FindFiles.net

FindFiles.net

allow

FindITAnswersbot search engine

IT Influentials

FindITAnswersbot

allow

Freespoke search engine

Freespoke

Freespoke

allow

FreespokeCrawler search engine

Freespoke

FreespokeCrawler

allow

Funnelback search engine

Squiz - FunnelBack

Funnelback

allow

fyndbot search engine

Fynd

fyndbot

allow

FyndSearchEngine-Crawler search engine

Fynd

FyndSearchEngine-Crawler

allow

FyndSearchEngine-ReCrawler search engine

FyndSearchEngine-ReCrawler

allow

GeedoProductSearch search engine

Geedo

GeedoProductSearch

allow

Gigabot search engine

Gigablast

Gigabot

allow

Google Images search engine

Google

allow

Google Scholar search engine

Google

Google

allow

Google Videos search engine

Google

allow

Google-Flights-Search-Spackle search engine

Google

allow

google-xrawler search engine

Google

google-xrawler

allow

GoogleBot search engine

Google

Googlebot

allow

Googlebot-IA search engine

Google

Googlebot-IA

allow

Googlebot-Image search engine

Google

Googlebot-Image

allow

Googlebot-Mobile search engine

Google

Googlebot-Mobile

allow

Googlebot-News search engine

Google

Googlebot-News

allow

Googlebot-Video search engine

Google

Greppr

HaosouSpider search engine

Qihoo 360

Hype Machine

Hype Machine search engine

Hype Machine

allow

IbouBot search engine

Babbar

IbouBot

allow

intelx.io_bot search engine

Intelligence X

JamBot

Jobiglo-Bot search engine

Jobiglo

Jobiglo-Bot

allow

jobswithgptcom-bot search engine

JobsWithGPT

jobswithgptcom-bot

allow

Jooblebot search engine

Jooble

Jooblebot

allow

Kagi Bot search engine

Kagi

allow

Kagibot search engine

Kagi

Kagibot

allow

Level9SearchBot search engine

Level 9

Level9SearchBot

allow

Linespider search engine

LINE Corporation

Linespider

allow

← Prev 1 / 2 Next →