Skip to main content
Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

73 results · Social Media

AppcastBot social media

Appcast

AppcastParsingBot
allow
Arena Bot social media

Arena.im

ArenaBot
allow
Artemis Web Reader social media

capjamesg

artemis
allow
Bazqux social media

Bazqux

BazQux
allow
Bing Preview social media

Microsoft

BingPreview
allow rDNS
Blogtrottr social media

Blogtrottr

Blogtrottr
allow
Bluesky Link Preview Service social media

Bluesky

allow
Buffer Link Preview Bot social media

Buffer

BufferLinkPreviewBot
allow
Buttondown RSS-Feed-Parser social media

Buttondown

Buttondown RSS-Feed-Parser
allow
ConvermaxBot social media

Convermax Corp.

ConvermaxBot
allow
discordBot social media

Discord, Inc.

Discordbot
allow
Embedly social media

Embedly

Embedly
allow
FacebookExternalHit social media

Meta

facebook
allow
Fastmail Bot social media

Fastmail

Fastmail Bot
allow
Feedbin social media

Feedbin

Feedbin
allow
Feeder social media

Really Simple AB

Feeder
allow
Feedly social media

Feedly

Feedly
allow
FeedOtter social media

FeedOtter

FeedOtter
allow
FeedWind Crawler social media

Mikle

Feedwind
allow
FIT Feeds Fetcher social media

follow.it

follow.it
allow
FlipboardProxy social media

Flipboard

FlipboardProxy
allow
FlipboardRSS social media

Flipboard

FlipboardRSS
allow
FullFact Ingest social media

FullFact

FullFact
allow
Google Calendar Importer social media

Google

Google-Calendar-Importer
allow
Google Feed Fetcher social media

Google

allow
Google Image Proxy social media

Google

GoogleImageProxy
allow
Google Page Renderer social media

Google

Google-PageRenderer
allow
Google Publisher Center social media

Google

Google Publisher Center
allow
Google Web Snippet social media

Google

Google-PageRenderer
allow
Hatena social media

Hatena

Hatena Antenna
allow
HEY Email Privacy Proxy social media

37signals

allow
HubSpot Feed Fetcher social media

HubSpot

HubSpot Feed Fetcher
allow
HubSpot Page Fetcher social media

HubSpot

HubSpot Page Fetcher
allow
Iframely social media

Iframely

Iframely
allow
IFTTT RSS Feed Service social media

IFTTT RSS Feed Servuce

IFTTT
allow
Innguma Fetcher social media

Innguma

Innguma Fetcher
allow
Innologica social media

Innologica

inoreader.com
allow
KakaoTalk Scrap social media

Kakao

kakaotalk-scrap
allow
LinkedInBot social media

LinkedIn

LinkedInBot
allow
LinksIndexerBot social media

Links Indexer

LinksIndexerBot
allow
Loomly Bot social media

Loomly

LoomlyBot
allow
MailRUBot social media

Mail Russia

Mail.RU_Bot
allow
Meta-WebIndexer social media

Meta

meta-webindexer
allow
Microsoft Preview social media

Microsoft

MicrosoftPreview
allow
MonitoRSS social media

MonitoRSS

MonitoRSS
allow
Muck Rack Scraper social media

Muck Rack

MuckRack
allow
NewsBlur social media

NewsBlur

NewsBlur Favicon Fetcher - 1692 subscribers - https:
allow
Nooshub social media

Nooshub

Nooshub
allow
Opengraph Bot social media

Opengraph

allow
OpenRSS social media

OpenRSS

OpenRSS
allow
Orlo Link Preview social media

Orlo

Orlo-LinkPreview
allow
OurFamilyWizard social media

OurFamilyWizard

Cozi-iCalendar-FeedReader
allow
Overcast social media

Overcast Radio

Overcast
allow
Ozon Web Grabber social media

Ozon

Ozon
allow
Pocket Casts Feed Parser social media

Pocket Casts

Pocket
allow
PressEngine Bot social media

PressEngine

PressEngine Bot
allow
Protopage social media

Protopage Ltd

Protopage
allow
RSS API social media

RSS API (by Tibush GmbH)

RSSAPI
allow
rss2tg_bot social media

Yellow Rubber Duck Consulting

rss2tg
allow
Scour RSS Bot social media

Scour

scour.ing
allow
← Prev 1 / 2 Next →