746 crawlers, AI bots and automated clients. Look up who runs a bot, what its user agent looks like and how to block or allow it.
Monitoring bots check uptime, performance and page changes on behalf of a site owner or a third party.
SEO crawlers map links, keywords and technical issues for SEO tools and backlink databases.
Search engine crawlers index your pages so they can appear in search results.
Security scanners probe sites for vulnerabilities, certificates and exposed services.
Webhook senders call your endpoints to deliver events from payment, messaging and SaaS platforms.
Advertising bots review landing pages, check ad placements and classify pages for brand safety and contextual targeting.
Link preview bots fetch a page when someone shares its URL in a chat app or social network, to build the preview card.
Feed fetchers poll RSS and Atom feeds for readers, aggregators and podcast apps.
AI crawlers collect web pages in bulk to build training data for large language models and other AI systems.
Aggregators collect articles and links from many sites to build news feeds, read later apps and discovery services.
AI assistants fetch pages on demand when a user asks a chatbot or an agent to read, summarise or act on a URL.
AI search crawlers index pages so that AI answer engines can cite and link to them in their responses.
Bots that do not fit the other categories.
Accessibility checkers scan pages for WCAG issues on behalf of site owners and auditors.
Research crawlers collect pages for university and non-profit studies of the web.
HTTP clients are programming libraries and command line tools. Developers use them for scripts, integrations and scrapers.
Archivers save copies of pages to preserve the history of the web.
Headless browsers run a full browser engine without a screen. They are used for testing, rendering and scraping sites that need JavaScript.
Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.
Social Media Marketing Bots
Social media marketing bots read pages for social scheduling, listening and app linking tools.