Search engine crawlers index your pages so they can appear in search results.
Keep the genuine ones allowed and verify them, because fake search crawlers are one of the most common disguises for scrapers.
Of the 33 search engine crawlers with a stated policy, 33 follow robots.txt and 0 do not. Bots that ignore robots.txt can only be stopped by user agent or IP at your server, CDN or WAF.
Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.
