Aggregators collect articles and links from many sites to build news feeds, read later apps and discovery services.
Some send referral traffic, others republish your content. Check what each one does with your pages before allowing it.
Of the 10 content aggregators with a stated policy, 10 follow robots.txt and 0 do not. Bots that ignore robots.txt can only be stopped by user agent or IP at your server, CDN or WAF.
Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.
