HTTP clients are programming libraries and command line tools. Developers use them for scripts, integrations and scrapers.
Real browsers never send these user agents, so treat them as automated traffic and block them on pages meant for people.
Of the 7 HTTP clients and libraries with a stated policy, 0 follow robots.txt and 7 do not. Bots that ignore robots.txt can only be stopped by user agent or IP at your server, CDN or WAF.
Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.
