Webpagetest is a bot in the monitoring and analytics bots category, operated by WebPageTest.
WebPageTest is one of the most popular and free tools for measuring webpage performance and enables you to run web performance tests on your site from a number of different locations across the world in a number of different browsers.
Monitoring bots check uptime, performance and page changes on behalf of a site owner or a third party. Allow the services you run yourself and exclude them from analytics so they do not inflate page views.
Sample user agent strings sent by Webpagetest:
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0 monitor/webpagetest PTST/190827.180809Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 monitor/webpagetest PTST/190827.180809Anyone can copy a user agent string. Transfon Security also checks where each request comes from, so a request that claims to be a known crawler but comes from the wrong network is treated as a fake, not as the real crawler.
We have not confirmed whether Webpagetest follows robots.txt. Add a rule, then check your logs to see that it stopped. robots.txt is a plain text file at the root of your domain, such as https://example.com/robots.txt.
User-agent: monitor Disallow: /
User-agent: monitor Disallow: /private/ Disallow: /search
User-agent: monitor Allow: /blog/ Disallow: /
User-agent: monitor Allow: / User-agent: * Disallow: /
User-agent: monitor group, Webpagetest ignores your User-agent: * rules, so repeat any that should still apply.monitor from Webpagetest's user agent; check WebPageTest's own documentation if it publishes a different robots.txt name.Treat robots.txt as a statement of your policy, and enforce it where requests arrive.
Allow Webpagetest if it is a service you use, and keep its visits out of page views and ad requests either way.
Transfon Security for publishersWebpagetest is not a person. Exclude it from campaign and conversion data so reports reflect real visitors.
Transfon Security for advertisersAllow Webpagetest only if it works for your store, and keep it away from cart and checkout.
Transfon Security for e-commerceIf Webpagetest is a service you use, allow it on the endpoints it needs and block it elsewhere.
Transfon Security for SaaS and APIsrobots.txt is a request, not an enforcement. Transfon Security shows Webpagetest by name in the Top Bots report, so you can see how often it visits and which pages it reads. Block or allow it with a User Agent rule, or slow it down with a rate limit decision rule.
Transfon Security
Webpagetest is a bot in the monitoring and analytics bots category, operated by WebPageTest. WebPageTest is one of the most popular and free tools for measuring webpage performance and enables you to run web performance tests on your site from a number of different locations across the world in a number of different browsers.
We have not confirmed whether Webpagetest follows robots.txt. Add a robots.txt rule, then check your logs to see that it stopped, and block it by user agent if it did not.
Add "User-agent: monitor" followed by "Disallow: /" to the robots.txt file at the root of your site. To enforce it, block requests whose user agent contains "monitor/webpagetest" at your WAF. In Transfon Security that is one User Agent block rule.
No. robots.txt is voluntary, so Webpagetest may not follow it, and anyone can send Webpagetest's user agent, so a fake Webpagetest ignores your rule. Enforce the rule where requests arrive, at your WAF, and check the source network.
Monitoring bots check uptime, performance and page changes on behalf of a site owner or a third party. Allow the services you run yourself and exclude them from analytics so they do not inflate page views.
Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.