ShieldSquare named a Leader in The Forrester New Wave™: Bot Management Report, Q3 2018. Click to know more.
Upon analyzing random samples from the top 1 million websites (Alexa ranking), our internal data shows that about 50% of the web traffic is comprised of bots. Of these, most of them are from bots with malicious intents. After categorizing and analyzing websites based on monthly page views, the following is what we have observed during peak scraping:
For websites having over 100 million pageviews per month, the impact is 120,000 bot hits per hour.
For websites with monthly pageviews between 10 million and 100 million, it’s about 20,000 bot hits per hour.
For websites having less than 10 million pageviews a month, they experience about 3,000 bot hits per hour.
These numbers are only indicative, but gives you an idea on the amount of bot traffic a website can get, if left unprotected. For the sake of generality, assume that these bot hits are fake page views, and consider the website with about 300 million pageviews a month:
120,000 (pageviews/hour) x 720 (hours/month) = 86,400,000 page views/month.
So, almost 29% of the website’s traffic is from bots. In other words, almost 29% of your server resources are wasted in processing those fake pageviews. Practically speaking, the bot hits per hour may not be a perpetual phenomenon, but when this happens, the website’s competitive advantage and brand perception is lost forever.
With easy access to computing power and sophisticated scraping tools, a scraper can scrape at least 4500 - 5500 pages per minute.
Suppose an eCommerce website is preparing for the shopping season, and the marketing teams have invested thousands of dollars running campaigns to acquire customers. And, suppose the marketing team sees a huge spike in traffic when the sale begins. The marketing and management teams, oblivious to bot traffic, may draw conclusions that their campaigns are working and that site traffic has increased 2x or 3x. The real problem comes up when these visits don’t convert to sales, or in many cases, end up ruining the end-user shopping experience.
In other words, their marketing campaigns didn’t succeed in bringing in real users. This is why it becomes important for websites to know how much of their traffic is from real humans. At least to change their strategies quickly when something doesn’t work, and quickly implement workarounds to attract new customers.