Introduction to Bots
Automated scripts, commonly known as bots, are widely used across the Web to perform a range of functions such as indexing web pages, monitoring websites, aggregating content, providing information and updates on demand, and much more. The GoogleBot is perhaps the best-known example, crawling websites to build the biggest searchable index on the Web. On the other hand, an exponentially-increasing number of bad bots are extant across the Web today, constituting over 50% of all Web traffic. These bad bots carry out malicious and fraudulent activities such as Account Takeover, Application DDoS, API Abuse, Carding, Content Scraping, Skewed Analytics, Ad Fraud, and Form Spam. Unchecked bot traffic, apart from the threats they directly pose to online businesses, also leads to poor user experience from website slowdowns and outages, non-availability of inventory, personal data breaches, and hurts brand reputation and revenue.
What is Bot Management?
Essentially, Bot Management is the real-time detection and classification of every bot (good or bad) that visits a website or app. Good bots are categorized based on their purpose, such as site uptime monitoring, backlink-checking, social networking, and so on. Bad bots are categorized depending on their sophistication and intent: basic, advanced, or with a specific intent. A Bot Management solution can be used to blacklist and block bad bots before they can cause any harm, but whitelist good bots, partner bots, and vital third-party applications and services.
Why is Bot Management essential?
It is crucial for businesses to understand the nature and intent of bots visiting their online properties. Appropriate responses should be taken depending on the bot’s intent and business requirements. Good bots and partner bots can be whitelisted, or selectively allowed. For example, if you switched one of your partner services to a new vendor, the old vendor may continue to access your site to read data.
For bad bots, you can decide what kind of actions can best reduce the probability of bot mutation and the impact of their activities:
- Basic bots can be directly blocked from accessing your website or app.
- Sophisticated bots with discernible intent can be challenged with a CAPTCHA to be solved so that the bot is fingerprinted before it visits again in a mutated form (such as changing IP address/ User Agent, using another human-like cookie pattern or a seemingly-genuine URL traversal path, and so on).
- Certain bots with similar characteristics can be throttled at the web or application server level to lower their impact.
- Some solutions (such as ShieldSquare) have a custom feature that provides the ability to continue blacklisting persistent bots by consistently analyzing the behavior of their signatures before their expiry.
Advanced Bot Management Options
A specialized Bot Management solution gives webmasters and security teams the ability to granularly manage every kind of bot and take the most appropriate action on each one. Listed below are some of the advanced management options that help organizations effectively manage bots:
URL/ Section-based Management
You may have critical URLs for your business such as your login page, product or listing page, checkout page, and so on. You can manage bots to best suit your business workflow. For example, you can block the bots visiting login URLs, show fake or misleading data in the product or listing page, and challenge with a CAPTCHA on the payment page. This form of bot management will drastically reduce the impact of bots on business critical sections. You can also selectively decide to have custom actions for customer APIs or URLs.
If you work with multiple traffic vendors to source traffic to your site, and want to block a vendor that sends a lot of bot traffic to your site, you can do so by blocking the traffic with a custom UTM variable. The same principle applies to prevention of affiliate fraud caused by bots.
You can choose to block bots during peak hours of your business, let’s say from 17:00 to 23:00 hours, and choose to challenge bots with a CAPTCHA during off-peak hours.
You can choose to challenge bots with a CAPTCHA with a predetermined threshold of hits per hour, beyond which you can choose to block them altogether. You can also decide to block bots after a certain percentage of traffic crosses a preset threshold (per hour or per day). Traditional IP-based rate limiting protocols are prone to a large number of False Positives, as many users can originate from the same IP address. Solutions such as ShieldSquare provide device-level rate limiting to greatly reduce the number of False Positives.
You can mitigate bots at your CDN level if you want to protect your web and application servers by either blocking or redirecting to alternate origin servers. You can also stop them at your web server (Nginx/Apache) if you want to protect your APIs or app servers from unwanted traffic. You can also decide if you want to block bots at the Edge level, Network level, or App level.
Protection for Web/ Mobile Web/ Mobile Applications
Ideally, Bot Management should be performed by a unified solution to protect all your web assets, including mobile-specific websites and mobile applications, and provide the flexibility to customize responses for each platform based on business needs.
Filtering Bots From Web Analytics Tools
Bot traffic inflates and distorts website traffic data, and prevents marketers from making accurate, informed decisions. A dedicated Bot Management solution can filter all invalid traffic from analytics dashboards to restore accuracy to skewed analytics data.
Integrating with SIEM tools
Bot Management solutions should have the ability to integrate with SIEM (Security Information and Event Management) tools to provide timely alerts and a holistic view of infrastructural security through existing SIEM tools.
Ability to Generate Custom Reports
Understanding the intent of bots, and how they cause harm to businesses, requires a management solution that can generate custom reports to provide a comprehensive range of insights to webmasters and security teams.
Machine Learning and AI Capabilities
A Bot Management solution should have self-learning capabilities that allow it to constantly assimilate data on bad bots, their behavior, and their tactics in order to provide the most accurate detection and blocking abilities.
Bottom Line — Bot Management Must Constantly Evolve to Counter Increasingly-Sophisticated Bots
As sophisticated bots get better at mimicking human behavior, bot management solutions that can detect and block them are critical for web and application security. Leading research organizations such as Forrester Research are increasingly advising security professionals to implement specialized bot management solutions to protect business-critical Web assets from bot threats. For more information on how ShieldSquare can help your business prevent bot attacks, please contact firstname.lastname@example.org.