Understanding Traffic Bots & How to Stop Them

TL;DR

Traffic bots are automated programs that account for about 40 percent of all internet traffic. While some bots serve legitimate purposes — such as powering search engine indexing, analytics, and copyright monitoring — others are weaponized for ad fraud, Distributed Denial-of-Service (DDoS) attacks, scraping, and spam. Website owners can defend against malicious bots by configuring robots.txt files, manually blocking suspicious Internet Protocol (IP) addresses, or deploying a dedicated bot management solution that uses machine learning to adapt over time.

How much of your website traffic is actually human?

Web bot hits originate from machines, not humans. The click, comment, or visit you believe comes from a person actually stems from a computer program instead.

Web bots aren't intrinsically good or bad. Computers can tackle repetitive tasks very quickly, and web bots can do many things critical to your performance and tracking. But web bot hits can also cost you quite a bit of money.

Let's dig deeper into what separates the good from the bad. We'll end with some tips you can use to both monitor and stop bot traffic.

What is a traffic bot?

Web bot hits stem from a programmed script. A developer creates a program that can click links, download images, or handle some other kind of simple task.

When a program like this is written, a developer can launch it on almost any website and wait for results with no more work required. Most bots run almost all the time.

How can you spot bot traffic on your site?

Chances are, your website deals with at least some kind of bot. About 85 percent of websites that analyze traffic use Google Analytics to do so. If you have this tool installed on your site, look for traffic that is:

• Bouncy. A bot needs mere seconds to look over a page. If your bounce rate is sky-high, you're likely dealing with a bot.
• Deep. A traffic bot will likely look over your entire site, working page by page.
• Foreign. If your website serves a local audience and you suddenly have plenty of visitors from far-off places, you're likely dealing with a bot. 

You may also spot traffic bots by behavior on your site. If something you post suddenly has hundreds of hits and your comment section is full of nonsensical phrases, a bot could be at work.

Good traffic bots

You've likely written your content with humans in mind, and it can be hard to think about computers taking up all of your traffic. But some website bots can be good for your performance.

These are four examples of good bots:

1. Aggregation: These website bots gather data from multiple websites and pop them into one page. A bot like this could put your website on the first page of Google as a so-called "featured snippet." That could result in thousands of hits from real people.
2. Analytics: These traffic bots monitor how many people visit your site, what pages they visit, how quickly your pages load, and more. This data could help sites like Google and Bing determine the value of your content.
3. Copyright: If someone steals your content and you decide to take legal action, you could get up to $150,000 in damages per infringement. Copyright bots help you spot the theft quickly and prove that you've been harmed.
4. Search engine optimization (SEO): Search Engine Optimization bots walk through each page of your website and determine what you're discussing. That helps sites like Google and Bing determine what search terms should trigger your pages.

Any or all of these tasks could be critical to your success.

Bad traffic bots

Programmers can use traffic bots to attack your site, steal your resources, or both.

These are six examples of bad bots:

1. Ad fraud: Traffic bots programmed to click your paid ads can drain your bank account. Experts say ad fraud like this costs companies about $35 billion annually.
2. Click fraud: Programmers can create scripts that repeatedly load a page on your website. This could make a web page seem more prominent than it really is. And if you have something embarrassing on your website (such as a press release you wish you hadn't written), it could make that page very visible.
3. Distributed Denial-of-Service (DDoS): Web bots can attempt to load your website repeatedly, and all of that traffic could overload your server and make your site unavailable. Sometimes, hackers ask for a payment before they will turn off their bots.
4. Scraping: Do you publish email addresses or contact information on your website? Scraping bots can pull that information and sell it to the highest bidder.
5. Spam: Website bots can fill out forms on your website. They could fill your inbox with nonsense messages or publish strange responses in your comments section.
6. Vulnerability: A programmer creates a web bot to crawl your pages and look for security problems. A report moves back to a hacker that can attack you.

Any or all of these bad bots could cost your company a lot of money.

How do you stop bad bots from harming your site?

It's not smart to keep all bots away from your website. You need some of the benefits they offer. But you should do all you can to keep bad bots away.

What steps can you take to block malicious bots?

Protecting your site requires a layered approach — here are the most effective methods.

Try these three steps:

1. Adjust your website settings. A tiny bit of code (a robots.txt file) tells bots what they should and should not crawl on your site. Use it to block some pages from inspection. But know that only good bots play by the rules. The bad bots will likely do whatever they want, regardless of whether you like it or not.
2. Hire a programmer. Ask an expert to walk through your reports and spot Internet Protocol (IP) addresses responsible for bot traffic. Block them manually with a filtering tool.
3. Use a tool. Bot management solutions use code to stop bad bots from causing chaos. These tools use machine learning, so they get more effective the longer you use them.

At Okta, we offer an excellent bot management solution. Our integration with Perimeter X keeps bad bots from threatening your business. Find out how it works.

Frequently asked questions

What percentage of internet traffic comes from bots?

Approximately 40% of all internet traffic originates from bots rather than human visitors, meaning a significant portion of the activity on any given website may be automated.

How can I tell if my website is being hit by bot traffic?

Common indicators include an unusually high bounce rate (bots leave pages within seconds), traffic that crawls your entire site page by page, and a sudden influx of visitors from geographic regions your site does not typically serve. Nonsensical comments or form submissions are also a telltale sign.

Are all traffic bots harmful?

No. Many bots perform essential functions, such as helping search engines index your content, monitoring site analytics, detecting copyright theft, and improving your search engine optimization. The key is distinguishing between these beneficial bots and malicious ones.

What is the financial impact of bad bots?

Malicious bots can be extremely costly. Ad fraud bots alone are estimated to cost businesses around $35 billion annually, and Distributed Denial-of-Service (DDoS) attacks can take your site offline entirely — sometimes with hackers demanding payment to stop.

Does a robots.txt file fully protect my site from bad bots?

While a robots.txt file is a useful first line of defense that signals which pages bots should avoid, it only works for compliant, well-intentioned bots. Malicious bots routinely disregard these instructions entirely.

What is the most effective way to stop malicious bots?

No single method is sufficient on its own. The strongest defense combines robots.txt configuration to guide compliant bots, manual Internet Protocol (IP) blocking to address known offenders, and a machine-learning-powered bot management tool that continuously improves its detection capabilities.

References

• Usage Statistics and Market Share of Google Analytics for Websites. W3 Techs.
• Bad Bots Now Make Up 20 Percent of Web Traffic. (April 2019). ZDNet.
• Copyright Infringement. Digital Media Law Project.
• CHEQ Report: Cost of Digital Ad Fraud Will Rise to $35 Billion Globally in 2020. (November 2020). Martech Series.