Bots now account for 42% of Internet traffic, known Good bots account for 15%, and Bad bots are 27%. Bad bots perform malicious tasks like hacking and spam.
Table of Contents
What is a Bot?
A Bot is a software program that performs pre-defined tasks that are repetitive and automated on behalf of a human. The word is short for robot. They imitate human behaviour on a website and can perform these tasks faster than humans.
There are good bots and bad bots. Good bots perform tasks often associated with understanding a website or posts. They do this by crawling the website and collecting information. An example of a good bot is Googlebot.
Bad bots perform malicious and intrusive tasks like spam, hacking or damaging websites. They do this by looking for weaknesses and entry points to behave maliciously.
There are different types of bots:
- Web Crawler
- Malicious bot
- Social bot
- Chatbot
A Social bot imitates humans on social media sites such as Facebook and Twitter. A Chatbot will simulate a human and attempt to provide answers to chat-based help requests.
Good Bots
One of the primary examples of good bots is Web crawlers. Search engines typically operate these. They can also come from SEO sites that rate websites and gather information. Web crawlers can be called a spider or spiderbot or abbreviated as crawlers. The primary purpose of Web crawlers is web indexing.
Web crawlers identify themselves to Web-servers using the User-agent field of an HTTP request.
Spam bots and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawlers.
A list of good bots (largely Web Crawlers):
- Googlebot (Google) – Google has 92.47% of the search engine market
- Bingbot (Bing MSN)
- Slurp Bot (Yahoo)
- DuckDuckBot
- Baiduspider (China)
- Yandex Bot (Russian)
- Sogou Spider (Sogou. China)
- Exabot (Exalead, France)
- Facebook external hit (Facebook)
- Petalbot, Petal search engine (Huawei)
Decide their usefulness based on their relevance to the region you live in or the audience you are targeting.
Googlebot
Googlebot is the generic name for Google’s web crawler. There are two different types of web crawlers:
- a desktop crawler
- a mobile crawler
A desktop crawler simulates a user on a desktop, and a mobile crawler simulates a user on a mobile device.
A Googlebot must access a WordPress website and perform its indexing tasks. It enables Google to rank your site and include it in relevant searches.
You can recognise the Googlebot User-agent as:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
Other Bots (The Ugly, so to speak)
There are ‘Other’ Bots that fall into a grey area. Some people and software will classify them as good bots because they are not technically malicious bad bots. However, they are the products of independent companies, and they crawl your website and collect data for that company. A number of them are SEO software companies with ranking and keyword databases.
One aspect that differentiates these Bots is they identify themselves in the user agent field. However, they can behave in a very intrusive manner.
You may decide some of these as a problem because they crawl a website too frequently and consume server resources at the expense of proper site visitors. Some examples of these bots are:
- Ahrefs – SEO database
- Semrush – SEO, Ranking and Keyword database
- Monitoring.Internet-measurement
- Paloaltonetworks
- Serpstatbot SEO link crawler (Germany)
- Babbar.tech SEO (France)
- NetcraftSurveyAgent (USA)
- Censysinspect (USA)
- Intelligence Network Online USA
Bad Bots
Bad bots engage in an activity that is detrimental to a website. It includes a variety of intentions, including:
- Credential stuffing
- Web/content scraping
- DoS or DDoS attacks
- Brute force password cracking
- Inventory hoarding
- Spam content
- Email address harvesting
- Click fraud
The volume of bot activity itself can be a problem for a website. Consequently, you will need a mitigation and security solution to protect a website. The starting point is to be aware of what is happening by monitoring traffic of unusual requests and blocked attempts.
They will target any of the following:
- Any Contact Form and Registration Forms
- Login
- Wp-admin
- Other WordPress files
The first step is to become familiar with what Bad bots are attacking your website. You can view this via your hosting provider, Cloudflare or WordPress security plugins. Understanding what is happening and what they are targeting is critical. It may be quite confronting at first, seeing the actual number involved and the relentless nature of the problem.
Contact Form and Registration Forms
WordPress Contact Forms and Registration Forms will get targeted by spamming bots, causing large amounts of spam in your email account.
The simplest solution is to incorporate a reCAPTCHA feature in each form. A bot cannot pass these. reCAPTCHA is a free Google feature installed via a Google/Gmail account.
There are two types of reCAPTCHA to choose from:
- reCAPTCHA v3 verify requests with a score.
- reCAPTCHA v2 verify requests with a challenge
With reCAPTCHA v2, there is then a choice of:
- “I’m not a robot” checkbox
- Invisible capture badge. Validate request in the background. A floating reCAPTCHA symbol will show in the bottom right-hand corner of the page.
Bots cannot respond to a reCAPTCHA and fail at this step.
Login Attempts
One of the predominant Bad Bot attacks on a WordPress site is Login attempts. They will attempt to log in with a known name that could be the username:
- name of the website
- name of the author of blog articles
When a Bad bot attempts many login attempts, they refer to it as a brute-force attack. The WordPress login page is easy to find as it has the same standard URL on each WordPress site.
There are several ways various sites recommend protecting against malicious login attempts (Hacking):
- Have a strong password
- Limit the number of logins (need a plugin or security plugin)
- Two-factor authentication login (need another app)
- Replace any easily identifiable usernames, e.g. admin
- Use Cloudflare and create a rule to limit access to the login page (Cloudflare Account)
WordPress Folder Queries
Bad Bots attempt to attack a WordPress site by looking for weaknesses in different WordPress directories and files. Those regularly targeted are the:
- wp-admin folder.
The following are also particular targets of Bad-bot attacks:
- xmlrpc.php
- wp-content folder or
- wp-includes folder.
Monitoring Bot behaviour in these folders and defining rules in Cloudflare will mitigate some of these threats.
How to Protect your WordPress site from Bad Bots – recap
- The problem and threat from Bad Bot attacks are genuine and enormous. Ignore it at your own peril.
- Start by reading posts like this and educating yourself about Good and Bad Bots.
- Invest time in observing the bot traffic that is visiting your website. In the early days of your site, they will vastly outnumber human traffic. You will need a security plugin and Cloudflare to do this.
- You must be careful in controlling bot traffic and attacks on your site. You have to stop Bad Bot attacks while allowing Good Bots. If you accidentally block Googlebots, then Google will not index your website, which is deadly.
- You need to have an overarching security approach. It is not sufficient to just add a security plugin and think that is it.
- You will need to add protection at the WordPress application level, such as a security plugin or firewall. In many cases hosting providers will have some form of Bot protection.
- Some Security plugins can impact page loading times and slow down your site. Reviews for many security plugins provide feedback that they are only partly effective. Research the impact and effectiveness of any Security plugin selected.
- Using products such as Cloudflare is a game-changer in protecting against Malicious Bots. Implementing Cloudflare Web Application Firewall rules will stop most Bad Bot attempts at Cloudflare – they never get to your website.
- Implement Spam Protection using software like Google’s reCaptcha and a Plugin like Akismet from Automattic. There is a small page load impact with reCaptcha.
