You’ve been hacked?
Don’t take it personally. Most websites are hacked randomly and with total disregard to your person, position, or title. It’s just business…
Kevin Mitnick once said, “The hacker mindset doesn’t actually see what happens on the other side, to the victim.” Born true a decade ago and still does today.
So why are you allowing your website to be “owned?”
Prevention is your best weapon, but most people seem oblivious to how easy it is to blend into the virtual masses and hide in the shadows online.
One method of blocking hackers involves blocking bots from even seeing your website. If a bot stumbles onto your website and is immediately repelled, you’ve just scored a big win security-wise. Don’t be a target, be a bot killer!
How to block bots connecting from other countries
Wish to “deny” visitors from other countries access to your website, try my favorite country blocking plugin IP2Location Plugin. See the picture setup note at right.
Other plugins useful for blocking bots from other countries or blocking connections from other countries are the Country IP Specific Redirections plugin and the Admin Block Country plugin.
How to prevent bots from visiting your website
Blocking bots is crazy-simple to do. First you’ll need to know how to locate your .htaccess file is a file most often found in your public_html directory. It’s just a text file. Don’t be scared… :).
Most web hosts have a basic File Manager, where the list of your website’s files can be viewed, along with an “Edit” button to make and save changes.
With your trusty File Manager or FTP File Editor in hand, you are less than one minute from being done with this tedious task.
Below is the bot blocking list I use. Feel free to edit it to your heart’s content:
Within the bad bot list you’ll see bot names like “1Noonbot.” If you are a 1Noonbot fan then remove from list, save file, and you’ll be all set.
Likewise, if your site is being hammered by a bot, just add your bad bot to the beginning of the list. For example:
RewriteCond %{HTTP_USER_AGENT} ^.*(1Noonbot|
with
RewriteCond %{HTTP_USER_AGENT} ^.*(Reallybadbot|1Noonbot|
In summary:
- First, see the list using the above link.
- Copy it to your computer’s memory (Control-C).
- Next open your .htaccess file via your editor.
- Paste this list at the very top of your .htaccess file (Control-V).
Save.
Done!
Seriously, you are done.
Check your website to see if it’s loading properly and without error. If so, then you’ve just built the poor man’s bot firewall in all of one minute flat, and can sleep better at night knowing you are super smart, accomplished, and can say with pride, “I am so smart!”
61 Comments
This information is impressive; I am inspired with your post writing style. Its a wonderful post and very helpful, thanks for all this information.
Found a new Bot that drained > 6GB of data last month:
Mozilla/5.0 (compatible; Adsbot/3.1)
Coming from:
216.18.204.151
173.231.59.207
173.231.59.197
I added |Adsbot| to to my htaccess
Found a general USER AGENT blocking valid traffic:
|Link|
Users accessing the site using the LinkedIn built browser: LinkedInApp
Are getting blocked – User Agent below;
Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [LinkedInApp]
“LinkedInApp” is triggering |Link|
Also having issues with og:image being picked up in content when shared on LinkedIn and their post inspector working https://www.linkedin.com/post-inspector/
Jakarta Commons-HttpClient is apparently used as user-agent by LinkedInBot. There also seems to be a block against the word link / Link .
Excellent Sherlocking!
Both bot entries now removed.
Found that the ‘Indy\ Library’ block was stopping transactional information being send back to ecommerce sites using Barclaycard EPDQ and GlobalIRIS.
Maybe is a coincidence or my mistake. Let me make a deeper review. If is your list, I will update with more information. Thanks for the list
by using the list, i can not to use vaulpress for the backup. How can we fix it?
thanks
I’m not aware of there being issues with Vaultpress.
Please provide them a link to my bad bot list and ask them if there is an entry I need to remove which relates to the Vaultpress service.
Hi, How do you input the Bot code if you are using a security plugin like ithemes? The ithemes code is at the top of the htaccess file with a “do not move this line” message. Best, H
iThemes Security already had a bad bot list option.
Otherwise, with a bit of trial and error, just paste one line of the list into your .htaccess then check your site, etc.
The setup will become more obvious with a bit of trial and error.
So I have the blacklist feature enabled in ithemes, but the bot list above looks much longer. I’m guessing if I add the list shown in your pastebin, it is a lot better than only using the ithemes blacklist right?
Yes, my list in Pastebin is more current.
You may uncheck/remove the bad bot list option in the plugin settings, then copy/paste my list in place of the prior list.
Found another overly general USER AGENT blocking valid traffic:
|DA|
Users accessing the site using:
Device: iPhone
Browser: Facebook
Network: Vodafone
Are getting blocked – User Agent below;
Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_1 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C153 [FBAN/FBIOS;FBAV/153.0.0.53.87;FBBV/84268146;FBDV/iPhone9,3;FBMD/iPhone;FBSN/iOS;FBSV/11.2.1;FBSS/2;FBCR/vodaAU;FBID/phone;FBLC/en_GB;FBOP/5;FBRV/0]
“vodaAU” (network ID) is triggering
Thank you.
And removed from my list as well.
I added the bad bot list to my .htaccess file, but then UptimeRobot started to consider my site down. I tracked it down to like #42 which starts with “42.RewriteCond %{HTTP_USER_AGENT} ^.*(Nessus|NESSUS::SOAP|nestReader …” but I can’t figure out what in that line is blocking UptimeRobot.
The UptimeRobot user-agent is: “Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)”
Any ideas?
You should be able to ID the culprit through a trial and error process. Delete large chunks of the user agents link, then add back smaller pieces until you find the offending agent.
I would be curious to hear your findings as well.
Firstly – clicking Social login loses everything that has been typed in this window – probably want to provide a warning about that… I will now retype…
Found another overly general blocker:
|RMA|
This can be found when using Instagram Browser on iPhone- the text that is triggered: [ gamut=normal ] i.e. noRMAl
Here is the full User Agent that is blocked:
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 Instagram 10.32.0 (iPhone5,4; iOS 10_3_3; en_AU; en-AU; scale=2.00; gamut=normal; 640×1136)
Done. https://pastebin.com/544Wqexn
|archive.org_bot|
This is a tricky one – It is effectively a bot but it is a useful bot – I like it to keep a history of my sites in case I break something – it’s always good to have a backup to go to in disaster situations.
True. And it’s easy enough to remove.
Just view your .htaccess and remove that bit of text and you should be set.
Found another problem string:
The referrer: COMODO DCV
– Comes from Comodo when validating the IP of the requesting server for SSL certificate
It is being blocked by |comodo|
Is not in my current bat bots list. Agreed, you should remove the comodo related line.
Found another one causing issues.
The referrer: Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) GSA/24.1.151204851 Mobile/14D27 Safari/602.1
– Comes from Apple iPhone Google App
It is being blocked by |gsa|
Nice catch. Now removed.
Hi – thanks for this. I’m a novice and i’m wondering when you say, paste it in before rules, is that totally before anything else, or between the #BEGIN Bad Bot Blocker and #END Bad Bot Blocker, or before the#BEGIN WordPress. Sorry to be so clueless. Thanks.
Before anything else should work fine.
Thanks Jim! I did it, nothing exploded and everything still works. Another new person question I’m afraid….does this mean that any sites I run from the cPanel are protected now or just my main site?
It will not protect other installs sharing the same public directory.
Okay, so I need to copy it into the ht.access file for each domain. Thanks for answering so many questions. I appreciate it.
Just found what was blocking images linked in emails:
1. Gmail referrer: “Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)” was being blocked by: |PROXY|
2. Microsoft Outlook referrer: “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; Media Center PC 6.0; Microsoft Outlook 16.0.7167; ms-office; MSOffice 16)” was being blocked by |OsO|
|NG|
|PROXY|
|OsO|
are all probably a little too broad.
|NG|
|PROXY|
|OsO|
All removed.
Thank you.
Hi, thanks for the info.
I have added the list in my .htaccess and now i get the error 200 on “google structured data testing tool”.
Hi,
Are you sure the 200 “error” is an error?
HTTP 200 normally means the connection is OK on the HTTP level. This is a good thing.
NOTE: This code is great and I use it on most sites HOWEVER, it has a bad bot listed as |NG| – which, can end up blocking a lot (any visitor with a referrer ending in ng (ing).
Took me a while to figure why my monitoring wasn’t working…
Removed from bad bot list.
Thanks for sharing. But this solution won’t stop smart bots.
They can spoof HTTP header and randomize IP addresses.
thanks for sharing…
Hi, thanks for the Useful list; but I’ve got a Doubt – Sorry for the Dumb Question but I’m just a Beginner – Does using this List Prevent other Bots like Googlebot & Bingbots from Indexing my Site?
Thanks.
The list in no way blocks Google.com search.
Just a FYI – the latest Blacklist can cause errors in WP if you use PHPbay to list eBay items.
The line –
RewriteCond %{HTTP_USER_AGENT} “^$” [NC,OR]
will cause a Forbidden error –
You don’t have permission to access /wp-content/plugins/phpBay/forms/form.page.header.php
on this server.
Nice catch. Thank you. Luckily the fix is as easy as removing that line.
THanks for this list, will give it a try and see how it works
I’m having an issue with one bot in particular which is not a malicious bot, but rather a bot used by an SEO tool. The name is MJ12bot from MajesticSEO. I can see it here:
RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC,OR]
I don’t even have the blacklist feature from hackrepair enabled on one site and it is still being blocked. Is there a work around for this? Can I edit this one line out somewhere?
Thanks
This bot is listed within my block list. If you do not wish to use my full block list then adding this one line should block that bot nicely:
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC,OR]
Sorry, I meant I need to unblock it, not block it.
If you want that bot to visit your site then just remove the line.
One of the drawbacks of using the list is that some hosting companies are now recommending disabling the wp-cron mechanism and using real cron jobs to trigger wp-cron instead. The recommended method I’ve seen is to wget cron.php and wget is one of the user agents disabled when you use the hackrepair list in iThemes Security. Cron never runs, and with the options recommended, you don’t see any errors.
I’ve stopped using the list because I don’t want to discover that my sites haven’t been backed up for months because the user agent I’m using to run the jobs has been added to the list.
wget is not part of the latest list (though it was last year).
If it is, then it’s a simple matter of removing that line from the .htaccess file to resolve.
If I’m misunderstanding your question please reply back and I’ll do my best to better answer your question.
Below is copied from my .htaccess file. Since the “RewriteEngine On” is already there, I commented out that
line from the list I copied from your site. Will this still work effectively? Is there any way for me to tell if the Bad Bots are being blocked?
Thanks
# BEGIN iThemes Security
# BEGIN Tweaks
RewriteEngine On
# Begin HackRepair.com Blacklist
# RewriteEngine on
# Abuse Agent Blocking
The only way you can tell if bad blocks are being blocked is via your server or account log file.
Hello, thanks for the list. I´m using I themes security, in wordpress. But I have some issues.
Since a couple of months I have really a lot of bad logins attemps., I have already ca 400 bad ip adresses.
1 Is this normal?
since yesterday there a 30 new ones. I have to put them all manually on the banned hosts list.
2.Do you know if there is a way to do this automatically?
3. Are you interested in the list?
I never got an answer on this in the wordpress forum.
thanks
greetings Marga
Receiving “many” bad login attempts?
That’s awesome!
It just means the plugin is working. Personally I would ignore these and give up blocking IP’s.
The plugin is doing its job in blocking the bad bots for you, so worrying over what IP attempted to login is totally unnecessary- well, unless you enjoy worrying (…I’m not judging – to each his/her own). 🙂
Ok, thanks, yes this makes sense, yes and easy is right…. Have a good day.
marga
How do you know you’re site is being spammed by bots? like for example I have a wordpress site and the iThemes Security (formerly Better WP security) plugin.
Well, that is a good question. Generally you can tell the difference between a person or a machine posting trash comments. As for whether people can write badly and post totally not relevant stuff as well–I suppose that’s true too. 🙂
Will using the bot blocker slow the server down?
I have not personally experienced slowness with any method of bot blocking I’ve tried.
Hi, the WordPress Plugin iThemes Security (formerly Better WP security) is offering your bad bot list saying: “As a getting-started point you can include the excellent blacklist developed by Jim Walker of HackRepair.com.”
What I was wondering is that are any drawbacks associated with blocking (bad) bots. I’d appreciate a detailed response/explanation.
Thanks.
I have received virtually no complaints over the past two years. The bad bot list is really quite minimal, blocking only the most well known and most egregiously aggressive bots.
I have been using the Wordpress iThemes Security plugin for months, but recently it appears that this option, after a lot of trial and error, is causing our site to not load json content appropriately. Do you know if/why your bot ban would be doing this now?
Our server is hosted by SiteGround, if that makes a difference, and the json content, which the json file is coming back NULL, is being prepped via a for loop and json_decode();. It work just fine when your plugin option is disabled, could our dedicate IP be no your ban list? We also use custom cron schedules, and have disabled wp_cron. Thanks for any advice you may be able to offer.