UPDATE – April 2020: It’s not escaped my notice that this old, old, old post from 2015 has been getting increased hits in the light of the new world ‘lockdown’ caused by COVID-19 (aka: Coronavirus). Please note that—if you’ve come here after a search for ‘lockdown’—then this is not the blog post you are looking for. This post details how I dealt with a large attack of spammers and hackers, back in 2015.
It may (or may not, depending how much you care!) have come to your attention that I’ve been bloody quiet over here for a couple of months. The reasons for that are twofold:
- Life happened
- Something far worse and more annoying happened
I’m going to throw this behind a cut, because it’s long and complicated. However, if you self-host your own WordPress blog (ie: you use your own domain and your blog isn’t hosted at wordpress.com) then you NEED TO READ THIS.
Follow me.
The first inkling I had that something was wrong was, well, how fucking s l o w the backend of the blog was for me. Honestly, the wp-admin pages were crawling. Sometimes my dashboard would take two minutes to load, and often it would time out. I figured that, since I’m on a shared server my site was hammering the cpu use a little bit and my webhost was throttling it.
Now, given that I was already running a caching plugin (W3 Total Cache) and serving the site through a CDN (Cloudflare) at my webhost’s request, I was scratching my head as to why it might still be causing problems. So I started digging a little deeper. And I didn’t like what I found.
Sometime toward the end of December, my regularly-scheduled Wordfence security scans started showing these:
Yeah. Big “Oh shit!” moment when you see that kind of thing. The malware URL only popped up two or three times, but my scans run twice daily and I was getting two of those suspected ‘malicious executable code’ warnings in my cache files every single time. Of course, I deleted them every single time, but that’s just a sticking plaster for the issue. I needed to find out what the fuck was happening and where they were coming from.
My chief concern was that my site may be compromising visitors’ computers, so my first port of call was the Sucuri Site Check. To my relief (and I ran this scan twice a day, every single day from the moment I first noticed this problem) my site was clear:
So, if you’ve visited Virtual Bloke recently, you have no need to worry. Nothing will have made its way onto your computer. (I do hope you run regular antivirus and anti-malware scans anyway though, regardless of what you do on the interwebs. It’s a dangerous place out there…)
They were always in my cache, and that meant I had to stop caching so I could find out what (if any) of the actual files on my site were potentially-infected. Now, if you’ve ever tried to remove W3 Total Cache, you’ll know that it integrates so deeply into WordPress that it’s a bastard to remove. In short, you have to:
- Enter W3TC settings and uncheck all the ‘enable’ options, then save all settings
- Go to your Plugins page and de-activate the plugin. Then delete it.
- FTP into your server and delete (if they’re there) the advanced-cache.php, db.php, and w3-total-cache-config.php files. You may also have to delete object.php
- Still on the server, delete the folders for: wp-content/cache, w3tc, and w3tc-config
- Download your .htaccess file and check it for any additions by W3TC. Remove them and re-upload it, if you find any.
- Download your wp-config.php file and either remove the line “define (WP_CACHE’, true);” or change it to ‘false’
So yeah, that’s a lot of hassle just to completely remove one plugin. I will say, though, that unchecking all of the ‘enable’ options, de-activating the plugin, and then deleting it (in that order) meant that most of those other things were already dealt with. I didn’t have to edit .htaccess (the text had been removed by the de-activation process), and the w3tc and w3tc-config folders had been removed from my server. I did have to remove the wp-content/cache folder and edit my wp-config.php file, though.
Once that was done, my cache was empty and I could run another Wordfence scan. Good news: not a single problem found. That went for the next scan and the next, etc. So the cache had been the backdoor into my site, and the suspect items weren’t part of the site itself; they’d been added there somehow. However, I wasn’t done yet.
The next thing I did was to force log-out anyone currently logged into my wp-admin, including myself. The way to do that is to edit wp-config.php and change your keys. The WordPress Key Generator will give you a new random set of keys every time you click it, and you can overwrite those in your wp-config.php file with those. This will force everyone currently logged into your blog to be logged out. Now, I’m the only person who has an account on my blog, but there was a slim possibility that some hacker might have been able to create a new user and obfuscate it with code so that I wouldn’t see it. He might have got in through that backdoor and once in the only thing to get him out would be changing the keys to forcibly log him out.
Now you log back in and change your password. For good measure you also change your FTP password, and the CPANEL password for your webhost.
More scans, and now everything seems fine. The repeated injection of those suspect files was in the cache folder and that’s now gone. So now it’s time to analyse stuff, and oh boy did I get an eye-opener.
I run the Bad Behavior plugin which tracks all requests to the blog and blocks dodgy-looking ones. It doesn’t catch them all, but combined with my other security plugins it does a good job. I regularly see scans like this:
At 08:53:04 he lands on the site. 0.01 seconds later he’s scanning for a list of administrator accounts. Another 0.01 seconds later he’s hitting the login page. Good job I have that locked down in Wordfence, huh? Bad Behavior let him through (he could, after all, be me), but Mr Greek Wannabe Hacker didn’t go any further.
As an interesting sidenote, the minute I removed W3 Total Cache, the backend of my site sped up incredibly, running at full speed again. Hrmm…
I then checked my Cloudflare account. I run the pro version, which includes a WAF (Web Application Firewall) which, in their own words:
CloudFlare’s Web Application Firewall stops real-time attacks like SQL injection, cross-site scripting (XSS), comment spam and other abuse at the network edge. Default settings include coverage for the OWASP core vulnerabilities.
And ahahaaaaa, here was my problem. These are the attempts blocked by Cloudflare’s WAF:
Not only was I being subjected to brute-force attacks, there had been almost two hundred attempts to drop a DDoS bot. Now, in case you don’t know, the WordPress platform – purely because of its popularity – is the perfect place for a DDoS botnet. Have a read of this article for more info. It states that trackbacks and pingbacks are the main way that botnets operate through WordPress.
How many of you have those enabled? Yep, same as I did. It’s a courtesy to other bloggers we might mention in our posts, plus it’s nice to see when we’ve been mentioned by others. But it’s also a way for hackers to use your site to bring down other websites without you knowing a bloody thing about it.
I knew I was going to have to disable those, and oh fuck it all, I had almost 200 posts on the blog. Luckily, there’s a quick way to do it.
1. Navigate to Posts > All Posts in your dashboard sidebar
2. Check the button at the top to select all the posts listed on that page.
3. Select ‘Edit’ from the ‘Bulk Actions’ menu:
4. Click ‘apply’. When you see the top of the page roll down to show bulk edit options for all of the posts listed on that page, look at the right hand side. You’ll see this:
5. Under ‘Pings’ select ‘do not allow’ and click ‘Update’
6. Repeat this for every page of posts until you’re done. (I had 10 pages to get through, and it was done in less than two minutes. Easy.)
Check any random post on your blog, and at the bottom you should see ‘allow trackbacks and pingbacks on this page’ is unchecked. If you don’t see that option, look at the top-right under your admin toolbar. Pull down the ‘screen options’ tab and check ‘discussion’.
Now, go into Settings > Discussion in the dashboard sidebar, uncheck the first two options, and save your settings.
Check any pages that you might have on your blog, too, and make sure those don’t have trackbacks and pingbacks enabled.
If you want to know how many times people you don’t even know about are attempting to leave trackbacks on your blog, this will give you an idea. Look at the number of blocked hits on the right.
And you wonder why I block China and Ukraine from accessing my site…
Just out of interest, I had downloaded two of those suspect files that Wordfence had found. They turned out to be .php files, and when I looked at them (use the free Notepad++ text editor for viewing and editing .php files, if you need to) I found a BIG RED TRAFFIC LIGHT:
Believe me, you do not ever want to see something like that in a .php file. It’s bad enough that – had I typed it here instead of showing you a screenshot – your antivirus software wouldn’t have even let you load this page. It would have told you that it contained unwanted malicious code.
My Cloudflare dashboard will give you some idea of the amount of attempts I’ve been dealing with. Remember: I didn’t post at all during December, and this is my first post in January. The site has been very quiet, and normally when that happens a site’s traffic slows down considerably.
Not mine.
- Dec 30th spike: 7,187 hits
- Dec 31st and Jan 1st spikes: 10,063 and 10,051 respectively
- Jan 1st second spike: 10,676
- Jan 2nd spikes: 9,599 and 10,602 respectively
January 3rd was when I first noticed what was going on, and look at the traffic. It’s moving back down to what it should be for a blog that hasn’t posted in over a month.
If you’re concerned about your own site my recommendations are:
WordPress plugins
- Wordfence
- Bad Behavior
- Anti-Malware and Brute-Force Security by Eli (and enable both the brute-force and Revolution Slider protection options)
- Cloudflare (if you have a Cloudflare account)
Also:
If you log in with the default ‘admin’ as your username, CHANGE IT! This is the most-targeted account on all WordPress blogs, because so many people don’t bother creating a new one. Create a new Administrator account with a different name (you’ll also need to use a different email account for it) and log in as that user. Then, delete the old ‘admin’ user. Make sure you have no user called ‘admin’ in your user list.
Check your site with Sucuri Site Check and do it regularly.
Learn how to harden WordPress. Make it as hard as you can for the bastards to get in.
And get yourself a free account at Cloudflare. Change your DNS records (check with your webhost how to do that server-side) to point to Cloudflare’s DNS, and all traffic will route through Cloudflare first. This will give them a chance to head off any suspicious traffic before it reaches your site. They’ll also cache your site and keep it online, even if you’re under heavy attack. A paid ‘pro’ account will give you the added security of the Web Firewall, which will stop the really nasty stuff.
Believe me, what I’ve had to do has been a nightmare. I thought I had my site locked down pretty hard, but it only takes one sneaky little backdoor in a cache folder to cause a fuckton of problems that it’s taken me three days to sort out.
Virtual Bloke is (and always has been) safe-to-visit. At no time have you been compromised if you’ve visited the blog (the number of scans I ran at Sucuri, and also the Google Site Checker have made me certain of that). However, I always recommend running a good antivirus scanner at least once a week; one with realtime protection. You don’t need to pay for one if you don’t have the funds. You can use Avast or AVG or any of the other recommended free AVs out there. Set it to auto-update and auto-scan and once a month run a boot scan with it, for good measure.
Make sure, too, that you run a decent anti-malware scanner. Malwarebytes Anti-Malware, or SuperAntiSpyware, or Spybot Search and Destroy are all good, and free.
Keep your site safe, and keep yourself safe. It’s like the fucking Wild West out there.
UPDATES
Jan 7th – Here’s the big catch by Cloudflare’s WAF yesterday – a null byte PUT request from some dude in the Russian Federation. nyet.gif indeed:
The WAF also blocked two SQLi attempts.
And overnight I had someone using a botnet of zombie computers all around the world to scan for any way into my admin pages. (These images aren’t in any order. They came at intervals over the course of about half an hour, so I screencapped them, cropped the rest of the stuff out of the way, and chucked them onto a composite image.)
Note all of the options he tries. ‘Old’ is an obvious one, as auto-updates for WordPress can leave your old – and unsafe – installation still on the server, in a folder called ‘old’. If you have WordPress set to auto-update, I strongly suggest that you FTP into your server and make sure you don’t have any .old versions of your blog there (Eg: if your blog database is ‘myblog’ then you’d see your current installation at ‘myblog’ and your outdated installation at ‘myblog.old’). Delete any .old folders you see there. They are usually unsafe, un-patched versions and are a prime way for hackers to get onto your server. Once they’re in that old installation they can attack any other installations or sites that you can see in your FTP client.