[SL Blogger Network: This post is not for consideration. Thank you!]
If you tried to visit Virtual Bloke between around 00:50AM and 11:30AM (GMT) this morning, you might have thought I’d given up on it, or forgotten to pay for the hosting (which is actually due in the next week, funnily enough) because it was simply not available.
In fact, the first I knew of the site downtime was when I checked my email this morning, and saw two alerts from the Jetpack Site Monitor: one at 01:09AM notifying me that the site was either offline or unresponsive, and another at 02:11AM notifying me that it was still offline or unresponsive. I was absolutely sparko in bed at that hour, and the first I knew of the issue was when I saw that email at around 10:50AM. I tried the site, and sure enough it was unresponsive.
So, off I toddled to log into my webhost’s cPanel and hit up the live support chat. There was an automated alert about emergency MySQL maintenance, but my server wasn’t listed, so I sat and waited for a response. In the meantime, I checked that I could still FTP into the server (I could) and twiddled my thumbs.
After checking with me to see if I’d done any work in the backend (I hadn’t – my last active work was to upgrade to the new WordPress a couple of weeks ago when it released – and I did mention that my installed plugins are all set to auto-update), the support dude did some scans but couldn’t figure out what had caused the issue. The site did, however, now load. He guessed it was possible a DNS hiccup, which… ehhh, I wasn’t so sure. Those are domain-specific, and two other domains that I run were absolutely fine, but I was just happy to have the site up and running again, so I didn’t question it…
… until my security plugin started screaming at me that I had CRITICAL ISSUES OMG!
Uh-oh.
It had found a change to one WordPress core file, and a publicly-accessible debug.log file. The latter was a simple deletion job (I figured the support dude had left it there) but the other was a concern. Upon comparing the changes, I found myself tumbling down one of those delightful rabbit holes (in the sense that “curiouser and curiouser” starts running through one’s head as more and more weirdness crops up) that can take up hours of your time.
Eventually, though, I did track it down, and of all things it was a check of my FTP files for ‘last modified’ that did it. At 00:49:19AM one of my plugins – the previously 100% reliable WP-Optimize – had automatically updated. And on my live traffic logs, the last visit to the site before it conked out was just five seconds before that, at 00:49:14AM. The next live traffic visit was at 11:36:18AM, and that was the support dude.
I hit up my plugins page and selected the reviews section for WP-Optimise. Hooboy, there are some unhappy people right now, complaining about critical errors, hundreds of websites down, all because of that plugin update.
So yeah. My apologies if the site was down when you tried to access it earlier today. Those of you who are grizzled old farts in SL (like wot I am) may remember this thing appearing on the website every Wednesday, as the whole Second Life grid went down for maintenance on that day every week way back when. And, while the banging on things in this case was kind of me figuring out WTF had happened after the fact, it still felt apt for the title of this post.
