The week the exam paper leaked

This has been the most stressful week in nineteen years of running Iteracy. On paper, it should have been a fairly dull security upgrade, but it's turned into one of the biggest issues in IT history.

How security fixes are supposed to work

When someone finds a serious flaw in software that millions of websites rely on, they don’t announce it. They tell the people who make the software, privately. They write a fix quietly, and on an agreed day and time, they release the fix and the announcement together. Web hosts like us install it within hours, and by the time anyone with bad intentions has read the details, the door is already locked.

It works a bit like an exam paper. Everyone opens it at the same moment, so nobody gets a head start.

This time, the paper leaked

On Tuesday 20th October, details of two serious flaws in PHP leaked online before the fix was ready. PHP is the programming language that runs all our websites, and by most counts around three-quarters of the websites on the internet. The flaws were nicknamed PHP2Shell. “Shell” is techie shorthand for being able to type commands on someone else’s server, which tells you how serious they were.

So for about a day, everyone knew roughly where the hole was, and nobody could fill it.

We spent that day doing what you can do when you can’t fix the real problem. We switched off the feature the leak pointed at. We wrote firewall rules ready to switch on. We checked every file on every website for signs of tampering. When the official fix came out at 3pm on Wednesday, we installed it straight away and checked every site again. We found no sign that anyone had got in, then or since.

That should have been the end of the story - a good news story, even.

The night from hell

Just after midnight, the bots arrived. By some estimates, around half of all traffic on the internet isn’t people at all. It’s automated programs: search engines, price checkers, AI scrapers, and a great many bots whose only job is to try every door on every website, looking for one that’s been left unlocked. Most of the time, a well-run server shrugs them off.

Bots clogged up our server faster than it could clear them, until there was simply no room left for real visitors. It’s a bit like a phone line jammed by robocalls. Nothing is broken, nobody has stolen anything, but your real callers can’t get through.

Why it was a long night, not a lost week

This is where the security work we’ve done over the last few months paid off. Not one of those bots got anything. Nothing was taken, nothing was changed, and no data was lost. And because we’d spent the summer building better logs, firewall rules and monitoring, we could see exactly what had happened. By mid-morning we’d traced it, and by lunchtime we’d made sure the same thing can’t jam things up for long if it ever happens again.

What’s next

This week has made us even more determined to stay right up to date. This year we've already released eight security updates for our content management system, and we're spending a huge amount of time and energy keeping your websites safe and running.