The week the exam paper leaked

Written by Mat


This has been the most stressful week in nineteen years of running Iteracy. On paper, it should have been a fairly dull security upgrade, but it's turned into one of the biggest issues in IT history.

How security fixes are supposed to work

When someone finds a serious flaw in software that millions of websites rely on, they don’t announce it. They tell the people who make the software, privately. They write a fix quietly, and on an agreed day and time, they release the fix and the announcement together. Web hosts like us install it within hours, and by the time anyone with bad intentions has read the details, the door is already locked.

It works a bit like an exam paper. Everyone opens it at the same moment, so nobody gets a head start.

This time, the paper leaked

On Tuesday 20th October, details of two serious flaws in PHP leaked online before the fix was ready. PHP is the programming language that runs all our websites, and by most counts around three-quarters of the websites on the internet. The flaws were nicknamed PHP2Shell. “Shell” is techie shorthand for being able to type commands on someone else’s server, which tells you how serious they were.

So for about a day, everyone knew roughly where the hole was, and nobody could fill it.

We spent that day doing what you can do when you can’t fix the real problem. We switched off the feature the leak pointed at. We wrote firewall rules that might help. We checked every file on every website for signs of tampering. When the official fix came out at 3pm on Wednesday, we installed it straight away and checked every site again. We found no sign that anyone had got in, then or since.

That should have been the end of the story - a good news story, even.

The night from hell

Just after midnight, the bots arrived. By some estimates, most traffic on the internet isn’t people at all. These are automated bots: search engines, price checkers, AI scrapers, and a great many bots whose only job is to try every door on every website, looking for one that’s been left unlocked. Most of the time, a well-run server shrugs them off. To give you an idea of scale, on some of our websites we're blocking 95% of all incoming traffic because they're bots looking for vulnerabilities.

Bots clogged up our server faster than it could clear them, until there was simply no room left for real visitors. It’s a bit like a phone line jammed by robocalls. Nothing is broken, nobody has stolen anything, but your real callers can’t get through.

Why it was a long night, not a lost week

This is where the security work we’ve done over the last few months paid off. Not one of those bots got anything. Nothing was taken, nothing was changed, and no data was lost. And because we’d spent the summer building better logs, firewall rules and monitoring, we could see exactly what had happened. By mid-morning we’d traced it, and by lunchtime we’d made sure the same thing can’t jam it up for long if it ever happens again.

What’s next

This week has made us even more determined to stay right up to date. This year we've already released eight security updates for our content management system, and we're spending a huge amount of time and energy keeping your websites safe and running. 



Tagged under: Troubleshooting   Build a better website   Hot topics   Hosting   Security   Hacking  

Nice things people have said about us

"Iteracy have that great combination of excellent technical understanding and ability to communicate in everyday language!"

Caitlin Dean, Pregnancy Sickness Support