Cyberwar never ceases

A great deal of our work as Linux system administrators is related to security. Each server we maintain is bombarded on daily basis in a never ending cyberwar. Some of our customers (e.g. the website of the former Finnish Minister of Foreign Affairs, government websites, high profile political organisations and media sites) are obvious targets but statistics show that really, any device with a public IP address gets targeted by a steady stream of attacks.

Our global political system seems quite toothless on this matter; much due to the technical fact that it is next to impossible to track down the real source of a well executed cyber attack. In a case where the attacker’s IP address seems to originate from China, all we can really conclude is that China is probably the last place the attack came from.

Cyber attacks are also very cheap, and you most often can’t tell if your harasser is a government funded organisation or just a 13-year-old boy with a laptop in his parents’ basement. This is asymmetric warfare, because proper defence for cyber attacks is certainly not cheap or easy to establish.

Case example: attack on WP-palvelu.fi on September 3rd, 2015

Most of the time, cyber attacks are just a cost of doing business we need to withstand; but unfortunately sometimes effective attacks arise from the constant bombardment. Last week our own WordPress optimised cluster at WP-palvelu.fi got some extra heat from somebody trying to DDOS it.

A regular denial of service (DOS) attack is easy to tackle by automatically blocking the IP source of the attack, but a distributed DOS is much harder as the requests come from many different IP addresses, so our automatic rate limiters can’t kick in. This attacker was also smart enough to have bypassed our downstream caches by making unique requests every time with requests like these:

[03/Sep/2015:10:01:37 +0300] "GET /superdry4.php?tmpl/detail.tmpl?designer=Superdry%2Fshop.tmpl%3Fmysize%3DL%2CXL;designer=Superdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSuperdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DBazlen%20accessoires;art_id=531863;pg=2044 HTTP/1.1" 404
[03/Sep/2015:10:01:37 +0300] "GET /superdry4.php?tmpl/detail.tmpl?designer=Superdry%2Fshop.tmpl%3Fmysize%3DL%2CXL;designer=Superdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSuperdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSisterhood;art_id=106485;pg=81 HTTP/1.1" 404
[03/Sep/2015:10:01:37 +0300] "GET /superdry4.php?tmpl/detail.tmpl?designer=Superdry%2Fshop.tmpl%3Fmysize%3DL%2CXL;designer=Superdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSuperdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSpeedo;art_id=506899;pg=1719 HTTP/1.1" 404
[03/Sep/2015:10:01:37 +0300] "GET /superdry4.php?tmpl/detail.tmpl?designer=Superdry%2Fshop.tmpl%3Fmysize%3DL%2CXL;designer=Superdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSuperdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DNikita;art_id=480217;pg=82 HTTP/1.1" 404
[03/Sep/2015:10:01:37 +0300] "GET /superdry4.php?tmpl/detail.tmpl?designer=Superdry%2Fshop.tmpl%3Fmysize%3DL%2CXL;designer=Superdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DSuperdry%2Ftmpl%2Fshop.tmpl%3Fdesigner%3DPlomo%20o%20Plata;art_id=554941;pg=107 HTTP/1.1" 404

Using this method, the attacker managed to trigger a fairly heavy PHP heap to generate the WordPress 404 pages on our client’s website, thus maximising the impact of the attack.

Our first defensive action was to quickly take the targeted site offline, and then re-launch it on a separate, more powerful host that could better withstand the massively increased load. We also took other defensive steps, like trying to block some of the attacker’s IP ranges during the attack, which proved not to be as effective fighting against the DDOS.

Remedy in this case

When the dust settled and we had time to analyse the logs, we discovered the details of this attack. Using these discoveries, we made the decision to apply more of a white-listing approach to the HTTP requests our customers’ WordPress applications receive.

The most important step was to only pass through requests for PHP files that actually exist, and then deny all other *.php requests, returning code 404 error pages instead of bothering the WordPress instances at all. This change is very conservative and has already been applied to our development version, and will be deployed to all WP-palvelu.fi sites during the next round of system upgrades.

What’s next

In the longer run we’ll start to apply even more white-listing principles, which are possible thanks to the fact that our WP-palvelu.fi hosts only WordPress sites, and we can safely assume that the sites have certain structures.

We also plan to start caching 404 responses for up to 1 minute to prevent this WordPress 404 page generation attack, even though caching 404 pages is generally discouraged. We’ve also made it easy for our client to purge the downstream caches if the 1 minute cache is just too large a window for their operation. Related to this, we’re also developing automated scanning of our clients’ access logs that could detect anomalies like large spikes in traffic so that we can notice attacks before they show any visible effects.

We will be sharing all of our upcoming cyber-fighting techniques in future blog posts. Stay tuned!