The Terra Galleria Blog – QT Luong

Frequent visitors to terragalleria.com may have noticed unusual, repeated periods of downtime over the past month, as well as a gap in updates. The quick explanation is that the server hosting the site was hacked. Sure, this has happened several times in the past, but each time I had been able to figure out which vulnerability the hackers had used, which (criminal) activities they had planned, and shut them out. Not this time, which opened to door to multiple, escalating attacks which eventually forced me to decommission the server and rebuild a new one. To get a sense of the aggravation I faced, read on the long explanation.

Let’s begin by addressing the most obvious concern. There isn’t much to steal on the server. For safety reasons, we do not store credit card numbers, simply passing them to our gateway. High-resolution digital files are valuable, by only if you can license them, a task which is nowadays quite difficult even for legitimate image creators. To the hacker’s credit, they did not vandalize the site contents. They seemed to be interested in the server only as a platform to try and compromise other systems and get access to them.

It started about a month ago. A reader reported to me that his anti-virus software had flagged the terragalleria.com website. An investigation based on time-stamps (to look at recently modified files) quickly showed that in one of the files containing Javascript utilities, the following code was appended (I’ve included it as an image because as text it is harmful !):

If this sounds like Hebrew to you, rest assured that it was the same for me until someone on G+ pointed me to the following page, which explains everything. The code was obfuscated to make it difficult to understand what it does: it creates a iframe – a special invisible frame around the webpage – which points to a website that tries to infect visitors with malware, which I can only assume (I didn’t download it, and nor should you ever download software with no clear purpose) allows them to take control of the victim’s computer.

Although getting rid of this particular code was easy, finding out how the hackers were able to plant it remained an unsolved challenge, as there are so many different security flaws which can be exploited to gain control over a computer. Even systems from the US Department of Defense are not immune to hacking. Since I didn’t know the extent to which the server was compromised, the safest thing to do was to rebuild the site. However, I was going to travel the following week, so I didn’t have time to do much. I also hoped that the intrusion was limited – as had been the case in the past. I left for Alaska.

In search of the Aurora, we drove north of Fairbanks on the Dalton Highway, passing the Arctic Circle. Although there were absolutely no services of any kind on the 240 mile stretch of road until Coldfoot -making it the most remote in North America – the nearby village of Wiseman (pop 13), has two B&Bs open year-round. Because our rental car was marginally equipped to deal with -30F weather, instead of camping further north, we stayed there. Unexpectedly, they provided for a few hours per day satellite-based internet.

That’s when I learned that my server woes were getting worse. The hackers were trying to use the compromised server to hack into other machines on the network, using what is called a “brute force attack” which consists of automatically trying to log on by trying thousands of different passwords (the reason you shouldn’t use simple passwords). As a result, the hosting company took the server offline after I had failed to respond to their warnings. Back in Fairbanks, we huddled in the public library to benefit from a reliable internet connection. At first we found themselves locked out from the system, but after assistance from the data center technicians, we had the main (root) password reset so we could do administrative tasks. My friend then found one of the backdoors they left and removed it.

All was (almost) quiet for a few days, but the following week they tried again to do brute force attacks against other computers. I was not able to locate the script nor the security hole, but at least I was able to stop the activity by disabling some system utilities.

A few days later, I discovered that they were back to trying to propagate malware using an iframe. But this time, they used more sophisticated methods. They had covered more carefully their tracks, since a time-stamp based search didn’t yield anything interesting. Instead of just modifying a single text file, they managed to get the webserver to append a malicious iframe for all web pages and all the domains hosted on the server. Protecting our visitors is the priority, so upon discovering that the server was trying to spread malware, I immediately shut down all web services.

I looked all day long at various configuration files, yet remained stumped. Because I hold a PhD in Computer Science, it is sometimes assumed that I am a wizard with computers. But in fact, I was mostly a mathematician studying the geometry of visual perception of three-dimensional space. I learned very little about system administration in grad school – there was no world wide web yet. I decided to simply abandon the server. This was the right decision, as in the following week, the situation degrated further. Not content with changing all the server passwords, they reconfigured the server so that even after reseting the root password it was not possible for me nor the data center technicians to log on. In the while, I received complaints from system administrators of large, well-known companies, into which they tried to hack using the server. I eventually put an end to this activity by wiping out all server contents with an OS reload.

Upon returning home from Alaska, I had already begun to plan for a new replacement server – kind of like burning your house to the ground and rebuilding it to get rid of vermin. I was looking at more complex solutions involving virtualization and cloud computing. However the server was now down, and every such day costs us quite a bit in lost business, so I decided to rebuid again a simple and cost-effective dedicated server. I rent a box running Linux on a brand new Quad Core Xeon 1270 – 3.40GHz (Sandy Bridge), 1 x 8MB cache w/HT, 2 GB DDR3 1333 RAM, 2x 750GB SATA II HDD with a 100 Mbps uplink port and a 3TB monthly data transfer allowance.

Besides the hardware upgrade – which greatly accelerate searches and dynamic loads – the main difference is that unlike on my previous servers, decided to forego a control panel (I used Plesk before), mostly for security reasons. The data center technicians had pointed out to specific Plesk-related vulnerabilities. A control panel provides an easy-to-use graphical interface to all services on your server. Without one, you administer your server by editing text files and typing commands.

I found out during installation that while a control panel makes configuration simpler and more foolproof (no messing all your web services because of a single missing directive or a mistyped command), its main benefit is that it provides web-ready software. In my case, all the installation went smoothly until I noticed that PHP (an active component of web pages) would not communicate with MySQL (the database). Since they are such basic and popular building blocks, you’d think that they would work right out of the box in a distribution such as Red Hat Enterprise Linux (RHEL) 5, but this wasn’t the case at all. After trying many things, I re-installed PHP from source, but in the process this happened to break httpd. At this point I got fed up with fiddling around without real understanding. I hired the services of a server administration company.

They fixed the problem promptly, and I was happy to get the site up again, first in an emergency instance without many functions. However, as I began to set-up my backup software, I found out that the hard drives were not configured properly. I had ordered the server with two hard drives, with the intention to use the second one for backup. However, the hosting company set up the drives as an LVM (Logical Volume Manager) volume group, which defeated the purpose. They recommended that I reload the OS and restart from scratch, although there exists a risky procedure to remove a drive from the LVM. As it had taken me more than a day to get to that point, I wasn’t too keen on doing it over again, so I asked my system administrator to try the procedure. Sure enough, it failed, leaving the server in a state where it wouldn’t even boot.

It quickly became clear that I had no choice but to reload the OS. I thought that I might as well upgrade to version 6 of RHEL as older versions sometimes lack critical security updates. It turned out to be even worse than RHEL 5. PHP didn’t even work properly. After trying to fix it without success, the system administrator recommended that instead I reload CentOS 5. I was surprised, since CentOS is the free – and supposedly functionally identical – version of the licensed RHEL, but indeed PHP and MySQL worked out of the box. This doesn’t mean that all my scripts worked, since software upgrades forced me to update my scripts, but at least the basics were there.

Acting as system administrator for two weeks was more than I cared for, but thanks to backups I didn’t loose any data. I just wished hackers would be more considerate in choosing their targets. Many companies who run websites with less traffic than terragalleria.com have full-time system administrators, but this is essentially a one-person operation. I have found that one of the main difficulties – and sometimes source of frustration – in running a small photography business is that one has to wear so many hats. Over the years, terragalleria.com has grown to become, amongst other things, a fully-featured stock photography website and digital wallpaper subscription site. That it needed to run on its own server, which was at the root of last month’s troubles, is something I suppose I should be grateful for, as it had provided me my own path to full-time photography.