Hetzner breach – 40,000 customers affected! Passwords stored in clear text and banking details leaked…

Let’s start off with the most obvious: Local IT websites such as MyBroadband do not cover one of South Africa’s largest ISP breaches with objectivity and without bias. What will you do if your core business is to report on South African IT news, but you host with the ISP which just has been hacked and to make matters worse, the same ISP is a frequent advertiser and Gold sponsor to various conferences (i.e. MyBroadband Cloud & Hosting conference in May 2017): Nothing or very little – perhaps just rehash company announcements, and not try to poke to much about asking some serious questions.

We should assume that all our customer data has been exposed. While we’re able to see where and how the data was accessed, there is no way for us to ascertain how the exposed data will be used.

The other ethical dilemma comes in, when the leak is known to insiders and then published to news before customers could be alerted.

What data was compromised and how?

According to Hetzner, their proprietary hosting management tool konsoleH was compromised via a SQL-injection attack which then lead to the access of and subsequent leak of over 40,000 ISP customers details:

• Customer details (name, address, government identity number, telephone numbers and email addresses)
• Domain names and FTP passwords in clear text
• Full bank-account details in clear text

While the main konsoleH password was stored encrypted, the company suggested to reset all passwords. While 40,000 ISP customers does not sound like a lot, just remember that those customers will run websites as businesses and the Hetzner breach would have allowed the attacker to also access any of those 40,000 ISPs websites and databases due to the FTP passwords being stored in clear text.

Hetzner Germany had a similar breach in 2011 and 2013

Tobias Huch found a similar vulnerability with Hetzner Germany several years back and that vulnerability also applied to Hetzner South Africa:

If the recent local Hetzner attack is due to the previous vulnerability not being patched or is something new is unclear. What is clear and comparable to the Hetzner Germany incident is that German IT publication Heise Onliine also did not cover the incident broadly as Hetzner was a key sponsor and advertiser.

We are all vulnerable, but how you deal with a breach counts

No company is immune to attacks. It is the nature of IT and requires constant focus to attempt to be vigilant and have the necessary processes in place to protect customers when a breach occurs.

While I have great sympathy for small companies being exposed to attacks, an ISP of the size of Hetzner should have the necessary security and mitigation protocols in place to deal with incidents properly and have absolute transparency with customers. What happened next was puzzling:

IT websites reported the breach before customers were informed

Hetzner was aware of the issue since at least 9am on 1st November 2017 as per Emergency Maintenance notice:

 

Customers affected by the breach where informed on 1st November via Twitter at 7:34pm.

MyBroadband reported on this at 06:08pm and Hetzner published a high-level incident minutes before it online. Customers received email notifications from 6pm onwards.

While for most an incident of this magnitude would create complete chaos in an organisation, this would be fairly “standard protocol” for an ISP, since attempted attacks, DDOS and other intrusion attempts happen all the time. What follows next is even more bizarre.

We will regenerate all your passwords and make them available as a FTP download

Most customers only found out about the breach on the 2nd of November when looking at their emails or had noticed the incident via Twitter. Let’s put in perspective that a single ISP customer can be a reseller who in turn manages or hosts websites for other people. Manually resetting a large number of accounts is time-consuming, but Hetzner comes to the rescue:

Let the above sink in for a moment: Hetzner thinks it is perfectly okay to bulk-reset passwords and then make them available via insecure FTP-download? How will this help a customer? The newly generated and random passwords are stored on FTP-servers in clear text. Wouldn’t you as the affected customer then want to reset those passwords anyway?

Why mention MasterDeeds and the leak of 60m South African identities?

If you missed the background about MasterDeeds and how South Africa’s entire citizen-database of identity-numbers and names (similar to leaking social security numbers in the USA) leaked, you should read my background story “Master Deeds leak contains 14 million records of children“.

I personally found it strange for Hetzner to mention this, especially since Hetzner hosted the server with the masterdeeds.sql.gz-file for a customer:

When Hetzner was asked by a reporter to take the file down, Hetzner allegedly did not assist – when I asked them for feedback during the Master Deeds leak, I got the following response:

It is unclear how long the Hetzner SQL injection issue was present and if the clear-text passwords had anything to do with Master Deeds, but it is certainly worrisome to have two leaks occur within two weeks with Hetzner South Africa being in the middle of it.

Previous Hetzner customers are also affected by the leak and might not even know

One would think that an ISP’s billing engine would be “decoupled” from the administrative interface and hence billing information would be stored elsewhere. In the case of Hetzner it went a step further that even customers who had terminated their accounts years ago had their data leaked:

Yes. As a standard procedure, Hetzner does not delete information when a customer terminates their service with us. However, should a customer terminating their service with us specifically ask for their information to be removed, we do so without question.

It is very likely, that previous Hetzner customers could not be contacted via email, since they probably hosted their emails with them and possibly felt it was not necessary to update Hetzner with a change of email after terminating their account.

This then leaves a number (no-one knows how many) customers who could not be notified about the breach and will have their banking details exposed which can be used for identity theft or running rogue debit orders – something which is quite common in South Africa as debit order requests are hardly ever validated by banks.

The Hetzner CEO writes an open letter to the customers and it gets worse

Amidst all the drama and uncertainty, Hetzner’s CEO publishes a letter to customers on 3rd November. With 3 days lapsed, I would have expected that the company has now employed a forensic team and they have consulted on a proper mitigation process and have now put plans in place to take this forward. Judging from the letter below it seems otherwise:

 

Let’s go over some of the key-points:

I would like to personally assure you that we have addressed the breach and are working around the clock to identify other similar vulnerabilities.

Wait? What? Other similar vulnerabilities? Are there more? Maybe Hetzner should explain how their Hetzner SA’s version of konsoleH differs from Hetzner Germany. And if so, does the vulnerability exist on both or only locally. And if only locally, is it the same one from 2011/2013 or a different one?

Due to the breach, we must unfortunately assume that our customers’ data has been compromised.

Perhaps a bad choice of words, but it is obvious that the data leaked. Hetzner has yet to disclose when first accounts have been compromised or when the first recorded incident occurred (provided that Hetzner’s data-retention policy stores access information for an extended period of time). While it is easy to change passwords, it is extremely difficult to change banking details, but would be my first priority as the most fraudulent actions will center around the leaked Identity numbers (comparable to US social security number) and the bank account information.

Why have we been storing FTP and database passwords in plaintext? So that our support team could assist our customers by having this information on hand.

As a secure hosting provider you store passwords in plain-text so that you can assist customers with access issues? The computer industry has literally written books about password policies, self-help reset mechanisms, security tokens, two-factor authentication etc etc.

In short: There is no reason in 2017 to store a password unencrypted. What made matters worse, is that Hetzner has not learned from the mistake and then offered to regenerate randomised passwords for customers so that they can download them via FTP text-files. Just to cover the obvious: It is also not save to email password lists! Or place them in the HTTP-root of a customer as “password.txt” – just saying!

We believed that the security measures we put in place were adequate to protect these passwords. We were wrong.

Okay, so you put the key to your house under the doormat and expected no-one to look there. Now your house has been robbed. You are still quite unsure how the criminals found the key, but too bad – everything is gone (and someone has used your toothbrush in ways it should not be used). This was quite a naive assumption for an ISP, who proudly announced:

PoPI does not require that Hetzner prevents all unauthorised access because this is impossible to do.

The elephant in the room “Although POPIA is not law, we are POPIA compliant!”

Although POPIA is not passed as law yet, I think it is very prudent to properly understand the act when Hetzner decided to make themselves compliant:

POPIA makes Safe-guarding of personal information very specific:

Personal information must be kept secure against the risk of loss, unauthorised access, interference, modification, destruction and disclosure.

Section 11 of POPIA is even more specific and this is where I believe that storing passwords in clear-text will fall under gross-negligence as it can never be reasonably expected that passwords in clear text can be maintained in a safe and secure fashion:

Section 11: A provider must at all times have and effectively employ the resources, procedures and appropriate technological systems that can reasonably be expected to eliminate as far as reasonably possible, the risk that clients, product suppliers and other providers or representatives will suffer financial loss through theft, fraud, other dishonest acts, poor administration, negligence, professional misconduct or culpable omissions.

Still not convinced? Hetzner shares their standby schedule with customer details

Earlier today I noticed that Hetzner also shares their data-center standbye schedule. I dropped them an email, since phoning them and requesting to speak to the security officer resulted in an “Please log a ticket if you are a customer”. Basically, Hetzner has shared their Google Apps calendar publicly, and I am sorry to say, Gerhardt Voster’s details are publicly available (this includes ID-number, waybills etc – so quite easy to re-route deliveries if one wanted to do so):

 

Who cares about data-centre schedules when server-information is publicly shared?

In addition to sharing data-centre schedules which disclosed IP addresses, customer names, ID numbers, telephone numbers and vehicle registration numbers, Hetzner also chose to make use of Stikked, an open-source web-application which allowed Hetzner support engineers and customers to share “pastes” publicly. Although Hetzner took down the domain after my Abuse notification, it allowed public RSS access via the URL https://pastebin.hetzner.co.za/lists/rss for anyone to view information shared between Hetzner’s internal support engineers.

This went as far as making server credentials publicly available:

Although Hetzner acted on my abuse notification (ref # 2498506), I was not deserved the decency or professionalism to explain why it happened or that it was taken down. The point however is that for an extended period of time (at time of take-down it contained over 90,000 “pastes”) information such as the above was publicly available and could have been data-mined by attackers via RSS feed or just paginating through the all records.

What can you do as a customer?

First action should be to change every password to a unique new password. This becomes even more important if certain passwords are shared. As an example: While Hetzner’s konsoleH password is encrypted, it is very possible that the same konsoleH password was used as the FTP password (which we all know was stored in clear text).

If passwords are shared outside Hetzner’s eco-system (i.e. using the same password for your banking or other servers) you need to urgently reset those passwords too. As far as I understand, the breach affected all hosting services and having access to a clear-text FTP password, an intruder will be able to download files from your Hetzner hosted server and gain access to MySQL passwords (for example stored in WordPress config files) or any other passwords of software you run.

If the FTP access allowed root access (or privileged access) to the server, it is very possible that backdoors or remote shells have been installed – all it takes is to drop an innocently named PHP file into your Apache folder and you would not be any wiser.

I personally would move ISPs as this will be safer and quicker than going through a full audit of your servers. I doubt that Hetzner will be able to tell which of their 40,000 hosting customers had their accounts accessed without authorisation and this is an answer an affected customer will not be able to give either.

Although POPIA is not in effect and since Hetzner has already announced that they will not compensate for any losses, I would (depending on the scale of damages) pursue civil action based on:

• Payment for damages as compensation for losses suffered as a result of the breach due to negligence;
• Aggravated damages;
• Interest;
• Costs associated with mitigation of the breach (changing bank accounts, informing customers, overtime etc)

I would hold off on any civil action until Hetzner has published a full forensic report about the breach. The report should at the minimum include full coverage of the incident (the WHEN, HOW, WHAT, WHO) as well as remedial actions implemented or planned.