First things first: your data is safe. 

Passpack runs on dedicated servers at a provider in Germany. Yesterday, that hosting provider was likely hacked into. Due to our application architecture, and the fact that we’ve completely isolated the servers from any access by the provider, Passpack has not been compromised. All user data is secure.

This announcement is simply because we believe in transparency.

Why Passpack was not affected

Fortunately I don’t trust anybody, not even our hosting providers (Passpack is, after all, built on the “Host-proof” Hosting pattern). As soon as our dedicated servers were delivered to us with the OS installed, the first order of operation was to make it so that our provider was completely unable to access our servers. Every default password was changed and (most importantly) the SSH setting only allows access via keys. Yes, that makes it more complex to handle eventual hardware problems, but it’s worth the trouble. Today, when I read the communication below, I knew it was the right choice.

This is the communication that we received today, like hundreds of others:

Dear Client,

We were informed yesterday, Wednesday 5 October, about an improper access to our internal system.
As far as we can presently reconstruct, the attackers could have been able to access internal customer data on [our] administrative systems.
[...] To our present knowledge we have no information regarding data abuse from customers.
Unfortunately, it is not possible for us to exclude this possibility completely and we would therefore ask that you change all passwords on your [Provider] system immediately as a precaution.
[...] To ensure complete and transparent clarification, we shall shortly be reporting this incident to the regulatory authorities.
[...]

As always, we’ve taken follow-up security available to us for good measure. We immediately updated the credentials to login to the the online account manager. Nobody has accessed the account manager, or changed any settings.

My biggest concern was that with access to the provider’s account management system, though they couldn’t have accessed any user data, a hacker could have been able to reset a server: starting a new installation while deleting all the current data. Fortunately, they didn’t. And the access codes have all since been changed. As you can imagine, this would have caused an interruption in service until we’d have reconfigured everything and restored the data from our remote backups.

A secondary concern would be that they could have gotten physical access to the servers while putting it into maintenance mode. Also in that case, there’d have been a noticeable downtime. There wasn’t. Anyway, as you know, our data are useless without hacking the entire distributed system.

Since we had no problems or outages, I could have easily not informed anyone about this. But I believe that transparency is the most important thing for a service like Passpack. So now you know.

Have a good day, and let me know if you have any questions.

7 Comments

  1. Full transparency is always appreciated. Thanks!

    Having said that, I just want to raise another concern. Paranoia is a virtue, isn’t it?

    I assume that the provider is actually using virtual machines. In that case, having access to the provider’s management system, they could have created a snapshot (no noticeable downtime here).

    Having physical access, full filesystem encryption is the only real mitigation.

    Cheers,
    L.

  2. Francesco

    @Luca, I still don’t trust cloud infrastructure for critical data. And, in fact, our servers are physical machines.
    Also, I prefer software raid for our disks, because you must turn off the server to take a snapshot.
    So: no downtime, no snapshot :)

  3. John

    I’m concerned about the Javascript that deals with my passpack password locally in my browser; did you store, and have you checked the hashes of the Passpack Javascript files that actually handle the local password interaction and decryption?

    Thanks.

  4. Francesco

    @John, the code injection is the primary risk for a Javascript application. So we constantly monitor all the files, with an automatic procedure that, if there is any anomaly, overwrites the files and immediately alerts us.

    Until now, I received alerts only during tests.
    And I hope that I will never receive further alerts :)

  5. Well done Francesco! ;)

  6. Dave

    Is there any point on you hosting on FireHost.com ?? It sounds as a perfect match for you. Please let us know.

  7. Francesco

    @Dave, Firehost is a good solution, but it is expensive. Currently our 4 dedicated servers cost about 300 dollars a month. On Firehost we will spend ten times this amount.

Leave a Reply