Blog

Our provider has been hacked, but Passpack is safe. Zero data compromised.

First things first: your data is safe. 

Passpack runs on dedicated servers at a provider in Germany. Yesterday, that hosting provider was likely hacked into. Due to our application architecture, and the fact that we’ve completely isolated the servers from any access by the provider, Passpack has not been compromised. All user data is secure.

This announcement is simply because we believe in transparency.

Why Passpack was not affected

Fortunately I don’t trust anybody, not even our hosting providers (Passpack is, after all, built on the “Host-proof” Hosting pattern). As soon as our dedicated servers were delivered to us with the OS installed, the first order of operation was to make it so that our provider was completely unable to access our servers. Every default password was changed and (most importantly) the SSH setting only allows access via keys. Yes, that makes it more complex to handle eventual hardware problems, but it’s worth the trouble. Today, when I read the communication below, I knew it was the right choice.

This is the communication that we received today, like hundreds of others:

Dear Client,

We were informed yesterday, Wednesday 5 October, about an improper access to our internal system.
As far as we can presently reconstruct, the attackers could have been able to access internal customer data on [our] administrative systems.
[...] To our present knowledge we have no information regarding data abuse from customers.
Unfortunately, it is not possible for us to exclude this possibility completely and we would therefore ask that you change all passwords on your [Provider] system immediately as a precaution.
[...] To ensure complete and transparent clarification, we shall shortly be reporting this incident to the regulatory authorities.
[...]

As always, we’ve taken follow-up security available to us for good measure. We immediately updated the credentials to login to the the online account manager. Nobody has accessed the account manager, or changed any settings.

My biggest concern was that with access to the provider’s account management system, though they couldn’t have accessed any user data, a hacker could have been able to reset a server: starting a new installation while deleting all the current data. Fortunately, they didn’t. And the access codes have all since been changed. As you can imagine, this would have caused an interruption in service until we’d have reconfigured everything and restored the data from our remote backups.

A secondary concern would be that they could have gotten physical access to the servers while putting it into maintenance mode. Also in that case, there’d have been a noticeable downtime. There wasn’t. Anyway, as you know, our data are useless without hacking the entire distributed system.

Since we had no problems or outages, I could have easily not informed anyone about this. But I believe that transparency is the most important thing for a service like Passpack. So now you know.

Have a good day, and let me know if you have any questions.

Planned Maintenance Next Saturday, Sept. 24

We will be performing a server maintenance next Saturday, September 24, from 8-9am GMT+1 (it was initially planned for 5-6pm PST). During that period, we will put the database in read-only mode. So, if you change something and you need to save it, you have to wait. When the operations will be completed you will save your data without problems.

We will move the system to new powerful servers. If everything will go like during the other migrations in the past, the most of the user will not have the perception of the transfer and all will seem as normal. He hope that also this time it will be so. For good measure, if you haven’t yet, please install Passpack Desktop to be sure that you have an offline backup of your data. This is very important because on the Internet it is always possible that a website is unreachable and you must have your passwords when you need them. Passpack Desktop solves this emergency case and using it is a best practice.

Fixed a Bug in the Billing System

When we first introduced the premium plans, we built a system that (surprisingly) wasn’t supposed to manage plan upgrades before the expiring date. So if you upgraded from a Pro to a Team after two months, you needed to contact us to get  pro-rated manually. On the other hand, there was a bug that applied the unused months from the Pro plan to your Team plan. In essence, you could pay for a Pro plan, quickly upgrade to a Team, and get the total of unused Pro months plus all 12 Team months as a Team account.

As of today, the bug is fixed. We have also expanded the system to allow for automatic pro-rating depending on how much time was available in your previous plan. And finally, for any user who didn’t contact us to be pro-rated and therefore accidentally paid the full price when upgrading, we have extended the renewal deadline to accomated the double-paid months.

All should work well. But, if you see any errors in your account, please let me know.

Shared Tags Are On, Also in Bulk Edit Mode

Today we’ve released two long awaited features: the ability to include tags along with the entry sharing and the ability to apply tags in bulk edit mode.

As a background: when we first introduced tags, we had considered them as a personal way to organize your data. Technically the Tags data was encrypted separately from the Entries data, in a different, simpler way and… couldn’t be shared the same as entries could be.

So once you all made it clear to us that you did indeed want to share your tags,  we were faced with the problem of how to introduce that feature securely, without completely overhauling the way entries work (Passpack is stable, why risk critical bugs?).

ScreenHunter_05-Jul.-19-23.30

Initially we were thinking of changing the logic of the Tags collection, but the risk of add critical bugs was too high. So in the end, we opted to use a sort of plugin to the entry to contain extra data. This extra data is perfectly sharable and is integrated dynamically in the Tags collection, without changing any of its rules.

This approach solved the immediate problem of how to share tags, but it also will allow us to expand the entries in the future too. I am sure that you have a lot of ideas about what can be added to standard entries. So let us know!

Oops, I was forgetting that if you want to transfer an entry to another user preserving your tags, set the tags as shared and you are done.

Bulk Tag Editing

ScreenHunter_10-Jul.-24-17.05

We also added the ability to apply tags in bulk edit mode. The options should be self explanatory. A couple of particular cases are:

  1. If you want to delete all the tags, leave the field empty and chose the merging option “No, substitute them instead” (the logic here is that you are substituting all existing tags with nothing – it’s the equivalent of deleting them).
  2. If you want to quickly convert the settings on a bunch of shared entries  to incude tag sharing all at once, without changing any of the actual tags, leave the field blank, leave the substitution checkbox unchecked, and select the “No, activate tag sharing on all of them instead.
For those of you who have been waiting for the shared tags feature, you should find #2 above very useful.
Don’t worry about mistakes, in bulk edit mode the autosave is disabled, so if something looks wrong you can safely undo the changes.

What else?

The version 7.7.7 of Passpack fixes some minor bugs and improves performance a bit. For example, the tags format has been optimized and now during encryption and decryption the process is faster.
Also, as you can see, the entry table has been enlarged and a smaller font size is used to allow for longer titles.
Finally, there are two new themes in Settings > Appearance & Themes. The Warm Grey theme is inspired by Google+ colors, and the Khaki Brown theme was requested by Allen – Allen, if you don’t like it, let me know :)

New Passpack Desktop Minor Version 2.2.2

Today we released a new minor version of Passpack Desktop that fixes a bug during synchronization with your online account. The new version also adds a post-sync alert about any entries deleted during the sync and allows you to recover them so that they can be resynchronized with your online version.

Added the support for a second Yubikey

Today I discovered that in the Knowledge Base we suggest to have two Yubikeys so that if you loose one of them, you can continue to use the other. But Passpack didn’t support more than one Yubikey. Oops… So, today, I decided to fix the issue adding the support for a second Yubikey.

I also modified the process of removing a second factor if you have only one Yubikey and you loose it. In fact, I added the ID of the Yubikey to the Second Factor configuration page. This will be required to unlock your account if necessary. So, please, save it in your records.

2011-10-02_11-39-442-e1317581942610

Passpack is not LastPass. We Have a Big Friend

A few weeks ago LastPass had a serious breach of data. I don’t like to talk about the competitors, but this unlucky event generated a lot of articles, concern and fear. It also caused an increase in emails from Passpack users who are seriously concerned about their security and want to understand if what has happened to LastPass can happen also to Passpack.

I think not, but I can’t be sure because I don’t know the architectural details of LastPass, and I don’t know what happened to their data. I can say however, that compared with what I understood from their post, we use a few different approaches. For example, it seems that they had all the services on the same network. We have always had totally separate sandboxes for the different Passpack properties. This eliminates the risk that a bug in the blogging or support ticket systems could open up the core application to attack.

But speaking more specifically to the data loss and the risk of a dictionary attack issue, we use a non-standard approach to work around this risk. I’ll explain that here, and at the bottom of the post is also some code so that anyone, LastPass included, can employ it if they’d like.

Let me tell you a story

When I was a child, I was fragile, thin, short and delicate. I was the typical victim of children bigger than me. I remember that during a Christmas vacation, after having been bullied yet again, my grandfather said to me: “if you can not defend yourself, you need a big friend that can defend both of you”. I took his advise, got a big friend, and I never had problem for the rest of my childhood.

Still now that I am a grown man, I am more at ease knowing that I can count on a big, trustworthy friend if necessary. We’ve used this “big friend” approach in Passpack as well. It’s one of the components we use in solving the hash problem that is the main focus of this post. The full details are very technical, so please be patient.

About openness and reactions

I don’t always agree with LastPass’ choices (see my post on how sharing “masked passwords” is a serious security hole), however they are a good company, and their choice to inform their users about what was happening in a quick and open way was the right one. They knew the risks of that choice, but they chose to do what was best for their users over what was best for their public image. I applaud them and would surely do the same.

The reaction of the press to their announcement was less impressive. I was surprised by the superficial knee-jerk reactions from some blogs, including the prestigious LifeHacker. Instead of investigating to understand if there could have been ways for an online password manager to have avoided such an incident, they simply assumed there was not and suggested users go back to offline tools.

This is not a solution. It is a step back of at least five years and doesn’t help people that need something that only the cloud can offer – collaboration and access. Every day we all connect to Facebook, Twitter, PayPal and many other services. If we refuse to persevere and solve important issues in the cloud, then we might as well you turn off our computers now. Problems need solutions, not avoidance. Maybe the current solutions are not ideal, but if we stop the progress we will never have a definitive solution.

Premise: your security is your choice

We can call the best engineers in the world to manage our server security, but if you join the name of your sons and use it like your Packing Key you are very lucky if nobody entered your account yet. So, the first thing to do, in any case, is to choose a really strong Packing Key. Believe me, it is not difficult. A good pass phrase could be something like: my grandma had a strange green bbq or when I jump I can not stop laughing. You simply need a sentence that is  easy to remember for you. You will discover that it will be also easy to write. And, most importantly, I can tell you that it will be very resistant to every attack.

About passwords and hashes

Here are some basics. Every web application that doesn’t use a federated authentication system (like Facebook connect) needs something form of password to identify the user. Old systems used to save the passwords directly in a database. Recently (fortunately) most will save a derivation of the password generated using an hash function instead. What is a hash?

cryptographic hash function is a deterministic procedure that takes an arbitrary block of data and returns a fixed-size bit string (via Wikipedia).

There are certain algorithms that can take text, and create a different text that identifies it – it doesn’t transform it, it just identifies it, like a fingerprint. The result is called a hash string, and that string can’t be reversed. For example, if you take the Wikipedia definition above and run it through a SHA1 hash function, you obtain the following result:

e98dc145444d8758633e73d175a1715e1cdb17e7

Now, if you change just one thing, anything in that Wikipedia definition, then you will get a completely different result. For example, if you remove the period at the end of the paragraph, the new hash would be:

e91090189005b016bd20bb05be5c4fb089fec8d3

As you can see the two hashes are the same length, but they are totally different.

Here’s another example. If you hash the word “password” you get another different result, but again, it’s the same length, 40 characters:

5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8

So every string that you hash with a SHA1 function has always the same size, being (almost) always different. It doesn’t matter how long or short the original text is, it will always produce the same size hash. Both your ATM PIN number and “War and Peace” would both have a hexadecimal 40 character hash. This makes hashes perfect for identifying something without knowing what that something actually is.

In fact, hash functions are used in your ATM and credit cards to protect the pin.

The problem with hashes

Unfortunately there are methods to discover the original string, starting from the hashed one, even though it’s not technically a reversal. The most used hacking method is the so-called Dictionary Attack. A dictionary attack simply means that the attacker takes an entire dictionary of words, and tries to guess your password. Alas, people tend to use always the same passwords or very common words. Some time ago Twitter banned 370 common passwords from use in  its accounts because it was really dangerous to allow users to choose a password like password, testingnaked, stupid, 123456, etc.

Since hash algorithms are known, an attacker can simply run all the dictionary words through the algorithm and compare the result with an existing hash (perhaps that he’s somehow stolen) and see if they match. When the two hashes match, he then knows what the original password was.

Of course, to make hacking more efficient, there are dictionaries of the hashes of the dictionary words (so called Rainbow Tables), available for the outputs of all the major algorithms (SHA256, MD5, etc.). Therefore,  the very first step in protecting users is to salt their hashes. This means that before running a string through a hash algorithm, some additional text is “mixed in” with it  (like adding a dash of salt to your hash browns :) so that the result is at least not so easily looked up in the dictionary of hashes.

At Passpack we use several different combinations with iterations of SHA256 adding in the user’s clientcode as salt. Since that is unique for every user, even if two users use the same Packing Key the hashes will be different. Does that completely solve the problem? No, not yet. It’s a first step to raising the bar a bit since it wards off mass attacks, forcing a potential hacker to try and match hashes one by one.

So at this point, if you’ve been paying attention, you should realize that if you chose a strong Packing Key, then you would not be susceptible to these dictionary attacks (and if you don’t have a strong Packing Key, now is as good a time as any to change it).

We can urge you all the time to create a good Packing Key, but it’s still our job to build the system to be as a secure as possible, regardess of your choices. That means that for all the work we do to build a secure server infrastructure, and make it as close to impossible as we can to penetrate it, we still need to plan for the rare case in which someone does, indeed, get in. Everyone works hard to make sure that this doesn’t happen, but sometimes it still does. This is what just happened to LastPass.

So what are the possible solutions to protect users?

A common solution: iterations

One of the methods used to minimize the risk of a dictionary attack is a strong key derivation function like PBKDF2. According to their post, this is the solution LastPass is looking into.

It works like this: instead simply saving the hash to the database once you receive it on the server, you first run it through another algorithm… many, many, many times. For example 50,000 times. Then you save that to the database. By doing so you slow down the potential hacker by the same number of times. This makes cracking the hash wildly inefficient for the hacker, if not impossible.

The problem with this method is that the server has to work intensively to derive the original hash and you’ll wind up with some serious performance issues. The result is that you can not use this method if you have a lot of users because you risk continuous server downtimes. PBKDF2 is therefore not a real solution for web applications.

The Passpack solution: distribution

Passpack is a small company. We do our absolute best to build the safest infrastructure possible, but we can not spend a lot of money to build our own bomb-proof servers. So I decided to find a big friend, someone that  can share with us the work and, in defending himself, defends us as well.

I wrote a super simple application on Google App Engine called Spino. Spino runs on a Google account that I use only for this app and for nothing else. Spino collaborates with Passpack to create a distributed solution for the hash problem.

How does it work? It’s quite complex, but here’s a simplified description:

  • The User types his Packing Key
  • The Browser applies an hash function to the Packing Key using the clientcode as salt
  • The Browser sends the hash of the Packing Key to Passpack
  • Passpack sends the temporary hashes to Spino
  • Spino verifies it is receiving the hashes from the real Passpack
  • Spino applies a secret hash function to the original hashes and sends it back to Passpack
  • Passpack saves this result in the database

Until some time ago, Spino was on another server and probably in the future we could move it on some other one. Also, to further strengthen the pattern we could distribute the hash manipulation across multiple, external apps on multiple servers using different technologies.

The risk in a solution like this is that if the external platform — in our case Google App Engine — goes down, it would be impossible to login to Passpack. But since we’ve implemented this solution, we’ve had no such complaints. On the other hand, the big advantage is that even in the rare event that an attacker were to grab the Passpack database, he would not be able to do anything with it unless he is also able to penetrate Google.  And Google is a really big friend.

To maintain good performance, once again, we use an encrypted cookie as a distributed cache. In fact, only the server can decrypt the cookie, but the server needs the browser to access the cookie itself.

As you can imagine, Passpack uses other techniques to strengthen the theoretically weak points, though I can not tell you all of them. Another thing that my grandfather used to say often was: “Always tell the truth, but keep a little something for yourself”.

Do you like Spino?

If you like Spino’s approach, feel free to use it. It is open to all.

When I decided to move it from the previous server to Google App Engine, in a week-end I learned Python and I ported the script, so I am sure that you can create and deploy your app from scratch in 20 minutes using the following code.

#
# Spino, a remote hasher
# version 1.2
# Copyright (c) 2009+ Francesco Sullo, Passpack SRL
# Licensed as Open Source under MIT License
#

import hashlib
import os

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

def hashx(val):
    salt = 'Wow! I am a super salt. Please change me!'
    # just a few interactions
    iterations = 20000
    cont = 0
    while cont < iterations:
        m = hashlib.sha256()
        m.update(salt + val)
        val = m.hexdigest()
        cont = cont+1
    return val

class MainPage(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        try:
            a = self.request.get('a')
            e = os.environ['REMOTE_ADDR']
            # set your authorized referer IP or IPs:
            if e == '100.100.100.100':
                ret = hashx(a)
                self.response.out.write(ret)
                return
        except:
            pass
        self.response.out.write('Hello, World!')

application = webapp.WSGIApplication(
    [('/', MainPage)],
    debug=True)

def main():
    run_wsgi_app(application)

if __name__ == "__main__":
    main()


This is a simplified version. But it works :)

Static files are now served from a new server

We completed the performance improvement of Passpack moving the static files on a new server. This improves also the general security because it is now much harder for an attacker to try to inject Javascript code.

Google Gears support has been discontinued

Passpack has discontinued Google Gears support. It was used with slow browsers like Internet Explorer 6 or 7. If you use one of these browsers, probably your account will become really slow without Google Gears. We strongly suggest to upgrade to IE 9 or to a totally different browser like Chrome.

Problems with port 8071

To improve the performance, we moved the static files to Nginx that is much faster then Apache. Unfortunately, we discovered that some users can not access non-standard ports, so we have reversed to the previous configuration.