A few weeks ago LastPass had a serious breach of data. I don’t like to talk about the competitors, but this unlucky event generated a lot of articles, concern and fear. It also caused an increase in emails from Passpack users who are seriously concerned about their security and want to understand if what has happened to LastPass can happen also to Passpack.

I think not, but I can’t be sure because I don’t know the architectural details of LastPass, and I don’t know what happened to their data. I can say however, that compared with what I understood from their post, we use a few different approaches. For example, it seems that they had all the services on the same network. We have always had totally separate sandboxes for the different Passpack properties. This eliminates the risk that a bug in the blogging or support ticket systems could open up the core application to attack.

But speaking more specifically to the data loss and the risk of a dictionary attack issue, we use a non-standard approach to work around this risk. I’ll explain that here, and at the bottom of the post is also some code so that anyone, LastPass included, can employ it if they’d like.

Let me tell you a story

When I was a child, I was fragile, thin, short and delicate. I was the typical victim of children bigger than me. I remember that during a Christmas vacation, after having been bullied yet again, my grandfather said to me: “if you can not defend yourself, you need a big friend that can defend both of you”. I took his advise, got a big friend, and I never had problem for the rest of my childhood.

Still now that I am a grown man, I am more at ease knowing that I can count on a big, trustworthy friend if necessary. We’ve used this “big friend” approach in Passpack as well. It’s one of the components we use in solving the hash problem that is the main focus of this post. The full details are very technical, so please be patient.

About openness and reactions

I don’t always agree with LastPass’ choices (see my post on how sharing “masked passwords” is a serious security hole), however they are a good company, and their choice to inform their users about what was happening in a quick and open way was the right one. They knew the risks of that choice, but they chose to do what was best for their users over what was best for their public image. I applaud them and would surely do the same.

The reaction of the press to their announcement was less impressive. I was surprised by the superficial knee-jerk reactions from some blogs, including the prestigious LifeHacker. Instead of investigating to understand if there could have been ways for an online password manager to have avoided such an incident, they simply assumed there was not and suggested users go back to offline tools.

This is not a solution. It is a step back of at least five years and doesn’t help people that need something that only the cloud can offer – collaboration and access. Every day we all connect to Facebook, Twitter, PayPal and many other services. If we refuse to persevere and solve important issues in the cloud, then we might as well you turn off our computers now. Problems need solutions, not avoidance. Maybe the current solutions are not ideal, but if we stop the progress we will never have a definitive solution.

Premise: your security is your choice

We can call the best engineers in the world to manage our server security, but if you join the name of your sons and use it like your Packing Key you are very lucky if nobody entered your account yet. So, the first thing to do, in any case, is to choose a really strong Packing Key. Believe me, it is not difficult. A good pass phrase could be something like: my grandma had a strange green bbq or when I jump I can not stop laughing. You simply need a sentence that is  easy to remember for you. You will discover that it will be also easy to write. And, most importantly, I can tell you that it will be very resistant to every attack.

About passwords and hashes

Here are some basics. Every web application that doesn’t use a federated authentication system (like Facebook connect) needs something form of password to identify the user. Old systems used to save the passwords directly in a database. Recently (fortunately) most will save a derivation of the password generated using an hash function instead. What is a hash?

cryptographic hash function is a deterministic procedure that takes an arbitrary block of data and returns a fixed-size bit string (via Wikipedia).

There are certain algorithms that can take text, and create a different text that identifies it – it doesn’t transform it, it just identifies it, like a fingerprint. The result is called a hash string, and that string can’t be reversed. For example, if you take the Wikipedia definition above and run it through a SHA1 hash function, you obtain the following result:

e98dc145444d8758633e73d175a1715e1cdb17e7

Now, if you change just one thing, anything in that Wikipedia definition, then you will get a completely different result. For example, if you remove the period at the end of the paragraph, the new hash would be:

e91090189005b016bd20bb05be5c4fb089fec8d3

As you can see the two hashes are the same length, but they are totally different.

Here’s another example. If you hash the word “password” you get another different result, but again, it’s the same length, 40 characters:

5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8

So every string that you hash with a SHA1 function has always the same size, being (almost) always different. It doesn’t matter how long or short the original text is, it will always produce the same size hash. Both your ATM PIN number and “War and Peace” would both have a hexadecimal 40 character hash. This makes hashes perfect for identifying something without knowing what that something actually is.

In fact, hash functions are used in your ATM and credit cards to protect the pin.

The problem with hashes

Unfortunately there are methods to discover the original string, starting from the hashed one, even though it’s not technically a reversal. The most used hacking method is the so-called Dictionary Attack. A dictionary attack simply means that the attacker takes an entire dictionary of words, and tries to guess your password. Alas, people tend to use always the same passwords or very common words. Some time ago Twitter banned 370 common passwords from use in  its accounts because it was really dangerous to allow users to choose a password like password, testingnaked, stupid, 123456, etc.

Since hash algorithms are known, an attacker can simply run all the dictionary words through the algorithm and compare the result with an existing hash (perhaps that he’s somehow stolen) and see if they match. When the two hashes match, he then knows what the original password was.

Of course, to make hacking more efficient, there are dictionaries of the hashes of the dictionary words (so called Rainbow Tables), available for the outputs of all the major algorithms (SHA256, MD5, etc.). Therefore,  the very first step in protecting users is to salt their hashes. This means that before running a string through a hash algorithm, some additional text is “mixed in” with it  (like adding a dash of salt to your hash browns :) so that the result is at least not so easily looked up in the dictionary of hashes.

At Passpack we use several different combinations with iterations of SHA256 adding in the user’s clientcode as salt. Since that is unique for every user, even if two users use the same Packing Key the hashes will be different. Does that completely solve the problem? No, not yet. It’s a first step to raising the bar a bit since it wards off mass attacks, forcing a potential hacker to try and match hashes one by one.

So at this point, if you’ve been paying attention, you should realize that if you chose a strong Packing Key, then you would not be susceptible to these dictionary attacks (and if you don’t have a strong Packing Key, now is as good a time as any to change it).

We can urge you all the time to create a good Packing Key, but it’s still our job to build the system to be as a secure as possible, regardess of your choices. That means that for all the work we do to build a secure server infrastructure, and make it as close to impossible as we can to penetrate it, we still need to plan for the rare case in which someone does, indeed, get in. Everyone works hard to make sure that this doesn’t happen, but sometimes it still does. This is what just happened to LastPass.

So what are the possible solutions to protect users?

A common solution: iterations

One of the methods used to minimize the risk of a dictionary attack is a strong key derivation function like PBKDF2. According to their post, this is the solution LastPass is looking into.

It works like this: instead simply saving the hash to the database once you receive it on the server, you first run it through another algorithm… many, many, many times. For example 50,000 times. Then you save that to the database. By doing so you slow down the potential hacker by the same number of times. This makes cracking the hash wildly inefficient for the hacker, if not impossible.

The problem with this method is that the server has to work intensively to derive the original hash and you’ll wind up with some serious performance issues. The result is that you can not use this method if you have a lot of users because you risk continuous server downtimes. PBKDF2 is therefore not a real solution for web applications.

The Passpack solution: distribution

Passpack is a small company. We do our absolute best to build the safest infrastructure possible, but we can not spend a lot of money to build our own bomb-proof servers. So I decided to find a big friend, someone that  can share with us the work and, in defending himself, defends us as well.

I wrote a super simple application on Google App Engine called Spino. Spino runs on a Google account that I use only for this app and for nothing else. Spino collaborates with Passpack to create a distributed solution for the hash problem.

How does it work? It’s quite complex, but here’s a simplified description:

  • The User types his Packing Key
  • The Browser applies an hash function to the Packing Key using the clientcode as salt
  • The Browser sends the hash of the Packing Key to Passpack
  • Passpack sends the temporary hashes to Spino
  • Spino verifies it is receiving the hashes from the real Passpack
  • Spino applies a secret hash function to the original hashes and sends it back to Passpack
  • Passpack saves this result in the database

Until some time ago, Spino was on another server and probably in the future we could move it on some other one. Also, to further strengthen the pattern we could distribute the hash manipulation across multiple, external apps on multiple servers using different technologies.

The risk in a solution like this is that if the external platform — in our case Google App Engine — goes down, it would be impossible to login to Passpack. But since we’ve implemented this solution, we’ve had no such complaints. On the other hand, the big advantage is that even in the rare event that an attacker were to grab the Passpack database, he would not be able to do anything with it unless he is also able to penetrate Google.  And Google is a really big friend.

To maintain good performance, once again, we use an encrypted cookie as a distributed cache. In fact, only the server can decrypt the cookie, but the server needs the browser to access the cookie itself.

As you can imagine, Passpack uses other techniques to strengthen the theoretically weak points, though I can not tell you all of them. Another thing that my grandfather used to say often was: “Always tell the truth, but keep a little something for yourself”.

Do you like Spino?

If you like Spino’s approach, feel free to use it. It is open to all.

When I decided to move it from the previous server to Google App Engine, in a week-end I learned Python and I ported the script, so I am sure that you can create and deploy your app from scratch in 20 minutes using the following code.

#
# Spino, a remote hasher
# version 1.2
# Copyright (c) 2009+ Francesco Sullo, Passpack SRL
# Licensed as Open Source under MIT License
#

import hashlib
import os

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

def hashx(val):
    salt = 'Wow! I am a super salt. Please change me!'
    # just a few interactions
    iterations = 20000
    cont = 0
    while cont < iterations:
        m = hashlib.sha256()
        m.update(salt + val)
        val = m.hexdigest()
        cont = cont+1
    return val

class MainPage(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        try:
            a = self.request.get('a')
            e = os.environ['REMOTE_ADDR']
            # set your authorized referer IP or IPs:
            if e == '100.100.100.100':
                ret = hashx(a)
                self.response.out.write(ret)
                return
        except:
            pass
        self.response.out.write('Hello, World!')

application = webapp.WSGIApplication(
    [('/', MainPage)],
    debug=True)

def main():
    run_wsgi_app(application)

if __name__ == "__main__":
    main()


This is a simplified version. But it works :)

21 Comments

  1. Allen

    This is an excellent write up. Thanks for the reassurance.

    (By the way, your English writing is really coming along! :)

  2. Francesco

    @Allen, thanks :)

  3. Pavel

    Passpack is really small and great company. Thank you for article.

  4. Noah

    Thank you for the write-up. Could you tell me how secondary authentication (like Yubi keys) would influence anything in this article?

  5. TwoHawks

    Good job, Francesco. It is appropriate for PassPack to post something in the wake of the LasPast problem, so I am very happy to see this.

    I strongly urge you to also add something to your Help Center (faq) about PassPack’s use of salted hashes (and you can reference this blog). I recently found that while searching online there were a lot of places criticizing PassPack without realizing it uses this, and very little information coming up about it even if you searched hard for it (so the criticism was no wonder).

    Keep up the good work – we need more companies with your kind of creative spirit and dedication.
    TwoHawks

  6. jonny

    This is a great article. You’ve just got yourself one (1) customer. Although I suspect I may be representative of many more.

  7. Ariel

    Thanks for explaining more about your company. I’m a happier user now.

  8. Dave

    Very interesting article. I didn’t even realize English was not your first language before I got to the comments. There were no more errors than I might commit via incomplete edits etc.

    It’s a shame some people (LifeHacker) paint all online password managers with the same broad brush, making this background information so much more important to explain.

    On a humorous note, the mention of PBKDF2 reminded me of a frequent diagnosis for problems experienced by poorly trained users: PEBSAK (problem exists between seat and keyboard).

  9. Francesco

    @Noah
    A second factor is important because also if someone discover you Password and Packing Key, he can not access your account without your Yubikey. I didn’t cited it because only a few thousand users have activated a second factor.

    @TwoHawks
    You are right. In these years we focused on the product without explain how it works. Passpack is more complex than it seems and we should explain it. The problem is that every component of the program requires more space than this post to be explained.

    @Dave
    Also for English I have… big friends :)

  10. Loughlin

    Excellent write up Francesco, and very interesting to see the python code there for Spino. This has reassured me and is a testament to how much you guys care about your customers.

  11. Philippe

    Very interesting article.
    Thank you for the good job you do and the information you provide to the users.
    I like to use Passpack more and more every day.

  12. Ivan Naitana

    Using Google App Engine resources for Spino is such a simple but good idea, since it both gives you cheap and affordable processing resources (until Google begins charging for CPU time!) and provide extra security through distribution, which is such a simple but overlooked strategy. It is great to know that you employ it to the max through the whole structure.

    Using a customised hashing iteration function could sound a bit of security-through-obscurity, but I can see that by iterating for an odd amount of times and adding some odd value to the salt (and maybe something which is variable in some way) in line 21 would build up a whole hashing function which would be so hard to reconstruct for an attacker, even by knowing existing plaintext/hash couples.

    What I wonder about is why you’d need to store an hash of the Packing Key at all.

    Actually I thought you used a totally different approach from LastPass, since from the “Data Privacy” page it looks like the Packing Key is never sent to the server but used only locally to decrypt the Pack.

    Now by analysing the traffic it looks like the Pack is sent to the client only after verifying the Packing Key server-side, which is anyway a good move for not giving away too easily Packs, even if they are encrypted.

    But wouldn’t it be possible to have a Packing Key (maybe a third one?) which is never sent to or known by the server, but only validated client-side by checking if it can decrypt the Pack? Wouldn’t it add an ultimate layer of security?

  13. Francesco

    @Loughlin, @Philippe thank you :)

    @Ivan, you write:

    But wouldn’t it be possible to have a Packing Key (maybe a third one?) which is never sent to or known by the server, but only validated client-side by checking if it can decrypt the Pack?

    It was so until the version 4 of Passpack. But this was not so secure how you can imagine. Because your security is a concept that has many shades.

    For example, people often forget their Packing Key after having changed it. In this case we can roll back the keys set of the user and he is safe. With the old approach your pack would have been definitely lost.

    Instead, now we can recover it because all the entries (and the other data in your pack) aren’t encrypted directly with the Packing Key, but with a set of several keys, depending on a lot of factors. This basic keys set is encrypted using a Master Key. In turn, this Master Key is encrypted using a derivation of the Packing Key.

    So when you change your Packing Key, a new Master Key record is created in the database. This allows to recover the old record, and it is able to recover the basic keys set if you have the old Packing Key.

    Anyway, imagine that we would validate the Packing Key trying to decrypt the Master Key on the client-side, also an attacker that raise access to the database can try to discover the Packing Key with a brute force dictionary attack to the encrypted Master Key. This would be harder, but it would be possible.

    How to avoid this? The solution — as you can guess — is a Spino’s brother that receives the Master Key encrypted by the browser, encrypts it again with a secret key and returns the double-encrypted Master Key so that it can be saved in the database.

  14. justin time

    I love this app and philosophy of the company. PERIOD.

    BUT, your gramps gave you some terrible advice. He should have thought you about self defense and self reliance because when the ‘big friend’ is gone you are now again exposed all over again.
    All it does is postpones the inevitable and gives the opposing side more time to even more carefully plan an attack. Don’t pass this advice to your kids and don’t let them hide in shadows of others…teach about better passwords :o

  15. Francesco

    @Justin, my grandfather taught me to use my brain to find creative solutions when the standard solution are not good. The “big friend” is mostly a metaphor. Passpack doesn’t live in shadow of Google, Passpack uses Google in order to help itself. That’s different.

  16. Ivan Naitana

    @Francesco:

    Well, I actually thought that the Packing Key couldn’t still be changed at all (I tried to some years ago, and didn’t check back lately).

    Anyway you’re right, I overrated the security of having the Pack encrypted with a key which is never known in any of its derivations by the server. After all, running a brute force attack against the Pack (or the Master Key) to discover the “private” encryption key wouldn’t be harder than running an attack against the hashed Packing Key (actually, I guess that thanks to Spino the latter could be even harder).

    But still I thought that you could use such an option for “political” reasons.

    You could say that it makes your system verifiably host-proof (everybody could verify the javascript and audition what’s being sent), and that even if you were an evil organisation which is able to somehow mystically reverse all its hashes (don’t take me wrong, I know you can’t), there would still be something that you had no way to know.
    Of course, if you were so evil you could run a brute force attack against the encrypted pack, but if the password was strong enough the pack would be safe from your wicked hands.

    Also, it could have been a good answer in cases like this, to all the superficial “don’t trust the cloud” criticism from LifeHacker and other media.
    You could have said that even if the cloud collapses, your whole system gets compromised, all the databases lost to attackers, and all hashes reversed, there would still be something that has never been in the cloud… and their passwords pack would be left encrypted in just the same status as it would have been if they used the desktop password managers which they love so much.

  17. Skip Franklin

    Outstanding writeup! Thanks for sharing a bit of your philosophy and implementation in this critical area of web security.

  18. Shawn Willden

    So… if the master key is derived by this call to the Spino service, and it is then used to encrypt the bundle of keys used to encrypt the contents of the pack, that means that the master key passes through Passpack’s servers. That, in turn, means that Passpack has the key needed to decrypt the keys needed to decrypt my pack.

    It seems like you’ve eliminated the Host-proof Hosting.

  19. Francesco

    @Shawn, no.

    1. The maskerkey is generated randomly in the browser
    2. The masterkey is encrypted using a 256-bit derivation of the Packing Key
    3. The encrypted masterkey is sent to the server
    4. The server send it to the remote service that encrypts it again
    5. The final double-encrypted masterkey is finally saved.

    So all depends on the Packing Key. Since the Packing Key is hashed in the browser and only the hash is sent to the server, the process is Host-proof Hosting.
    Obviously, as Ivan says, we can performe a brute force attack to our system. But… why we should do it?

    @Ivan, I think that the best solution is always to put your data not in only one place. In 2008 we introduced Passpack Desktop in order to offer a desktop application that can be used stand-alone or like a full local backup of the online account. It is a must, in my opinion. But, unfortunately, only a small percentage of the users seems to use it.

  20. Jon

    I love this post. I’ve read it several times and it has really helped me to understand Passpack. I think a good packing key would be “francesco taught me about hashes and distribution”

    Now that’s strong!

  21. Francesco

    @Jon :)

Leave a Reply