I’ve recently started doing some web application development again after stumbling about in the world of Information Security for the past two years of my career, and I’ve come to notice the many flaws in the way that I have always written code. One particular area of weakness in my ever-evolving application development methodology involves my use, or rather, non-use of encryption when storing sensitive user information in the database. Don’t worry though. That was back in college and I assure you that you won’t find any of the insecure systems that I have written deployed in the wild.
Table of Contents
- The Common Approach
- Problem: Single Point of Failure
- Solution: Encrypt Sensitive Data Using User Passwords
- Problem: Changing Passwords
- Solution: Use An Intermediate Encryption Key
- Problem: Resetting Passwords
- Solution: Recovery Codes and Security Questions
- Problem: Sharing Data With Other Users
- Solution: Use Asymmetric Encryption Scheme
- Problem: Multiple Invocations of Asymmetric Algorithms
- Solution: Perform Encryption Using Shared Key
- Problem: User Password Compromise
- Solution: Encrypt With Application-Wide Key
- What Data to Encrypt and How to Encrypt Them
The Common Approach
After researching about this topic for days on end, I’ve come to notice that what seems to be the “normal” practice is to store a clear-text application-wide encryption key in a configuration file somewhere in the web application server and then use that key to encrypt data using some symmetric encryption algorithm (usually AES-256-CBC) before pushing the data onto the database. And then when you have to retrieve the encrypted data, you just pull it off the database and run it through the decryption routine of your chosen algorithm using the application-wide encryption key that you have stored on the web server.
Problem: Single Point of Failure
The approach above may be adequate for most cases, but I wanted to take it a bit further. I wanted the appropriate users to be the only ones to have the ability to decrypt the data that they should be able to view. If possible, I wanted the attacker to not have the ability to decrypt sensitive data from the database even after compromising both the web server and the database.
As you may have noticed, the approach described in the section above suffers from a single point of failure issue because there’s only one key for all the encrypted data in the entire application and having it compromised may allow some hack to decrypt our precious data in case they get a hold of our database (so make that two points of failures, but you get my drift). However, that should be unlikely because you, dear developer, are guarding our database against injection attacks by using prepared statements or some sort of escaping routine right? And you did apply appropriate access control to our database by using a different user account per type of transaction that our application can perform right? And you did apply appropriate table/function/view/procedure access permissions to those user accounts right? Good! But you can never be too careful.
Solution: Encrypt Sensitive Data Using User Passwords
To eliminate that single point of failure. Why don’t we delegate the management of encryption keys to our users? I mean, aren’t they already holding onto something that’s a key of sorts?
If I wasn’t clear enough, I’m talking about their passwords.
Think about it. If we run our user’s passwords through some Key Derivation Function (KDF) and use the result to encrypt the data that only they should be able to access, then we won’t have to store any clear-text encryption keys anywhere. The encryption keys would be stored in the brains of our users and any key leakages will just expose the stuff of those users involved in the leakage.
Anyway, in this scheme, we just ask the user to enter their password whenever they need to perform some operation involving their encrypted data, run their password through our chosen KDF, decrypt their data using the key generated by the KDF, perform the operation using the clear-text form of their data, and re-encrypt again when we need to store it somewhere else.
Nice and clean!
Do note that we would have to avoid storing the key generated by the KDF into the database because we’d just be providing the encryption key to the adversary in the case that they be able to get a hold of the database. I’m pointing this out because PBKDF2 is a common way to derive encryption keys from a password while also being a common way to store password hashes in the database. If you’re using PBKDF2 to store user password hashes AND derive encryption keys from user passwords, you should probably use a different salt for the password hash of your user and a different salt for their encryption key.
So that would be:
encryption_key_salt = generate_salt() encryption_key = pbkdf2(encryption_key_salt, password) password_hash_salt = generate_salt() password_hash = pbkdf2(password_hash_salt, password)
Again, do not store the encryption_key into the database as you will be using that to encrypt and decrypt user data. Instead, you will just have to compute it every time you need to access or write encrypted user data from and to the database. This would however require that the user enter their password for every transaction involving their encrypted data.
You should only store the encryption_key_iv, password_hash_iv, and password_hash into the database. If you’re asking why we’re storing the password hash but not the encryption key, it’s because we use the hash to determine whether or not the user has entered the correct password. Reversing a hash is computationally infeasible so it should be safe to store it in the database.
Anyway, according to what I’ve read, you’d generally want to store passwords as BCrypt instead of PBKDF2, but using the results of BCrypt as an encryption key isn’t such a good idea because it produces a fixed-length result and you wouldn’t be able to adjust it to encryption algorithms that would require a length of more or less than that of the results produced by BCrypt.
By this point, our database should look something a bit like this:
Table: User
id- Primary key identifying a user
username_sha256- Unsalted SHA256 hash of the user’s username
password_salt- Salt used to compute the user’s BCrypt password hash
password_bcrypt- The BCrypt password hash of the user
- Used for authenticating the user
pbkdf2_key_salt- Salt used for computing the user’s PBKDF2 password-based encryption key
Table: DataEntity
id- Primary key identifying a data entity
user_id- The identifier of the user owning the data within the data entity
value_iv- The Initialization Vector for encrypting and decrypting the value of the data within the data entity
value_aes256cbc- The encrypted value of the data within the data entity
- Encrypted using AES-256-CBC and uses the user’s password-based encryption key for encryption
Problem: Changing Passwords
But what if the user wants to change their password? Wouldn’t that prevent us from accessing the data already encrypted with their passwords unless we re-encrypt each of them one-by-one. Do we really have to re-encrypt all their data? That would be terribly inefficient once the amount of data scales up!
Yes. That would be a very troublesome consequence indeed if we do decide to use user passwords to encrypt our users’ data. Fortunately, there is a work around to this and it involves encrypting our users’ data with an intermediate key rather than using a derivation of the user password itself.
Solution: Use An Intermediate Encryption Key
Basically, what we would do is: we randomly generate an encryption key for each of our users and then use that encryption key to encrypt their data. We then encrypt each user’s encryption key with a derivation of their password (using a KDF) and then store the encrypted user encryption key in the appropriate row in the user table. Whenever we need to encrypt or decrypt some data for a specific user, we just ask them for their password, run it through the same Key Derivation Function we used earlier, use the result to decrypt the user’s encrypted encryption key, and then use the now-decrypted encryption key to encrypt or decrypt user data.
So that would be:
encryption_key_pbkdf_salt = generate_salt() encryption_key_aes_iv = generate_iv() encryption_key = aes256( encryption_key_aes_iv, pbkdf2(encryption_key_pbkdf_salt, password), generate_encryption_key() )
Instead of:
encryption_key_salt = generate_iv() encryption_key = pbkdf2(encryption_key_salt, password)
Using this approach, we would only need to re-encrypt the user’s intermediate encryption key whenever the user decides to change their password instead of having to re-encrypt all data involving a specific user, thus, saving us a valuable amount of computing resources.
Do note that we would have no choice but to store the encrypted encryption key in the database unlike the previous approach, but this would eliminate the difficulty with changing user passwords.
By this point, our database should look something a bit like this:
Table: User
id- Primary key identifying a user
username_sha256- Unsalted SHA256 hash of the user’s username
password_salt- Salt used to compute the user’s BCrypt password hash
password_bcrypt- The BCrypt password hash of the user
- Used for authenticating the user
pbkdf2_key_salt- Salt used for computing the user’s PBKDF2 password-based encryption key
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- The encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the user’s password-based encryption key for encryption
Table: DataEntity
id- Primary key identifying a data entity
user_id- The identifier of the user owning the data within the data entity
value_iv- The Initialization Vector for encrypting and decrypting the value of the data within the data entity
value_aes256cbc- The encrypted value of the data within the data entity
- Encrypted using AES-256-CBC and uses the user’s intermediate encryption key for encryption
Problem: Resetting Passwords
So we now have the capability to allow users to change their passwords. But what if the user forgets their password and they now have to reset it? We’re in a bit of a bind here since we would need the derived encryption key from the user’s password in order to re-encrypt their encryption key. We don’t exactly have the user’s password stored in a manner where we would be able to derive the original password or its derivation so we’re pretty much stuck.
Solution: Recovery Codes and Security Questions
The next best thing would be to have other encrypted copies of the user’s encryption key using alternative “passwords” in the form of randomly generated recovery codes or answers to security questions.
When using randomly generated recovery codes, you’d want to provide it to the user upon registration and then provide them the ability to randomly generate a new set on demand. You are going to need access to the user’s encryption key when generating recovery codes and so you are going to have to ask the user for their password when generating a new set. What you’d basically do is: randomly generate a recovery code, hash it with BCrypt for later authentication, encrypt the user’s encryption key using a derivation of the recovery code that was just generated, and then store the hash of the recovery code and encrypted copy of the user’s encryption key in a recovery codes table in your database.
In a nutshell, that would be:
recovery_code = generate_recovery_code() recovery_code_salt = generate_salt() recovery_code_hash = bcrypt(recovery_code_salt, recovery_code) encryption_key_pbkdf_salt = generate_salt() encryption_key_aes_iv = generate_iv() encryption_key = aes256( encryption_key_aes_iv, pbkdf2(encryption_key_pbkdf_salt, recovery_code), encryption_key_clear )
As stated above, recovery codes are a form of alternative “passwords” and so we would have to follow the rules of password storage when storing recovery codes, and those are: 1) never store them in clear text, and 2) store them as a hash (preferably using BCrypt) so you can authenticate later. You are going to have to store the encrypted encryption key though because it needs to be recovered and re-encrypted with the user’s new password later on when the user resets his or her password.
Upon provision of the recovery codes, the user should be instructed to store them in a safe place such as a password manager or an actual physical safe. When users forget their password, they would need to provide one of the recovery codes in order to decrypt their encrypted encryption key and then re-encrypt it with the derivation of the new password of their choosing. You should also invalidate the old recovery codes and generate a new set when the user resets his or her password.
When using security questions, you can either ask users for common personal questions or give them the ability to provide their own set of security questions. In the case of common personal questions, you’d want to use multiple ones and then have combinations of answers serve as the user’s recovery codes. In the case of custom user-provided questions, you might be able to use the answers as recovery codes individually given that you require the users to provide a minimum length for each of their answers.
As for storing answers to security questions, you’d pretty much handle them in the same manner as you would recovery codes, so you may refer to the instructions above when storing the answers to security questions.
By this point, our database should look something a bit like this:
Table: User
id- Primary key identifying a user
username_sha256- Unsalted SHA256 hash of the user’s username
password_salt- Salt used to compute the user’s BCrypt password hash
password_bcrypt- The BCrypt password hash of the user
- Used for authenticating the user
pbkdf2_key_salt- Salt used for computing the user’s PBKDF2 password-based encryption key
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- The encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the user’s password-based encryption key for encryption
Table: DataEntity
id- Primary key identifying a data entity
user_id- The identifier of the user owning the data within the data entity
value_iv- The Initialization Vector for encrypting and decrypting the value of the data within the data entity
value_aes256cbc- The encrypted value of the data within the data entity
- Encrypted using AES-256-CBC and uses the user’s intermediate encryption key for encryption
Table: RecoveryCode
id- Primary key identifying a recovery code
user_id- The identifier of the user owning the recovery code and user key contained within the recovery code entity
recovery_code_salt- Salt used to compute the recovery code’s BCrypt hash
recovery_code_bcrypt- The BCrypt hash of the recovery code
- Used for authenticating the recovery code
pbkdf2_keysalt- Salt used for computing a recovery code-based encryption key using PBKDF2
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- Encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the recovery code-based encryption key for encryption
Problem: Sharing Data With Other Users
But wait. What if a piece of encrypted data created by one user needs to be shared with another? Surely the method of having an encryption key encrypt sensitive data per user won’t work since in order to decrypt a piece of encrypted data, the data owner will have to enter his or her password. Other users do not have access to the data owner’s password so they won’t be able to read the data. An approach one might try is to obtain an unencrypted copy of the owner’s data when the owner invokes the action of sharing his or her data to another party (where we would have to ask the data owner for their password), but then we won’t be able to encrypt the data upon receipt of the second party because then we would need access to their password. So using this, scheme, both parties will have to do the process of sharing in real time. This would greatly reduce the usability of your system. However, there is hope, and it’s called Asymmetric Encryption.
Solution: Use Asymmetric Encryption Scheme
Asymmetric Encryption in a nutshell is where you have a Public Key and a Private Key. Things you encrypt with your Public Key can only be decrypted by your Private Key, and things you encrypt with your Private Key can only be decrypted by your Public Key. In an Asymmetric Encryption scheme, you share your Public Key with everyone while you keep the Private Key to yourself. Everyone then uses your Public Key to encrypt data they want to share with you, and then you use your Private Key to decrypt it.
If we want to apply this scheme to our system, we would have to assign a Public Key and a Private Key to each user. We then encrypt everyone’s Private Key using the intermediate symmetric encryption keys assigned to them as described in the earlier sections. You may also encrypt user Private Keys using the derivation of your users’ passwords directly but you will have to re-encrypt two keys when the users change or reset their passwords. You may alternatively forgo the use of the symmetric intermediate encryption key and just encrypt Private Keys with the derivations of your users’ passwords directly though I would suggest having both in case we would need to store data that only the data owner should be able to access.
So in a nutshell, what’s going to happen is:
public_key, private_key = generate_key_pair() private_key_aes_iv = generate_iv() private_key_encrypted = aes256( private_key_aes_iv, encryption_key, private_key )
If a user then wants to share their data with another user, we won’t need to have access to the recipient’s password in order for the owner to share their data with the recipient. We just obtain the recipient’s Public Key (which we store in clear-text), encrypt the data that we want to share with them using said Public Key, and then give them access to the data encrypted with their Public Key which they will then decrypt with their Private Key.
By this point, our database should look something a bit like this:
Table: User
id- Primary key identifying a user
username_sha256- Unsalted SHA256 hash of the user’s username
password_salt- Salt used to compute the user’s BCrypt password hash
password_bcrypt- The BCrypt password hash of the user
- Used for authenticating the user
pbkdf2_key_salt- Salt used for computing the user’s PBKDF2 password-based encryption key
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- The encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the user’s password-based encryption key for encryption
public_key- The unencrypted public key of the user
private_key_iv- The Initialization Vector for encrypting and decrypting the user’s private key
private_key_aes256cbc- The encrypted value of the user’s private key
- Encrypted using AES-256-CBC and uses the user’s intermediate key for encryption
Table: DataEntity
id- Primary key identifying a data entity
user_id- The identifier of the user owning the data within the data entity
value_iv- The Initialization Vector for encrypting and decrypting the value of the data within the data entity
value_aes256cbc- The encrypted value of the data within the data entity
- Encrypted using AES-256-CBC and uses the user’s intermediate encryption key for encryption
Table: RecoveryCode
id- Primary key identifying a recovery code
user_id- The identifier of the user owning the recovery code and user key contained within the recovery code entity
recovery_code_salt- Salt used to compute the recovery code’s BCrypt hash
recovery_code_bcrypt- The BCrypt hash of the recovery code
- Used for authenticating the recovery code
pbkdf2_keysalt- Salt used for computing a recovery code-based encryption key using PBKDF2
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- Encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the recovery code-based encryption key for encryption
Table: SharedDataEntity
id- Primary key identifying an instance of shared data
data_id- The identifier of the data entity whose value is being shared
recipient_id- The identifier of the recipient user
value_rsa- Encrypted value of the data being shared to the recipient user
- Encrypted using RSA and uses the Public Key of the receiving user for encryption
- Decrypted using RSA and uses the Private key of the receiving user for decryption
Problem: Multiple Invocations of Asymmetric Algorithms
This isn’t really a problem to be honest but it would be best if we could limit the number of times that encryption and decryption takes place using an Asymmetric algorithm because it’s known to be very very slow. We can get around this limitation by allowing two communicating users to exchange a shared encryption key with each other and then use that encryption key to encrypt and decrypt the data that should be shared between the two users.
Solution: Perform Encryption Using Shared Key
So basically, what would happen is: we randomly generate an encryption key that will be shared between two users who want to communicate, create two copies of it, encrypt one copy with the Public Key of one user and encrypt the other with the Public Key of the other user, and then give each user the copy which was encrypted with their Public Key. And then, to acquire the shared key, each user would then decrypt the copy of the shared key which was encrypted with their Public Key using their Private Key and then re-encrypt it using their intermediate encryption key so they wouldn’t need to decrypt the shared encryption key over and over again using their Private Key thus avoiding the repetitive use of slow Asymmetric Encryption and Decryption.
So that would be:
shared_key = generate_encryption_key() shared_key_rsa_user0 = rsa_encrypt(public_key_user0, shared_key) shared_key_rsa_user1 = rsa_encrypt(public_key_user1, shared_key)
Again, don’t store the shared_key in the database. Store only the copies of the shared key that was encrypted by the Public Keys of the users.
And then on the side of each user:
shared_key = rsa_decrypt(private_key, shared_key_rsa) shared_key_aes_iv = generate_iv() shared_key_aes = aes256( shared_key_aes_iv, encryption_key, shared_key )
The interesting thing about this approach is that you could scale it up and have a shared key between not just two users, but an entire group of users. Anyone with access to the shared key can potentially add new users to the group, but you would most likely want to limit this capability to certain users by enforcing it through your application’s code.
By this point, our database should look something a bit like this:
Table: User
id- Primary key identifying a user
username_sha256- Unsalted SHA256 hash of the user’s username
password_salt- Salt used to compute the user’s BCrypt password hash
password_bcrypt- The BCrypt password hash of the user
- Used for authenticating the user
pbkdf2_key_salt- Salt used for computing the user’s PBKDF2 password-based encryption key
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- The encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the user’s password-based encryption key for encryption
public_key- The unencrypted public key of the user
private_key_iv- The Initialization Vector for encrypting and decrypting the user’s private key
private_key_aes256cbc- The encrypted value of the user’s private key
- Encrypted using AES-256-CBC and uses the user’s intermediate key for encryption
Table: DataEntity
id- Primary key identifying a data entity
user_id- The identifier of the user owning the data within the data entity
value_iv- The Initialization Vector for encrypting and decrypting the value of the data within the data entity
value_aes256cbc- The encrypted value of the data within the data entity
- Encrypted using AES-256-CBC and uses the user’s intermediate encryption key for encryption
Table: RecoveryCode
id- Primary key identifying a recovery code
user_id- The identifier of the user owning the recovery code and user key contained within the recovery code entity
recovery_code_salt- Salt used to compute the recovery code’s BCrypt hash
recovery_code_bcrypt- The BCrypt hash of the recovery code
- Used for authenticating the recovery code
pbkdf2_keysalt- Salt used for computing a recovery code-based encryption key using PBKDF2
user_key_iv- Initialization Vector for encrypting and decrypting the user’s intermediate key
user_key_aes256cbc- Encrypted value of the user’s intermediate key
- Encrypted using AES-256-CBC and uses the recovery code-based encryption key for encryption
Table: SharedKey
id- Primary key identifying a shared key
shared_key_salt- Salt used to compute the BCrypt hash of the shared key
shared_key_bcrypt- The BCrypt hash of the shared key
- Used for authenticating decrypted versions of associated copies of the shared key
Table: SharedKeyCopy
id- Primary key identifying a copy of a shared key
key_id- The identifier of the shared key that an instance derives from
owner_id- The identifier of the user owning the copy of the shared key
shared_key_rsa- Encrypted value of the copy of the shared key
- Encrypted using RSA and uses the Public Key of the owning user for encryption
- Decrypted using RSA and uses the Private key of the owning user for decryption
shared_key_iv- Initialization Vector for encrypting and decrypting the copy of the shared key which was encrypted using AES-256-CBC
- Initially empty at first, but is filled up once the user first accesses the clear-text value of the shared key
shared_key_aes256cbc- Encrypted value of the copy of the shared key
- Encrypted using AES-256-CBC and uses the user’s intermediate key for encryption
- Initially empty at first, but is filled up once the user first accesses the clear-text value of the shared key
- The application should check whether or not this value exists first before trying to decrypt the copy of the shared key encrypted using RSA
Table: SharedDataEntity
id- Primary key identifying a shared data entity
key_id- The identifier of the shared key that was used to encrypt the value of the data contained within the entity
value_iv- Initialization Vector for encrypting and decrypting the value of the data contained wihtin the shared data entity
value_aes256cbc- The encrypted value of the data within the data entity
- Encrypted using AES-256-CBC and uses the shared encryption key for encryption
Problem: User Password Compromise
Of course, there’s always the chance that the user’s password can be compromised by an attacker. If an attacker gets access to the user’s password and also gets a hold of the database, they would be able to decrypt the user’s data. However, they could also just walk right in through the front door and login using the user’s account in the application and then access the user’s data from there, but I believe that it would be best practice to ensure that decryption of user data can only be done with the consent of both the application and the user, so I would suggest that you optionally add another layer of encryption to your users’ data by additionally encrypting them using an application-wide encryption key.
Solution: Encrypt With Application-Wide Key
You don’t really have to do this, but if you want to ensure that your data can only be read by the appropriate user using the appropriate application, you can add another layer of encryption on top of the encryption you’re already doing with user-specific encryption keys.
Honestly, we’re really just back to square one here. But instead of solely relying on the common approach, we’d be applying it on top of the approach we’re already doing with user-specific keys. So, basically, as described earlier, the common approach is having a symmetric encryption key stored on a configuration file on the application server, and then using that to encrypt every piece of sensitive data that we store in the database. This way, the adversary would have to compromise both the application server and the user’s password in order to be able to read data directly from the database. It won’t stop them from just waltzing into the application and logging in as the user though, as stated earlier on.
What Data to Encrypt and How to Encrypt Them
Here’s a short taxonomy of what kind of data you should encrypt, how to encrypt them, and which keys you should use when encrypting.
- Secrets
- Password
- Algorithm: BCrypt
- Key: N/A
- Intermediate Key
- Algorithm: AES-256-CBC
- Key: PBKDF2 of Password
- Private Key
- Algorithm: AES-256-CBC
- Key: Intermediate Key
- Shared Key
- Non-cached Copy
- Algorithm: RSA
- Key: Public Key
- Cached Copy
- Algorithm: AES-256-CBC
- Key: Intermediate Key
- Non-cached Copy
- Password
- Sensitive Data
- User-only
- Algorithm: AES-256-CBC
- Key: Intermediate Key
- Shared with Specific User
- Non-cached Copy
- Algorithm: RSA
- Key: Recipient Public Key
- Cached Copy
- Algorithm: AES-256-CBC
- Key: Shared Key (between owner and recipient)
- Non-cached Copy
- Shared with a Group
- Algorithm: AES-256-CBC
- Key: Shared Key (between members of the group
- User-only
One final note: don’t go nuts and just encrypt everything. You’d want to store as much data in unencrypted form as your security policy will allow because 1) encryption consumes computing resources and 2) you can’t perform queries based on the contents of an encrypted column.
/john.peñafiel/ 