How to securely store data? / How to securely encrypt a file?

Question

How do I safely encrypt a file?

or formulated differentely:

How do I safely store the data of my application in a file?

Note: Using established tools isn't an option for this. The tasks need to be done at program level (using libraries and such).

Answer 1

1. Ask yourself if you even can securely store the user's data.
   This includes to check that the user won't get sniffed by the root / admin user.
   Also make sure that your implementation is secure.
   Employ standard code review mechanisms, make sure that your implementation has countermeasures against timing attacks (like against AES).
   Also make sure that you use appropriate data types.
   Using a byte* allocated using malloc usually isn't very secure. If your library provides mechanism for data storage, use them!
   Your OS may also provide means to protect data while in memory, for example the data protection API in Windows only relies on the current user and can be used to conceal sensible keys in memory. Reading Schneier's Cryptography Engineering may be a very good idea for you, to ensure you don't have side-channel leaks.
2. Ask yourself what you can use to authenticate the user. The options are (it is explicitely allowed and recommended to combine as many of them as convenience and availability allow):
   1. Passwords: Your user can supply a password or a passphrase. Do not limit him in length of this, to allow him to use good and long passphrases. This usually is a good choice, as it requires the user to know something and this hence can not be lost like an usb-stick.
   2. Keyfiles: Allow the user to provide keyfiles. These files may look arbitrary, but should be rare. Allow the user to use the (cryptographical) random number generator of your application to generate strong keyfiles (of 64 bytes length or more). Examples for keyfiles are rare documents, like personal (word/) text documents or on-purpose generated files just containing random data (created using the application's random number generator).
   3. Custom Hardware: If you are deploying in large scale you can force the user to be required to use a tamper-resistand hardware device. This device should support symmetric encryption with a device-held key.
   4. User's Hardware: Allow the user to use his PKCS#11 or similar smartcard or cryptographic token, which provides public key encryption, to be used. Also allow the user to protect the key using the TPM (if available)
   5. OS data: Allow the user to make the decryption dependant on something stored in the current OS, like the user credentials or OS provided secure key storage.
3. Inspect what algorithms you have at your disposal. You'll need (at worst) five classes of algorithms: Password-based key derivation functions, encryption algorithms, modes of operation, message authentication codes and cryptographically secure random number generators.
   At the time of writing the following algorithms are recommended (in descending order). If functionality is already provided more top-level don't use it.
   1. Cryptographically secure random number generators:
      1. Fortuna
      2. An approved algorithm. This may either be the result of a contest or some approved algorithm by some trusted institute (e.g. ANSSI, NIST, BSI, ...). Avoid algorithms with "provable" security where the institute provides the parameters, these parameters may be chosen on purpose to get back-door access. For seeding use the cryptographically secure random number generator of your OS. They are usually a bit hidden.
      3. Some "weak" standard approach. (i.e. run a blockcipher in CTR mode and delete key and IV as soon as possible) Get the key and the IV from the cryptographically secure random number generator of your OS. They are usually a bit hidden.
   2. Password based key derivation functions:
      1. The winner of the PHC-competition. If there are multiple winners, ask for the differences and use the one most suited for your needs.
      2. scrypt
      3. bcrypt
      4. PBKDF2, using an efficient and secure hash function (on your platform)
      5. implement your own PBKDF2!
   3. Ciphers:
      1. The winner of the CAESAR-competition (2018+). If there are multiple winners, ask for the differences and choose the appropriate one.
      2. AES. Note: For data that needs to be confidential unti 2070 or earlier, it is recommeded to use AES-128. In every else case use AES-192 or AES-256.
      3. ChaCha(preferred) or Salsa20.
      4. One of the following three: Threefish, Serpent or Twofish
      5. 3DES with 3 keys. (needs 168 bits of keying material)
   4. Modes of operation (this is obsolete for CAESAR-winners):
      1. OCB (If the patent has expired)
      2. GCM
      3. EAX (preferred) or CCM
      4. CTR, below this a message authentication code is required (including)
      5. One of the following: CFB or OFB
      6. ECB, avoid at all cost, rather implement your own CTR!
   5. Message authentication codes:
      1. Poly-1305 using a standardized construction
      2. HMAC, using a secure standardized hash function, preferably SHA-3 or a member of the SHA-2 family. Encrypt the data first, then authenticate the ciphertext!
      3. implement your own HMAC!
4. Ask yourself on how to design the header. The header should contain:
   • The salt, a random 64 byte or more value that will be newly chosen on each single creation of a new file. This will be fed into the password-based key derivation function. The salt can be open, but may be authenticated using the authentication mechanism.
   • The parameters for the password-based key derivation function. These parameters need to be open. They should be chosen such that key derivation needs approximiately 100ms to 1s on the target platform. A value indicating the password based key derivation function used is considered a parameter.
   • A nonce or an IV for the cipher / mode. This value should be chosen at random and be stored openly.
   • The remainder of the header should be encrypted and authenticated.
     The before-mentioned nonce/IV should be used along with the key derived from the password using the password-based key derivation function and the stored salt and parameters.
   • The master-keys for the data part of the file. These should be random and maximal length. This is part of the encrypted part of the header
   • A nonce / IV for the data part. This value should be random. This value can (and should) be part of the encrypted part of the header.
5. The actual data is now encrypted using the above chosen cipher/mode/authentication triple and the key/IV/nonce pair of the encrypted part of the header.

Now some question that may arise:

1. Is it worth it to use multiple-encryption?
   No.
   The benefit by double encryption is only 1-bit due to the meet-in-the-middle attack. Streamciphers may be combined easier, but it is absolutely unneccessary as all ciphers are unbreakable for the forseeable future (50 years+)
2. What authentication methods should I use?
   As many as possible, but at least password
   You need to have a very good reason not to use passwords because everything else can be attacked with admin rights or can be stolen.
3. You didn't mention the other four authentication methods in the header part, how should I incorporate them?
   2. Keyfiles: Assume that each keyfile has low entropy. That being said, you can't really do alot against it. Derive a key from every keyfile using some digest-expansion function and your favorite secure hash function (SHA-2/3) and construct a big XOR consisting of all file's hashes and the user's password. This would look like this: $Key = PBKDF(PW) \oplus H (File_1) \oplus H(File_2) \oplus ...$, where $\oplus$ denotes bitwise XOR.
   3. Custom Hardware: At the creation of the file let the device generate a symmetric key that will be kept inside the device. Perform everything up to the keyfile / password level (see directly above point) and then send this to the device for encryption. The key being used doesn't have to be unique for each file. It is suggest that one key is being used for all files. It is also possible to add an identifier to the file to identify the associated hardware key.
   4. User's Hardware: Let the device generate an OpenPGP key or a valid S/MIME encryption certificate. Send the key being the result of the step directly above (or 2 steps above) to the device for decryption. It will decrypt it and return the key used to decrypt the master-keys.
   5. OS data: If you have user credentials (maybe some random strings), hash them and incorporate them at the same time in the same way as the keyfiles. If your OS provides you secure storage (or your TPM does), store a key and pass it in as "keyfile", thus hash it and incorporate it in the XOR step.
4. Should I use an open source library or some closed-source library?
   Open-Source
   Chances are very good that there aren't any security issues in well-known open-source libraries, whereas closed source puts restriction in regard of when you can use it, usually you have to pay to be able to use and you can't usually inspect the code on your own to make sure it's secure.
5. Why did you use XOR to include keyfiles?(relates to question 3)
   Convenience.
   If you don't use something that is commutative, like XOR, users will run into problem because files have to be supplied in the exact same order. This gets even harder if files are just re-named but their contents are kept. And from a security perspective this is no less secure as one basically constructs a "stream cipher" with the password based key derivation function output, the hashes of the keyfiles and the other outputs (TPM, credentials,...) which is secure as long as one source is secure.