1

Given an encrypted file, what are possible ways to find its encryption algorithm?

Heuristic ways are okay as well.

Azad
  • 191
  • 1
  • 5
  • There is no general answer. – Steve Peltz May 06 '15 at 19:35
  • I asked for possible ways. I know there won't be any general, work-all-the-times, exact answer. – Azad May 06 '15 at 19:38
  • Is it safe to assume you are only interested in modern ciphers? Also, do you want to place a limit on the size of the encrypted file? With unlimited data, there may be some things you can do. Also, by "encryption algorithm" do you mean "AES vs Serpent vs 3DES", that sort of comparison, or are you interested in modes of operation too? – mikeazo May 06 '15 at 19:42
  • You could look at the size of the file, you could look at characteristics at the beginning or end (e.g. markers indicating algorithm, length, signature method and encoding), but that would only cover a small subset of possible file formats and algorithms. For example, I could simply encrypt the file by XOR with a stream cipher with the key based on file name and password. You could check to see if decrypting it with a given cipher and password yields a file that makes sense, but using a different password-derived key algorithm or parameters effectively changes the cipher as well. – Steve Peltz May 06 '15 at 19:46
  • Thanks. Between 1 to 10 MB. I think it should be modern. I've already checked byte frequency and two byte histogram. No particular pattern almost flat with white noise. Number of bytes not a multiple of any special number eg 16 (actually not even 2). I'm interested in mode of operation too. – Azad May 06 '15 at 19:59
  • The problem is that algorithms don't encrypt files, and the number of encryption programs that are known (available publicly, with a known method of operation) is huge, and the number of unknown programs is much larger. Many (of both kinds) can only be verified if you actually guess the correct password (and even then, for some you'd still need to determine if the file makes sense for each guess, e.g. run it past the Unix "file" command). Someone determined to keep you from even knowing what was used to encrypt it could simply run it through two such programs. – Steve Peltz May 06 '15 at 20:29
  • 1
    One possible clue that would reduce the possibilities might be if you know the decrypted size of the file. Also, keep in mind that it may not even be a password, just a random 256-bit key. There's essentially no way to determine the difference between having the wrong key or the wrong algorithm if there are no structural clues. – Steve Peltz May 06 '15 at 20:30

1 Answers1

2

One of the important properties of a modern cryptographic algorithm is that it resist distinguishing attacks. Distinguishing attacks are any way to tell the encrypted data apart from random data. So if you can tell what the algorithm used was simply by looking at the ciphertext the algorithm is vulnerable to this attack.

Unless there is a header or such that tells you the algorithm and mode used, or a weak cipher was used, determining the algorithm used from the ciphertext is impossible.

SAI Peregrinus
  • 5,836
  • 19
  • 26
  • 1
    impossible is a strong word, maybe something more along the lines of "not significantly easier than trying all possible keys with all possible algorithms" – Mike Ounsworth May 06 '15 at 23:57