What does it mean when two encrypted files, encrypted with the same method but with different keys, when xor'd together, produce a repeating pattern?

Question

Let's say I have fileA and fileB, both encrypted somehow, but in the same method with different keys.

If I xor them together, they will make a repeating pattern of 32 bytes length.

What does that say about the repeating pattern, the encryption pattern, and the likelihood of the unencrypted files to be identical?

Is there any way to extract the keys used to encrypt them?

Is there a way, other than to open up Ollydbg and try to reverse engineer the decryption procedure?

The program contacts a server, sends a certain unique code, and the server replies back with an md5 hash. Then the server sends back the encrypted file, which is easily decrypted by the program. I want to reverse engineer this program and encryption method.

This is not a homework question, this is a real situation that is tormenting me.

The decrypted plaintext is either Lua source or Lua VM bytecode, with a small possibility of being XML.

Edit: I now realize that the repeating sequence is probably two different MD5 hashes XOR'd with each other. Is there a method to analyze and try to extract the individual MD5 hashes?

Answer 1

The two plaintexts are almost certainly identical. At the very least, any difference between them must be representable in 32 bytes, since that's as much information as the XOR of the encrypted files contains.

Assuming that the plaintexts are indeed identical, we can also see that changing the encryption key has a very simple and predictable effect on the ciphertext: every 32nd byte is affected the same way. Clearly, this cannot be a semantically secure encryption scheme.

Indeed, the most likely explanation of the phenomenon you observe is that:

1. the two plaintext files are identical, and

2. the encryption simply consists of taking a 32-byte key (or possibly the 32-byte hash of a key) and XORing it with each 32-byte block of the plaintext.

If true, this is a very weak cipher, and it should not be hard to break. A couple of methods to try would be:

• If you can guess some part of the plaintext, you can immediately get the corresponding part of the key by XORing the guessed plaintext with the ciphertext. For example, if the plaintext is XML, there's a good chance it'll begin with something like:

  <?xml version="1.0" encoding="utf-8" standalone="no"?>
  

  With a bit of luck, this can reveal the entire key! Similarly, I believe Lua bytecode files should have a more or less fixed header that should also reveal at least part of the key.

• Even if you can't directly guess the plaintext, you can make use of the fact that the key repeats, essentially making the cipher a many-time pad.

  One way to attack such a many-time pad is to use crib dragging. Basically, you guess an $n$-byte string that's likely to occur somewhere in the plaintext (such as, say, function in Lua code) and XOR it with each $n$-byte substring of the ciphertext. Each XOR will yield a candidate $n$-byte substring of the key, which you then XOR with the corresponding parts of the ciphertext. If the guess was wrong, you'll mostly get garbage, but if and when you guess right, the decrypted plaintext suddenly starts making sense.

  Another way to attack such a cipher is to split the ciphertext into 32-byte blocks, take one of the blocks and XOR all the other with it (which cancels out the key, leaving you with an XOR of the plaintext blocks), print out the XORed blocks as a hex+ASCII dump (as with xxd -c 32) and examine the columns. With the help of an ASCII table, you can infer a lot about the plaintext block you chose to XOR with the others (and thereby about the key).

Alternatively, if you're feeling lazy, you can just download xortool and feed one of the ciphertexts to it. With a bit of luck (and the right options), it'll spit out the key and the plaintext (or at least something close enough for you to finish the job by hand) for you.