How to check that you got the right key when brute forcing an encryption?

Question

How do you know when you have the right key when brute-forcing? Let's say that they test the right key. They then have to check that the decrypted text makes sense. To do so, they can test whether there is a word in the dictionary.

But then let's say that the message is encrypted with Enigma then with RSA encryption. They try to brute force the RSA encryption but how do they do to know that they got the right key, since the text itself is encrypted with Enigma and has no word contained in the dictionary ?

Answer 1

First, you do not break RSA through brute force. RSA is an asymmetric encryption algorithm, with a public/private key pair. The public key has a strong internal structure, and unravelling it yields access to the private key (basically, the main component of the public key is the modulus, which is a big composite integer, and the private key is equivalent to the knowledge of the prime factors of the modulus). This can be done a lot more efficiently than trying out possible private keys; in fact, trying prime factors one at a time is a method known as trial division, and is about the least efficient of all integer factorization methods.

The bottom-line is that the attacker perfectly knows when he has found the private key, and he does not need to see any encrypted text for that.

---

Now let's suppose that you are not talking about RSA encryption, but a symmetric encryption algorithm such as AES. In that case, your question makes more sense, because:

1. In a symmetric encryption algorithm, there is no public key to work with, so the attack must proceed from some intercepted ciphertexts.

2. In symmetric encryption algorithms (at least the "secure" ones), the best known attack method is, indeed, brute force. Brute force is the generic method that necessarily works (at least in theory -- in practice we use keys that are large enough to make it totally infeasible); an encryption algorithm is said to be secure if no better attack method is known.

Enigma uses as input and output sequences of latin letters -- the 26 letters from 'A' to 'Z'. Modern encryption systems work on bits and octets, i.e. "binary data". If you cascade AES with Enigma, then you need some conventional representations of letters into bits, e.g. ASCII. You would, for example, map each letter of the Enigma output into an 8-bit byte, to be used as input to AES. Since 26 is not a power of two, it is unavoidable that this mapping will have "holes": there necessarily are some sequences of bits that could be a valid AES input, but are not possible as an encoded Enigma output.

This thus yields a valid strategy for our attacker: for each possible AES key, decrypt the AES layer, and see if the result is a valid encoded Enigma output. If not, the AES key is wrong, and proceed with the next one. If you used ASCII encoding, then a 16-byte output from AES decryption is a valid Enigma output encoding only with probability 1 in 7803076729492126.75 (approximatively). This comes from the fact that there are 256¹⁶ possible sequences of 16 bytes, but only 26¹⁶ possible sequences of 16 uppercase letters. This is an awful lot of power of recognition of bad keys by the attacker. If the attacker has access to two encrypted blocks (32 bytes, corresponding to 32 characters of Enigma output), then he will be able to pinpoint the single AES key that "makes sense". In all of this, the attacker never has to even begin to consider how he will break through the Enigma.

---

Of course, the above is quite artificial. You may imagine cascading algorithms that live in the same world; e.g. two symmetric encryption algorithms that both work over bits.

Cascading algorithms is a method of construction of bigger encryption algorithms. It is not a very good one, though. For instance, suppose that you cascade AES with another block cipher such as Serpent; or maybe you use two AES instances with two keys chosen independently. The hope may be to thereby produce a synthetic block cipher that will have twice the key length. But it does not deliver all these promises: the double-encryption method falls prey to the Meet-in-the-Middle attack (not to be confused with the unrelated Man-in-the-Middle attack). This schema may be enlightening:

So, by cascading two algorithms, you double the operational cost, but you do not get your money worth in security increase. To really get extra security, you must go to triple-encryption. This is why Triple-DES is triple-DES, not double-DES: the latter would not be substantially stronger than simple-DES.

---

It still is much simpler not to use weak algorithms at all. If you cascade AES with anything else, then either the "anything else" is completely useless, because the AES is already strong enough to defeat brute force attacks; or you use the AES really poorly and that is what you should fix first.

Answer 2

Let's presume the attackers are fully aware of how this information is being encrypted. The attackers know that (in your example) the plaintext is being encrypted with Enigma and then with RSA.

When attempting a bruteforce attack, they will first test a key against the RSA encryption. At this point, the attackers can check whether the output data makes sense assuming enigma encryption (I'd assume they could check to see if the data out putted is all ASCII characters for instance). Then they can move on to trying to attack the enigma encryption.

Answer 3

In modern cryptography attackers are considered that are allowed to change the ciphertext. It's because of this fact, that the decryption always either outputs an error or the correct decryption. So you can verify your key guess by observing that the authenticated decryption succeeds.

This is especially true for RSA(-OAEP) which must have this sort of mechanism or otherwise you can recover any encryption by exploiting it's homomorphic property.

So in your example the attacker guesses the RSA key until he hits that delivers a valid decryption and uses this as the basis for breaking the enigma ciphertext.

Answer 4

but how do they do to know that they got the right key since the text itself is encrypted with Enigma and has no word contained in the dictionary ?

They usually don't.
Of course you could combine several encryption algorithms in a way that you could find out if you cracked one level of encryption, like by adding a checksum which is only valid if the encryption got broken. But a properly designed encryption scheme would not allow for such information leakage because this would weaken the encryption.

Answer 5

You suggest combining two encryption methods. In principle, either an attacker can figure out how to effectively split your encryption into two independent parts that can be cracked independently, or they can't. An infamous case is the encryption used for encyrpting DVDs: It's a 40 bit encryption, but it could be split into one 16-bit encryption, plus one 25-bit encryption (not 24 bits, there was some overlap). That split made it easily crackable ten years ago.

Answer 6

As the others have stated in their answers, the answer to this interesting question is basically: Known Plaintext.

Basically, the attacker must know some of the properties of the plaintext beforehand to decide if his decryption yielded a plausible result. The assumption that only ASCII letters are found in the plaintext already is knowledge about the plaintext. Assuming that most plaintext fragments must contain words from a dictionary is also helpful.

This leads to the question if and how this type of attack can be countered. Double- or triple-encryption increases the effort the attacker has to spend but does not mitigate the original problem.

One could, however, for certain use cases conceive an algorithm that can map a give plaintext to a certain encoded representation and when reversed will map any input to plausible plaintext. Using this pre encryption and post decryption avoids the known plaintext attack vector mentioned above.

Example:

I have an account number, say 1234567890. This means, that any valid plaintext can only consist of the digits 0-9. Lets say that those digits are represented by 8-bit values between 0 and 9. This plaintext has the inconvenient property that in every 8-bit value the upper 4 bits are always 0 (4.83 bits on avarage).

Now, before encryption, I preprocess my plaintext by adding a random number from {0*10, 1*10, 2*10, ..., 24*10} to each of my plaintext's bytes; then encrypt.

The decoding of the decrypted text is just b mod 10. This decoding can be applied to any (random) input and will always produce plausible account numbers. If the output is the correct account number cannot be inferred from any result of a decryption attempt.

The same could be used to store encrypted passwords. Your mapping algorithm would decode any input to a sequence of characters from e.g. a-zA-Z0-9.!,+- without disclosing anything about the validity of its input.

(All the above is of course assuming that there's no other known pattern in the plaintext of the encoded representation. For the account number example, byte values 250...255 will not be present in the encoded form, which gives the attacker an advantage of 2.59 bits per byte on avarage; but that's still better for us than the 4.83 bits he had prior to the encoding.)