2

I know that a simple monoalphabetic substitution cipher is considered extremely weak, on account of linguistic frequency-analysis attacks. However, assume the following:

  • cleartext is encrypted (with 256-bit AES, for example), and the resulting cipher is in base64 format
  • the client has a key of 64 chars (a-z, A-Z, "+/"), where each character is randomly mapped to another character (i.e., 'a' -> 'Y', 'b' -> 'f', 'c' -> '/' etc)
  • a simple monoalphabetic substitution cipher is applied to the base64 ciphertext

Aside from the practicalities of implementing this approach, would this add much security? I argue that it would, taking into consideration the following points:

  • neither base64 encoded text nor the raw encrypted data it represents are subject to frequency-analysis attacks
  • the 'key space' (to use the word loosely) of the base64 character set, represented as a factorial (64!) = 1.2688693e+89, or roughly 2^296 (if I calculate correctly).

So, in the case of a brute force attack, an attacker would need to try 2^296 key combinations, and for each one of those, they would need to go through the normal process of brute-forcing a 256-bit AES cipher (already unfeasible).

In the case of a break in the underlying algorithm (AES, in this case a break would be highly unlikely) the attacker would still need to search the 2^296 keyspace to find the original ciphertext.

There has been plenty of discussion about double-encryption, and how it adds little more security (in terms of orders of magnitude), but I would like to know people's thoughts about the approach described above. I DO understand that it's complete overkill, and that 256-bit AES encryption is more than sufficient by itself, but I'm interested to hear opinions at least from a theoretical point of view. Does this potentially add much security?

hunter
  • 3,965
  • 6
  • 28
  • 42
  • 3
    your systems seems to be vulnerable to a meet-in-the-middle attack and is therefore only as strong as the strongest cipher (at best). – mikeazo Dec 31 '12 at 12:50
  • @mikeazo: I'm sceptical that simple monoalphabetic substitution used in combination with a block-cipher like AES would be vulnerable to a meet-in-the-middle attack. But thanks for pointing it out - it's worth the consideration. – hunter Jan 02 '13 at 17:28
  • Why wouldn't it be? It's the exact same construct. – Stephen Touset Jan 02 '13 at 18:47
  • @mikeazo (and StephenTouset) - after doing some further reading, I stand corrected. Thanks for bringing it to my attention. – hunter Jan 03 '13 at 13:37
  • Your assumption: "the client has a key of 64 chars (a-z, A-Z, "+/")" is a pretty bad assumption due to the fact that a key itself should be a binary key generated by using a (pseudo) number generator or - if a password is the only input - an output of a password based key derivation function (PBKDF) such as PBKDF2. – Maarten Bodewes Jan 03 '13 at 21:10
  • @owlstead: the key for the initial AES encryption would be, of course, as you've described. The 64-char key is for the monoalphabetic substitution cipher. – hunter Jan 03 '13 at 21:38
  • @hunter That explains. I must have been brainwashed by the many questions on stackoverflow, where this is common... – Maarten Bodewes Jan 03 '13 at 21:40

2 Answers2

6

The fact that a given cipher has a key length of 296 bits doesn't mean at all that it provides 296 bits of security or even that a brute force attack would take $2^{296}$ steps. The problem of mono-alphabetic substitution cipher is the ridiculously small block size (in this case, barely $\log 64 = 6$ bits).

If absolutely nothing about the plaintext is known, every cipher is unbreakable, as there's no way of knowing if you got the right result.

In practice, you always know something about the plaintext. If that's not enough to launch a known-plaintext attack, your method might add some security form a theoretical point of view, but it's rather pointless in practice, since brute-forcing a 256-bit key is just as feasible as brute-forcing a 552-bit key.

However, if you do know enough plaintext, the mono-alphabetic substitution cipher doesn't add any security at all. After encrypting the known plaintext with an AES key, all you have to do is check if matching characters get mapped into matching characters. If they don't, discard the key; no further tests are required.

Dennis
  • 2,151
  • 15
  • 21
  • 3
    The trick of encrypting the text with the known key works only if the encryption process is deterministic. If, say, CBC with unpredictable IVs are used, it's somewhat trickier. You could iterate through all $2^{128}$ IVs; however, it's cheaper to look through the known ciphertext and find the block that has the fewest unique base64 digits; for a ciphertext of 10,000,000 blocks, there's a good chance of a block somewhere with only 10 unique characters; that reduces the effort to about $2^{59}$ cipher operations per key checked. – poncho Jan 01 '13 at 19:26
  • @poncho - thanks - your explanation of how to perform an attack in this scenario has helped me to understand its vulnerabilities. I had indeed intended to use CBC mode with a CSPRNG-generated IV. – hunter Jan 02 '13 at 17:44
  • @Dennis - thanks for pointing out the small block-size of mono-alphabetic substitution, I hadn't considered that - you're absolutely right. As noted by poncho, using an IV somewhat mitigates the risk of known plain-text attacks. – hunter Jan 02 '13 at 17:55
3

Encrypting the AES key does not actually make a brute force search any harder: an attacker doesn't need to know the encrypted key to decode messages, they only need to know the actual AES key. Thus, the attacker only(!) needs to search the 256 bit AES keyspace, not the roughly 296+256 = 552 bit encrypted keyspace.

Besides, even if the attacker did try an exhaustive search of the 552-bit keyspace, those keys still fall into 2256 equivalence classes, such that every encrypted key in a class maps to the same AES key. Thus, the attacker can still be expected to find the right key after only(!) about 2256 trials.

As for super-encrypting the ciphertext with a monoalphabetic substitution, as Dennis notes in his answer, this adds no appreciable security as long as the attacker has enough known plaintext / ciphertext. (70 bytes of plaintext should be just about enough, but let's make it an even 80 or 96 just to be sure.) All the attacker needs to do to test a given AES key is to encrypt the plaintext with it, base64-encode the output and check whether all matching pairs of characters occur at the same positions as in the known ciphertext.

(One efficient way to do that would be to canonicalize each base64-encoded ciphertext by mapping the first distinct character in it to 0, the second to 1, the third to 2, and so on, but so that any previously seen character is always mapped to the same value it was originally assigned. This way, two ciphertexts map to the same canonical sequence if and only if they can be transformed into each other by some monoalphabetic substitution.)

Having found the correct AES key, and still having the known plaintext and ciphertext used to do that, determining the monoalphabetic substitution table is then completely trivial.

Ilmari Karonen
  • 46,120
  • 5
  • 105
  • 181
  • encrypting the AES key was never really up for discussion, but thanks for considering it. As for further encrypting the ciphertext, your attack is one that I'd considered, but (as mentioned above by poncho), the risk of known-plain-text analysis is mitigated by the use of a strong IV. It is, however, feasible. – hunter Jan 02 '13 at 18:01