How to exchange keys despite a MITM, using some short known password?

Question

The simple scenario is two devices on one WiFi network sitting side by side. The user can copy a short code from one device to the other to be the "password" however this code can't be too long. Let's assume it's at most 12 decimal digits. The devices will only allow one attempt so any MITM attacker will have to take a chance with only 1 in a trillion for success, which will also alert the user that he (the attacker) is there, so that would be secure "enough" for me. This is all there is, that is, there is no server which both devices have its public keys.

But how would this work? I thought of something like the following but ran into a problem. If there's a standard solution for this key-exchange scenario, please feel free to ignore the following:

I thought device d1 can create a random number in the above mentioned range and use it as the key for Rijndael encryption of its public encryption key but without a need for padding so that all ciphertexts can be "decrypted" so that the MITM won't be able to brute force which is correct by being alerted that a wrong one is wrong. The second device d2 then uses that password to decrypt the ciphertext (which is transmitted over wifi). It then uses it to encrypt and send d1 its (d2's) keys. The idea is that the MITM won't know which of the trillion possible deciphered texts is correct. And even when he (the MITM) sees the message back from d2, won't be able to brute force it, because the plaintext of the sent ciphertext will be random-like (because it's a public key of an unknown key pair) and will be encrypted like the first one, where there is no padding and therefore an attacker can't see whether they got the correct plaintext or an error.

Unfortunately this might not work because the MITM might be able to preemptively craft a set of keys (- a trillion decipherings of some key) where the key used can be detected from the ciphertext, and send that key to d2 and then discover the "password" from d2's response. Can this be done?

Also, the usual rule of not rolling out my own encryption scheme.

So, is there a safe way to exchange keys despite a MITM, using some short known password? (Preferably with forward secrecy such that brute forcing, if it takes a month, won't give the attacker the ability to start impersonating one of the devices from then on.)

Answer 1

This is called the Socialist Millionaire Problem. A solution for it exists, as published in this paper.

According to Wikipedia:

It is often used as a cryptographic protocol that allows two parties to verify the identity of the remote party through the use of a shared secret, avoiding a man-in-the-middle attack without the inconvenience of manually comparing public key fingerprints through an outside channel. In effect, a relatively weak password/passphrase in natural language can be used.

The protocol allows two parties to know whether or not they both share the same secret, such as a password, without transmitting it. If the other side, which may be an MITM attacker, is not able to prove that it has the same secret, then the session can be terminated before sending any sensitive information. Neither side learns any information about the secret in the process, other than the fact that the shared secrets did not match. If it succeeds and they do match, then they're MITM-free.

This technique is used in the OTR protocol. Unauthenticated Diffie-Hellman key exchange is done first, after which SMP is performed within the encrypted channel. Further communication is denied unless the other side can prove that it shares the same secret and thus is not an MITM attacker. In order to successfully perform an MITM attack, the attacker would need to either correctly guess the shared secret in its entirety by performing multiple connection attempts and re-trying on each failure without alerting anyone to the attack, or solve the discrete logarithm problem (DLP) that underlies Diffie-Hellman security, which is currently thought to be hard when correct parameters are chosen.

A simplified explanation of the protocol and its properties as used in OTR is available here.

Answer 2

I thought device d1 can create a random number in the above mentioned range and use it as the key for Rijndael encryption of its public encryption key but without a need for padding so that all ciphertexts can be "decrypted" so that the MITM won't be able to brute force which is correct by being alerted that a wrong one is wrong.

Relying on the indistinguishability from random for a public key is a highly unusual security property and one that requires careful consideration of the schemes involved to not be violated.

Luckily, there's another alternative. Either you can use a balanced PAKE where one device generates the password and the user transmits it to the other - which is what the question tried to build. Or given that these "passwords" actually have no good reason to be picked beforehand we can use a Short-Authentication-String based key-exchange. Sylvain Pasini's PhD thesis (PDF) is an excellent resource for that. In particular it offers a pre-instantiated authenticated key exchange in Figure 10.5:

• Parameters: A secure hash function $h$ like SHA3-256 or SHA-256, a Diffie-Hellman group $\mathbb G$ with a generator $g$, e.g. x25519 or secp256r1 or secp256k1, and a commitment scheme, e.g. Pedersen Commitments, a function $f$ that maps a 256-bit string to a human readable and verifiable (shortened?) representation while preserving as much entropy as possible, e.g. by interpreting the input as a 256-bit integer and performing a modular reduction $\bmod 10^{12}$.
• Alice picks a private DH key $x_A$ and computes $y_A\gets g^{x_A}$ - the DH public key. Alice then also picks a 256-bit uniformly random bit string $K$ and commits to both $y_A$ and $K$. Finally, Alice sends the commitment and the public key to Bob.
• Bob also picks a private DH key $x_B$ and computes $y_B\gets g^{x_B}$ and also picks a 256-bit string uniformly at random, called $R$. Bob transmits $y_B$ and $R$.
• Alice then sends the opening of the commitment to Bob. Alice displays the short authentication string (SAS) $f(R\oplus h(y_B,K))$.
• Bob now learns $K$ and confirms that the commitment was to $K$ and $y_A$, if it wasn't he aborts. Bob displays the SAS $f(R\oplus h(y_B,K))$.
• The user confirms that both displayed SAS strings match (on both devices). If a device doesn't receive a confirmation or if the user indicates different SAS to a device, it aborts.
• Alice outputs $y_B^{x_A}$ as the shared secret.
• Bob outputs $y_A^{x_B}$ as the shared secret.