Sign MD5 hash with RSA

Question

I have to hash a message with MD5, digitally sign it, and then verify.

I have looked around, but I can't find a step-by-step example. I understand how RSA works when a message is given as number (not hashed). But I am confused when I have to sign a hash.

For example: $n=33, d=7, e=3$.

I have a message: Hi
I compute the MD5 hash, $H(m)$, which is c1a5298f939e87e8f962a5edfc206918.
Now I have to sign it with $S = H(m)^d \bmod n.$

I get confused here: How can I perform this $H(m)^d$? What will the result be after that?

You should tell your instructor that MD5 is COMPLETELY BROKEN and shouldn't be used even for a class example. You should only be using SHA256, and even then signatures require padding. I know it's only an exercise but it's teaching bad practice. — Yehuda Lindell, Dec 29 '17 at 10:02
@Sari: I'm happy to have the duplicate flag contested (that is the point after all), but the answer to the linked question seemed to me to provide sufficient details: it explains the importance of padding and points to working code to do this. — Marc, Dec 29 '17 at 14:56

score 2 · Answer 1 · edited Jun 17 '20 at 08:17

First read all the other answers about how this naive approach to RSA signatures is dangerous.
Have you read them all yet? No? You missed one? Go back and read that one to the end.
Now that you've read them we can start.
$H(m)$ is a string of 16 bytes, or 128 bits: 11000001 10100101 00101001 10001111 10010011 10011110 10000111 11101000 11111001 01100010 10100101 11101101 11111100 00100000 01101001 00011000. Interpret this as the binary expansion of an integer between $0$ and $2^{128} - 1$: if $b_i$ is the $i^\mathit{th}$ byte, the integer you choose can be $$x = \sum_{i = 0}^{16 - 1} b_i 2^{8i}.$$ (Exercise for the reader: Is this little-endian or big-endian? How would you express the other one? If you're not sure what these words mean, look them up—they're important to know!)
Once you have an integer $x$, don't compute $x^d$—that's much too big an integer to work with. Instead, divide $x$ by the modulus $n$, and keep only the remainder $r = x \bmod n$.
Once you have an integer $0 \leq r < n$, still don't compute $r^d$—compute the modular exponentiation $s = r^d \bmod n$. That is, instead of computing the $d$-fold product $r\cdot r\cdots r$, and then dividing the result by $n$ and keeping only the remainder, use the square-and-multiply algorithm for computing exponentiation, and divide every intermediate quantity by $n$ and keep only the remainder, so that you never work with integers greater than $n$.

(If you're not sure what the square-and-multiply algorithm is, look it up! Hint: $r^5 = r\cdot r^4 = r\cdot (r^2)^2$, which involves only one multiplication and two squarings, while $r\cdot r\cdot r\cdot r\cdot r$ requires four multiplications.)
You didn't really read all the answers about why this naive approach to RSA signatures is dangerous, did you? Really do read them, and read about MD5, and why MD5 can't be used safely for standard signature schemes. This is more important than understanding the details of the computation for a cheesy classroom exercise.

At step 5, if $r=x\bmod n$ is not $x$, we have a devastating potential factorization weakness. That step and its "that's much too big an integer to work with" might be causing some confusion. On the other hand, my own answer likely cause more confusion (which makes this one welcome). — fgrieu, Dec 29 '17 at 19:00
@fgrieu Can you elaborate on the factorization weakness? How does revealing the reduction modulo $n$ of a string not even under attacker control (assuming no preimage attacks on MD5) enable factorization? — Squeamish Ossifrage, Dec 29 '17 at 19:02
If $r=x\bmod n$ is not $x$, then $n<2^{128}$ and is easy to factor. If we don't care about security, we might as well tell that the signature of any message is 0 and call it a day! +1 for the quantum computer reference.. — fgrieu, Dec 29 '17 at 19:05
@fgrieu Oh, well, yes. That's why I exhorted the original poster to read everything about why this is a BAD IDEA first. I thought you were talking about a generic problem with RSA-FDH reducing (say) a double-size hash modulo $n$! In this case, $n = 33$, which is possibly even factorable on a quantum computer today. — Squeamish Ossifrage, Dec 29 '17 at 19:06

fgrieu · Answer 2 · 2018-01-01T08:28:48.950

Adjust the problem if at all possible.

The parameters envisioned in the question's

For example: $n=33, d=7, e=3$

are terminally insecure, because $n$ is way too small: only 6-bit, when 512-bit is routinely broken my amateurs, 768-bit was experimentally broken in a public attack 8 years ago, 1024-bit is considered breakable by well-funded and capable adversaries, and 2048-bit the recommended baseline. See this. As an aside, publishing $d$ makes $n$ insecure.

As noted by Yehuda Lindell in comment, MD5 is very seriously broken, and using it is mostly unjustifiable.

For example, we know how to prepare different PDF files with the same MD5 but showing about any text desired (including chosen after the hash is frozen). With most common RSA signing schemes, this means a signature for a PDF chosen by the adversary (at least in its beginning) and telling something is also a signature for a PDF telling anything else, that the adversary can adjust as wanted (without practical limit for something decided before signature, subject to size limitation of the PDF for something decided after: the best current attacks require an additional 512 octets in the PDF per octet in the text displayed that must be chosen after signature, or a few times less with compression).

Use SHA-256 instead, if that's allowed.

When $n$ is at least 1-bit wider than the hash, we can hash then textbook-RSA-sign. That's considering the hash as an integer (per e.g. big-endian convention), computing the signature as $S=\big(H(M)\big)^d\bmod n$ , with verification by $S^e\bmod N\;\stackrel{?}{=}\;H(M)$. But even with large $n$ and an otherwise secure hash, this is vulnerable to the Desmedt-Odlyzko attack, if some messages legitimately signed can be chosen by the adversary. See this for the method and references.

The correct way is to use a much wider $n$; not publish either its factorization or $d$; use a better hash; and use proper RSA signature padding. A modern such scheme with a security argument, available from a library in most modern programming languages, is RSASSA-PSS of PKCS#1v2.

As noted here with references, coupled with MD5, RSASSA-PSS has been practically attacked in some contexts. But it can be modified to become quite secure in practice until a better break of MD5, or we consider somewhat brute-forcing MD5 for preimage (hard but not inconceivable). In RSASSA-PSS with MD5 we can replace $\operatorname{MD5}\big(\mathrm{padding_1}\|\operatorname{MD5}(M)\|\mathrm{salt}\big)$ with $\operatorname{MD5}(\mathrm{padding_1'}\|\mathrm{salt}\|M)$ or $\operatorname{MD5}\big(\mathrm{padding_1}\|\operatorname{MD5}(\mathrm{padding_1'}\|\mathrm{salt}\|M)\|\mathrm{salt}\big)$ (say, with $\mathrm{padding_1'}$ of 384-bit so that $M$ starts on a block boundary); the random unpredictable $\mathrm{salt}$ before the message $M$ mitigates the worst known weaknesses of MD5.

Addition: RSASSA-PKCS1-v1_5 suggested by this answer is slightly simpler, also easily available in libraries, and is secure as far as we know when used with a secure hash; but needs more transformation to be practically secure with MD5, and does not come with a security argument.

score 1 · Answer 3 · edited Oct 07 '21 at 07:34

You're trying to use real life signing with a toy set of RSA keys. The modulus, which determines the raw payload of RSA, is too small to do anything interesting with it. So to do this you first need to use a set of keys of at least 512 bits.

If you want to implement RSA signing it is strongly recommended to simply follow PKCS#1 v1.5 signature generation and verification. This simply creates a header before the hash (indicating the type of hash used, MD5), performs some simple padding and then performs the RSA modular exponentiation. The ASN.1 / DER encoded header is always static, so you can just prefix the static header bytes in front of the hash.

Instead of describing the signature generation step by step I'll just point you to PKCS#1 v1.5 signature generation and verification and the encoding required; the static header is at the end of section 9.2. If you have trouble with the theory then please ask any follow up questions here. If you have trouble implementing the theory then stackoverflow is the place to ask (but don't forget to check for dupes first).

Notes:

If you don't want to produce a number but also bytes you can optionally perform the I2OSP primitive as well, which simply encodes the signature as a statically sized byte array (unsigned big integer).
MD5 and SHA-1 are broken for signing, I have no idea why they insist on using that. Use SHA-2 instead (unless you need to implement the hash algorithm as well).
512 bits RSA is not secure either, you need to use 4096 bits at the minimum. 512 bits is however a nice low number that supports the required padding while being efficient so you don't need to have a fast RSA operation.
I'll not point to PSS as it is hard to implement, even if it is more secure. PKCS#1 v1.5 padding is still considered secure and is much easier to implement.

Sign MD5 hash with RSA

3 Answers3