Differences between two RSA signatures using OpenSSL

Question

I want to sign a file called data.txt and store its signature in data.txt.sig. To create the signature, I want to use an existing RSA private key, stored in the PEM format within the file RSAPrivate.key. I use the following openssl command:

openssl dgst -sha256 -out data.txt.sig -sign RSAPrivate.key data.txt

But I would have expected, that the following two commands yield the same result. Yet the result is a different signature.

openssl dgst -sha256 -binary -out data.txt.hash data.txt
openssl rsautl -sign -pkcs -in data.txt.hash -out data.txt.sig -inkey RSAPrivate.key

As long as I use a signature scheme which does not incorporate a salt, the results should be deterministic and binary exactly equal. I am pretty sure my understanding of at least one of the OpenSSL commands is flawed. Can someone please explain me why the results are different?

Answer 1

There are several ways to build a signature scheme on top of RSA. The two main schemes are PKCS#1v1.5 (named after the version of the document called PKCS#1 that codified it) and PSS (also codified in PKCS#1, but only since version 2.1 of this document). Both the commands you ran follow PKCS#1v1.5 (so I won't discuss PSS further in this answer), but only one follows it fully, the other ones uses a popular minor variation.

PKCS#1v1.5 follows the hash-and-sign paradigm: the first step in signing or verifying a message is to hash it, independently of the key. The rest of the computation takes the hash and the key as inputs.

openssl dgst -sha256 -out data.txt.sig -sign RSAPrivate.key data.txt

This command creates a PKCS#1v1.5 signature using the hash algorithm SHA-256. The process is fully described in PKCS#1 §8.2, with substeps in other sections. For your question the most important part is the encoding process described in §9.2. To summarize the relevant parts of the process:

1. Hash the message using the chosen digest algorithm.
2. Embed the hash in a structured message called DigestInfo that contains the hash value as well as metadata encoding which hash algorithm was used.
3. Pad the DigestInfo structure to the size of the RSA key.
4. Apply the RSA private-key operation to the padded data.

Verification reverses these operations except that hashing the message always happens in the same direction: apply the RSA public-key operation, check and strip padding, parse the DigestInfo structure, and finally calculate the hash of the message to verify and compare it with the hash value from the DigestInfo structure.

openssl rsautl -sign -pkcs -in data.txt.hash -out data.txt.sig -inkey RSAPrivate.key

This command uses a variation of PKCS#1v1.5 that skips step (2) above. At step (3), instead of using a DigestInfo structure, this command uses the hash directly. Notice that this command does not know which hash algorithm you used to produce data.txt.hash.

This variant of PKCS#1v1.5 is not formally standard, but it's well-established. Notable, it was used in TLS versions up to 1.1. It's present in many cryptography APIs, often called something like “raw RSA PKCS#1v1.5 signature”. It's as secure as the standard version as long as there's no risk of confusion between hash algorithms (for example, because a key is only ever used with a single hash algorithm).

You can view the differences by looking at the inputs to the private-key operation (i.e. the results of step 3 above), which you can see by applying the public-key operation to the final result.

openssl rsautl -inkey RSAPrivate.key -raw -encrypt -in data.txt.sig -out data.txt.padded

Compare the result for the two data.txt.sig files you created. You'll notice that they're the same except for a chunk in the middle. That chunk is the last byte of padding and the DigestInfo structure apart from the hash itself.