Store hashed email and compare hash values

Question

I have a number of different systems sending me email addresses, but I don't actually need the underlying email, just a hash of the email address. I know I can compare hash values to find matches across the senders, which is all I want to do, but what if each sending system uses a different salt or hash method. Is there some way to compare the resulting hash values I have?

Eg. Sender 1 sends me emails using sha1 and no salt. Sender 2 sends me email addresses using md5 and a salt (which they also send)

Can I take md5(sha1) and sha1(md5) and compare that hash output? Or do I need to specify that all emails should be hashed using the same method? Is there a better way that will allow each sender to select the hash method they want to use but still allow me to compare email addresses across senders?

Answer 1

With normal cryptographic hash functions, two different ones, or the same hash function with different salts (which in effect are different hash functions too, for each salt) are supposed to be independent of each other, i.e. there should be no way (other than brute-forcing the message) to know from the results if they have the same input.

There are some special constructions of "different" hash functions which do not quite behave that way, but when there are independent salts, you even then have no chance.

As you used the homomorphic-encryption tag on your question, this might be a way to go. For example, with modular exponentiation modulo a prime (or analogously in a elliptic curve group), each sender would have to convert the mail address to a number $x$ (everyone using the same method for this), then calculate $f_i(x) = g_i^x$, where $g_i = g^i$ is a number assigned to each sender (and you know the $i$).

To compare the results $f_i(x)$ and $f_j(y)$, you calculate $f_i(x)^j$ and $f_j(y)^i$. If they match, then we have $(g_i^x)^j = (g_j^y)^i$, i.e. $(g^x)^{i·j} = (g^y)^{i·j}$, which usually means that $x = y$. (This is a variation of the Diffie-Hellman key exchange.)

The $i$ works as a kind of salt here, so for listeners who don't know $i$, the mail addresses of different systems are not comparable. But I don't see how we could incorporate some kind of sender-chosen random salt for each message.

Answer 2

As you suspected, md5(sha1(x)) is not equal to sha1(md5(x).

If each sending system uses a different salt or hash method, there is no way to compare the resulting hash values. To see if two email addresses are the same from their hash values, both hash values must have been computed using exactly the same method and the same salt.

Have you considered maybe asking these people to use the standard email hashing method used for the "suppression list"?

Technically you don't really need all the senders to use any one particular hash function. For example, say

• Sender 1 sends you one list of 100 hashes computed using md5, and a separate list of 100 hashes for the same 100 email addresses computed using sha256.
• Sender 2 sends you a similar list of 150 hashes computed using sha256, and a separate list of 150 hashes for their 150 email addresses computed using sha3.
• Sender 3 sends you a list 2000 hashes computed using md5, and a separate list of 2000 hashes for the same 2000 email addresses computed using sha3.

(The lists don't need to be in the same order -- in fact, it's probably best if each list of hashes is sorted by the hash value).

Then you can find a hash value for any email address that any two senders have in common, even through there is no one standard hash method used by all three senders.

One minor exception is that there exist a few hash function families that work by repeated hashing -- something like f(x) = sha2(sha2(sha2(sha2(x))). Say one sender gives you the hash of an email produced by hashing once by sha256. And another sender gives you the hash of an email that is repeatedly hashed in that way 128 times by sha256. Then you could take the hash from the first sender and re-hash it the remaining 127 times. If the resulting hash is the same as any hash that the second sender sent you, then they have the same email address. This exception isn't particularly useful in your case, because I doubt that your senders are using such a family of functions.