0

I have circa 1 million datapoints, each with a unique integer ranging from 6 to 24 digits: 

exampleDataPointOne: 655092
exampleDataPointTwo: 333402221 
exampleDataPointThree: 332021
...
exampleDataPointN: 903232211

I want to run each of these unique integers through a hash function, with the objective of 1) maintaining the uniqueness, 2) obscuring the underlying integer value and 3) having a hash output with a length of <= 34.

I've looked around for potential hash algo's that are a good fit, such as BLAKE, md5 and the sha family of hashes.

Given my requirements, what is the most advisable hash algorithm to use?

Community
  • 1
Babra Cunningham
  • 101
  • 3
  • When you say <= 34, is that 34 bytes? – mikeazo Jan 03 '17 at 14:50
  • Apologies for lack of clarity, no thats 34 digits – Babra Cunningham Jan 03 '17 at 14:57
  • In theory if you use a 128-bit or maybe even a 64-bit hash function you probably won't see any collisions because the interesting threshold for this is around $2^{32}$ respective $2^{64}$ unique inputs. BTW you can fit about 112 bit into 34 digits so BLAKE2 with truncation to 14 bytes / 112 bits sounds reasonable and fast. – SEJPM Jan 03 '17 at 14:58
  • Is there a malicious adversary at all in your scenario? – mikeazo Jan 03 '17 at 15:37

2 Answers2

2

I want to run each of these unique integers through a hash function, with the objective of 1) maintaining the uniqueness, 2) obscuring the underlying integer value and 3) having a hash output with a length of <= 34.

You need to be very clear about what you mean by "obscuring." The basic problem here is that your values look like they're likely to be subject to brute force guessing attacks—just like passwords are. This means that if you apply a public, unkeyed hash function to your values, you subject yourself to the same sorts of attacks that are very successful against passwords—particularly attacks against unsalted passwords.

If by "obscure" you really mean that an attacker should not be able to figure out which number corresponds to each hash, then your solution needs to incorporate a secret key, and that key needs to be protected. This is much, much, much, much more important than whether you use SHA-2 or Blake2. (PS: Don't use MD5 for anything! And if you need to incorporate a secret into the computation, don't do it in an ad-hoc manner—use HMAC-SHA2 or Blake2's optional key support).

Luis Casillas
  • 14,468
  • 2
  • 31
  • 53
1

Note: this answer previously recommended using the SHAKE functions standardized as part of SHA-3 but has since been edited because I misunderstood the notation used in FIPS-202

I'd recommend using a keyed MAC, truncated to the appropriate length. The key would serve as your "domain separator" for this use case and prevent dictionary attacks against the small input space if that is a concern. You can share the key with third parties if they need to compute the hashes as well for your application.

If you're dealing with recent x64 hardware I'd choose HMAC-SHA2-256 truncated as needed due to the hardware support for SHA2-256 on-chip.

Keyed BLAKE2 might be a more performant choice on platforms without hardware SHA2-256 instructions (BLAKE2 can be keyed directly and used as a secure MAC without using the HMAC construction).

Community
  • 1
rmalayter
  • 2,297
  • 16
  • 24
  • 2
    I don't see how using SHAKE is any better than using SHA-256 and truncating down to the desired number of digits. – mikeazo Jan 03 '17 at 15:00
  • 1
    The output length is part of the input in SHAKE, so requesting a 128-bit output with input "abc" produces completely different output than requesting a 256-bit output with input "abc". This matters in a lot of use cases (not sure it matters here given the "small" input space) – rmalayter Jan 03 '17 at 15:22
  • That is a good point. You can easily get similar behavior with SHA-2 though. Suggesting SHAKE just seems strange to me as it is so new and the library support will not be as good. – mikeazo Jan 03 '17 at 15:36
  • I have not read up on SHAKE, so I'm curious about your statement that different output lengths affect the output value, given this question. – mikeazo Jan 04 '17 at 02:37
  • I hadn't read that appendix, and was confused by the notation of FIPS-202 which has the output length in SHAKE being passed into the Keccack core function. Clearly my answer is wrong; the output length doesn't prevent one output from being the prefix of another. Should I edit or delete it? – rmalayter Jan 04 '17 at 02:57
  • Edit is fine I'd say. – mikeazo Jan 04 '17 at 03:03
  • Edited to recommend a keyed hash/MAC – rmalayter Jan 04 '17 at 03:19