10

I'm confused.

I thought that tossing a 6-sided die 100 times had a greater than 256-bit entropy because $6^{99} < 2^{256} < 6^{100}$. (A similar concept appeared in this XKCD comic, where choosing four random words from a dictionary of presumably 2048 words has a 44-bit entropy, presumably because $2048^4 = 2^{44}$.)

On the other hand, the Shannon entropy of a 6-sided die tossed 100 times is $-6 × 1/6 × \log_2(1/6) = 2.5849625007$ bits. (According to the comments by Sakamaki Izayoi to my other question.)

Are these two different concepts of "entropy" entirely? If so, what's the difference and if not, what am I missing?

I read this thread but I'm still confused.

otus
  • 32,132
  • 5
  • 70
  • 165
cryptonamus
  • 233
  • 1
  • 5
  • 2
    log2(6^100) = log2(6)*100 –  Aug 19 '15 at 11:30
  • Can someone explain what the point of this formula is? -61/6 == -1. -1log(1/6) = log(6). Under what circumstances are the terms different? – Random832 Aug 20 '15 at 02:54
  • @Random832 $H(X)=\log_2n$ is a simplified formula for the entropy of a random variable $X$ which is uniformly distributed over $n$ possible values. In the general case (i.e. for non-uniform distributions), the formula is $H(X)=-\sum_{x\in A}(\log_2\Pr[X=x])\cdot\Pr[X=x]$, from which the complicated expression for $\log_26$ can be obtained. – yyyyyyy Aug 20 '15 at 12:19

2 Answers2

20

On the other hand, the Shannon entropy of a 6-sided die tossed 100 times is $-6 × 1/6 × \log_2(1/6) = 2.5849625007$ bits.

That is wrong: $-6\cdot\frac16\cdot\log_2\frac16$ is the entropy of a single die roll. Assuming the $100$ die rolls are independent, you can simply sum the entropies of the individual rolls to obtain $$ 100 \cdot\left(-6 \cdot\frac16\cdot\log_2\frac16\right) \approx 258.49625 \text, $$ which is precisely $\log_2(6^{100})$ as expected.

otus
  • 32,132
  • 5
  • 70
  • 165
yyyyyyy
  • 12,081
  • 4
  • 47
  • 68
12

It seems that with your shannon entropy, you are using 100 tosses to estimate the shannon entropy of a single die toss. If it is a fair die, that would be $\log_2{6}\approx 2.58$.

This is different from rolling a die 100 times to generate a cryptographic key, for example. Each roll of the fair die would contain $2.58$ bits of entropy, so in total you would have approximately $2.58\cdot 100=258$ bits of entropy, or $\log_2{6^{100}}=\log_2(6)\cdot 100$ as mentioned in the comment.

mikeazo
  • 38,563
  • 8
  • 112
  • 180