The explanations about Shannon's theorem speed up all of a sudden when they should tell us why he introduced the log term. They usually range from 'it's useful' to 'it's because there are two choices, then log2 is suitable'. Am I right in thinking that what the log does is 'give a context' to the probability you are measuring the entropy of? If 0.125 was one of two outcomes it's different than if it was one out of ten. Hence, choosing different logs puts this specific option (0.125) in the right ballpark. Am I correct? And is there a more formal proof for this (maybe for the use of log in probability in general). thanks!
Asked
Active
Viewed 203 times
0
-
Are you asking why a logarithmic function, or why the choice of log base 2? – Jul 15 '17 at 16:51
-
why the function. thanks – magnolia1 Jul 15 '17 at 16:58
-
2Have you seen these previous questions? What is the role of the logarithm in Shannon's entropy?, Intuitive explanation of entropy? I don't know of any interpretation in terms of contexts or ballparks -- a probability of 0.125 contributes the same term $-0.125\log 0.125$ to the entropy no matter how many other outcomes there are -- so I don't think you are on the right track. – Jul 15 '17 at 17:37
1 Answers
2
Why are there squares in Pythagoras' theorem? Long and deep thinking led Shannon to this formula. The proof is in the pudding: You could, in the heuristic process, try other functions, like $\arctan$ or something, but only with the correct "Ansatz" you obtain the mighty theorem.
Christian Blatter
- 226,825