14

In many articles regarding computational models of some particular phenomenon, there seems to be a consensus: "the smaller the number of 'free parameters' in the model, the better". So, what is meant by "free parameter", and why is it less desirable to model something with such a parameter?

Thanks!

dnbwise
  • 143

2 Answers2

14

A free parameter is one that can be adjusted to make the model fit the data. If I make a model that says $A$ is proportional to $B$, there is one free parameter, the proportionality constant. If my model has a specific value of the proportionality constant, there are no free parameters.If I say that $A$ is a quadratic function of $B$, there are three free parameters, $a,b,c$ in $A=aB^2+bB+c$. That makes it easier to fit the data, even if my model is not correct, so it is less impressive.

Ross Millikan
  • 374,822
  • What prevents one from simply having a single "free parameter" $d \in \rm I!R$ that encodes $a, b$ and $c$? e.g. if $a = 0.a_1a_2a_3...$, etc, then let $d = 0.a_1b_1c_1a_2b_2c_2....$. Are there 3 "free parameters" or only one? – Him Apr 22 '20 at 06:31
  • @Scott: That is still three in the sense that you have a three dimensional space of possible solutions. You are correct that the cardinalities of the two approaches as reals or sets of three reals are the same, but you can define the topological dimension of the space of functions and find it to be three. – Ross Millikan Apr 22 '20 at 13:54
  • So the "number of free parameters" relates to the topological dimension of the parameter space? Why would this be the case? Is it somehow related to the fact that your model is continuous? – Him Apr 22 '20 at 14:10
  • It really reflects the number of knobs you have available to turn to make your model fit the data. The more knobs, the closer a fit you can get even if your model does not reflect the underlying science. The topological space is just to show why combining three reals into one does not change the number of parameters. – Ross Millikan Apr 22 '20 at 14:53
  • "The more knobs, the closer a fit you can get even if your model does not reflect the underlying science." So everyone keeps claiming, but I have never seen any rigorous demonstration of this, and I've certainly never seen a link established to the topological dimension of the parameter space. If anything, I would imagine that it is related to the information entropy on the number of decimal places, but that's just speculation. – Him Apr 23 '20 at 15:19
  • @Him: and yet there are many. (https://quasar.as.utexas.edu/papers/ockham.pdf)[here is one I like] – user1963 Jun 11 '22 at 12:37
  • @user1963 this is a nice paper, but only gives an example of this "more knobs" theory, not a general principle. Note my original objection above. – Him Jun 11 '22 at 19:48
13

The following marvellous quote from Freeman Dyson's "A meeting with Enrico Fermi" contains both an example for your first question and an answer to your second.

[Enrico Fermi] delivered his verdict in a quiet, even voice. . . . "To reach your calculated results, you had to introduce arbitrary cut-off procedures that are not based either on solid physics or on solid mathematics."

In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, "How many arbitrary parameters did you use for your calculations?" I thought for a moment about our cut-off procedures and said, "Four." He said, "I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk." With that, the conversation was over. . . .

Community
  • 1