Tuesday, November 4, 2014

Why am I skeptical about curve fitting?

It continues to amaze and frustrate me how some people will do the following.
Take experimental data for a specific quantity [e.g. resistivity vs. temperature].
Fit the data to a function from some exotic theory X involving N free parameters.
Claim that the "successful" fit "proves" that X is the correct theory.

Why am I skeptical? What would it take to convince me X is actually valid?

1. Have N < 4, remembering the elephants wiggling trunk.
2. With the same set of parameters also fit at least one, and preferably several,  other experimental observation [e.g. thermopower vs. temperature].
3. Show that the fit parameters are physically reasonable and consistent with estimates from independent determinations. Science is all about comparisons.
4. Also fit the data to the predictions of mundane theory M, and alternative exotic theory X2, and clearly show they cannot fit the data. i.e., apply the method of multiple alternative hypotheses.

Finally, there is a more profound philosophical point, the underdetermination of scientific theory. We can never be sure there are not alternative theories we have not considered.

When is curve fitting valid and useful? What does it take to convince you?

5 comments:

  1. I agree but I would like to add something a little bit different with respect to curve fitting.

    There is one set of methods that I would put as a class of curve fitting, although computer scientists found a fancy name for it, that tend to work very well, even with it being an interpolation scheme in an abstract space with a very large number of dimensions: Machine Learning.

    Images recognition where it is now possible to detect very specific details in complicated new images is a good example.

    Hopefully, we can also do something good in physics too.

    ReplyDelete
    Replies
    1. Thanks for the comment.

      Please give an example in physics or chemistry where you think Machine Learning will work in a definitive way.

      Here is one recent study

      http://arxiv.org/abs/1410.5447

      Delete
  2. Are you just fishing for a Bayesian enthusiast here? ;-)

    ReplyDelete
    Replies
    1. HI Jess, Please expand for those of us who still struggle to understand exactly what Bayesian methods and philosophy are.

      Delete