Earlier I posted about a basic skill: take initiative! Don't wait for someone else to tell you what to do. Try stuff.
It is exciting when you think that you have finally obtained some research results. It is even more exciting if they seem interesting and potentially important. But, don't fool yourself. They may be wrong! Mistakes happen in research. More often than many want to admit. Furthermore, the more complicated the technique and the system under investigation, the more likely something will go wrong. Murphy's law!
So how do you check your results? I am not sure. There is no simple universal procedure to check results. Just repeating the experiment or calculation is not good enough. You [or the instrument or software...] may be making the same mistake.
Learning to check results is an art and requires patience, discipline, and creativity.
Furthermore, different individuals and different research fields often have quite different standards as to how many different checks one should perform. Some seem to rush to publish once they get an "interesting" result. Others, are very cautious and careful and perform multiple checks.
I am very thankful that many of my collaborators over the years have been more conscientious than me.
For students: here are a few ideas as to some basic checks that one should do.
Compare your results to relevant published work. Make sure you can reproduce earlier work. If not, do you have a good reason to believe you are right and they are wrong.
Computational work.
Compare your results to limits [e.g. weak or strong coupling, for which one can obtain analytical results].
Use different versions of software or numerical methods.
For short programs, write two codes from scratch.
Analytical calculations.
Compare your results to Mathematica or a numerical calculation.
Experiments.
Change the sample, device, material, instrument, or procedure.
Computational chemistry.
Try different basis sets and levels of theory. Don't just do DFT! When possible, benchmark it against smaller systems.
Curve fitting.
Have different individuals do it independently and see if they get the same result.
What do you think are good procedures for checking results?
When should you quit checking?
Subscribe to:
Post Comments (Atom)
From Leo Szilard to the Tasmanian wilderness
Richard Flanagan is an esteemed Australian writer. My son recently gave our family a copy of Flanagan's recent book, Question 7 . It is...
-
Is it something to do with breakdown of the Born-Oppenheimer approximation? In molecular spectroscopy you occasionally hear this term thro... -
If you look on the arXiv and in Nature journals there is a continuing stream of people claiming to observe superconductivity in some new mat...
-
I welcome discussion on this point. I don't think it is as sensitive or as important a topic as the author order on papers. With rega...
from an experimentalist: I fully agree with the basic idea of this post. But the recommendations in your "Experiments" section are not general. If you change material or procedure, you should expect different results. Physics reproduces, but in identical procedures with identical materials.
ReplyDeleteSo while changing an experimental procedure may shed light on the underlying physics, it would do so most likely because of the changes in results.
Reproducibility should only be expected for identical procedures.
Unfortunately many experimentalists do not properly describe their experimental procedures, making it hard to check whether you can get the same results as there are often many parameters that are still "free" after studying a paper.
pcs,
Deletethanks for the helpful comment. it is great to hear from an experimentalist.
Let me expand a bit on some of my point with a specific example. Suppose one is measuring the electrical conductivity of a metal as a function of temperature. Changing procedure might include varying the cooling rate, the magnitude of the electrical field, the configurations and spacing of the contacts. Different samples could mean ones which ostensibly have the same chemical composition. All of these variations should produce no significant variation in the conductivity versus temperature. This is what I mean by reproducibility.
I agree with your concern that many experimentalists under-report the details of their experiments, making reproducibility difficult. This is unacceptable in an age of supplementary on-line material.
your points are well-taken.
DeleteIn the end it depends on the physical mechanism behind the observation one wants to understand. Your (correct!) examples reflect your frame of mind; in reality many materials are not ideal/perfect. Changing parameters would then not necessarily (perfectly) reproduce data.
Some specific examples
In e.g. the phase separated manganites, cooling rates do matter, and changing the speed would not (perfectly) reproduce R(T) curves. But it could teach something about domain formation.
Changing the magnitude of the electric field would change the current and the (possible) Joule heating through narrow constrictions in the domain configuration. Again data would not perfectly reproduce, but it could teach about domain nonuniformities in the material.
I think it is really hard to define general guidelines in this respect. Possibly that is one reason why experimentalists under-report details - though in all honesty I'm more sceptical, and think many do not think carefully about publication requirements in view of the greater scientific good.
Nevertheless, your points are well-taken, and it would be good if on both the undergrad and graduate level, and specifically in "first guided" but real experiments, this would be imprinted on students. Excitement about the work at hand is important, but "good old boring" carefully considering mundane issues as reproducibility and the influence of different parameters often get buried by the rush towards sexy new results.
/end of my cynicism.../ :-)
I guess this depends on what sorts of students/what level etc, but one method I always advocate is simple dimensional analysis. Its also a good way of understanding if the basic physics is right, provides reasonable estimates for magnitudes and a reality check.
ReplyDeleteGood point.
DeleteDimensional analysis is particularly good for checking analytical formulas.
Order of magnitude estimates are also valuable for checking whether proposed explanations really are relevant.
I remember our undergraduate professor telling us the corollary to Murphy's law i.e. Murphy was an optimist!
ReplyDelete