Previously I posted about learning how to critically read experimental papers.
A theory paper may claim
"We can understand property X of material Y by studying effective Hamiltonian A with approximation B and calculating property C."
Again it is as simple as ABC.
1. Effective Hamiltonian A may not be appropriate for material Y.
The effective Hamiltonian could be a Hubbard model or something more "ab initio" or a classical force field in molecular dynamics. It could be the model itself of the parameters in the model that are not appropriate. An important question is if you change the parameters or the model slightly how much do the results change. Another question, is what justification is there for using A? Sometimes there are very solid and careful justifications. Other times there is just folklore.
2. Approximation B may be unreliable, at least in the relevant parameter regime.
Once one has defined an interesting Hamiltonian calculating a measurable observable is usually highly non-trivial. Numerous consistency checks and benchmarking against more reliable (but more complicated and expensive) methods is necessary to have some degree of confidence in results. This is time consuming and not glamorous. The careful and the experienced do this. Others don't.
3. The calculated property C may not be the same as the measured property X.
What is "easy" (o.k. possible or somewhat straightforward) to measure is not necessarily "easy" to calculate and visa versa. For example, measuring the temperature dependence of the electrical resistance is "easier" than calculating it. Calculating the temperature dependence of the chemical potential in a Hubbard model is "easier" than measuring it.
Hence, connecting C and X can be non-trivial.
4. There may be alternative (more mundane) explanations.
The experiment was wrong. Or, a more careful calculation of a simpler model Hamiltonian can describe the experiment.
Theory papers are simpler to understand and critique when they are not as ambitious and more focused than the claim above. For example, if they just claim
"effective Hamiltonian A for material Y can be justified"
or
"approximation B is reliable for Hamiltonian A in a specific parameter regime"
or
"property C and X are intricately connected".
Finally, one should consider whether the results are consistent with earlier work. If not, why not?
Can you think of other considerations for critical reading of theoretical papers?
I have tried to keep it simple here.
Subscribe to:
Post Comments (Atom)
From Leo Szilard to the Tasmanian wilderness
Richard Flanagan is an esteemed Australian writer. My son recently gave our family a copy of Flanagan's recent book, Question 7 . It is...
-
Is it something to do with breakdown of the Born-Oppenheimer approximation? In molecular spectroscopy you occasionally hear this term thro... -
If you look on the arXiv and in Nature journals there is a continuing stream of people claiming to observe superconductivity in some new mat...
-
I welcome discussion on this point. I don't think it is as sensitive or as important a topic as the author order on papers. With rega...
Your examples are written from a situation after the property has been measured (we can understand property X - i.e. the fact that property X exists is known, i.e. it has been measured).
ReplyDeleteA big concern I have is with theoretical papers that predict properties (sometimes in new materials not yet synthesized) that have not yet been observed. This may be very valuable, or may be very wrong.
While in a properly functioning scientific community (including a healthy ratio of theorists/experimentalists ...), discoveries should be coming from both sides, i.e. sometimes new properties are first predicted, and sometimes they are first observed, there are many theoretical papers nowadays proposing properties that later turn out to not exist, or proposing properties in materials that simply can't be made (due to thermodynamic or kinetic constraints).
This will never be not a problem, and wrong predictions may spur good science. However, I think there is a cottage industry of (DFT-based) work predicting all kinds of things, and getting "impact-points" without actually turning out to be relevant in the real world.
So when I review theoretical papers, and especially when such papers frame their relevance in terms of practical applications (see your functional electronic materials post) I always like to see at the very minimum a justification showing that the material and/or the property is present in experimental conditions that are accessible.
This is not to say that predictions in conditions inaccessible in the real world do not have value (they may help understand accessible properties). I.e. it's not a black and white weighing of value.
Bottomline: can it be done? (and is it thus falsifiable?!) If not, why is the result still important, or maybe more appropriate, useful to be known?
Thanks for the helpful comment.
ReplyDeleteThis is an excellent point that I had not thought of. I agree it is an issue. Some theory papers consider completely unrealistic parameter regimes. This is barely mentioned or hidden in the paper. Sometimes it is not clear to me if the authors are trying to hide it or if they are oblivious to the problem.
A critical reader should try and estimate the order of magnitude of the effect being discussed.
One specific case is that of high magnetic field effects. I mentioned this in an earlier post
http://condensedconcepts.blogspot.com.au/2011/03/strongly-correlated-electron-systems-in.html