Wednesday, November 9, 2011

When the data is "too good to be true"

Remember Hendrik Schon! A decade ago he published a string of very impressive Nature and Science papers that eventually turned out to be "too good to be true". It seems a similar thing has been happening in the field of social psychology. The AP reports
 three graduate students grew suspicious of the data Stapel had supplied them without allowing them to participate in the actual research. When they ran statistical tests on it themselves they found it too perfect to be true and went to the university's dean with their suspicions.
In the future, the university plans to require raw data from studies to be preserved and made available to other researchers on request - a practice already common in most disciplines.
Nature News reports
The commission found that co-authors of Stapel's papers seem to have been unaware of the fraud, naively trusting in Stapel's reputation and fooled by elaborate preparations for tests that were never actually carried out..... Stapel and a colleague or student came up with a hypothesis, and then designed an experiment to test it. Stapel took responsibility for collecting data through what he said was a network of contacts at other institutions, and several weeks later produced a fictitious data file for his colleague to write up into a paper. On other occasions, Stapel received co-authorship after producing data he claimed to have collected previously that exactly matched the needs of a colleague working on a particular study.....
The data were also suspicious, the report says: effects were large; missing data and outliers were rare; and hypotheses were rarely refuted. Journals publishing Stapel's papers did not question the omission of details about where the data came from. 
This is part of a Nature News piece which has the misleading title "Report finds massive fraud at Dutch universities". A more responsible and accurate title would be "Report finds massive fraud by one Dutch professor of social psychology". In  the comments section several Dutch researchers rightly object to the title.

No comments:

Post a Comment