Some of the problems that need to be addressed are:
- journals are wasting a lot of time and money
- rubbish and mediocrity is getting published, sometimes in "high impact" journals
- honorary authorship leads to long author lists and misplaced credit
- increasing emphasis on "sexy" speculative results
- metrics are taking priority over rigorous evaluation
- negative results or confirmatory studies don't get published
- lack of transparency of the refereeing and editorial process
- .....
It is always much easier to identify problems than to provide constructive and realistic solutions.
Here is my proposal for a possible way forward. I hope others will suggest even better approaches.
1. Abolish journals. They are an artefact of the pre-internet world and are now doing more harm than good.
2. Scientists will post papers on the arXiv.
3. Every scientist receives only 2 paper tokens per year. This entitles them to post two single author [or four dual author, etc.] papers per year on the arXiv. Unused credits can be carried over to later years. There will be no limit to the paper length. Overall, this should increase the quality of papers and will remove the problem of "honorary" authors.
4. To keep receiving 2 tokens per year, each scientist must write a commentary on 2 other papers. Multiple author commentaries are allowed. These commentaries can include new results checking the results of the commented on paper. Authors of the original can write responses.
5. When people apply for jobs, grants, and promotion they will submit their publication list and their commentary list. They will be evaluated on the quality of both. Those evaluating the candidate will find the quantity and quality of commentaries on their papers very useful.
The above draft proposal is far from perfect and I can already think of silly things that people will do to publish more... However, for all its faults I sincerely believe that this system would be vastly better than the current one.
The first step is to get the arXiv to allow commentaries to be added. But, this will only become really effective when there is a career incentive for people to write careful critical and detailed commentaries.
So, fire away! I welcome comment and alternative suggestions.
I like this idea. I have been thinking of (and discussing with others) something similar to your 2. above, but 3. and 4. are very interesting and useful additions. What I could suggest is to allow any number of archive submissions but to categorize them as A and B, say. Restrict the number of A submissions to 2 per year, as you suggest, but allow for any number of B publications. Also, all evaluations for career advancement etc. should only consider A publications. I'd support something like this.
ReplyDeleteThe advantage that I see is that scientists who are highly creative and prolific, say in multiple areas, can make their work known to others, though the 'B' route if the 'A' route is exhausted.
Gautam,
DeleteThanks for the feedback.
The A/B option is a possibility. But, I worry that in end people may still get assessed on the B's and so there will be pressure/temptation to just keep going with the system as is.
Indeed there may be some highly creative/prolific people out there. But, who are they? I would like to see some historical examples of people who over a 4 year period produced more than 8 papers/author units of high value.
The biggest potential problem with your proposal is point 3. As written, your suggestion would seem to require radical institutionalisation of the Arxiv, sufficient to grant and enforce a sanctioned monopoly by the Arxiv and scientific publishing. The danger is that the monopoly could then be used for nefarious purposes. How would other rival sites, based on the Arxiv model, but perhaps offering more benefits to authors, be prevented from subverting the monopoly? How will the proper oversight be provided to Arxiv to ensure that the monopoly is not used in contravention of the goals of science.
ReplyDeleteAn officially sanctioned monopoly by Arxiv may not be strictly necessary, but since the token-based publication rate control you suggest doesn't seem implementable without some form of centralised authority to keep it followed*.
W.R.T. the token-based scheme you suggest, would it be possible to trade tokens between scientists (for example, if I knew I was going to take parental leave for a year, and may not use my tokens, could I defer them or -- even better -- trade them? Such a "scientific emissions trading scheme" seems like a natural extension of what you suggest.
*the emergence of Bitcoin does suggest that the need for a central authority may be circumvented with clever engineering
(note: last post had a grammar error; this is the corrected version)
Seth,
DeleteThanks for the comment.
I agree there are dangers with monopolies. Perhaps there could be several competing sites, all who are accredited by funding agencies and universities. Tokens can then be used at any. arXiv is already institutionalised, being part of the Cornell library, and has a board and other governance structures.
As proposed, unused tokens can be saved for later years. This solves the parental leave problem. I don't think a trading scheme is desirable. Junior people will be under pressure to give credits to the powerful.
A limit of two single-author papers a year? Do you really not see the problem with this? Do you honestly believe that (a) all scientists are equally creative/productive and (b) no scientist has "good years" and "bad years"? Way to ruin an otherwise good idea!
ReplyDeleteJess,
DeleteI don't appreciate your combative tone.
Obviously, I don't think all scientists are equally creative/productive. Two paper/author tokens is my upper bound on creative productivity. The proposal as stated allows for tokens to be saved for later years and so allows for good/bad years.
Sorry. I do tend to express my critiques "explosively".
DeleteBut why do you feel a need to place any "governor" on the number of papers one can publish? This seems to me a propagation of the myth of scarcity that journals now use to keep scientists "hungry" and thus controllable. There is no problem having more literature than anyone can ever read -- in fact I consider that inevitable, as we've been there for half a century that I know of. The problem is having a way to guess which publications are WORTH reading. I say, "Publish them all!" But then submit them to an ongoing filter that gradually identifies which are (a) correct; (b) readable; (c) significant; (d) original; ... and as many other rankable attributes as we want to define. The algorithm for this filtering is going to take many years to refine, and it will never be perfect; but the debate will be over the algorithm, not the individual disputes, and therefore worth debating for all concerned.
Sorry, I get preachy too....
Let me be more constructive and offer my own version of the same idea: first, don't bother trying to "abolish journals". You can't abolish journals without first providing a working alternative, and if you provide a working alternative they will just become irrelevant. Here is my proposal in some detail: http://jick.net/~jess/misc/OPeer-essay.pdf
ReplyDeleteHi Jess,
DeleteI agree we need to work towards making journals irrelevant. They already are for doing physics. They remain purely for career evaluation purposes. Thanks for sharing your alternative proposal. I am nervous about introducing a new metric, one for "credibility". On the other hand, I think evaluating people for the quality of their commentaries [reviews] will work towards this.
Absolute egalitarianism is wishful thinking. Some people's opinions ARE more credible (and valuable) than others' in the public's desperate search for ways to filter the wheat from the chaff. The trick is to identify them by some more objective and democratic means than "trusting the Editors". SlashDot-style democracy could easily implement the "quality of commentaries" criterion, but that doesn't necessarily tell us whether the comments are CORRECT. Cultural relativism notwithstanding, in science it is possible to be brilliantly, persuasively, eloquently WRONG.
DeleteThere are issues with the token system, I think. For instance it seems to me that the trading system would increase honorary authorship, not deplete it.
ReplyDeleteHowever, I think this is a great system. What I particularly like is the peer review structure. How amazing would it be if reproducing someone's results could be constructively reported, and alternatives/counter-demonstrations and the like could as well.
What this system reminds me of is the complicated network direction that everything is going. There's no longer encyclopaedias, but multiply-linked wikis that can take you deeper and on tangents and whatever else you want. Why should journals remain so linear? In a way, appendices are an archaic version of this very concept.
Similarly, the peer review system is so crap at the moment, that I think I agree, this system could genuinely be a very good one.
So, Ross, you mention a date in the title which you address. You have some thoughts on how this transition may take place? Is this for a later post?
ReplyDeleteOne of the difficulties is how to adopt such a scheme (of course). In particular, my initial train of thought was "if I eventually get an ongoing position, I'd be completely happy to only post on the arxiv, but I can't possibly do that now. But then, I'll have students, and they'll need to publish in journals. So then, only single author manuscripts could be arxiv only?" This doesn't get anywhere. So you'd really need a whole lot of people to all agree that this is a valid choice, and over time, if it's good, more people will sign on?
Anyway, I really do assume you have thoughts for a post on where your figure 2030 comes from, and what happens over the following 17 years!
Incidentally, Kitaev is a famous example of someone who has some arxiv only papers, as well as some papers which aren't even on the arxiv, but just do the rounds through email (I don't know the latter for a fact). His 1D chain paper had hundreds of citations before it got a journal reference (Phys. Uspek...something).
Secondly, Scientific Reports has a "comments" section, where you can critique a paper or whatever. And in principle, referees are only supposed to accept/reject based on correct-ness. In my experience this didn't happen, and they cared greatly about significance and innovation. Nevertheless, it seems to me that Sci. Rep. is actually moving in some of these directions.
sorry - which you 'never' address, first sentence
Delete