Like most faculty I have to evaluate the scientific "performance" and "potential" of applicants for jobs, promotion, prizes, and grants. This is a difficult task because we are often asked to evaluate people who we don't know, are unfamiliar with their work in our (somewhat related) field of expertise, or are working in completely different fields we know nothing about. For example, I have been on a committee where I had the ridiculous task of evaluating people in fields such as veterinary medicine, geography, and agriculture! This is one reason why metrics are so seductive and deceptive.
Here I want to focus on evaluating applicants who work in an area that is close enough to my own expertise, according to the following criteria. I can read one of their papers and make a reasonably informed assessment of its value, significance, and validity. This is important because it is easy to loose sight of the fact that there is only ONE measure that really matters: the ability of a person to produce valuable scientific knowledge. All the metrics, invited talks, grants, hype, slickly presented grant applications, enthralling presentations, .... are not what really matters. They are not research accomplishments. This measure can only be assessed from the actual content of papers.
I am embarrassed to admit that I am finally trying to work harder at ``practising what I preach.'' When I need to assess a credible application I try to identify just one paper that the applicant identifies as significant or for a grant application a paper that is central to the proposed project. I then look at this paper and then go back to the (copious) paperwork of the submitted application. You might think that this takes more time. But, actually it may save time because it can be so definitive to my view (positive, neutral, or negative) that I have to read and agonise less about my assessment.
I have always done this before when assessing applicants for postdocs to work directly with me, but much more irregularly for other situations.
Ideally, this should not be necessary. Rather, a good applicant would be someone whose papers we have already read because we wanted to. However, that is not the world we live in.
Is this a reasonable approach? Do you do something like this? Any other recommended approaches?
Subscribe to:
Post Comments (Atom)
From Leo Szilard to the Tasmanian wilderness
Richard Flanagan is an esteemed Australian writer. My son recently gave our family a copy of Flanagan's recent book, Question 7 . It is...
-
Is it something to do with breakdown of the Born-Oppenheimer approximation? In molecular spectroscopy you occasionally hear this term thro... -
If you look on the arXiv and in Nature journals there is a continuing stream of people claiming to observe superconductivity in some new mat...
-
I welcome discussion on this point. I don't think it is as sensitive or as important a topic as the author order on papers. With rega...
Your method is good, and I usually do the same, but there is always the danger that the applicant did not really contribute much to the paper in
ReplyDeletequestion. For job applications, letters and face-to-face conversation can reveal a lot. Even so, a few years ago we nearly hired a very impressive guy who turned out to have fabricated the data in his best paper.
Thanks for the comment.
DeleteYes. I agree for multi-author papers it is hard to know the contribution of the specific individual. However, if the paper is dubious or mediocre then it does not matter! I also agree that letters and face-to-face conversations should play a key role. But for grants one usually does not have this option.
Teaching ability is it important when applying ? How does one judge? This question is great botheration for the younger generation of applicants esp after many have said that teaching does not matter. Here is one sample.
Deletehttp://www.preposterousuniverse.com/blog/2011/03/30/how-to-get-tenure-at-a-major-research-university/
"Don’t worry about teaching, leadership, organizing, etc. I don’t think being good at these things actively hurts you, although I did once hear a senior faculty member say that he was negatively predisposed to candidates who had good teaching evaluations. (He was joking, I think.) Why? Because you’re spending time on something that isn’t research. But generally it won’t hurt, it just won’t help. You will typically be told (as I was) something like “teaching isn’t really important, but if your case is very close, it can help put you over the top.” Everyone agreed my case was very close, and my teaching was among the best in the department; it didn’t help. The point is simple: this stuff is not research"
The above para from the website coming from Caltech an Ivy league school and from the department where the legendary Richard Feynman taught ( Am I right ?). Do you think the grant system ( it is an compulsive obsessive order (COO) or is it technology which has disrupted the rigour of classroom undergrad teaching ( not talking about online)
All I can say is a wholehearted "yes, this is a good way". The only "beef" I have is that reading only one paper (out of 20?) gives a snap-shot view.
ReplyDeleteI'd ask for the candidate to identify the best paper from each of their career stages (PhD student, postdoc 1, postdoc 2) - noting that "best paper" could (should??) very well be a PRB, and not a flashy paper originating in some lucky serendipitous event.
Obviously this adds to the burden of a committee member. However, I'd say the additional 5 hours studying two more papers should be worth that, especially when this is a hiring decision.
(But this too is more easily preached than practiced...)
Thanks for the comment. I agree. The more significant the decision (e.g. tenure track faculty vs. reviewing ten small grant applications) the more thorough the review should be.
DeleteWhen the academic world is grappling as to who is the right candidate with the near good PhD thesis for being selected as an academic , one has go back in time and just look at some famous PhD thesis. There are 12 listed here.
ReplyDeletehttp://www.mrgeek.me/lists/12-most-famous-phd-theses-in-history/
Two are very interesting PhD thesis. John Nash and Debroglie. The full thesis of John Nash is not available in the site. DEBroglie's thesis is about seventy pages. Bibliography only 8. Full thesis is available on the above web site. John Nash PhD thesis is 29 pages with a bibliography of 2 !!!. Thesis is below.
https://www.scribd.com/doc/4800022/John-Nash-s-Ph-D-Thesis
Since evaluating the right PhD candidate is becoming more difficult in recent times, many PhDs and Postdocs look out for " connections" through the " club" approach. This leads to " to whom I know rather than what I know " becoming the driving force for getting selected. One alternative would be to make a PhD student publish a single author paper during the period of their PhD in their area of research, which would confirm the candidates ability.
ReplyDeleteIf you make that a metric, then it will be gamed. I know of a few instances where the advisor was under significant pressure to graduate a student, and they ended up writing large portions of the thesis. What is to stop the same thing from happening in this instance? In many cases, professors are judged by the placement of their students.
DeleteLong ago (well actually only fifty years ago..) a single author paper was required in the best Ph.D programs in the USA.
Deletehttps://www.nature.com/nature/journal/v423/n6939/full/423479a.html
ReplyDeleteThis was written some 12-14 yrs back in the above web site.
"A useful method for job interviews that has been used in our department is to ask candidates to nominate their best three or four papers, then question them on the content of those papers. This selects against publication of over-condensed reports in high-impact journals (unless it is one of the relatively few genuinely important papers of this type). It also selects against 'salami slicing', and is a wonderful way to root out guest authors, another problem of the age. Experience has shown that candidates can have astonishingly little knowledge of the papers on which their names appear"
Thanks for the link. Good article. It also pointedly says
Delete"Candidates can judge institutions by the questions they ask, rather than the other way round. Any selection or promotion committee that asks you for impact factors is probably a second-rate organisation."