How does one evaluate the quality of ones own research and that of others?
First, acknowledge that this is subjective. Different people have different values, personalities, and background. Here are my values. Most are somewhat interconnected. Not only is the selection of criteria below subjective but how one evaluates each one is subjective and personal.
Ultimately science is about truth and reality. How confident can I be that the results (but also conclusions) actually are correct? Are the results likely to be reproduced? Are they consistent will earlier work and general principles?
Of course, one can never be completely confident, but particularly with experience, once is able to weigh up the probability that results will stand the test of time. The more extra-ordinary the claims the greater the evidence must be.
Ultimately, theories must have something to do with real materials and be experimentally testable in some broad sense. Research that claims to have technological relevance must have some chance of being actually realisable on the time scale of a decade or two and not wishful thinking and marketing [science fiction].
There is a well defined problem. The research presents a well defined answer. For some papers/talks/grants I encounter it is not clear what the problem and answer are. It just seems like a bunch of measurements or a bunch of calculations. Uncontrolled approximations and poorly understood experimental techniques have some role, but do not have the same value as definitive techniques and results where one actually knows what one is doing.
One measure of this is the breadth of application [or scientific relevance] and implication. Will this change how we think about a field? Will it make possible new measurements or new calculations? Does it solve an outstanding problem that has previously confounded excellent scientists?
For some work I am just never really clear what is being done, whether it is valid, or why it is important. On the one hand, this might just be because I don't have the relevant background to appreciate the work. On the other hand, I fear some scientists just don't make the effort to explain what they are really doing, the context, the assumptions, or the significance of their work.
I think that technical prowess, originality and priority are over-rated. There are exceptions. Sometimes they are very important, but they don't have intrinsic merit, and sometimes fail on the importance criteria.
Note that I do not consider funding, status [journals and institutions], or citations as good measures of research quality. They are sometimes a consequence of some of the above, but they are neither a necessary nor a sufficient condition for high quality research.
I welcome comments and discussion.