Researchers are science accelerators (or: The citation-impact paradox)

Tags: ResearchScience
Cite 
@misc{frieder2024scienceaccelerators,
  title={Researchers are science accelerators},
  author={Simon Frieder},
  year={2024}
  month={July}
  url={https://www.friederrr.org/blog/researchers-are-science-accelerators}
  howpublished = {\url{https://www.friederrr.org/blog/researchers-are-science-accelerators}}
Is your research impact bigger if you are cited more? Not necessarily! This is a post about resource allocation in science and how to really make sure your contribution as a researcher is meaningful.
THE QUEUE is all the ideas that are "up in the air" at the moment, but haven't been manifested in a publication or a piece of software. The (long-hanging) fruits that are up for grabs. The things researchers are talking about behind the scenes at conferences. Since THE QUEUE is essentially public information, there is a slow trickle of ideas from it being transformed into papers. THE QUEUE also serves as an orientation to where the field is moving.

THE QUEUE has a very different character, depending on the scientific subfield. In machine learning, THE QUEUE is volatile and can change fast, yet its ideas are very well specified. A perfect example comes from the currently very hyped and trendy subfield of language model research: After ChatGPT came out, the first papers appeared that benchmarked it and sometimes showed truly impressive performance (yours truly provided a benchmark on mathematical capabilities just two months after ChatGPT’s release (arXiv), although that paper also contains general on how to make datasets for mathematics), it became clear that benchmarking language models was a thing. THE QUEUE suddenly got chock-full of ideas for various specific benchmarks to be given to language models. In fields such as mathematics, THE QUEUE is much more stable, and researchers who pursue some idea from THE QUEUE will rarely drop their research because of a radical paradigm shift. Unlike in machine learning, after the deep learning revolution happened in 2012 with AlexNet, a perspective shift happened, and feature engineering ideas were flushed out of the queue. Processing ideas off of THE QUEUE is also more difficult. Even if experts generally have a good feeling about what technique can work on which type of problem, technical issues can arise.

So, how do we measure the impact of a researcher? There are two ways:

Option Q)

The queue way: How fast you can transform ideas from THE QUEUE into papers.

Option nQ)

The non-queue way: Proceed counterfactually and examine how likely it is that someone else might develop the same ideas you would develop hadn’t you published.

Being cited often correlates in the short to medium term much more strongly with choosing option Q. Citations counts really are only combined measure of how active your field is and how fast you were able to process ideas off of THE QUEUE.

In both cases, you, as a researcher, serve as a science accelerator. Working in Q), your own individual contribution might not really be of much value: You could just wait for other people to do that. If your field, like machine learning, is already highly accelerated and thus has many people working on THE QUEUE, this will happen fairly quickly. Your contribution isn’t strictly needed for the field to advance; your contribution will, at most, advance your career by proving that you are a capable THE-QUEUE-processor. (Being such a processor is not solely negative though: A certain amount of people are needed to keep THE QUEUE moving.)

Another blog that explores this in great detail in the field of psychology is “I’m So orry For psychology’s loss”. Go read it; it dramatically illustrates that papers worth entire careers could be removed from the field without it changing much.

If you are working on nQ), your ideas and papers will make a difference since no one else could have written them. But no one might read them or cite you. This is a constant risk if you work in (very) pure mathematics, such as model theory, and only a few groups exist worldwide that follow up on each other’s work.

I wish you the best of luck in achieving a balance in your relationship with THE QUEUE and in your stream of decisions about when to work on it or off it.