Do the best papers have the highest probability of

Scientometrics manuscript No.

(will be inserted by the editor)

Do the best papers have the highest probability of being cited?

J. A. Garc´ıa, Rosa Rodriguez-S´ anchez, and J. Fdez-Valdivia

the date of receipt and acceptance should be inserted later

Abstract If scholars suffer from imperfect attention, they will not always cite the

best paper on a particular topic. The most chosen scholarly works may merely be the most cited ones, not the best articles. Here, a paper is chosen when someone cites it, after paying attention to it. Manuscripts’ authors might affect preferences by using salience to influence what scholars pay attention to. In our work, paying attention to an article is when someone reads it. For instance, authors can submit the research works to top-tier journals in the discipline, and thus enter the salient papers of the readers. However, do such competitive forces tend to correct choice errors caused by reader’s imperfect attention? In this short communication, we study about how the competition between research works for publication ensures that the best paper is the one having the highest probability to be cited. According J. A. Garc´ıa, Rosa Rodriguez-S´ anchez, J. Fdez-Valdivia, Departamento de Ciencias de la Computaci´ on e I. A., CITIC-UGR, Universidad de Granada, 18071 Granada, Spain. Address correspondence to J. A. Garc´ıa at [email protected] Jose A. Garcia ORCID iD https://orcid.org/0000-0001-7742-7270

2

Garcia, Rodriguez-S´ anchez, and Fdez-Valdivia

to the model, the best papers are the ones published in the journal with the highest citation impact. Therefore, these papers are also the ones that have the highest probability to attract attention and the highest probability of being cited. Keywords: Scholars; Imperfect Attention; Article Quality; Journal Citation Im-

pact; Paper Salience; Competitive Forces.

1 Introduction

Scientists read for many purposes, including current awareness, teaching, and research (Tenopir et al., 2009). When seeking a leading scholarly article, scholars choose journal papers that are ranked by quality to represent a better publication in order to integrate and synthesize information on a topic (Tenopir and King, 2007). Whenever several works are noticed, those with a better quality are preferred over others of lower quality. There, the quality of a paper is its position in the preference ranking on the basis of its originality, importance, accessibility, and conclusions. However, if scholars suffer from imperfect attention, they will not always pick the best quality paper on a particular topic (Garcia et al., 2018). The most chosen scholarly works may simply be the most cited ones, but not the best articles. This happens because preference is focused on the salient set that the scholar pays attention to, not necessarily the whole set of papers on the topic. The failure to consider all related works may stem from the scholar’s imperfect attention (Garcia et al., 2018). In our model, we assume that “paying attention” refers to “read the article”, while “choosing an article” refers to “citing the article”.


3

The manuscripts’ authors might affect preferences by using salience to influence what scholars pay attention to. For instance, authors can submit the research works to top-tier journals in the discipline, and thus enter the salient papers of the scholars if they get published and achieve higher citation impact (Heintzelman and Nocetti, 2009). In this way, authors can influence the subset of works that the reader actively considers. In this short communication, we study when such competitive forces ensure that the best papers are the ones that have the highest probability of being cited.

2 Publication games

Following Manzini and Mariotti (2016), we present a basic model of competition between scholarly articles for reader’s attention. In our model, the scholar (she) with imperfect attention wishes to choose one article in a finite set A = {a1 , . . . , an } on a topic in the field. The scholar evaluates papers a1 , . . . , an using a preference order on the basis of their originality, importance, accessibility, and conclusions. The position of a paper ak in the preference ranking is its quality: Quality (ai ) > Quality (aj ) iff 1 ≤ i < j ≤ n.

Let the salient papers be the subset of articles that the scholar pays attention to. We assume that she maximizes the preference order Quality (·) on the salient papers. The probability that a research work ai belongs to the salient papers depends on its journal citation impact σi , with σi ∈ S . The journals’ impact factor is a measure of the frequency with which the average article in a journal has been cited in a period of time. The author is only able to choose the citation impact of the journal where the paper is published. The publication strategy set for each

4


paper ai is S ⊂ R. Here, to simplify, we suppose that the variable σi that quantifies the journal citation impact for article ai also indicates the ability of ai to attract attention. For the finite set of research works A = {a1 , . . . , an }, a publication profile σ denotes the list of respective journal citation impacts σ = (σ1 , · · · , σn ) ∈ S n .

Let pi (σ ), with 0 < pi (σ ) < 1, be the probability that article ai belongs to the salient papers (i.e., article ai is read by the scholar) when the list of journal citation impacts is σ . The effectiveness of the publication profile σ = (σ1 , · · · , σn ) is described by probabilities pi (σ ). Each pi (σ ) associates a publication profile σ with the probability of membership of the set of salient papers for article ai . The scholar chooses the preferred article among those she considers (i.e., the salient papers). Therefore, it follows that the probability that article ai is chosen at a publication profile σ is the probability that ai is in the salient papers (it is read by the scholar) and that none of the better articles ak , with k < i, is also in the salient papers (they are not noticed): P i (σ ) = p i (σ )

Y

(1 − pk (σ )).

(1)

k 2). A publication game is denoted (A, S, P ay ), where A is the set of articles {a1 , . . . , an } on a topic in the field, S is the publication strategy set for each paper, and


5

P ay = (P ay1 , P ay2 , · · · , P ayn ) represents the payment to each article. Given a

publication game, we say that the game is well-behaved as follows. A well-behaved publication game: Increasing ai ’s journal citation impact σi strictly increases the probability of being noticed by the scholars, i.e., pi (σ ).

Here, a well-behaved publication game describes the natural competition between research works to be chosen by one scholar with imperfect attention. In this game, manuscript’s authors can set the paper salience by submitting it to a journal of a given impact factor. However, if readers suffer from imperfect attention, do competitive forces tend to correct choice errors? To answer this question, in what follows, we study the pure strategy Nash equilibria of publication games. In game theory, the Nash equilibrium is a solution concept of a non-cooperative game involving two or more players. In the publication game involving two or more manuscripts’ authors, if each author has chosen a strategy (regarding the manuscript submission to a journal of certain impact factor) and no author can benefit by changing strategies while the other authors keep theirs unchanged, then the current set of strategy choices constitutes a Nash equilibrium. Stated simply, scholarly articles ai and aj are in Nash equilibrium if ai ’s author is making the best decision s/he can (regarding the manuscript publication), taking into account aj ’s author decision while aj ’s author decision remains unchanged, and aj ’s author is making the best decision s/he can (regarding the manuscript publication), taking into account ai ’s author decision while ai ’s author decision remains unchanged. The Nash equilibrium is one of the foundational concepts in game theory, (Nash, 1951). In economics, the reality of the Nash equilibrium of a game can be tested using experimental methods, (Osborne, 2004; Bordalo et al., 2016; Manzini and Mariotti, 2016).

6


3 Does the probability of being cited reveal quality in a well-behaved publication game?

We illustrate a simple scenario where, when authors have access to the same journals, the best paper on a topic is guaranteed to have the highest salience in equilibrium. In a well-behaved publication game, when articles are ex-ante symmetric except for the difference in manuscript quality, the game is symmetric as follows. A symmetric publication game: (i) holding the salience of all articles fixed except for research works ai and aj , if ai and aj were published by the same journal of a given impact factor, the effectiveness of this publication profile for getting noticed is the same for each one; (ii) the costly effort to publish a paper in a given academic journal is the same for each article ai and aj .

Regarding (ii), Garcia et al. (2015) provide a formal study on manuscript quality control in peer review. Garcia et al. (2015) show that the effects of editors’ bias on authors’ satisfaction and motivation cause sorting in the authors who submit manuscripts to scholarly journals, and therefore, match authors and journals with similar quality standards. Following Manzini and Mariotti (2016), to describe those simple scenarios where the best paper is guaranteed to have the highest probability of being cited, we need two more conditions that impose positive spillovers of own salience on the other papers: ‘Weak Supermodularity’ acting on the first differences of the probabilities pi , and ‘Cross Monotonicity’ acting on the absolute salience levels of papers.


7

Cross Monotonicity: The article’s author cannot harm the visibility of other papers on a topic by choosing for publication journals of higher impact factor.

This is verified in publication games when new citations of a paper draw attention to similar papers (the research works that study similar problems) and hence to the whole group of articles on the topic. Weak supermodularity: The effectiveness for getting noticed of an increase in the citation impact of the journal where the paper is published increases with the salience of the other articles on the topic.

This condition might be verified in a situation in which the publication effort of an article on a little-known problem is likely to be less effective than when other articles treating the same problem are well-known by the scholars in the field. Cross monotonicity and weak supermodularity are external benefits since they are the positive effect an article imposes on another one on the same topic. Because the cost function is common and because of the external benefits (of cross monotonicity and weak supermodularity) and symmetry assumptions, the scientific discipline looks the same from the perspectives of two different authors, except for the structure of the benefits: the best paper has a higher benefit from raising journal citation impact than an inferior work because there are more situations in which it is chosen conditionally on being noticed. So the author of the best paper has overall a stronger incentive to invest in salience by publishing it in a top-tier journal of the discipline. Now, in the following, we present the best-is-the-most-salient result. Proposition. Let σ be a pure strategy equilibrium of the symmetric well-behaved publication game which exhibits the external benefits of weak supermodularity and cross monotonicity.

8


Let σ1 be the publication strategy of the best quality paper a1 in the whole set A of papers on the topic. Then, in equilibrium, there is no more salient paper than a1 in A. Therefore, the best papers are also the ones that have the highest probability to attract attention and the highest probability of being cited.

Proof: The proof is similar to that of Prop. 2 in Manzini and Mariotti (2016). Thus, we skip it in this short communication. The reason why lower quality articles never produce more salience in equilibrium does not derive either from lower levels of resources or lower unit costs of salience production, both types of asymmetry having been ruled out. In fact we assume that everyone can choose from the same set of journals (to get published) at exactly the same cost or benefit. The result is purely a function of the competitive forces that counterbalance the imperfect attention of the reader. However, note that we have also assumed (through the condition of a well-behaved publication game) that salience is a directional attribute of academic articles: the higher the impact of the journal the paper is published, the more likely the paper would be noticed by scholars in the discipline. Why do the conditions on the probabilities that the article is noticed, pi , turn out to be important for the result? Authors of good articles do not care whether bad research works are noticed or not. Their payoff only depends on the probability that even better quality articles are noticed by the reader. In the extreme case, the author of the best paper on a topic only cares about its own probability of getting noticed. If the probabilities pi were decreasing in the salience of the other articles, an increase in own salience would tend to be the more profitable the lower the quality of a competitor. However, the external benefit of cross monotonicity removes this tendency.


9

Therefore, the only reason why the best paper might not want to follow worse works in raising salience is that raising own salience becomes less useful for becoming more noticeable when the worse research works raise their salience. However, the property of weak supermodularity eliminates precisely this adverse effect. The proposition above ensures that if there is an article that is uniquely maximally salient in equilibrium, then that paper must also be the best one, but it does not exclude that lower quality papers tie with the best for salience. However, the reader could still infer the identity of the best article from citation impact, provided that this paper suffers no technological disadvantage regarding the salience strategy. Competition between research works pushes the best papers to raise their salience sufficiently to overcome the distortive effects of reader’s imperfect attention on the relative popularity of the competitors.

4 Conclusion

Whenever several works are noticed, those with a better quality are preferred over others of lower quality. There, the quality of a paper is its position in the preference ranking on the basis of its originality, importance, accessibility, and conclusions. So we assume that the scholar evaluates papers using a preference order based on their quality. In our model, the information in the publication game is assumed to be perfect: all manuscript’ authors know the quality of all manuscripts. We have also considered the manuscript quality as a fixed characteristic of research works. However, better reviews in a top journal could increase the posterior expected quality of the manuscript, or the quality perceived by the reader could be impacted by the impact factor of the journal. As future research line, an additional

10


condition on how manuscript quality may benefit from submission to top-tier journals could thus be necessary to keep a “revealed-quality” Nash Equilibrium. Also, we have assumed that the reader responds to article’s salience in a simple way. Even though other sources may help create more visibility for own research works (e.g., Google Scholar, OrcidID, ResearchGate, or Twitter), for simplicity we have only considered journal citation impact. Researchers both consume and produce papers. If citing an article increases its salience then we would have a negative externality when citing an article of higher quality than ours. That is precisely the reason why we only consider the extreme case, when the author of the best article on a topic only cares about its own probability of being noticed. However, when the conditions of symmetry, weak supermodularity or cross monotonicity fail, or the publication game is not well-behaved, the equilibrium properties may break down, and the most salient paper might not faithfully reveal its quality ranking. But, how realistic and adapted are the hypotheses of the model to describe the academy? In fact, a well-behaved publication game describes the natural competition between research works. This is so because, in the academy, increasing the journal impact factor strictly increases the probability of being noticed by the scholars and manuscript’s authors can set the paper salience by submitting it to a journal of a given impact factor. Moreover, in a symmetric publication game the costly effort to publish a paper in a given academic journal is the same for different authors. This could be realistic since the effects of editors’ bias on authors’ satisfaction and motivation cause sorting in the authors who submit manuscripts to different scholarly journals, and therefore, match authors and journals with similar quality standards. Furthermore, by cross monotonicity, the article’s author cannot


11

harm the visibility of other papers on a topic by choosing for publication journals of higher impact. This is also well adapted to describe the academy because in a large number of subfields new citations of a paper draw attention to research works that study similar problems. On the other hand, the condition of weak supermodularity might also be verified in a realistic situation in which the publication effort of an article on a little-known problem is likely to be less effective than when other articles treating the same problem are well-known by the scholars in the subfield. In this short communication we assumed the reader only chose one article, so an interesting question would be whether it is the equilibrium stable when we allow readers to pick their k-best papers. Even though here we do not give more details on this issue, it could be an interesting line of future research. Acknowledgments. This research was sponsored by the Spanish Board for Sci-

ence, Technology, and Innovation under grant TIN2017-85542-P, and co-financed with European FEDER funds. Sincere thanks are due to the reviewers for their constructive suggestions and help.

References

Bordalo, P., Gennaioli, N., and Shleifer, A., (2016). Competition for Attention. Review of Economic Studies, 83, 481-513, doi:10.1093/restud/rdv048 Garcia, J.A., Rodriguez-Sanchez, R., and Fdez-Valdivia, J., (2015). The author-editor game. Scientometrics, 104(1), 361-380, https://doi.org/10.1007/ s11192-015-1566-x

Garcia, J.A., Rodriguez-Sanchez, R., and Fdez-Valdivia, J., (2018). Competition between academic journals for scholars’ attention: the ‘Nature effect’ in scholarly

12


communication. Scientometrics, 115(3), 1413-1432, https://doi.org/10.1007/ s11192-018-2723-9

Heintzelman, M., and Nocetti, D. (2009). Where should we submit our manuscript? An analysis of journal submission strategies. The B.E. Journal of Economic Analysis and Policy, 9(1), 39, https://doi.org/10.2202/1935-1682.2340 Manzini, P., and Mariotti, M. (2016). Competing for attention: is the showiest also the best? The Economic Journal, 128, pp. 827-844. https://doi.org/10.1111/ ecoj.12425

Nash, John (1951). Non-Cooperative Games. The Annals of Mathematics, 54(2), pp. 286-295. Osborne, M., (2004), An introduction to game theory, Oxford University, ISBN 978-0-19-512895-6. Tenopir, C. and King, D.W. (2007), Perceptions of value and value beyond perceptions: measuring the quality and value of journal article readings, Serials, 20(3), pp. 199-207. Tenopir, C., King, D.W., Edwards, S., and Wu, L. (2009). Electronic journals and changes in scholarly article seeking and reading patterns, Aslib Proceedings, 61(1), pp. 5-32, https://doi.org/10.1108/00012530910932267