Several alternative tools for citation / impact factor analysis: ○ Commercial: •
Thomson ISI Web of Science: Journal Citation Report. • Scopus by Elsevier ...
Ranking Universities, Researchers, and Journals : Review of the Tools Mika Vaihekoski / 24.6.2009
Background | |
|
Ranking is becoming increasingly important in competitive academic world Universities, journals and researchers are increasingly ranked z to measure one’s impact & influence z to help recruiting (both parties) z to get prestige (e.g. schools and journals) z to raise money (corporate & government financing) z citation / impact factor analysis is one of the key criterion used Several alternative tools for citation / impact factor analysis: z Commercial: • Thomson ISI Web of Science: Journal Citation Report • Scopus by Elsevier – includes also journals from other publishers) z
2
Community / open source / free: • SSRN • RePEc • Publish or Perish Analysis (Google Scholar)
Thomson Reuters ISI‐database |
Citation Index: Science, Social Science and Humanities z researcher (and school/department) can analyze who and how many have cited NN’s articles or how many has cited article Y z NN’s citations available • from ISI‐journals to ISI‐journals (default) • from ISI‐journals to any journals (choose 2nd tab (?) in Web of Science); note that citations to non‐ISI‐journals are accounted only to the first author z
3
h‐index is also available
Thomson: new additions |
| |
4
New 2008: Conference Proceedings Citation Index (+ Unified Citation Index) z also several proceedings in economics and business (list of conferences available) New 2008: Scientific WebPlus New 2008: researcherID.com z service for researchers to make a list of one’s articles available for peers on the internet z counts citations automatically if in ISI
Thomson: JCR |
5
Journal Citation Report (JCR) Æ Impact Factor z uses citation data z IF(journal X, year 2008) = number of citations in 2008 to articles published in journal X during 2006‐7 / number of citable articles in published in 2006‐7 z key criteria around the world for “how good the journal is” z released once a year in May/June Æ impact factor for 2008 will be published in June 2009 z new journals added every year (takes two years before IF for a new journal released) z new 2009: 5‐year impact factor
Thomson: JCR (2) z
NEW 2009: Eigenfactor Metrics • Eigenfactor™ Score and Article Influence™ Score • available only for JCR years 2007 and later. • The Eigenfactor Score measures the number of times articles from the journal published in the past five years have been cited in the JCR year • Differs from IF: • Counts citations to journals in both the sciences and social sciences. • Eliminates self‐citations. Every reference from one article in a journal to another article from the same journal is discounted. • Weights each reference according to a stochastic measure of the amount of time researchers spend reading the journal.
• The Article Influence Score calculates measures the relative importance of the journal on a per‐article basis. It is the journal's Eigenfactor Score divided by the fraction of articles published by the journal. That fraction is normalized so that the sum total of articles from all journals is 1. • The mean Article Influence Score is 1.00. A score greater than 1.00 indicates that each article in the journal has above‐average influence. A score less than 1.00 indicates that each article in the journal has below‐average influence. 6
Free eigenfactor analysis
7
Scopus | | | | |
8
newcomer but clearly positioned against / to complement Thomson’s JCR by Elsevier but not restricted solely to their journals social sciences covered better, history a bit shorter (1996‐) includes also papers accepted for publications and from conference proceedings some new tools; e.g. one can easily search with institution’s name for all output done by researchers at the institution (e.g. for TSE, one can find 356 documents, 221 authors)
Scopus: Journal Analyzer |
Journal Analyzer gives for each journal: 1. 2. 3.
z
benefits: •
9
Total number of citations during a particular year to articles published at any time in the journal in question number of articles published during a year Trend Line = 1. / 2. ≈ impact factor (Scopus‐IF) Note: as a result, the Scopus‐IF can be quite high if compared to Thomson‐IF. E.g. the Journal of Finance got 95.68 citations per article published in 2008. updated every two months, older citations are also taken into account, ongoing year can be analyzed
Scopus‐IF vs. Thomson‐IF z
Impact Factor (IF) for journal Y for year X: •
•
10
Thomson‐IF = Average number of times articles from the journal published in the past two years have been cited in the JCR year. ≈ (short‐term) importance of the articles published in the last two years Scopus‐IF = number of citations during X made to articles in Y ≈ importance of the journal’s articles during year X
SSRN | | |
SSRN is a “warehouse” for working papers individual authors can submit their WPs to SSRN (minimal pre‐screening) benefits: z z z
|
negative sides: z
11
papers become available to wider public SSRN has subject oriented “newsletters” Æ further exposure to the paper offers “counter” for downloads and citations author loses part of the anonymity (some journal even require that papers online are taken down)
SSRN in ranking | | | | |
12
ranks schools in the USA and internationally, researchers and papers globally based on downloaded number of papers during the past 12 months SSRN rank measures the immediacy of research produced and its popularity Measures the impact of the contemporary research (and its future potential) university ranking is affected by the size of the faculty (researchers publishing papers at the SSRN), their activity (number of working papers), the exposure of the research in the net (school’s wp series, used in courses, home‐pages etc.)
SSRN / University ranking |
Top International Business Schools z z
Ranked by Total New Downloads (last 12 months) TOP‐5 (based on 10.12.2008 numbers) 1. London BS (71 542 downloads vs. Harvard 138 328) 2. Erasmus (70 867) 3. IESE Spain (58 641) 4. Tilburg University Netherlands (42 738) 5. City University London, BS (38 936) … 13. Stockholm SE (22903) 30. Copenhagen BS (13658), 48. NHH, 50. Aarhus SB And Finnish Universities…
13
SSRN and Finnish Universities 107. HANKEN Swedish School of Economics (3493) 127. TSE (2807) 195. Lappeenranta (1495) 309. Jyväskylä 360. Helsinki U of Tech / Institute of Strategy 657. HSE / Dep. of Marketing 657. University of Tampere ‐‐‐ Vaasa ‐‐‐ Oulu 14
15
HANKEN Swedish School of Economics
16
SSRN in 2009 … |
17
In 2009, SSRN will expand the number of Rankings: z to include those based on citations, and, z will add an Eigenfactor measure of importance for papers, authors, and institutions. z The Humanities Research Network will expand beyond the initial subject areas of Classics (CRN), English and American Literature (LIT), and Philosophy (PRN), and z we will continue to work with other content providers, such as ProQuest, to get broader readership for papers in the SSRN eLibrary.
RePEc ‐‐ Background | | |
|
18
RePEc = Researh Papers in Economics strong emphasis on economics and finance, but other closely related disciplines increasingly represented in the population community driven project (i.e. non‐commercial) that aims to: z collect list of all publications (articles, books, patents, and working papers) in economics (widely understood; e.g. business administration, statistics etc. included) into a database which can be searched z the main aim is disseminate information to fellow researchers z provides also other useful services (e.g. CitEc, NEP service) z collects information on published articles and working papers 18.2.2009: z 10800 institutional contact listings with 18900 registered authors z 279000 working papers, 415000 articles z 766,586 file downloads and 2,756,978 abstract views in January 2009
Background 2 |
info on articles come from publishers z z z
|
commercial publishers provide just the authors, title, and sometimes abstract and in few cases even the references non‐commercial publishers (e.g. Finnish Economic Papers) provides complete information (even article content & references) one way to promote the journal
info on working papers comes from different archives (kept by different organizations such as BOF, ETLA, Hanken, Turku: ACE, NBER, Harvard etc.; universities can set up their own) z z
complete contents (including e.g. citations) sub‐archives (e.g. Econpapers, Nordic S‐WoPA) • e.g. Bank of Finland discussion papers, Hanken working papers series are also included in S‐WoPA and Econpapers series through which they are also included in the RePEc analysis
z |
19
if a school does not have a WP archive, one can submit papers to the MPRA archive http://mpra.repec.org/
typically the published version is not downloadable, but the wp‐version is
Author registration |
RePEc also gives authors a chance to register and it is worthwhile to do so. Why? z
easy, two step process at http://authors.repec.org/ 1. provide your e‐mail, select a password & give your university 2. RePEc searches its database and gives a list of articles and papers that are potentially written by the author; use check‐boxes to pick those that are really yours; then the same for citations; and that’s it!
z
z
once registered, every time researcher’s article comes up from RePEc (Econpapers etc.), there is a link to author’s public profile page listing all author’s articles in the system Æ disseminates info every month registered users will also get an email (one email, nothing else ‐‐ no span, ads etc.) listing author’s new articles, citations, and his/her ranking among researchers based on several criteria (which is always fun to follow) • ranking not public unless one belongs to top‐5 (even then the details of the ranking is not public)
z 20
the ranking data is also used to create a ranking for the school, country, continent, men vs. women, top‐5%, journals rankings etc. • one way to promote school and research
Search You can check out if your papers are already included in the database: http://econpapers.repec.org/
21
CitEc |
RePEc uses the data in many ways, one of which is to find out citations and references z
z z z
22
note that since articles do not provide citations, they are taken from working papers Æ RePEc takes it a standard approach that researchers first publish their works in working papers series and then as articles this project has been named as CitEc it uses automatic algorithms ‐‐ not 100% correct RePEc can combine author’s papers and articles • e.g. if the paper is first published in the university’s own series, later in NBER series, and finally as an article, RePEc recognizes that they are actually the same articles and tells that the article is also available as … (given that the title has stayed the same)
RePEc‐ranking |
CitEc‐data can be used in many ways; such as to provide free (cf. Scopus, Thomson ISI) author ranking service z
|
ranking based on 32 criteria (smaller is better i.e. 1 = ranked the best) z z
| 23
registered authors receive monthly an email with a list abstract views and downloads + analysis of the ranking
criteria includes e.g. number of articles, number of pages, number of citations, number of downloads etc. self‐citations excluded (e.g. authors A and B write an article, and authors B and C cite it; only author A is credited in the ranking analysis).
ranks institutions, researchers and provides even an ranking for the journals
Example
24
Example continued
25
RePEc ranking |
Top institutes by the average score are: 1. 2. 3. 4. 5. 6. 7.
26
NBER = National Bureau of Econ Res, USA (1) CEPR = Centre for Econ Policy Res, UK (2.13) IZA, Germany (2.92) Harvard Princeton Berkeley LSE
RePEc ranking |
| |
|
| 27
Nordic countries: z 94. Bank of Sweden (116.09) z 97. Aarhus University z 131. University of Copenhagen z 198. Oslo University (199.51) Finland: http://ideas.repec.org/top/top.finland.html Top 20% institutions in Finland: z BOF, RUESG, HY, HECER, ETLA, VATT Top 20% authors in Finland: z Honkapohja, Lanne, Poutvaara, Kanniainen, Widgrén, Vilmunen, Maliranta etc. TSE at RePEc http://edirc.repec.org/data/tukkkfi.html
Harzing’s Publish or Perish (POP) |
POP differs e.g. from Thomson ISI Web of Science z freely available: http://www.harzing.com/ z analysis based on Google Scholar • material differs (e.g. wider coverage of journals, includes also books, conference papers etc.) • Butler (2006) analyzed the distribution of publication output by field for Australian universities between 1999‐2001. She finds that whereas for the Chemical, Biological, Physical and Medical/Health sciences between 69.3% and 84.6% of the publications are in ISI listed journals, for Social Sciences such as Management, History Education and Arts only 4.4%‐18.7% of the publications are published in ISI listed journals.
• Typically gives higher (typically twice the) number of citations • ISI includes only citations from ISI journals • ISI calculates non‐ISI citations only towards the 1st author z z 28
produces more statistics for more details, see http://www.harzing.com/pop_gs.htm
29
30
Results: Vaihekoski 6.2.2009 Papers: Citations: Years: Cites/year:
56 77 14 5.5
Cites/paper: 1.38 Cites/author: 60.00 Papers/author: 45.53 Authors/paper: 1.43
h-index: 5 g-index: 6 hc-index: 5 hI-index: 3.57
AWCR: AW-index AWCRpA: hI,norm
10.06 3.17 7.38 5
Hirsch a=3.08, m=0.36 Contemporary ac=1.60 Cites/paper 1.38/0.0/0 (mean/median/mode) Authors/paper 1.43/1.0/1 (mean/median/mode) 36 paper(s) with 1 author(s) 18 paper(s) with 2 author(s) 1 paper(s) with 3 author(s) 1 paper(s) with 5 author(s) |
NOTE: one should be careful when conducting searches. E.g. z z z z
|
31
Vaihekoski, Mika + Business, Adm, Econ, Finance only gives 78 citations (2.4.2009) Vaihekoski, Mika + All areas gives 81 citations (2.4.2009) Vaihekoski, M + All areas gives 97 citations (2.4.2009) vaihekoski, m + All areas gives 103 citations (2.4.2009)
the more common name (e.g. Smith, John), the more detailed/narrow search one should use + carefully uncheck unrelated results (Vaihekoski happens to be a safe one as there are only four people with the same lastname and I am the only one doing research)
Explanation of the results |
|
|
32
Hirsch’s h‐index z
A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np‐h) papers have no more than h citations each.
z
In other words, h=4 means that the author has four papers each of which has been cited at least four times; to get h=5, one need five papers each cited at least five times
z
Provided also in ISI and Scopus
Hirsch a, m z
a is the associated proportionality constant for h‐index (from Nc,tot = ah2)
z
m is the rate parameter (from h ~ mn, where n is the number of years since the first publication).
Egghe’s g‐index z
Given a set of articles ranked in decreasing order of the number of citations that they received, the g‐index is the (unique) largest number such that the top g articles received (together) at least g2 citations
z
It aims to improve on the h‐index by giving more weight to highly‐cited articles. Source: Harzing (2007, web-site)
Explanation of the results |
|
|
Zhang's e‐index (new 5/2009) z
The e‐index is the (square root) of the surplus of citations in the h‐set beyond h2, i.e., beyond the theoretical minimum required to obtain a h‐index of 'h'.
z
The aim of the e‐index is to differentiate between scientists with similar h‐ indices but different citation patterns.
Contemporary h‐index (hc‐index) z It adds an age‐related weighting to each cited article, giving less weight to older articles. z The weighting is parametrized; the Publish or Perish implementation uses gamma=4 and delta=1. This means that for an article published during the current year, its citations account four times. For an article published 4 years ago, its citations account only one time. For an article published 6 years ago, its citations account 4/6 times, and so on. Contemporary ac z Measure related to the hc‐index parametrization
33 Source: Harzing (2007, web-site)
Explanation of the results |
Individual h‐index (hI‐index and hI,norm) z It divides the standard h‐index by the average number of authors in the articles that contribute to the h‐index, in order to reduce the effects of co‐authorship; the resulting index is called hI. z
Publish or Perish also implements an alternative individual h‐ index, hI,norm, that takes a different approach: instead of dividing the total h‐index, it first normalizes the number of citations for each paper by dividing the number of citations by the number of authors for that paper, then calculates hI,norm as the h‐index of the normalized citation counts. • Claimed to be a better approximation of the per‐author impact
|
Multi‐authored h‐index (hm‐index) (new 5/2009) z method uses fractional paper counts instead of reduced citation counts to account for shared authorship of papers, and then determines the multi‐authored hm index based on the resulting effective rank of the papers using undiluted citation counts
34 Source: Harzing (2007, web-site)
Explanation of the results |
AWCR z z
|
AW‐index z
|
Age‐weighted citation rate The AWCR measures the number of citations to an entire body of work, adjusted for the age of each individual paper. It is an age‐ weighted citation rate, where the number of citations to a given paper is divided by the age of that paper The AW‐index is defined as the square root of the AWCR to allow comparison with the h‐index; it approximates the h‐index if the (average) citation rate remains more or less constant over the years.
AWCRpA z
The per‐author age‐weighted citation rate is similar to the plain AWCR, but is normalized to the number of authors for each paper.
35 Source: Harzing (2007, web-site)
36
Top 5 in Accounting and Finance in Finland |
Using POP, one could identify the following top‐5 (21.3.2009) in accounting and finance among Finnish professors (searched using lastname, firstname) using # of citations: z z z z z
|
Matti Keloharju, HSE/Finance, 1510 citations Kari Lukka, TSE/Accounting, 770 Teppo Martikainen, Finance, 719 Teemu Malmi, HSE/Accounting, 601 Eero Kasanen, HSE/Finance, 595
cf. result for two Swedish finance professors: z
Per Strömberg 1835, Mike Burkart 1649
Good or bad? | |
rankings are here to stay good: z "If you cannot measure it, you cannot improve it." — Lord Kelvin z they give information on good journals/universities/articles/authors z they give feedback on one’s work z they help internationalization and bring attention z different rankings give alternative measures • c.f. not just GDP (number of citations) but also GDP/person (e.g. number of citations / years) z
they bring info on citations • Berger (2008): ”Citations are the foremost yardstick by which scientific research is measured. They are seen as an objective measure of scholarship (Goodall 2006; Hamermesh & Schmidt, 2000) and are used to determine things like hiring, tenure, journal and department prestige, society membership, and even grant funding (Garfield 1999; Hargens & Schuman 1990; Leibowitz & Palmer, 1984; Lowy, 1997).”
|
bad: z z z z
38
z
can become self‐serving measure to some degree prestige, not the real scientific value “badly measured” issues and discussion issues in impact factors problems in comparison (c.f., 1 article w/100 citations vs. 5 articles w/on average 20 citations vs. 20 articles w/on average 5 citations: who is the best?)
Remember: aim is good research, not citations
39
