By Ruth A Pagell*
(29 Jul 2024) Ruth’s Rankings started in the middle of the current history of world university rankings. This is the story of how I became a player in the rankings game and the concerns I have about the future. I have tried to be a reporter rather than a critic, which was my previous role in evaluating business databases. The article is what is important to me. The Addendum contains the background methodologies and data tracking changes over the past 20 years.
Written before I began Ruth’s Rankings, Phil Baty, THE’s Chief Global Affairs Office, stated that “We are aware that higher education institutions are extraordinarily complex organisations. They do many wonderful, life-changing and paradigm-shifting things that simply cannot be measured. Data on some of their most valuable endeavours simply do not exist or cannot be meaningfully compared on a global scale; many of the proxies commonly used are less than satisfactory.” (Baty 2011; highlighting is mine). This is something I keep in mind as I evaluate the different rankings.
I first thought that my retrospective would cover the decade of Ruth’s Rankings, beginning in 2014. I realized I needed to look back to the inception of world rankings in 2003. My interest in university rankings grew out of the convergence of three streams of my professional life. I attended library school and then worked a few blocks from the Institute for Scientific Information. Citations were always part of my library tool kit. I used the citation indexes from their inception as print volumes, microfiche, and a website on the old internet on Dialog, and through all of its corporate changes. Before bibliometrics, citations were used as indicators for faculty hiring and tenure.
Stream two took me 14,000 kilometers from Philadelphia to Bangkok Thailand where I spent 1989. I met and kept in touch with Mr. Lee who had a company called Book Promotion and Service, now called iGroup. I also met and kept in touch with one of his clients, my editor Clive Wing, who at that time was working for IDRC in India. Coming back from Thailand I was asked to write a book on international business information. During the interim between living in Thailand and eventually moving to Singapore, I kept in touch with Mr. Lee, through international conferences. When Clive settled in Southeast Asia and became the editor of Access, I or my staff wrote occasional articles for Access. In 2011 I moved to Honolulu and temporarily taught as an adjunct faculty in their library program through 2014. When I finished teaching, I looked for a new challenge.
Stream three was professional expertise in business information in the period between Thailand and Singapore. I recognized similarities among rankings whether it was for the largest food company in Europe or the best university in the world. Issues identifying the largest companies are similar to the differences in measuring citations. The different company datasets used different time periods, categorized companies under different subjects, and had issues with currency conversion. (Pagell, 1991, 1994). I even wrote an article specifically on the concept of rankings in 1997.
World University Rankings
The first global ranking was ARWU, Academic Ranking of World Universities, published by the Center for World-Class Universities at Shanghai Jiao Tong University. The first release, in 2003, included 500 universities. Since 2009 the rankings have been published by ShanghaiRanking Consultancy, a separate company. 2004 saw the launch of the joint venture between Times Higher Education (THE) and Quacquarelli Symonds (QS) that included 200 world universities (US Dominates; Praphamontripong & Levy). The partnership between THE and QS ended in 2010. Each continued with the format that they had introduced after the initial ranking. Each serves a mixed target audience of researchers and students. These three rankings continue to be the most popular.
2014 marked the beginning of Ruth’s Rankings. Originally planned for four or five articles it is now up to 60 articles with a series of updates. 2014 was a time when rankings were growing in importance. Students, faculty, administrators, and governments looked to the rankings for more than they were designed to do. I began with the assumption, based on experience, that people want rankings and the rankings are never going to be perfect. What often goes unmentioned is the users of the rankings are not perfect either. They often do not look beyond the usual questions, “Who is number one?” and “Where is my university ranked?” With that in mind, I will highlight other features of the rankings, such as by subject or region, important for universities that will never make the world top 200. I have also pointed out the differences among universities’ ranks and their scores, which can have very small or very large gaps.
Before analysing the rankings, I introduced what were then the two sources of bibliometric data, Thomson-Reuters, now Clarivate, and Elsevier. As the rankings expanded, it became necessary to report on the role that journal publications played in the ranking, looking at Journal metrics such as the Journal Impact Factor (JIF) and predatory publishing.
By RR 23, I alerted readers to problems with the rankings, some based on the rankings themselves and some based on how they were being used and misused. It became obvious to me that one size does not fit all. I investigated the other rankings approved by IREG, the International Ranking Expert Group, known as the IREG Observatory on Academic Ranking and Excellence. In addition to reviewing these individual rankings in comparison to the more popular rankings, I have always been on the lookout for new metrics and other factors that impact the universities in different countries. The rankings have covered specific countries, such as Malaysia, subjects such as business, and indicators such as international students and reputation rankings for further research. Ruth’s Rankings has gone into the area of geopolitics and examined the different socio-economic factors that affect countries’ education environments.
Some universities have always been at the top, regardless of years or metrics. Table 1 lists only the universities that have been in the top 10 and top 25 in 2003/2004 and 2023/2024.
See this table(in excel) for original and current metrics by the three rankers, subject rankings, country coverage, and data on the original 200 universities, their countries, and their rankings through May 2024.
THE PRESENT
THE was the first to expand its scope by adding a separate Impact Ranking, based on SDGs. This allowed universities that did not qualify for the world rankings to participate and in some circumstances shine. Many of these universities are now included in the world rankings as “reporters”. THE World Rankings has grown from the 200 original universities in the inaugural ranking to over 2345, including reporters, and universities that have appeared in other rankings but have not met the world rankings criteria. Two new rankings will be released in 2024, Interdisciplinary Science and Online Learning. Data on the top 200 will appear in the Addendum.
QS integrated sustainability into its world rankings and made changes to its metrics in 2024. QS has expanded its regional rankings and disaggregated regions to allow universities to benchmark universities in their geographic area. For example, users can search for universities just in South-Eastern Asia (2024).
ARWU has not changed its methodology but has expanded coverage. Including its subject rankings GRAS, Global Ranking of Academic Subjects. it covered 1900 universities in 2023.
Even before the current trend to criticize the rankings, Altbech and Hazelkorn suggested that “most universities should quit the rankings” not only because they are flawed but also because they are misused”. Despite the changes made by THE and QS, universities are pulling out of the rankings. Examples are the University of Zurich and Utrecht University (Brent) which withdrew from THE’s ranking. South Korean universities suffered from the changes QS made in percentages of existing metrics and adding new metrics in its 2024 rankings. ). They have pulled out of QS. See RR 56 Part 2 (Jung & Shama). QS uses third-party data so Korean universities will not disappear. I will check the status of Korean universities in the recently released 2025 rankings.
THE FUTURE
Leiden’s experimental use of OpenAlex (RR 59) is the most prominent attempt to see the effect of using open data on rankings, making raw data available to users. RUR, Round University Rankings, the former Russian-based ranking, now located in the country of Georgia, has used open access data from LENS for its 2023 rankings. A whole new industry which I have labeled the open access industry comprised of many organizations, is trying to make major changes in university rankings.
University rankings are not a popular topic in scholarly journals. However, Ayhem analysed the 100 most cited articles on university rankings from 2017- 2021 and concluded “whether they like it or not, higher education institutions are and will continue to be ranked by different ranking systems whose methodology, scope and implications will definitely continue to be criticized in the future.” Richard Holmes, author of University Rankings Watch shared his thoughts on the changes ahead in a recent University World News article (Holmes).
It will be interesting to see what the next few years will bring. My concern, no matter what the data is being used, is one of quality control. I may be old and old-fashioned but I see a difference between academic publications for use by scholars and articles similar to mine, which are targeted for a different audience.
- What impact will this have on commercial bibliometric companies such as Clarivate, Elsevier, and Dimensions?
- How will the other world rankers respond to the changing world?
- Will the changes realistically be able to democratize the rankings?
- Where will AI fit into this picture and will the rankings of the future be created by AI?
Rankings’ metrics, subjects, countries, and the top 200 from 2003/04 to May 2024 will appear in the addendum to RR 60.
RESOURCES
Altbech and Hazelkorn, “most universities should quit the ranking game.” https://www.universityworldnews.com/post.php?story=20170105122700949
Ayham, I. (20 Feb 2024). Research trends in university rankings: A Scoping review of the top 100 most cited articles in academic journals from 2017-2021. European Review.https://www.cambridge.org/core/journals/european-review/article/research-trends-in-university-rankings-a-scoping-review-of-the-top-100-most-cited-articles-in-academic-journals-from-2017-to-2021/C1780A57BE48EC011A21E3D7B7FCA2FE DOI:10.1017/S1062798723000595
Baty, P. (3 Jan 2024). One ranking to rule them all? Absolutely not. https://www.timeshighereducation.com/world-university-rankings-2010-11-methodology
Baty, P. (1 Oct 2011). World University Rankings 2010-2011 https://www.timeshighereducation.com/world-university-rankings-2010-11-methodology
Brent, T. (12 Oct 2023). Utrecht University withdraws from global ranks as the debate on quantitative metrics grows. Science/Business.
Hazelkorn, E. (2013) How Rankings are Reshaping Higher Education. in Climent, V., Michavila, F. andRipolles, M. (eds) Los Rankings Univeritarios . Mitos y Realidades. https://arrow.tudublin.ie/cgi/viewcontent.cgi?article=1023&context=cserbk
Holmes, R. (3Feb 2024). Global rankings: the age of deference is coming to an end. University World News https://www.universityworldnews.com/post.php?story=20240202121114196
Jung, U. & Sharma Y. (4 Jul 2023). Korean universities unite against QS rankings changes. University world news, https://www.universityworldnews.com/post.php?story=20230704195008557
Praphamontripong, P. and Levy, D. (2005). World University Rankings 2004. PROPHE. https://prophe.org/cache/0723021_WorldUniversityRanking2004_ModifiedFromTHES.pdf
Siwiński, W. & Federkell, G., ed. (Dec 2023) IREG Guidelines for Stakeholders of Academic Rankings 2023, IREG Observatory on Academic Rankings and Excellence, https://ireg-observatory.org/en/wp-content/uploads/2023/12/IREG-guidelines2023.pdf
University of Zurich withdraws from international university ranking. (13 Mar 2024). SWI Swissinfo.ch. https://www.swissinfo.ch/eng/education/how-many-international-students-study-at-swiss-universities/72921721
US dominates worldwide league tables (5 Nov 2004). The Times Higher Education Supplement. https://www.timeshighereducation.com/news/us-dominates-worldwide-league-tables/192159.article
My background publications that contributed to this article.
Pagell, Ruth. (1991). It’s Greek to Me! Exchange Rate Translations and Company Comparisons, Database, February 14(1), 21-27.
Pagell, Ruth A. (1990). What’s for Dinar: Foreign Exchange Rate Data Sources, Database, 13(6), 46-51
Pagell, R.& Halperin, M. ( 1994,1997). International Business Information, How to find it how to use it. Oryx Press
Pagell, R. (Jun 1997). Rankings and comparisons: the top lists. Business Information Review. 14(2) 71-79
Ruth’s Rankings
A list of Ruth’s Rankings and News Updates is here.
*Ruth A. Pagell is emeritus faculty librarian at Emory University. After working at Emory, she was the founding librarian of the Li Ka Shing Library at Singapore Management University and then adjunct faculty [teaching] in the Library and Information Science Program at the University of Hawaii. She has written and spoken extensively on various aspects of librarianship, including contributing articles to ACCESS – https://orcid.org/0000-0003-3238-9674
