By Ruth A Pagell*
(27 Sep 2023) Ruth’s Ranking 56 Part 1 raised concerns over existing rankings and showed how THE Impact was addressing some of them. Part 2 continues the discussion. QS 2024 World Rankings changed metrics and weightings, causing controversy over changes in countries’ performance. Nature added a new subject, Health Sciences and it continues to promote the rise of Chinese universities (Baker). CWTS Leiden is exploring ways to make their future rankings more transparent, and it is promoting good practices. What impact have the changes had on university ranks?
QS 2024 WORLD RANKINGS
QS added three metrics to its World Rankings, International Research Network (IRN), Employment Outcomes, and Sustainability. It changed the weightings for three existing metrics.
Adding metrics and changing weighting simultaneously resulted in changes to university rankings and in some cases to overall countries’ performance. Mitchell provides a good overview of the changes from the perspective of QS Senior Vice President Ben Sowter.
From my perspective, Employment Outcomes does not address the concerns expressed by professional schools or match the values of Generation Z. Law schools pulled out of the US News Law rankings because “it discourages law schools from providing critical support for students seeking public interest careers and devalues graduates pursuing advanced degrees” (Gerkin)¹. QS Employment Outcomes excludes “further study, family formation, military service and voluntary work” (Ruby & Lin).
An article in QS’s own Insights Magazine recognizes the different values of Gen Z students, noting “For Gen Z, success encompasses personal, professional, financial, environmental and relational dimensions. It is multifaceted, multidimensional and tailored, with significant implications for higher education institutions, who must adapt to meet the evolving needs and aspirations of this generation”(Patel).
Based on the performance of Korean universities in the 2024 ranking, Korea plans to withdraw from QS (Jung). They can refrain from providing their own data. Universities cannot opt-out. India is looking at relying on its own rankings (Smart). I focused my research on changes in rankings from 2023 to 2024. In order to maintain a similar ranking in 2024 as in 2023, a university has to make up the differentials in the new metrics. Only seven universities fell out of the top 100. Western universities moved up and Asian universities moved down. The most important takeaway is that one size does not fit all. Users of rankers should consider looking at a variety of options, both within QS and across other rankings.
See Appendix A for details and comments on the metrics added to QS World and Table 56.2.2 for the changes in the top 10 between 2023 and 2024 rankings for the World, Asia, and Korea and attached Table 56.2.3 (excel), which lists the top 100 for 2023 and 2024, downloaded from QS, with my calculations on changes in rankings.
RESEARCH OUTPUT RANKINGS
The rankings from Nature Index Annual Tables and CWTS Leiden are targeted toward the scholarly community. Institutions are included based on the size-dependent metric, the number of articles per institution, from curated lists of journals.
NATURE INDEX 2023 Annual Tables uses a package of 146 high-end research journals. There is no fixed number of institutions. Any institution with an author who contributed to an article is included. Over 200 geographic regions are represented in 2023. See Ruth’s Rankings 7 for details of the methodology and the interactive interface. The main indicator is Share, calculated by the percentage of authors from the nation or institution that contributed to the paper.
In addition to “All”, tables are arranged under five sectors, Academic, Government, Corporate, Healthcare, and NPO/NGO. They are further segmented into five subjects, Biological Sciences, Chemistry, Earth & Environmental Sciences, Physical Sciences, and the new Health Sciences. Adding Health Sciences as a subject added 64 medical journals. The articles from these new journals comprise 17% of all articles. Click here for a complete list of Nature journals.
The US has over twice the share of Health Sciences output than the next four countries, China, the UK, Germany, and Canada. (Nogrady). The addition of Health Sciences to the overall score, surprisingly to me, did not improve the US overall top rankings relative to China (Conroy & Plackett). The US still leads in share when all sectors are considered and China is close behind. See Appendix B Table 56.2.5
Nature Index can answer the question ‘What university in Thailand is number one in Healthcare share using 2022 data (Chulalongkorn). An interesting feature of Nature Tables is a section on collaboration for every institution in the rankings. For example, it lists the top ten international collaborators for Chulalongkorn.
Nature clearly states that the data represent a small portion of scholarly output. “Nature Index metrics alone should not be used to assess institutions or individuals.” See notes at the bottom of each ranking table.
CWTS LEIDEN Ranking made no changes for 2023. It is exploring ways to make the ranking more transparent by replacing Clarivate data with open data sources such as Open Alex or Crossref. (van Eck, et. al.). The 2023 ranking has 1411 Institutions from 72 countries, fewer locations than other rankings. The default rank for all indicators is the number of publications. Harvard is number one in all categories (van Eck, et.al.)
It has been covered in several RR columns. It has similarities to Nature Index Annual tables in terms of target audience and inclusion based on the number of publications. It is the only ranking that has made the underlying dataset available to the public.
Eight Chinese universities are in the top 10 for total output, Hunan University, ranked 117 worldwide, is the top Chinese university in the size-independent category, with papers in the top 10 percent of their field. See Appendix B Table 6.
There are four types of indicators, Scientific impact, Open access, Gender, and Collaboration. It can answer the question “What university in Thailand has the most collaborative publications in biomedical and health sciences with collaborators located over 5000 kilometers away (Mahidol).
Leiden provides information on the Responsible Use of any rankings, has an interesting blog for those interested in rankings and scholarship, and advises students not to use the Leiden Ranking to choose where to study.
Although Nature and Leiden use different datasets and time periods, it is interesting to compare their rankings since they are the two top rankings that focus only on publications.
CONCLUSION:
I stand by my conclusion from a 2017 article on the problems with rankings. I acknowledged “that the metrics used in rankings are flawed and the rankers keep tweaking their methodologies. It is not the fault of the metrics if they are misused and misinterpreted (RR 23)
- Rankings are here to stay
- Rankings have a responsibility to present non-biased data
- Users of rankings have a responsibility to look beyond the rank, and look at the score, the individual metrics, the overall methodology, and more than one ranking
- There will be initiatives for more specialized and localized rankings since one size does not fit all, which will be covered in Article 57.
NOTES and RESOURCES
¹US News includes universities that did not provide data for the ranking
Previous relevant RR articles
RR 23 (25 Jan 2017) Are higher education rankings flawed or misunderstood? https://librarylearningspace.com/ruths-rankings-23-global-higher-education-rankings-flawed-misunderstood-personal-critique/
RR 7(2 Feb 2015). Asian institutions grow in Nature. https://librarylearningspace.com/ruths-rankings-7-asian-institutions-grow-nature/
RR 8 (27 Feb 2015). Something for everyone, https://librarylearningspace.com/ruths-rankings-8-something-everyone/, introduction to CWTS Leiden
News Flash (03, 2019). CWTS Leiden adds new rankings for Open Access and Gender: https://librarylearningspace.com/ruths-rankings-news-flash-2019-03-cwts-leiden-2019-adds-new-metrics-for-open-access-and-gender/
Update (14 Aug 2020). Drowning in ranking from Nature, CWTS,…and QS https://librarylearningspace.com/drowning-rankings-nature-cwts-qs-ruths-rankings-updates-2nd-quarter-2020/
Update (23 June 2021) Includes Nature, CWTS Leiden, and Qs. https://librarylearningspace.com/news-update-june-2021-the-end-of-the-2020-release-cycle-and-the-beginning-of-2021-2022/
Ruth’s Rankings 46 Part 1 (27 Jul 2023). (New metrics for old rankings,. https://librarylearningspace.com/ruths-rankings-56-part-1-new-metrics-for-old-rankings-the-impact-rankings-with-enhanced-sustainability-questions/
RESOURCES
Baker, S. (19 May 2023). China overtakes United States on contribution to research in Nature Index. https://www.nature.com/articles/d41586-023-01705-7
Conroy, G. & Plackett, B.(16 June 2022). Nature Index Annual Tables 2022: China’s research spending pays off, https://www.nature.com/articles/d41586-022-01669-0
Fischer, K. (12 Jul 2023). What’s the future for international rankings, Latitudes. https://www.chronicle.com/newsletter/latitudes/2023-07-12?utm_source=Iterable&utm_medium=email&utm_campaign=campaign_7236447_nl_Latitudes_date_20230712&cid=lt&source=&sourceid=&sra=true (sign up to read the article)
Health sciences added to Nature Index (15 Jun 2023). https://www.nature.com/nature-index/news/health-sciences-added-to-nature-index
Gerken, H.K. (16 Nov 2022). Dean Gerken: Why Yale Law School is leaving U.S. News & World Report. https://law.yale.edu/yls-today/news/dean-gerken-why-yale-law-school-leaving-us-news-world-report-rankings
Jung, U. & Sharma Y. (4 Jul 2023). Korean universities unite against QS rankings changes. University world news. https://www.universityworldnews.com/post.php?story=20230704195008557
Mitchell, N. (28 Jun 2023). Changes in methodology drives lean in QS world rankings, University World News https://www.universityworldnews.com/post.php?story=20230628085549571
Nogrady, B. (15 June 23, corrected 22 June 23). Nature Index Annual Tables 2023: first health-science ranking reveals big US lead. https://www.nature.com/articles/d41586-023-01867-4
Patel, N. (16 Aug 2023). What Gen Z’s vision means for Higher Ed, Insights Magazine. (Click on the link and create your own free account). https://magazine.qs.com/qs-insights-magazine-7/view-gen-z-and-gme
Priem, J., Piwowar, H., & Orr, R. (2022). OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. ArXiv. https://arxiv.org/abs/2205.01833
Ruby, A. & Lin, J. (15 Jul 2023). QS global university rankings revamp is a leap backwards. University World News. https://www.universityworldnews.com/post.php?story=20230710212521782
Smart, P. (29 Jun 2023). QS World University Ranking 2024: Decoding changes in ranking methodology and its impact on Indian varsities. The Indian Express. https://indianexpress.com/article/education/qs-world-university-rankings-2024-decoding-changes-in-ranking-methodology-this-year-impact-indian-varsities-iit-bombay-jee-advanced-2023-jee-main-8688658/
van Eck, N.J. , Waltman, L., Anli, Z. et. al. (21 Jun 2023). Leiden Madtrics. https://www.leidenmadtrics.nl/articles/the-cwts-leiden-ranking-2023
Ruth’s Rankings
A list of Ruth’s Rankings and News Updates is here.
*Ruth A. Pagell is emeritus faculty librarian at Emory University. After working at Emory, she was the founding librarian of the Li Ka Shing Library at Singapore Management University and then adjunct faculty [teaching] in the Library and Information Science Program at the University of Hawaii. She has written and spoken extensively on various aspects of librarianship, including contributing articles to ACCESS – https://orcid.org/0000-0003-3238-9674
