By Ruth A Pagell*
(30 Jul 2024) The first part of my retrospective (RR60) is personal. It focuses on the broader picture of university rankings, their role in higher education, and the current criticisms of their shortcomings. This Addendum examines the data. It tracks the first two world rankings, the Academic Ranking of World Universities (ARWU), published by the Center for World Class Universities at Shanghai Jiao Tong University in 2003 and the Times Higher Education (THE) and Quacquarelli Symonds (QS) joint ranking in 2004. It covers the methodologies from those first two rankings plus the separate QS ranking starting in 2010 to methodologies in 2024. It also compares subject rankings and country coverage. I see my Ruth’s Rankings role as a reporter. For the Addendum, I have added personal comments.
THE-QS METHODOLOGIES and METRICS
The Times Higher Education Supplement on November 5, 2004 (“US dominates”) introduced the THE-QS rankings. The rankings are based on a survey sent to 1300 academics in 88 countries who were asked to provide their opinions on universities in their fields. 200 universities are ranked. 29 countries are represented. The US dominates the rankings with 62 universities, followed by the UK with 30, Germany with 17, and Australia with 14. 24 universities are from eight Asian countries, with Japan leading with six. The data were chosen to represent strengths in teaching, research, and international reputation.
THE-QS Metrics (2004 – 2009)
- Peer Review – 50% of the final score from QS survey, similar to current reputation rankings
- International Faculty and International Students, 5% each, with data from the QS survey
- Citations per faculty – 20% “highlighting the international clout of institutions ” with data from Essential Science indicators
- Faculty/Student ratio – 20% “reliable indicator on commitment to teaching” from QS
I find this questionable. It makes assumptions about the interaction of faculty with students in the classroom and the varying abilities of faculty as teachers..
Source of bibliometric data:
- 2004-2006: Thomson- Reuters Essential Science Indicators
- 2007-2009: Scopus based on more publications
- Rankings for 2004 to 2009 can be found on the University Rankings.ch
Between 2004 and 2009 QS collected the data and THE reported it in Times Higher Education Supplement. The two companies split after the 2009 rankings. Data for the individual 2004 metrics are available (Praphamontripong and Levy).
THE WORLD UNIVERSITY RANKINGS: METHODOLOGY and METRICS (2010-2024)
For the 2010-2011 to 2013-2014 rankings, THE used Thomson Reuters data. Starting with the 2014-2015 rankings, THE brought all data in-house and reverted to Scopus for its bibliometrics. THE metrics remained the same until 2024 although changes were made in calculations. The most notable change in 2024 is disaggregating the Citation component from one to four indicators. Reputation rankings for Research and Teaching for 2023 rankings came from 68,000 responses from surveys in 2022 and 2023 from 166 countries. Universities submit data, Table 1 lists the categories and the number of indicators for each category. There is no public access to the individual indicator data. THE has 14 releases planned for the calendar year 2024. The rankings are labeled 2023, 2024, or 2025.
THE 2013-2014 https://www.timeshighereducation.com/world-university-rankings-2013-14-methodology ;
THE 2023-24 https://www.timeshighereducation.com/world-university-rankings/2024/world-ranking
Ross, D. (Sep 20,2023). World University Rankings 2024, https://www.timeshighereducation.com/world-university-rankings/world-university-rankings-2024-changes-our-methodology
US dominates worldwide league tables. (4 Nov 2004). Times Higher Education Supplement. https://www.timeshighereducation.com/news/us-dominates-worldwide-league-tables/192159.article (registration required).
Comments: Universities opt in or out of the rankings. THE added Impact Rankings on SDGs in 2009 and rankings for Sub-Saharan Africa in 2023. Universities in these two rankings that do not meet the criteria for receiving a score in World Rankings are added as “reporters. In searching for universities in countries such as France or Germany, the name must include the diacritics, as in “École.” THE offers fee-based consulting services See RR 57 for more information on the new metrics.
QS WORLD UNIVERSITY RANKINGS: METHODOLOGY AND METRICS (2010-2024)
When QS split from THE, it continued to use a methodology similar to the initial joint rankings until 2023. Start at https://support.qs.com/hc/en-gb to find the methodology and other supporting information. QS suggested not making comparisons when they changed their methodology for the 2024 rankings. Comparisons are important to me for analysis. See RR 56 Part 2. Since QS uses third-party data, universities can choose not to participate. They will still appear in the rankings.
All years of overall rankings from 2004 are available through Universityrankings.ch
See https://librarylearningspace.com/wp-content/uploads/2023/09/Appendix-A-QS-2024-PDF.pdf for changes in QS methodology; Qs 2024 rankings: https://www.topuniversities.com/world-university-rankings/2024
Comments: Positive features of the world rankings are the ability to search by region and the inclusion of a sustainability metric in the World ranking. QS has reduced the value of the Reputation rankings from 40% to 35%, which I think is still too high. The new methodology was accompanied by a new search interface, which displays 15 universities per screen without a way to jump ahead. There is the ability to download the dataset and to re-rank offline. It is difficult to track changes since only the current four years of world rankings are displayed. QS is also a consultancy. QS 2025 rankings have been released with the same methodology as 2024. Changes will be covered in the next RR update.
ShanghaiRanking ACADEMIC RANKING OF WORLD UNIVERSITIES (ARWU) (2003-2023)
This was the first world rankings. ARWU uses the year in which the rankings are released, unlike THE and QS, which use the following year. The 2003 edition has 500 research universities. In 2009 China created a new company to manage the rankings, Shanghai Ranking Consultancy. It is not affiliated with any Chinese university or governmental organization. All universities having faculty, staff, or articles in the categories below are considered for inclusion. The methodology has remained consistent over the past 20 years. In 2023, 2500 universities were ranked and 1000 are in the World Ranking. Universities not in the World Rankings are included in the subject rankings. Harvard remains number one on all but the per capita performance metric, the only metric that is not size-dependent. Chinese universities have moved up, based on the number of publications in Nature and Web of Science. Tokyo was the only Asian university in the top 25 in 2003 and Tsinghua is the only Asian university in the top 25 in 2023.
ARUW Metrics and Methodology
- Quality of education – Nobel Prizes (10%) and Field Medals (20%) won by Faculty and staff
- Quality of faculty- Highly cited researchers from Clarivate, formerly Thomson in 2003, ( 20% )
- Research Output – Number of articles in Nature and Science ( not included for social science universities) (20% ), and Number of articles in Science Citation Index Expanded and Social Sciences Citation Index from the Web of Science TM (20%)
- Per capita performance – based on the number of full-time equivalent academic staff (where available) or country average (10%)
All metrics are from third parties and the websites are listed. There is easy access to definitions, full explanations, and the web link to their sources (https://www.shanghairanking.com/methodology/arwu/2023). See RR 5 for the introduction to ARWU.
Comments: ARWU’s target audience is research institutions. Rankings are available from 2003 for easy comparisons. The metrics are size-dependent. The prize metrics grow slowly. In 2023, about 300 universities out of 1000 have a score for Alumni (Nobel prizes) and even fewer for Awards. This limits the mobility of smaller universities to move up. Chinese universities make up for not having a history of awards and prizes with the number of publications they publish. They do not do as well in the highly cited category. A positive feature is a list of all universities in ARWU (https://www.shanghairanking.com/institution). Subject ranks are included with each university entry.
RANKINGS BY SUBJECT:
Each ranking has a separate module for subject rankings. THE includes only those universities that are in their world rankings. It uses the same metrics but with different weightings. In 2017 THE added Computer Science, a sub-category under Engineering in QS and ARWU. In 2018 it added Education, Law, and Psychology as stand-alone subjects for a total of 17. QS and ARWU’s GRAS (Global Rankings of Academic Subjects) include universities that are best in their categories, even if they are not included in the world rankings. They modify their metrics by subject. QS’ strength is Arts & Humanities. An interesting example is the Performing Arts. Hong Kong University of the Performing Arts is tops in Asia. These rankings are based only on reputation. ShanghaiRankings’ first subject rankings were in 2009. In 2017, it updated its methodology with GRAS. The methodology is explained in detail. Its strength is engineering. Table 3 lists the fields with their number of specific subjects.
Backfiles but not current fields for THE, QS Universityrankings.ch Shanghai before GRAS: https://www.universityrankings.ch/fields
THE Subjects: https://www.timeshighereducation.com/world-university-rankings/by-subject
QS Subjects https://www.topuniversities.com/subject-rankings
GRAS : https://www.shanghairanking.com/rankings/gras/2023
TOP COUNTRIES
Table 4 lists the top-performing countries. It includes the countries from the initial 200 THE-QS rankings and the top 200 from ARWU’s 2003 ranking, which included 500 universities. There were no Chinese universities in ARWU’s initial top 200. Tsinghua was in the 201-250 range. In 2024, there were 28 mainland Chinese universities in the top 200 with Tsinghua coming in at 22. Hong Kong has four universities in the top 200. The number of countries included in these rankings has grown. THE has the most universities but it has fewer countries in 2024 top 200 than in 2004. The number of US and UK universities in the top 200 has dropped while the number of Chinese universities has grown from zero to 28.
NOTE: Selecting “China” as a country will include the mainland, Hong Kong, Macau, and Taiwan. The other three are identified individually as well.
TOP 200 UNIVERSITY RANKINGS
The Addendum focused on the three top world rankings and the initial top 200 universities from THE-QS. I have tracked these universities’ performance from 2003-2004 until May 2024. Table 5 includes the 200 universities listed in the THE-QS rankings and their rankings in 2004, 2014, and 2024. The rankings for those 200 universities in ARWU are also included in the table. Over 65 percent of the initial universities are still in the top 200 in all three rankings. Mapping university names across three rankings and datasets was a challenge. Issues included English or local language names, use of diacritical symbols, abbreviations or full names, and mergers. The most difficult has been France which reorganized its higher education system(Brooks). When in doubt I checked university websites. See this table for a list of the 200 universities. After spending weeks on this part of the project, I discovered that Webometrics uses the new open-access naming protocol from the Research Organizations Registry (ROR). I will examine Webometrics in the next article along with other rankings that were released during the past few months. It will be interesting to see how new open technologies, new metrics, and changes in support of higher education will affect the new rankings.
RESOURCES:
Brooks, J. (13 Sep 2023). Ranking agencies struggle to keep up with French mergers. Research Professional News. https://www.researchprofessionalnews.com/rr-news-europe-france-2023-9-ranking-agencies-struggle-to-keep-up-with-french-mergers/
Praphamontripong, P. and Levy, D. (2005). World University Rankings 2004. PROPHE, https://prophe.org/cache/0723021_WorldUniversityRanking2004_ModifiedFromTHES.pdf
RR 57 (Oct 2023). The’s rankings world and its regions. https://librarylearningspace.com/ruths-rankings-57-the-ranking-world-and-its-regions-highlighting-sub-saharan-africa/
US dominates worldwide league tables. (4 Nov 2004). Times Higher Education Supplement. https://www.timeshighereducation.com/news/us-dominates-worldwide-league-tables/192159.article
A list of Ruth’s Rankings and News Updates is here.
*Ruth A. Pagell is an emeritus faculty librarian at Emory University. After working at Emory, she was the founding librarian of the Li Ka Shing Library at Singapore Management University and then adjunct faculty [teaching] in the Library and Information Science Program at the University of Hawaii. She has written and spoken extensively on various aspects of librarianship, including contributing articles to ACCESS – https://orcid.org/0000-0003-3238-9674
