What College Rankings Do and Do Not Measure

Ever hear of a Cargo Cult? It’s a term that describes the Pacific Islanders who faithfully awaited the never-to-return U.S. Navy of 1945 and their cargoes that made their lives easy. Families that await the U.S. News & World Report College Edition have a similar mindset. They fervently hope, in vain, that the complexities of college selection can be made easy by the arrival of a magazine’s rankings.

Students must roll up their sleeves and do the hard work necessary to identify the colleges that fit them best from among the more than 4,000 in the U.S. There are tools that they may rely on to make the task less burdensome, but there are no shortcuts that wouldn’t do harm to the student’s educational potential.

Since 1983, U.S. News & World Report (U. S. News) has published an annual College Edition that ranks American colleges in various categories. Although U. S. News is the most popular, they are not alone in this market niche. There are other publications that release annual college rankings, including Forbes, Princeton Review, Money, Barron’s, Kiplinger’s, College Atlas, the Economist, and the New York Times. They are reputable, unbiased sources that share the same public data (the Common Data Set) but generate different results based on their distinctive methodologies.

The Current Rankings Controversy

The U.S. News College Edition is published annually in September. Each year, this sets off the avalanche of criticism that The Atlantic’s Derek Thompson calls a “National Carpfest.”  The 2023 rankings have caused more carping than usual, mainly from law and medical schools, which aren’t our concern here. However, two undergraduate colleges, the Rhode Island School of Design and Colorado College, have announced that they will no longer cooperate with U.S. News because the rankings reinforce social inequities. There are other schools that have expressed dissatisfaction with U.S. News and college rankings in general.

This mini-movement against rankings has prompted speculation that colleges will abandon them altogether. This is unlikely. By and large, the rankings are useful to colleges. However, U.S. Education Department Secretary Miguel Cardona essentially called for abandonment at a recent conference. He said that colleges should:

“Stop worshiping at the false altar of U.S. News and World Report. Rankings disincentivize the wealthiest institutions from enrolling and graduating more underserved students. That’s because doing so harms their selectivity, a factor in the U.S. News formula. Colleges, not some for-profit magazine, should set the higher education agenda.”

To some, it’s incongruous that Secretary Cardona blames the magazines for the fact that college rankings are misused. His position underscores a common criticism not of the magazines but of college administrations. They are often accused of placing more importance on rankings than on any other consideration. It’s fair to say that many institutions adopt certain practices and policies based on the likelihood that the result will raise their position in college rankings.

The ability of the rankings to influence decisions is due to the fact that a college’s data can be gamed in a way that generates better metrics for ranking calculations. Many administrators do this routinely because a rise in rank helps them justify tuition and salary increases. It is not illegal, but it is unethical.  There are rarely consequences, but occasionally a college gets called out for it in the media, as have Baylor and Columbia. This is not a good look for a high prestige institution.

In their response to Secretary Cardona, U.S. News observed that colleges simply don’t like to be compared to each other by objective third-parties. They commented that he should require colleges to be more transparent with their data, stating:

“More openness from colleges would allow prospective students and their families to make meaningful comparisons between institutions based on factors such as financial information, admissions data, and outcome statistics.”

Criticisms of Rankings

Rankings are first-amendment protected free press and the publishers can print virtually anything they want to about colleges. However, there are legitimate concerns with the way that many families perceive college rankings. The blame for this falls on families who are competing with the “Jones’s” or are seeking an easy way out of the stress of college selection.

The soundest criticism of rankings is that they cannot take into account the qualitative factors that matter most to students as individuals. Each student has their own combination of financial resources, talents, preferences, experiences, goals, and personal characteristics. Finding a student’s best-fit colleges should be the result of a subjective analysis that accommodates these factors. It can’t be done properly by a formula-driven calculation.

When purchasing a product like a car or TV, a publication that assists decision-making like Consumer Reports can come in handy. They conduct research, analyze, and test products. Then they rank the products from the best-buy on down.

That choosing a college is not like buying a TV is an obvious understatement. The best school for Student X is seldom the best school for Student Y. Yet the magazines rank colleges as if they were three-dimensional products. Key non-quantifiable features that are important to students can’t be compared in this manner. There are other valid criticisms of college rankings, but this one is paramount.

The Usefulness of Rankings

There is a positive side to college rankings. They are handy reference sources. U.S. News  devotes substantial effort every year to compiling and analyzing information relating to more than 3,000 colleges — about 75% of the U.S. total. All of the rankings publications generate comparisons of institutions of similar types on a level playing field! These comparisons are based on formulas that incorporate the factors that the rankings publications have determined contribute most to the quality and value of a college education.

Another positive aspect of rankings is that they adapt to changing circumstances. For example, U.S. News made a change to adjust for test-optional admissions. They were concerned that some colleges were reporting large pools of SAT/ACT scores in proportion to the size of their freshman class while other were reporting small pools. Starting next year, U.S. News will use test scores as a factor only for colleges that provide them for at least half of their freshmen class.

Understanding Methodologies

Families should decide how much credence to place on college rankings. To help make this decision, they should review a publication’s methodology. For example, a summary of the indicators used in the U.S. News formula for undergraduate colleges and the weights assigned to them is provided below in Table A.

Table A

U.S. News Ranking Indicators & Weights

RANKING INDICATOR	INDICATOR WEIGHT %
GRADUATION AND RETENTION RATES	22.0
AVERAGE SIX-YEAR GRADUATION RATE	17.6
AVERAGE FIRST-YEAR STUDENT RETENTION RATE	4.4
SOCIAL MOBILITY	5.0
PELL GRANT GRADUATION RATES	2.5
PELL GRANT GRADUATION RATE COMPARED WITH ALL OTHERS	2.5
GRADUATION RATE PERFORMANCE	8.0
UNDERGRADUATE ACADEMIC REPUTATION	20.0
PEER ASSESSMENT SURVEY	20.0
FACULTY RESOURCES FOR 2021-22 ACADEMIC YEAR	20.0
CLASS SIZE INDEX	8.0
FACULTY COMPENSATION	7.0
PERCENT FACULTY WITH TERMINAL DEGREE IN THEIR FIELD	3.0
PERCENT FACULTY THAT IS FULL TIME	1.0
STUDENT-FACULTY RATIO	1.0
STUDENT SELECTIVITY FOR THE FALL 2022 ENTERING CLASS	7.0
MATH AND READING/WRITING PORTIONS OF SAT/ACT SCORES	5.0
HIGH SCHOOL CLASS STANDING IN TOP 10%	2.2
FINANCIAL RESOURCES PER STUDENT	10.0
AVERAGE ALUMNI GIVING RATE	3.0
GRADUATE INDEBTEDNESS	5.0
TOTAL	100.0
Source: U.S. News & World Report – College Edition 2023

It should be noted that not all of the indicators in Table A are objective. The third Indicator is Undergraduate Academic Reputation. It is result of the Peer Assessment Survey, in which the President, Provost and Dean of Admissions at colleges are asked to provide a subjective rating of the quality of the academic programs of the colleges in their ranking category, including their own.