We’re number one!

Each year the University is appropriately restrained when it graciously acknowledges the results, pointing out that Princeton is happy to be highly ranked, but that there are plenty of good schools and prospective students should not base their choices solely on rankings. Of course everyone is quietly pleased to have Princeton’s greatness recognized.

There are many such rankings, from a veritable smokestack industry of suppliers. One of these is the National Research Council, a highly respected nonprofit institution that has provided “science, technology and health policy advice” to the federal government ever since the first of its constituents, the National Academy of Science, was established by Abraham Lincoln in 1863. Along with advising, from time to time the NRC ranks graduate programs at U.S. universities. The last study was published in 1995, so it was more than a bit dusty when the NRC undertook an update some years ago. The new results were due to appear in 2005, but for whatever reason were delayed until the end of September, just a month ago.

Princeton did very well in this exercise, with many departments in the top handful in their respective cohorts. Computer science, the department that I know best, went from sixth in 1993 to second. Wonderful! We’re No. 2!

Sober second thought, however, makes one a bit cautious. Princeton computer science is a fine department, but is it better than, say, Massachusetts Institute of Technology, which came in third? MIT was second in 1995, after Stanford and followed by University of California, Berkeley; Carnegie Mellon University; and Cornell — a quite reasonable result. Other computer science departments were jerked around wildly; for instance, computer science at the University of Washington went from ninth to 38th, a most unlikely outcome.

I asked friends in other departments about the rankings. One colleague here calls the results “completely nuts.” In his discipline, the top school has no faculty in several major areas of the field, while the three schools that are Princeton’s main competitors for graduate students are well down the list. Naturally those who move up are happy; another school congratulates one of its departments on rising from 81st to “in the top 12,” a dramatic leap that might be taken with a grain or two of salt.

How do such wild results happen? In the 1980s, statisticians at Bell Laboratories studied the data from the 1985 “Places Rated Almanac,” which ranked 329 American cities on how desirable they were as places to live. (This book is still published every couple of years.) My colleagues at Bell Labs tried to assess the data objectively. To summarize a lot of first-rate statistical analysis and exposition in a few sentences, what they showed was that if one combines flaky data with arbitrary weights, it’s possible to come up with pretty much any order you like. They were able, by juggling the weights on the nine attributes of the original data, to move any one of 134 cities to first position, and (separately) to move any one of 150 cities to the bottom. Depending on the weights, 59 cities could rank either first or last!

To illustrate the problem in a local setting, suppose that US News rated universities only on alumni giving rate, which today is just one of their criteria. Princeton is miles ahead on this measure and would always rank first. If instead the single criterion were SAT score, we’d be down in the list, well behind MIT and California Institute of Technology. If the criterion were ZIP code, which is nonsensical but objective, Princeton would be near the bottom (though still above Harvard and Yale!), and no school outside the northwest could compete with the University of Washington’s 98195.

To its credit, the NRC makes its data and methodology freely available, so you can create your own lists with any criteria you like. I often ask students in COS 109: Computers in Our World to explore the malleability of rankings. With factors and weights loosely based on US News data that ranks Princeton first, their task is to adjust the weights to push Princeton down as far as possible, while simultaneously raising Harvard up as much as they can.

The exercise is meant to demonstrate convincingly that such rankings don’t mean much. Most things in life are far too complicated to rank in linear order — almost everything is a messy combination of properties, and those properties are differently important to different audiences. The next time you hear that Princeton is number one, be grateful for the compliment, but remain a little skeptical.

Brian Kernighan GS ’69 is a computer science professor and a Forbes faculty adviser. He can be reached at bwk@cs.princeton.edu.