Mo' Rankings, Mo' Problems

The Washington Post published a skeptical look at college rankings today. Colleges are of course eager to use the lists in their marketing materials when the results are flattering. But when one list claims that Johns Hopkins University is one of the 15 "least rigorous" in the country and that Georgetown is a hipster mecca, then (according to a Johns Hopkins University spokesman) “the reaction around here was a collective scratch of the head and a unanimous ‘Huh?’ ”

College Rankings (Part III): What Should We Do Instead?

As I explained in my last post, I don't like the effects of the U.S. News rankings on students and colleges, and I would just as soon see those rankings disappear.  But the reams of data about the inner workings of colleges that the rankings have strong-armed into the public eye are certainly a good thing for students who wish to compare schools.  Those 10,000,000 hits that the U.S. News website received when the 2011 rankings came out go to show that there is an insatiable hunger for the information.  Are there alternatives that might help students and parents sift through colleges without the onerous effects of the Best Colleges rankings?  There are three types, two of which exist now and a third that exists in prototype but not in definitive form.

1. Competing aggregate rankings.  U.S. News isn't the only rankings game in town.  Washington Monthly offers its own rankings with a stubbornly alternative vision.  They reward colleges that foster social mobility, public service, and groundbreaking research.  The result is that Berea College (KY) and Morehouse College (GA) beat out big-name schools such as Amherst and Swarthmore.  I wonder, though, whether the social agenda promoted by these rankings really is of interest to students who are struggling to get a sense of what their four years at these various schools will be like.

The website "What Will They Learn?", which a reader of this blog brought to my attention, gives schools a letter grade (A-F) based on the extent to which they require classes in seven core subjects that are central to a liberal arts education.  While I'm not surprised to see the St. John's Colleges at the top of the list, it is certainly eye-opening to see that Baylor, Kennesaw State, and the Coast Guard Academy have more rigorous core curricula than Harvard, Princeton, or Yale.  I looked in vain on this list, though, for any assurance of educational quality beyond the bare requirement of classes.  What good is a core course in math, for instance, taught by a grad student who can barely speak English (an experience I have had more than once)?

I do think that both the Washington Monthly and "What Will They Learn" lists are useful correctives to U.S. News.  My common concern with all three -- and really with any unilateral ranking system -- is that they are chiefly useful insofar as a student shares the viewpoint of the editor of each list.

2. Rankings in individual categories.  College Prowler offers rankings that make it easy for students to compare schools in individual categories, from academics to co-ed attractiveness.  The data come mainly from student surveys, which means that they should be taken with a grain of salt.  For those of an old-school bent, there's also Rugg's Recommendations, which ranks the various undergraduate programs at each college separately.  These methods cede control to the student, who gets to choose which criteria are most important to him or her.  The drawback is that it's difficult to get a sense of how the schools that excel in individual categories stack up overall.

What I would like to see is:

3.  Personalized aggregate ranking.  A ranking system that allows students to choose their own priorities.  I have in mind something akin to those balance-the-budget simulators that have been making the rounds lately.  A student would be able to browse among the various categories in which data are available -- hard data like the student/teacher ratio or graduation rate, but also student survey data -- and assign a level of importance to each category.  Additionally, the student would answer a series of simple questions about their preferences (Rural, urban, or college town?  Deep south or west coast?) that would help narrow the field of choices.  So if a student wants really small classes, doesn't care much about athletics, is looking for a good political science department, and craves gourmet food in the dining halls, she can design her own personalized set of rankings that takes those things into account.  Students unsure about what they want could experiment with the list to see which schools excel in various categories, and then narrow it down once they know more.

Googling turns up a few nascent options along these lines.  The website URanker lets students select the importance of various qualities along a sliding scale.  The options are somewhat limited, and it would be nice if the website displayed the data that go into the rankings more openly, but the clean design makes it extremely easy to use.  StudentsReview offers many more options -- "friendliness" side-by-side with "not being treated as a number" -- but the data seems to be based on a fairly limited number of student surveys, and is thus of questionable reliability.  These websites are excellent starting points, but there is still room for the definitive personalized college ranker.  In fact, given how much colleges like to dump on the Best Colleges list, I'd bet there would be a lot of educators willing to support such a service.  Internet, get on it!

Until this smorgasboard of college data becomes a reality, the closest approximation is to mine the various available ranking systems (this list at College Confidential has most of them) for general ideas, then head to a good college guide or a college counselor you trust.  Also, Washington Post columnist Jay Mathews's book from a few years ago, Harvard Schmarvard, contains a list of schools that are overlooked by the national rankings, but strongly recommended by college counselors.


Please feel free to let me know what you think about the U.S. News rankings (or about this post) in the comments!

(Edit: It would be remiss of me not to link to Malcolm Gladwell's article on the U.S. News rankings from earlier this year.  He takes a closer look at the statistics than I do.  You need a subscription to read the article on the New Yorker website... but you can also (shh!) read it at this link for free.

College Rankings (Part II): Bad (U.S.) News

In my first post on this subject, I wrote a little bit about how influential the U.S. News college rankings (which will be published online tomorrow) are.  I also did my best to come up with some nice things to say about them.  In this post, I take off the kid gloves and lay out a half-dozen reasons why I think the rankings -- at least in their present form -- ought to be canned.  I stop at six only because the post threatened to get out of hand.  If you want to see some of the reasons I left out, check here.  If you want to hear what the other side in this debate has to say, here is U.S. News's defense of their rankings.  And if all this negativity is getting you down, don't worry -- tomorrow I'll offer some armchair generalizing about alternatives (some of which already exist) to the current system!


Six Reasons Why the U.S. News College Rankings Stink


1. Colleges have a perverse incentive to juke the stats.  Even if they're not fabricating data outright, colleges are doing their damnedest to massage the numbers.  As College Confidential reports, "some schools have submitted data that excludes scores from 'special admissions' (e.g., athletes, students identified as learning disabled, etc.).  One school reportedly left out the verbal scores of international students but kept the math scores."  Admissions officers play the numbers game to the hilt because their jobs are on the line.  The Atlantic published an article a few years back that offered a disturbing glimpse of this:

"There's pressure for rankings," says Tom Green, the associate vice-president for enrollment services at Seton Hall. "There's no doubt about it. Presidents get pressure from board members, from alumni. They'll say, You're number eighty-seven. How are you going to get to be number eighty-five?" Eugene Trani, the president of Virginia Commonwealth University, a Tier III school in the U.S. News rankings, carries a laminated card in his pocket to remind him of the school's strategic goal of making it to the next tier. For every year the school stays in the higher tier he will receive a $25,000 bonus—a fact first reported by AGB Priorities, an industry newsletter. A vice-president at Hobart and William Smith was fired when she failed to report current data to U.S. News and the school's ranking dropped from Tier II to Tier III.

Me College Monster!  Me gobble up more students
for better rankings!  OM NOM NOM NOM

2. Students are punished.  Colleges try to boost their average SAT/ACT scores and push down their acceptance rates in order to look good in the rankings.  They encourage students to apply who have almost no chance of getting in, since every mailed rejection letter makes the school look a tiny bit better in the Best Colleges methodology.  As a result, students are getting turned away in droves from schools that, just a decade ago, were safety schools.  The winners are U.S. News (Best Colleges is their cash cow) and the colleges that go up in the rankings; the losers are the students who don't sufficiently boost the stats at their first-choice colleges.

3. Poor students are disproportionately punished.  The Atlantic article from above also reports that "low-income students often suffer in this process. They drag on both revenue and academic profile, and it's hard for outsiders to tell when their numbers are reduced."  As an example of this, observe the importance of SAT and ACT scores in college rankings.  Those tests count for a larger percentage (7.5%) in the Best Colleges metric than class rank (6%), despite their inferiority to high school GPA at predicting academic performance in college.  SAT and ACT scores correlate much, much better with family income than with academic performance for many reasons.  When colleges try to boost their SAT and ACT scores for the sake of a boost in the rankings, poorer students are therefore the first to suffer, and the benefit to the learning environment is questionable at best.


4. The rankings foster cutthroat competition.  The win-at-all costs mentality of Wall Street has spread to high schoolers and their parents.  They spend thousands of dollars on SAT and ACT test prep, some of which goes to yours truly.  A few (I'm sure you've met them) obsess over the rankings and the prestige of colleges, rather than figuring out which schools might actually be a good fit.  This is understandable, given the extreme competitiveness of admissions at any of the top tier schools on the U.S. News list.  I think, however, that we ought to push back against the notion that college admissions are a (deadly serious) game to be won.  For starters, a study tracking the long-term success of students who were accepted at and rejected from "elite" colleges found no difference in their earnings 20 years on.  The only exceptions to this rule were poor and minority students -- another way in which amped-up college competitiveness hurts the disadvantaged disproportionately.  Jay Mathews of the Washington Post wrote a whole book about the silliness of the reverence we have for brand-name schools, arguing that we make an already stressful process even more so.  And Mark Speyer, a college counselor at a prep school in Manhattan, laments the effect of high-stakes admissions on his students' very souls:

I believe that the new obsession with numbers is counter-educational: that it makes for a less educated and less educable kind of student; a less thoughtful, more cynical, more boring, and more exhausted kind of student, not at all the kind of student that colleges on a clear day know they want and need. Pumped-up SATs do not mean better students.

More rejection letters = $$$$$$$

5. The mania for college rankings has spread beyond students and parents.  And I'm not just talking about crotchety alumni.  Bond rating agencies, offered a treasure trove of data about college selectivity, have incorporated it into their risk analysis for institutional borrowing rates.  In other words, colleges that reject a higher percentage of their applicants get to borrow money more cheaply because they are considered a safer investment.  That means colleges would just love for you and all your friends to apply... so they can reject you and then dive, a la Scrooge McDuck, into a vault full of borrowed doubloons.  Do we really want the same guys who helped to cause the recession by green-lighting bad mortgage loan packages influencing how colleges choose whom to admit?  According to William Shain, formerly the Dean of Undergraduate Admissions at Vanderbilt University, "institutions fail or default on financial obligations very infrequently, and I find the reliance on admissions data for these purposes to be hard to justify beyond extreme situations."

6. The rankings are not actually objective.  Those who call the the U.S. News rankings "objective" -- including Mr. Morse -- are suckered by what I call the "Objective Yardstick Fallacy".  A single measure used to assess a complex situation only merits the term "objective" if people can generally agree on the value of the results.  For instance, a yardstick is good at measuring length because we agree on the definition of length and on the units we use to measure it.  When we're measuring weight or electrical current, we reach for a scale or an ammeter and set the yardstick aside without qualms.  If we need to measure a length, a weight, and a current all at the same time, it makes as little sense to combine those results into a single number as it does to indicate by a single number whether one college has more "academic quality" than another.  Not only do we not all agree on what constitutes "academic quality", but the aggregate ranking necessarily embodies the prejudices of the aggregator: it is the polar opposite of an "objective yardstick".  Most students probably don't agree with Best Colleges that alumni giving rate (5% of the total) is more important than student/teacher ratio (1%), for example.  (I plan to write a post at some point about how the Objective Yardstick Fallacy applies to the use of SAT scores in particular.)

What Do You Think?

That's what the U.S. News rankings look like from my perspective (which, in brief, is that of an SAT tutor who loathes the SAT).  I'd love to know what your experience of Best Colleges as a parent, a student, or a teacher has been.  Please let me know in the comments!

College Rankings (Part I): Rank, Indeed

10 million hits in one day.  No, I'm not talking about my blog (yet).  That's how many hits the U.S. News and World Report website received on the day their annual Best Colleges guide dropped last year.  The number is likely to be matched, if not roundly thrashed, next Tuesday, when the 2012 edition makes its debut.  Over the weekend the Washington Post ran an enlightening article on the mild-mannered man who determines the algorithm for ranking colleges.  The author aptly concluded that Bob Morse "is to colleges what Robert Parker is to wine": a man who compelled an entire industry to bow down before the altar of statistics.

In some ways that's a good thing.  Before rankings came along, colleges could handle their business just as they pleased, and no one was the wiser.  They were under no obligation to let prospective students know about student/faculty ratios, graduation rates, or average SAT scores.  The information that did trickle out was fitfully reported and not standardized, so that it was almost impossible to compare across institutions.  Then the rankings came along.  They were a hit, and they put the fear of God (or rather, Bob) into deans everywhere.  A college that failed to supply the requested information could see its ranking drop or get left out of the rankings altogether.  As a result, there is now more information about the internal workings of colleges available to students and parents than ever before.

And that's where my praise for the Best Colleges guide stops.  The rankings, despite the good intentions of Mr. Morse, have contributed substantially to the present overheated climate in college admissions.  Not only do colleges waste time and money each year in jockeying for position -- why wouldn't they, if a modest bump in the rankings can mean a substantial pay raise for the Dean? -- but students suffer as well.  In my next post, I'll outline my principal objections to the rankings.  In a third post, I will then offer my own suggestions for alternatives.

P.S.  Why am I writing about the U.S. News college rankings in a blog that's ostensibly about the SAT and the ACT?  Well, SAT/ACT scores count for 7.5% of a college's rank in the U.S. News methodology.  That's about 7.5% too much, in my opinion.  The use of the SAT and the ACT to rank colleges is just one of a panoply of misuses of those tests that I aim to examine in the coming months.