The Cost of Bucking College Rankings

By Michele Tolela Myers

Published in The Washington Post, Sunday, March 11, 2007

Like most college presidents, I have seen many prospective students and their parents show up on campus in recent months, clutching their well-worn copies of U.S. News & World Report's rankings issue. U.S. News has smartly tapped into students' need to sort out colleges and universities in a rational way. Parents, who face increasing college costs, understandably want to know where best to make that expensive investment.

U.S. News benefits from our appetite for shortcuts, sound bites and top-10 lists. The magazine has parlayed the appearance of unbiased measurements into a profitable bottom line.

The problem is that the U.S. News college rankings are far from reliable.

Turns out that some of their numbers are made up. I know that firsthand. Two years ago, we at Sarah Lawrence College decided to stop using SAT scores in our admission process. We didn't make them optional, as some schools do. We simply told our prospective students not to bother sending them. We determined that the best predictors of success at Sarah Lawrence are high school grades in rigorous college-prep courses, teachers' recommendations and extensive writing samples. We are a writing-intensive school, and the information produced by SAT scores added little to our ability to predict how a student would do at our college; it did, however, do much to bias admission in favor of those who could afford expensive coaching sessions.

Since we dropped the SAT altogether, we no longer provide SAT information to U.S. News & World Report. Our two years' experience with this practice has been very good. Faculty members report that our students continue to be terrific. Their average high school grades, high school ranks and grades in Advanced Placement courses have not changed.

But this principled decision has put us in jeopardy. I was recently informed by the director of data research at U.S. News, the person at the magazine who has a lot to say about how the rankings are computed, that absent students' SAT scores, the magazine will calculate the college's ranking by assuming an arbitrary average SAT score of one standard deviation (roughly 200 points) below the average score of our peer group.

In other words, in the absence of real data, they will make up a number. He made clear to me that he believes that schools that do not use SAT scores in their admission process are admitting less capable students and therefore should lose points on their selectivity index. Our experience, of course, tells us otherwise.

But the story does not stop here. When I reported this conversation at Sarah Lawrence, several faculty members and deans suggested that perhaps it was time to stop playing ranking roulette and opt out of the survey. A few colleges explore this option each year, but most don't follow through (Reed College is one of the few exceptions I know of), because, like unilateral disarmament, unilateral withdrawal from the U.S. News ranking system is dangerous.

We discovered how dangerous it can be through a presentation U.S. News made at the 2006 meeting of the North East Association for Institutional Research. There, the magazine indicated that if a school stops sending data, the default assumption will be that it performs one standard deviation below the mean on numerous factors for which U.S. News can't find published data. Again, making up the numbers it can't get.

The message is clear. Unless we are willing to be badly misrepresented, we had better send the information the magazine wants. We haven't yet decided what we will do. But if we don't go along, we understand we will be harmed because many students will assume that Sarah Lawrence is much less selective than it actually is.

The reality is that the magazine's rankings issue has a large circulation and that parents and students rely on these rankings to make a college choice that has enormous educational and financial implications. This gives the magazine the power to keep colleges playing the game it sets and controls.

Why should we care if we lose our place in their rankings? Because ultimately, so many people take these rankings seriously. I would at least like to let them know how misleading the whole affair is.

See Also:

The op-ed and reader commentary in The Washington Post

An article in Inside Higher Education prompted by the op-ed, along with more reader discussion

An editorial in the San Francisco Chronicle

A follow-up article on CBSNews.com from The Christian Science Monitor