"As presently constituted, it's quite possible that the GRE physics subject test does more harm than good, and we should either fix it, or seriously consider getting rid of it altogether,"
A quote from Jennifer Siders in this article at aps.org, that really we ought to take seriously. I doubt we will, though, because the Physics GRE is well entrenched at most graduate programs across the country, and making changes like that is always tough. Indeed, the article I linked to (as a result of seeing it in Pamela Gay's Facebook status) was written 13 years ago, and yet the Physics GRE is still going strong.
I've been grouchy about standardized tests for some time. When it comes to things like the general GREs and the SATs, I believe that it does correlate with overall academic performance. Whether or not it's testing the right stuff, there seems to be some correlation between what it tests and what we'd really want to test. But, it's not perfect. That is, for (say-- I'm making this number up) 80% of students, the SAT and general GRE might a good indicator of how successful they'll be in college. As such, from a mercenary college admissions' point of view, it's worth keeping using them. Most of the time, they get the right students, and damn but it's really easy to cut down on the number of applications you actually have to put work into thinking about by sorting on a simple number. Of course, from an individual fairness and a humanity point of view, it's pretty sad to think that the other 20% (or whatever) who would have thrived at a certain college aren't even considered because of a bad test....
The Physics GRE, however, has bothered me since I started as an assistant professor. Now, mind you, this is not personal sour grapes. My Physics GRE score back in 1990 was 89th percentile. At the time, I felt a little bad about that; I was one of those geeks who always did well on standardized tests, and thought that I should get over 90% on anything math/science related. Much later, I realized that 89th percentile is damn good for the Physics GRE. I did not personally suffer as a result of the Physics GRE, so I'm not posting this out of bitterness.
But, there is evidence that the Physics GRE does not correlate very well with how you do in Physics grad school. It seems completely unsurprising. In grad school, you do well by doing well at research. Yeah, you have to pass your classes, but even there it's very different from what the Physics GRE tests. The Physics GRE tests your ability to think uberfast (which may be relevant in conference arguments, but is not terribly relevant for most research), your ability to recall things you've memorized, and your ability to quickly go through canned problems about basic physics. It's not completely irrelevant, but it's not testing what is most important about graduate school.
Of course, all the hand-wavy justifications for why it's the wrong test only mean so much. As I said, there is evidence that the Physics GRE does not correlate very well with how you do in Physics grad school. What's more, there's evidence that women who do just as well as men in grad school on average score lower on the Physics GRE. In other words, either because of societal conditioning or because of intrinsic differences, the Physics GRE is more unfair for women, on average, than it is for men. Given that we've got a recruiting and retention problem for women in Physics, we should take this very seriously.