The role of statistical competence in peer review

When Paige and I first opened the target article in the Journal of Controversial Ideas (Original article HERE, our response HERE), we couldn’t believe what we saw. Yes, it was openly racist, ridiculously essentialist about IQ, and filled with clumsy political argumentation. But the statistics!

Right at the outset, after obtaining measures of brain size and skin color by having a “student” look at a roughly sketched map published 50 years ago, they proudly announced they had conducted a “factor analysis” of the two variables. For the uninitiated: it is not possible to conduct factor analysis of two variables. This is not an esoteric statistical point. Everyone knows it: I would immediately return a first-year graduate student paper if it included such a ridiculous error.

Then, they proceed to enter every variable they can dig up into a chaotic box and arrow path analysis. Once again, the execution is well below grad student level, and the first version contained no standard errors whatsoever. (After we had completed the first draft of our comment, a new version of the paper mysteriously appeared, with at least some standard errors sprinkled in.) Anyway, it was not possible to figure out what they had done from the results provided, so we asked to see the Mplus output that generated the results. They kindly provided it, and as we mentioned in our reply, the output made it clear why they weren’t reporting standard errors— the model wasn’t correctly specified and Mplus couldn’t estimate them accurately. In fact, we had to shorten the error report in the interest of brevity. Here is the whole thing:

WARNING: THE MODEL ESTIMATION HAS REACHED A SADDLE POINT OR A POINT WHERE THE OBSERVED AND THE EXPECTED INFORMATION MATRICES DO NOT MATCH. AN ADJUSTMENT TO THE ESTIMATION OF THE INFORMATION MATRIX HAS BEEN MADE. THE CONDITION NUMBER IS -0.113D-03. THE PROBLEM MAY ALSO BE RESOLVED BY DECREASING THE VALUE OF THE MCONVERGENCE OR LOGCRITERION OPTIONS OR BY CHANGING THE STARTING VALUES OR BY USING THE MLF ESTIMATOR.

THE STANDARD ERRORS OF THE MODEL PARAMETER ESTIMATES MAY NOT BE TRUSTWORTHY FOR SOME PARAMETERS DUE TO A NON-POSITIVE DEFINITE FIRST-ORDER DERIVATIVE PRODUCT MATRIX. THIS MAY BE DUE TO THE STARTING VALUES BUT MAY ALSO BE AN INDICATION OF MODEL NONIDENTIFICATION. THE

CONDITION NUMBER IS -0.156D-16. PROBLEM INVOLVING THE FOLLOWING PARAMETER:

Parameter 16, %BETWEEN%: IQALL19K

THE NONIDENTIFICATION IS MOST LIKELY DUE TO HAVING MORE PARAMETERS THAN THE NUMBER OF CLUSTERS. REDUCE THE NUMBER OF PARAMETERS.

Stop and think for a second. Can you imagine, having been asked by an editor for output supporting your results, sending off something with that error report, with no explanation or excuse? They obviously just don’t care, don’t have any commitment to doing things correctly. In fact, throughout the “race science” literature, there is a certain Trumpy contempt for the very idea of doing things correctly. Worrying about the details of your statistics is for the “elites.” We’ll just have our press conference at Four Seasons Total Landscaping.

I will mostly save my concerns about what this means about the review process for a future post. Suffice it to say that in the course of submitting our reply, an editor, a very prominent scholar, assured us that they, “sent it to someone competent in statistics.  He suggested minor revisions.” Really? I know it is always tricky to evaluate peer review post hoc. Mistakes leak through. But someone “competent in statistics” thought a factor analysis of two observed variables was OK? On the off chance that anonymous statistician reads this, would you consider defending your opinion? I just can’t imagine.

Looked at another way, you could say none of this matters. It isn’t as though, had they conducted competent analyses, they would have reached different conclusions. Yes, desperate immigrants get lower scores on tests requiring them to complete abstract matrices in their head. Go figure. You don’t need a well-done factor or path analysis to tell you that. But in normal peer review, competence in statistics plays another role, that of certifying some minimal level of expertise. If I am peer reviewing a paper that makes a blatant statistical error on the second page, it’s game over. I don’t think, oh would they maybe have gotten similar results if they had done this correctly? I think, get back to me when you know what the f*** you are doing. I’m a busy person.

The authors of this paper aren’t using statistics to reach principled answers to questions. They use statistics to dress up their anti-immigrant racism in something that has the superficial appearance of science. That’s a definition of pseudoscience.