I'm working with a Mathematica script that a colleague developed that assesses various machine-learning classifiers by calculating AUC for simulated test data sets. AUC is cacluated from ROCFunctions["AUROC"] in the code.

For a simulated null model where we specifically generate data with no association between predictor variables and the response, I am still getting AUC > 0.5, at least when feature selection using LASSO is applied to the training set. Because any association between prediction and outcome variables is stochastic, the expected AUC is of course 0.5.

Overfitting can generate AUC < 0.5. Some ROC programs, e.g. R's proc, will by default reverse the classifier's outcome categories when AUC < 0.5 due to overfitting and thus generate anomalous AUC > 0.5 for a null model. I am curious if AUROC in Mathematica does the same. If it does, that would explain my results. If it does not perform this reversal, then there is some other error in the code that must be found and addressed.