I don't know whether this information is useful to anyone but me, but on the off chance that it is -- and also to memorialize it -- I have grouped the properties of a ClassifierMeasurementsObject into those that are invariant to the value of the UtilityFunctions option and those that will vary depending on that value.

<|"not invariant" -> {"Accuracy", "AccuracyRejectionPlot", 
   "BatchEvaluationTime", "ClassifierFunction", "ClassRejectionRate", 
   "CohenKappa", "ConfusionDistribution", "ConfusionFunction", 
   "ConfusionMatrix", "ConfusionMatrixPlot", 
   "CorrectlyClassifiedExamples", "DecisionUtilities", "Error", 
   "EvaluationTime", "F1Score", "FalseDiscoveryRate", 
   "FalseNegativeExamples", "FalseNegativeNumber", 
   "FalseNegativeRate", "FalsePositiveExamples", 
   "FalsePositiveNumber", "FalsePositiveRate", 
   "IndeterminateExamples", "MatthewsCorrelationCoefficient", 
   "MeanDecisionUtility", "MisclassifiedExamples", 
   "NegativePredictiveValue", "Precision", "Recall", "RejectionRate", 
   "Report", "ScottPi", "Specificity", "TopConfusions", 
   "TrueNegativeExamples", "TrueNegativeNumber", 
   "TruePositiveExamples", "TruePositiveNumber"}, 
 "invariant" -> {"AccuracyBaseline", "AreaUnderROCCurve", 
   "BestClassifiedExamples", "ClassMeanCrossEntropy", "Examples", 
   "GeometricMeanProbability", "LeastCertainExamples", "Likelihood", 
   "LogLikelihood", "MeanCrossEntropy", "MostCertainExamples", 
   "Perplexity", "Probabilities", "ProbabilityHistogram", 
   "Properties", "ROCCurve", "WorstClassifiedExamples"}|>