UtilityFunction of Classify

Questions
,
1

I understand that taking into account ClassifierMeassurements info on a test set, you may want to tweak how probabilities are translated to prediction, or maybe you want to prevent some misclassifications for practical reasons for example diseased being classified as healthy. However I do not understand how the utility matrix and probabilities are computed to calculate the prediction.
Please could somebody explain?

2

Hi Pablo,

There are several online resources that explain how probabilities are calculated. Take a look at the Google Machine Learning Crash Course. Probability computation in logistic regression is described here. More details in the classification section.

3

When you apply a ClassifierFunction to a piece of data, the “decision” is the one that maximizes the expected utility of the result. Here’s a Computational Essay showing how it all works. If the Community Managers could turn this into an embedded notebook I would be grateful; for some reason I can’t get it to work this morning.

The effect of changing utility functions.nb (88.2 KB)

4

Many Thanks Seth, I’m studying your notebook it looks very interesting, however when:
Create three RegionPlots with varying utility functions , does not execute and Mathematica reports:

Value of option UtilityFunction → <|A-><|A->a,B->0.,C->0.|>,B-><|A->0.,B->b,C->0.|>,C-><|A->0.,B->0.,C->1.|>|> is not valid.

Please could you check it?, this code uses several functions I have never used, and it is taking me a while to fully understand it.

5

Interesting. It works in the 12.2 beta. It does not work, as you note in 12.1.1. Let me see if I can find a backwards compatible way of implementing the idea.

6

Thank you, interesting resource.

7

Here’s an amended notebook that shows how you can get an (inferior) visualization in version 12.1

The effect of changing utility functions.nb (103 KB)

8

Now I understand, thank you very much for your excellent work and quick answer Seth

9

Just in case somebody else is interested, this is a function based on Seth’s answer, that computes utility values from probabilities and utilityFunction of Classify function, basically extracts values for the utility matrix and the probabilities vector, does dot product and then rebuilds the final association

utilityValues[probs_Association, uf_Association] := 
 Association[
  Thread[Keys[probs] -> (uf // Values // Values).(probs // Values)]]