Message Boards Message Boards

Threshold and ROC points from NetMeasurements

If one runs NetMeasurements[net,data,"ROCCurve"] one often gets a 2 x 140 matrix as the output. The first part represents false positive rate values and the second part represents true positive rate values. Each part appears to have 140 values regardless of the length of the data. That is, the Wolfram ROCCurve property generates 140 values regardless of whether the data has 8 rows of 80,000.

For each point on the ROCCurve there must be a corresponding threshold value (or, I suppose, interval of threshold values). That is, what the ROC curve is -- for all possible thresholds between 0 and 1, the false positive rate and true positive rate if the classifier treats as positive only those instances in which it predicts a probability greater than the threshold. But there doesn't seem to be any obvious way of figuring out the mapping between any of the 140 values and the threshold that generated it. It doesn't seem, for example, that the threshold values are {0,1/139, 2/139. 3/139 ... 1}. And it doesn't seem as if the thresholds derive in any obvious way from the probability values generated by the network.

So, does anyone inside Wolfram or outside have any insight on this problem? I would like to generate some sort of interpolating function that swiftly mapped from threshold to corresponding point on the ROC Curve and have thus far failed.

Footnote: Depending on the structure of the neural net, however, the output from NetMeasurements[net,data,"ROCCurve"] can also be an Association in which the keys correspond to possible positive class designations and the values are the corresponding 2 x 140 matrices described above. But the problem of figuring out the relationship between threshold and ROC point is then the same.

POSTED BY: Seth Chandler
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract