Your Classifier Is Damaged, However It Is Nonetheless Helpful | by David Lindelöf | Jan, 2025
While you run a binary classifier over a inhabitants you get an estimate of the proportion of true positives in that inhabitants. This is named the prevalence.
However that estimate is biased, as a result of no classifier is ideal. For instance, in case your classifier tells you that you’ve got 20% of constructive instances, however its precision is thought to be solely 50%, you’ll anticipate the true prevalence to be 0.2 × 0.5 = 0.1, i.e. 10%. However that’s assuming good recall (all true positives are flagged by the classifier). If the recall is lower than 1, then you already know the classifier missed some true positives, so that you additionally have to normalize the prevalence estimate by the recall.
This results in the widespread system for getting the true prevalence Pr(y=1) from the constructive prediction fee Pr(ŷ=1):
However suppose that you just need to run the classifier greater than as soon as. For instance, you would possibly need to do that at common intervals to detect developments within the prevalence. You possibly can’t use this system anymore, as a result of precision will depend on the prevalence. To make use of the system above you would need to re-estimate the precision frequently (say, with human eval), however then you could just as well also re-estimate the prevalence itself.
How can we get out of this round reasoning? It seems that binary classifiers produce other efficiency metrics (moreover precision) that don’t depend upon the prevalence. These embrace not solely the recall R but additionally the specificity S, and these metrics can be utilized to regulate Pr(ŷ=1) to get an unbiased estimate of the true prevalence utilizing this system (generally referred to as prevalence adjustment):
the place:
- Pr(y=1) is the true prevalence
- S is the specificity
- R is the sensitivity or recall
- Pr(ŷ=1) is the proportion of positives
The proof is simple:
Fixing for Pr(y = 1) yields the system above.
Discover that this system breaks down when the denominator R — (1 — S) turns into 0, or when recall turns into equal to the false constructive fee 1-S. However keep in mind what a typical ROC curve appears to be like like:
An ROC curve like this one plots recall R (aka true constructive fee) towards the false constructive fee 1-S, so a classifier for which R = (1-S) is a classifier falling on the diagonal of the ROC diagram. This can be a classifier that’s, primarily, guessing randomly. True instances and false instances are equally more likely to be categorised positively by this classifier, so the classifier is totally non-informative, and you’ll’t study something from it—and positively not the true prevalence.
Sufficient concept, let’s see if this works in follow:
# randomly draw some covariate
x <- runif(10000, -1, 1)# take the logit and draw the end result
logit <- plogis(x)
y <- runif(10000) < logit
# match a logistic regression mannequin
m <- glm(y ~ x, household = binomial)
# make some predictions, utilizing an absurdly low threshold
y_hat <- predict(m, sort = "response") < 0.3
# get the recall (aka sensitivity) and specificity
c <- caret::confusionMatrix(issue(y_hat), issue(y), constructive = "TRUE")
recall <- unname(c$byClass['Sensitivity'])
specificity <- unname(c$byClass['Specificity'])
# get the adjusted prevalence
(imply(y_hat) - (1 - specificity)) / (recall - (1 - specificity))
# examine with precise prevalence
imply(y)
On this simulation I get recall = 0.049 and specificity = 0.875. The expected prevalence is a ridiculously biased 0.087, however the adjusted prevalence is basically equal to the true prevalence (0.498).
To sum up: this reveals how, utilizing a classifier’s recall and specificity, you’ll be able to adjusted the anticipated prevalence to trace it over time, assuming that recall and specificity are secure over time. You can not do that utilizing precision and recall as a result of precision will depend on the prevalence, whereas recall and specificity don’t.
