Measuring A Classifier With Sensitivity and Specificity
Suppose you are training your own classifier, and you wish to measure its performance using the well-accepted metrics of sensitivity and specificity. It may be instructive to compare your classifier to a random classifier that flips a coin based on the prevalence of a disease. Suppose that the probability a person has the disease is and the probability that they do not is . Suppose then that we have a random classifier that guesses that you have the disease with that same probability and guesses you do not with the same probability .
The probability of a true positive is the probability that you have the disease and the random classifier guesses that you do, or . With similar reasoning, the probability of a false negative is . From the definitions above, the sensitivity of this classifier is . With more similar reasoning, we can calculate the specificity as .
So, while the measure itself is independent of disease prevalence, the performance of this random classifier depends on disease prevalence. Your classifier may have performance that is like this random classifier, but with a better-weighted coin (higher sensitivity and specificity). So, these measures may be influenced by disease prevalence. An alternative measure of performance is the Matthews correlation coefficient, for which any random classifier will get an average score of 0.
Read more about this topic: Binary Classification
Famous quotes containing the words measuring a, measuring and/or sensitivity:
“As an example of just how useless these philosophers are for any practice in life there is Socrates himself, the one and only wise man, according to the Delphic Oracle. Whenever he tried to do anything in public he had to break off amid general laughter. While he was philosophizing about clouds and ideas, measuring a flea’s foot and marveling at a midge’s humming, he learned nothing about the affairs of ordinary life.”
—Desiderius Erasmus (c. 1466–1536)
“By measuring individual human worth, the novelist reveals the full enormity of the State’s crime when it sets out to crush that individuality.”
—Ian McEwan (b. 1938)
“...some small boys came out of the city and jeered at him, saying, “Go away, baldhead! Go away, baldhead!””
—Bible: Hebrew, 2 Kings 2:23.
Elisha--proving that baldness has been a source of sensitivity for centuries, Elisha cursed them and they died.