Quantifying the Info Gain from a Classifier
The goal for this section is to quantify the information gain of this model and compare it on an apples-to-apples basis to a hypothetical competing model.
Model Characteristics
I previously established the following performance characteristics for this model:
Information gain measures are discussed here. The Venn diagram defining the types of information is linked below.
Mutual Information
Mutual information between default and the test is calculated as follows. This is the entropy of the original base rate minus the conditional entropy of default given the test classification.
Then,
Recall that the units for this value is mutual information, or “information gain,” in average bits per event.
Percent Information Gain (P.I.G.)
The Percent Information Gain (P.I.G.) is the ratio
Savings Per Bit
Between the savings-per-event value and the bits-per-event value, just calculated, it is possible to measure a savings-per-bit value. This concept is powerful, because it places a financial value on the information content of a model or data source.
The savings-per-event value of $337 was calculated previously.
The bits-per-event value, or mutual information, value of 0.0860 was calculated above.
Thus the savings-per-bit is given by
Alternative Model Characteristics
A hypothetical competing model has the following performance characteristics. It outperforms my model.
Mutual Information
Percent Information Gain (P.I.G.)
Savings Per Bit
The cost per event for this alternative schema is $838. The savings per event is calculated as
Points of Comparison
The following table establishes the important points of comparison between the competing models.
Parameters | My Model | Alternative Model |
---|---|---|
Mutual Information | 0.0860 | 0.1205 |
Percent Information Gain | 10.6% | 14.9% |
Cost per Event | $913 | $838 |
Savings per Bit | $3919 | $3419 |
A few important differential comparisons are possible.
The incremental information gain of the alternative model over my model is
If my model was available to an organization, the maximum price that the organization should be willing to pay for the alternative model is
At this maximum break-even price per score, the incremental value per bit from the alternative model is
AUC_Calculator-and-Review-of-AUC-Curve.xlsx
Data_Final-Project.xlsx
Information-Gain-Calculator.xlsx
Some other content is taken from my notes on other aspects of the aforementioned Coursera course.