Kim XNN TODOs

[jphall663]

Features

• Input feature list (hopefully informed by @navdeep-G using GBM Shapley)

For simulated data

• Unconstrained feedforward ANN trained w/ 5-fold CV with training/CV and test AUC, Accuracy, RMSE, logloss
• XNN trained w/ 5-fold CV with training/CV and test AUC, Accuracy, RMSE, logloss
• Mean local feature importance values across quintiles of predictions (by Shapley or gradient-based) for XNN for top 5 features
• Ridge function curves for XNN for top 5 features
• ICE curves at quintiles of predictions for XNN for top 5 features

For mortgage data

• Unconstrained feedforward ANN trained w/ 5-fold CV with training/CV and test AUC, Accuracy, RMSE
• XNN trained w/ 5-fold CV with training/CV and test AUC, Accuracy, RMSE
• Mean local feature importance values across quintiles of predictions (by Shapley or gradient-based) for XNN for top 5 features
• Ridge function curves for XNN for top 5 features
• ICE curves at quintiles of predictions for XNN for top 5 features

Fairness

• Pandas frame of predictions and row IDs for the test data for @nickpschmidt to conduct discrimination testing for XNN

[jphall663]

from 12/6 meeting: (supercedes above)

• test accuracy, auc, logloss, RMSE
• try for shapley values in the margin\logodds space: model_output=margin?
• provide csv of shapley values, preferably in both spaces, margin and prob (patrick will make global shap importance and local shap plot)
• provide CSV of ridge functions for top 3 features (patrick will make histograms)
• provide frame of probs for nick