Do I need to use AUPRC for reporting classification results on an imbalanced dataset when the model was trained using upsampling and CV

Question

I am working on a binary classification problem which dataset has about 5% of positive class samples. I split the dataset, 70% for training and 30% for testing. I used the test data only once for reporting the performance of the model.

Due to this imbalance, I used SMOTE to upsampling the minority class in the training dataset. In addition I used CV and grid search to optimize model performance following the suggestions from how to upsample, CV, and gridsearch to avoid data leakage.

Assuming that I am handling the training procedure correctly, I wonder how to report the classification results in the test data. My understanding is that for imbalanced datasets you should used AUPRC (see nice explanation here). So, if I address the imbalanced problem in training, do I need to report results using AUPRC or it is ok to used traditional ROC-AUC?

Thanks for your help in advance.

Answer 1

Your training and test data are supposed to come from the same distribution, therefore if the training dataset is unbalanced so the test one (probably) will.

The fact you oversampled the minority class in the training dataset does not affect the unbalance in the test dataset. So yes, you should use specific metrics to account for the unbalancedness also when reporting results on the test set as traditional ones may not reflect a lot the differences in performances between various predictors.