Fix mislabeled multiclass metrics in image learner report - #176
Conversation
Ludwig 0.10.1's category 'accuracy' inherits torchmetrics MulticlassAccuracy's default average='macro', so it is macro-averaged per-class recall (balanced accuracy), not sample-level accuracy; 'accuracy_micro' is the true accuracy. Relabel both for multiclass reports (binary labels unchanged), label multiclass roc_auc as Macro ROC-AUC, and drop from the test summary: - avg_precision_weighted / avg_recall_weighted: computed with average='micro' upstream (Ludwig eval_utils.py bug), silently duplicating micro values - token_accuracy: exact duplicate of accuracy_micro Also update the metrics help modal accordingly.
| """ | ||
| if metric_key == "hits_at_k": | ||
| return f"Hits@{int(top_k)}" | ||
| if output_type == "category" and metric_key in MULTICLASS_METRIC_DISPLAY_NAME_OVERRIDES: |
There was a problem hiding this comment.
This new category-aware branch is bypassed by the experiment configuration table: format_config_table_html() still calls format_metric_display_name() for validation_metric without passing its already-available output_type (around lines 190–192). As a result, a multiclass run configured with Ludwig’s raw accuracy metric still reports Validation Metric: Accuracy, while the performance tables correctly call the same metric Balanced Accuracy (Macro Recall). I confirmed the mismatch directly on this PR head. Please pass output_type at that call site and add a multiclass configuration-table regression test.
There was a problem hiding this comment.
Good catch. Fixed it.
format_config_table_html() already received output_type but did not forward it to format_metric_display_name(), so a multiclass run validated on Ludwig's raw 'accuracy' reported 'Validation Metric: Accuracy' while the performance tables called the same metric 'Balanced Accuracy (Macro Recall)'. Pass output_type at that call site and add a multiclass configuration-table regression test covering the override, agreement with the performance-table label, accuracy_micro, binary/unset output types, and top_k resolution.
build_train_validation_plots() hardcoded 'Accuracy across epochs' and 'ROC-AUC across epochs', so multiclass reports charted Ludwig's macro-averaged accuracy and ROC-AUC under labels that contradicted the performance tables directly above them. Pass output_type from ludwig_backend and reuse the shared multiclass overrides for the curve titles, the y-axis labels and the overfitting-gap plot. Also plot accuracy_micro, which was collected in training statistics but never charted, so the true sample-level accuracy curve is now shown alongside balanced accuracy. Binary and regression titles are unchanged. Adds regression tests for both the category and binary/unset cases.
|
Also fixed: learning-curve plots hardcoded "Accuracy"/"ROC-AUC" titles, so multiclass reports charted macro values under labels that contradicted the tables above them. build_train_validation_plots() now takes output_type and reuses the same overrides. Additionally plots accuracy_micro, which was collected but never charted. |
2 participants
Fix mislabeled multiclass metrics in the Image Learner report
Problem
For multiclass models, the report's "Accuracy" and "Micro F1-score" disagreed (e.g. 0.8044 vs 0.8947 on HAM10000), which is impossible for single-label multiclass classification, where micro-F1 ≡ accuracy.
An audit traced this to Ludwig 0.10.1, whose stats the tool relabels verbatim:
accuracyis not accuracy. Ludwig'sCategoryAccuracysubclasses torchmetricsMulticlassAccuracywithout anaverageargument, inheriting the defaultaverage="macro". The reported value is therefore macro-averaged per-class recall (balanced accuracy). Confirmed empirically: "Accuracy" always equals the "Macro Recall" row to 4 decimals.accuracy_microis the true sample-level accuracy (average="micro"), identical to micro precision/recall/F1 andtoken_accuracy.avg_precision_weighted/avg_recall_weightedare miscomputed upstream. Ludwig'sConfusionMatrix.stats()(ludwig/utils/eval_utils.py) passesaverage="micro"for both, so these rows silently duplicated the micro values instead of being support-weighted.roc_auc(MulticlassAUROC) is macro-averaged but was labeled plain "ROC-AUC".Changes
constants.py: addMULTICLASS_METRIC_DISPLAY_NAME_OVERRIDES— for category outputs,accuracy→ "Balanced Accuracy (Macro Recall)",accuracy_micro→ "Accuracy",roc_auc→ "Macro ROC-AUC".html_structure.py:format_metric_display_name()acceptsoutput_typeand applies the overrides only for multiclass; threaded through all three summary tables. Help modal updated (Balanced Accuracy entry added, obsolete Token Accuracy entry removed).utils.py: for multiclass test stats, excludeavg_precision_weightedandavg_recall_weighted(wrong values, per the Ludwig bug above) andtoken_accuracy(exact duplicate of accuracy); exclusions now also apply to the flattenedoverall_stats.No metric values are recomputed — only labels and row selection change. Binary and regression reports are untouched.