Feature: Allow custom metrics via a configuration file

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,24 @@
+---
+name: Bug Report
+about: Report a bug in the system
+title: "[Bug]: "
+labels: bug
+---
+
+### Description
+A clear and concise description of the bug.
+
+### Steps to Reproduce
+Give a detailed description of a minimal example to reproduce the bug.
+
+### Expected Behavior
+Explain what you expected to happen.
+
+### Screenshots
+Add screenshots if applicable.
+
+### Environment
+Please describe your environment, like for example the OS you're using, your python version and your package versions.
+
+### Additional Context
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,17 @@
+blank_issues_enabled: false
+issue_templates:
+  - name: "Bug Report"
+    description: "Report a bug in the system."
+    title: "[Bug]: "
+    labels: ["bug"]
+    body: "./ISSUE_TEMPLATE/bug_report.md"
+  - name: "Feature Request"
+    description: "Propose a new feature or improvement."
+    title: "[Feature]: "
+    labels: ["enhancement"]
+    body: "./ISSUE_TEMPLATE/feature_request.md"
+  - name: "Documentation"
+    description: "Suggest updates or additions to the documentation."
+    title: "[Docs]: "
+    labels: ["documentation"]
+    body: "./ISSUE_TEMPLATE/documentation.md"

.github/ISSUE_TEMPLATE/documentation.md

@@ -0,0 +1,18 @@
+---
+name: Documentation
+about: Suggest updates or additions to documentation
+title: "[Docs]: "
+labels: documentation
+---
+
+### Documentation Update
+What part of the documentation needs to be updated or added?
+
+### Why Is This Needed?
+Explain the importance of this update.
+
+### Suggested Changes
+Provide a detailed description of the changes.
+
+### Additional Context
+Include any related resources.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,18 @@
+---
+name: Feature Request
+about: Suggest a new feature or improvement
+title: "[Feature]: "
+labels: enhancement
+---
+
+### Feature Description
+What feature would you like to see?
+
+### Why Is This Needed?
+Explain the problem or need for this feature.
+
+### Suggested Solutions
+Describe how this feature could be implemented.
+
+### Additional Context
+Add any relevant screenshots, links, or resources.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,19 @@
+* **Please check if the PR fulfills these requirements**
+- [ ] New and existing unit tests pass locally with my changes
+- [ ] Tests for the changes have been added (for bug fixes/features)
+- [ ] Docs have been added / updated (for bug fixes / features)
+
+
+* **What kind of change does this PR introduce?** (Bug fix, feature, docs update, ...)
+
+
+* **What is the current behavior?** (You can also link to an open issue here)
+
+
+* **What is the new behavior (if this is a feature change)?**
+
+
+* **Does this PR introduce a breaking change?** (What changes might users need to make in their application due to this PR?)
+
+
+* **Other information**:

conf/metrics/example.conf

@@ -0,0 +1,17 @@
+train_metrics:
+  - name: AUROC
+    params:
+      thresholds: 10
+      average: macro
+
+valid_metrics:
+  - name: AUROC
+    params:
+      thresholds: 10
+      average: macro
+
+test_metrics:
+  - name: AUROC
+    params:
+      thresholds: 10
+      average: macro

docs/hyperparameter-optimization.md

@@ -6,9 +6,9 @@ We aim to keep the document brief and point to code for deeper understanding.
 
 Relevant sources:
 - Optimizer logic: [`dpdl/hyperparameteroptimizer.py`](../dpdl/hyperparameteroptimizer.py)
-- Search space example: [`conf/optuna_hypers.conf`](../conf/optuna_hypers.conf)
-- Ordered search space example: [`conf/optuna_hypers_ordered.conf`](../conf/optuna_hypers_ordered.conf)
-- Manual trials example: [`conf/optuna_trials.conf`](../conf/optuna_trials.conf)
+- Search space example: [`conf/optuna_hypers.conf`](../conf/optuna/optuna_hypers.conf)
+- Ordered search space example: [`conf/optuna_hypers_ordered.conf`](../conf/optuna/optuna_hypers_ordered.conf)
+- Manual trials example: [`conf/optuna_trials.conf`](../conf/optuna/optuna_trials.conf)
 
 ## Overview
 
@@ -31,14 +31,14 @@ Supported types:
 
 In the config, we can use `-1` to indicate the [maximum batch size](../dpdl/hyperparameteroptimizer.py#L146) (i.e. full dataset).
 
-See [`conf/optuna_hypers.conf`](../conf/optuna_hypers.conf) for an example.
+See [`conf/optuna_hypers.conf`](../conf/optuna/optuna_hypers.conf) for an example.
 
 ## Manual trials (`conf/optuna_trials.conf`)
 
 It is also possible to configure manual trials.
 These define hypers that Optuna will try before starting its own algorithm for suggesting the next hypers.
 
-In [`conf/optuna_trials.conf`](../conf/optuna_trials.conf), each entry is a dictionary of hyperparameters and the respective values to try.
+In [`conf/optuna_trials.conf`](../conf/optuna/optuna_trials.conf), each entry is a dictionary of hyperparameters and the respective values to try.
 
 ## Ordered discrete type (`ordered`)
 
@@ -48,7 +48,7 @@ This enables, for example, using discrete learning rate values while still being
 
 Gotchas:
 - For convenience, for `batch_size` with `ordered` type, the values are auto‑generated as powers of two from `min` up to the dataset size, plus `-1` for full batch.
-- For other ordered parameters (e.g., `learning_rate`), you must provide `options` explicitly (see [`conf/optuna_hypers_ordered.conf`](../conf/optuna_hypers_ordered.conf) for an example).
+- For other ordered parameters (e.g., `learning_rate`), you must provide `options` explicitly (see [`conf/optuna_hypers_ordered.conf`](../conf/optuna/optuna_hypers_ordered.conf) for an example).
 
 ## CLI flags and behavior
 

docs/trainer.md

@@ -24,6 +24,32 @@ We support [torchmetrics](https://github.com/Lightning-AI/torchmetrics) through
 
 Metrics live in the [model](../dpdl/models/model_base.py) (`train_metrics`, `valid_metrics`, `test_metrics`) and the Trainer(s) update them during different training phases.
 
+### Built-in metrics
+
+`MetricsFactory` always initialises a task-appropriate baseline set:
+
+- **Classification** (`ImageClassification`, `SequenceClassification`): macro and micro `MulticlassAccuracy` per class, and a `ConfusionMatrix` on the test split.
+- **Language modelling** (`CausalLM`, `InstructLM`): micro `MulticlassAccuracy` and `Perplexity`.
+
+### Custom metrics
+
+Additional metrics can be declared in a YAML config file and passed via `--metric-config`. The file has three optional sections (`train_metrics`, `valid_metrics`, `test_metrics`), each a list of entries. An entry is either a plain metric class name or a mapping:
+
+```yaml
+train_metrics:
+  - Accuracy                   # shorthand: just the class name
+
+valid_metrics:
+  - name: AUROC                # torchmetrics class name
+    params:
+      thresholds: 10
+      average: macro
+```
+
+See [`conf/metrics/example.conf`](../conf/metrics/example.conf) for a working example.
+
+`CustomMetricsFactory` resolves each name against the torchmetrics submodule hierarchy (`torchmetrics`, `torchmetrics.classification`, `torchmetrics.text`, `torchmetrics.regression`, `torchmetrics.image`, `torchmetrics.audio`). Task-level defaults such as `num_classes` and `task` are injected automatically where the metric accepts them; `sync_on_compute` is also set to match the train/eval setting. Params declared in the config always take precedence over these defaults.
+
 ## Task adapters
 
 The trainers use **task adapters** to keep task-specific logic out of the training loop.

dpdl/cli.py

@@ -564,6 +564,13 @@ def cli(
                 rich_help_panel='Bayesian optimization (Optuna) options',
             )
         ] = 'conf/optuna_hypers.conf',
+        metric_config: Annotated[
+            Optional[str],
+            typer.Option(
+                help='YAML file path containing custom metrics for train/validation/test',
+                rich_help_panel='Logging options',
+            )
+        ] = None,
         optuna_manual_trials: Annotated[
             Optional[str],
             typer.Option(

dpdl/configurationmanager.py

@@ -103,6 +103,7 @@ class Configuration(BaseModel):
     optuna_target_metric: str = 'loss'
     optuna_direction: Literal['minimize', 'maximize'] = 'minimize'
     optuna_config: str = 'conf/optuna_hypers.conf'
+    metric_config: Optional[str] = None
     optuna_manual_trials: Optional[str] = None
     optuna_journal: str = 'optuna.journal'
     optuna_resume: bool = False
@@ -254,6 +255,7 @@ def __str__(self):
             ('Enable callback debug logging', self.verbose_callback),
             ('Fairness-style subsampling class', self.fairness_imbalance_class),
             ('Random seed for creating dataset subsets', self.split_seed),
+            ('Metric configuration', self.metric_config),
             ('LLM use', self.llm),
             ('Task', self.task),
         ]