Skip to content

Model Evaluation Workflow

Jump to bottom
Amit edited this page May 29, 2026 · 1 revision

Model Evaluation Workflow

Overview

The IBF-SLM evaluation pipeline benchmarks small language models (SLMs) against a curated corpus of impact-based forecasting (IBF) tasks. Each evaluation run produces five dimension scores per model, stores results in PostgreSQL, and surfaces comparisons on the dashboard.

The evaluation loop has four stages:

  1. Register — declare the model endpoint and metadata.
  2. Run — submit each corpus item to the model and collect raw responses.
  3. Score — apply the 5-score rubric to each response.
  4. Persist — write results to the evaluations table and update the dashboard.

Registering a Model

Before a model can be evaluated, it must be registered via the FastAPI backend.

Endpoint

POST /api/models

Request body

{
  "name": "phi-3-mini-4k",
  "provider": "local",
  "endpoint": "http://localhost:11434/v1/chat/completions",
  "context_window": 4096,
  "parameters_billion": 3.8,
  "notes": "4-bit GGUF quantisation, CPU inference"
}

Response

{
  "model_id": "mdl_7f2a91c4",
  "name": "phi-3-mini-4k",
  "registered_at": "2026-05-29T08:14:03Z"
}

The returned model_id is used in all subsequent API calls for this model. Registration is idempotent on (name, endpoint); re-registering the same pair returns the existing record.

Running an Evaluation

Endpoint

POST /api/evaluations

Request body

{
  "model_id": "mdl_7f2a91c4",
  "corpus_version": "ibf-v2.1",
  "batch_size": 8,
  "temperature": 0.0,
  "max_tokens": 512,
  "label": "phi-3-mini baseline run"
}
Field Required Description
model_id Yes Registered model to evaluate
corpus_version Yes Pinned corpus tag; determines which items are used
batch_size No Concurrent inference requests (default 8)
temperature No Inference temperature (default 0.0 for deterministic output)
max_tokens No Per-item generation budget (default 512)
label No Human-readable run tag shown in the dashboard

Response (202 Accepted)

{
  "run_id": "run_3b9e204a",
  "status": "queued",
  "total_items": 340,
  "estimated_seconds": 420
}

The run executes asynchronously. Poll status at GET /api/evaluations/{run_id} or watch the dashboard run monitor. Once status transitions to completed, scores are available.

What happens during a run

  1. The backend fetches all corpus items for corpus_version from the corpus_items table.
  2. Items are dispatched to the model endpoint in batches.
  3. Each raw response is stored in evaluation_responses alongside the item ID and run ID.
  4. The scoring engine processes responses and writes per-item dimension scores.
  5. Aggregate scores for the run are computed and committed to the evaluations table.

Scoring Methodology — the 5-Score System

Each model response is scored across five dimensions drawn from the IBF conceptual framework. Scores are integers on a 0–4 scale per dimension; the run-level score for each dimension is the mean across all corpus items.

# Dimension What it measures
1 Hazard Recognition Does the response correctly identify the meteorological or hydrological hazard type?
2 Impact Reasoning Does the response link the hazard to plausible societal, agricultural, or infrastructure impacts?
3 Uncertainty Handling Does the response acknowledge forecast uncertainty appropriately rather than overstating confidence?
4 Action Relevance Does the response suggest or reference anticipatory actions consistent with the hazard context?
5 Geographic Specificity Does the response correctly ground its reasoning to the stated region, country, or administrative level?

Per-item rubric (each dimension, 0–4)

Score Descriptor Criterion
4 Excellent Fully correct, specific, and contextually appropriate
3 Good Correct with minor omissions or imprecision
2 Partial Partially correct; key elements missing or confused
1 Poor Mostly incorrect but shows marginal relevant content
0 Absent No relevant content, refusal, or hallucinated response

Aggregate metrics

Two summary values are computed per run:

  • Mean Dimension Score (MDS) — arithmetic mean of all five dimension means; single-number headline quality indicator.
  • Threshold Pass Rate (TPR) — percentage of corpus items where all five dimension scores are ≥ 2 simultaneously; measures minimum-viable reasoning consistency.

Interpreting Results

Per-run view (GET /api/evaluations/{run_id})

{
  "run_id": "run_3b9e204a",
  "model_id": "mdl_7f2a91c4",
  "model_name": "phi-3-mini-4k",
  "label": "phi-3-mini baseline run",
  "corpus_version": "ibf-v2.1",
  "status": "completed",
  "completed_at": "2026-05-29T08:27:11Z",
  "total_items": 340,
  "scores": {
    "hazard_recognition": 3.12,
    "impact_reasoning": 2.47,
    "uncertainty_handling": 1.83,
    "action_relevance": 2.91,
    "geographic_specificity": 2.68
  },
  "mean_dimension_score": 2.60,
  "threshold_pass_rate": 0.54
}

Reading the numbers

  • MDS below 2.0 indicates the model is unsuitable for IBF-adjacent use; consider excluding from the dashboard comparison.
  • MDS 2.0–2.9 indicates partial capability; useful for narrow sub-tasks.
  • MDS 3.0+ indicates strong performance suitable for assisted forecasting workflows.
  • Low uncertainty_handling scores relative to other dimensions are common for SLMs and typically indicate overconfident generation; prompt engineering or system-prompt constraints can partially address this.
  • TPR below 0.4 means fewer than 40% of responses meet even partial correctness on every dimension simultaneously — treat such a model as unsuitable for end-to-end IBF tasks regardless of its MDS.

Storing and Comparing Runs

PostgreSQL schema (simplified)

-- One row per evaluation run
CREATE TABLE evaluations (
    run_id               TEXT PRIMARY KEY,
    model_id             TEXT NOT NULL REFERENCES models(model_id),
    corpus_version       TEXT NOT NULL,
    label                TEXT,
    status               TEXT NOT NULL DEFAULT 'queued',
    total_items          INTEGER,
    hazard_recognition   NUMERIC(4,2),
    impact_reasoning     NUMERIC(4,2),
    uncertainty_handling NUMERIC(4,2),
    action_relevance     NUMERIC(4,2),
    geographic_specificity NUMERIC(4,2),
    mean_dimension_score NUMERIC(4,2),
    threshold_pass_rate  NUMERIC(5,4),
    created_at           TIMESTAMPTZ NOT NULL DEFAULT now(),
    completed_at         TIMESTAMPTZ
);

-- Per-item raw responses and scores
CREATE TABLE evaluation_responses (
    response_id          BIGSERIAL PRIMARY KEY,
    run_id               TEXT NOT NULL REFERENCES evaluations(run_id),
    corpus_item_id       TEXT NOT NULL,
    raw_response         TEXT,
    hazard_recognition   SMALLINT,
    impact_reasoning     SMALLINT,
    uncertainty_handling SMALLINT,
    action_relevance     SMALLINT,
    geographic_specificity SMALLINT,
    scored_at            TIMESTAMPTZ
);

Comparing runs via API

GET /api/evaluations/compare?run_ids=run_3b9e204a,run_9c1d88f0,run_0a7c553e

Returns a side-by-side payload:

{
  "runs": [
    {
      "run_id": "run_3b9e204a",
      "model_name": "phi-3-mini-4k",
      "mean_dimension_score": 2.60,
      "threshold_pass_rate": 0.54,
      "scores": { "hazard_recognition": 3.12, "impact_reasoning": 2.47, "uncertainty_handling": 1.83, "action_relevance": 2.91, "geographic_specificity": 2.68 }
    },
    {
      "run_id": "run_9c1d88f0",
      "model_name": "mistral-7b-instruct-v0.3",
      "mean_dimension_score": 3.21,
      "threshold_pass_rate": 0.72,
      "scores": { "hazard_recognition": 3.54, "impact_reasoning": 3.10, "uncertainty_handling": 2.88, "action_relevance": 3.22, "geographic_specificity": 3.31 }
    },
    {
      "run_id": "run_0a7c553e",
      "model_name": "gemma-2b-it",
      "mean_dimension_score": 1.94,
      "threshold_pass_rate": 0.28,
      "scores": { "hazard_recognition": 2.31, "impact_reasoning": 1.77, "uncertainty_handling": 1.40, "action_relevance": 2.05, "geographic_specificity": 2.17 }
    }
  ]
}

The dashboard renders this as a radar chart (one polygon per model) and a sortable table. Runs can be filtered by corpus version to ensure comparisons are like-for-like.

Example Output

Terminal (CLI run trigger)

$ curl -s -X POST http://localhost:8000/api/evaluations \
    -H "Content-Type: application/json" \
    -d '{"model_id":"mdl_7f2a91c4","corpus_version":"ibf-v2.1","label":"phi-3-mini baseline run"}' \
  | jq .

{
  "run_id": "run_3b9e204a",
  "status": "queued",
  "total_items": 340,
  "estimated_seconds": 420
}

$ watch -n 5 'curl -s http://localhost:8000/api/evaluations/run_3b9e204a | jq .status'

# After ~7 minutes:
"completed"

Example corpus item and scored response

Corpus item (ibf-v2.1, item ci_0814)

Context: Flood alert issued for the Lower Zambezi basin, Mozambique.
Forecast lead time: 72 hours. Peak discharge expected at 8,200 m³/s.

Question: What anticipatory actions should humanitarian actors prioritise
in the next 48 hours, and what uncertainties should they communicate to
local authorities?

Model response (phi-3-mini-4k)

Humanitarian actors should prioritise pre-positioning relief supplies
in Tete and Sofala provinces and coordinate with local disaster management
authorities on evacuation route planning. Given 72-hour lead time,
cash-based pre-payments to at-risk households are feasible if systems
are in place. Key uncertainties include discharge forecast accuracy
at ±15% and the flood extent model's sensitivity to upstream dam releases,
which should be communicated clearly so local authorities can make
go/no-go decisions for early evacuation without waiting for confirmation.

Scores assigned

Dimension Score Rationale
Hazard Recognition 4 Correctly identifies riverine flood with discharge metric
Impact Reasoning 3 Names affected provinces and humanitarian impacts; misses crop/infrastructure specifics
Uncertainty Handling 3 Quantifies forecast uncertainty and identifies dam-release dependency
Action Relevance 4 Cash pre-payments, pre-positioning, and evacuation coordination are all standard EAP actions
Geographic Specificity 3 Names Mozambique and two provinces; does not reach district level

Item MDS: 3.40 — item passes TPR threshold (all dimensions ≥ 2)

Rerunning and Versioning

  • Each call to POST /api/evaluations always creates a new run_id; existing runs are never mutated.
  • To re-evaluate the same model against a different corpus version, submit a new request with the updated corpus_version field.
  • Runs tied to a deprecated corpus version are retained in the database and remain visible on the dashboard under a "legacy corpus" filter tag.
  • To delete a run (e.g., test noise), call DELETE /api/evaluations/{run_id}. This hard-deletes both the evaluations row and all associated evaluation_responses rows.

Clone this wiki locally