Skip to content

Dataset Preparation

Jump to bottom
Amit edited this page May 29, 2026 · 1 revision

Dataset Preparation

This page describes the full lifecycle of a dataset in the IBF-SLM Application — from raw source ingestion through validation, annotation, and final JSONL export for SLM fine-tuning and evaluation.

Pipeline Overview

Raw Source Files
      │
      ▼
  Ingestion       ←  POST /data/upload
      │
      ▼
  Validation      ←  schema check · deduplication · range checks
      │
      ▼
  Annotation      ←  human reviewer assigns 5-score classification
      │
      ▼
  JSONL Export    ←  data/annotated/<dataset_id>/corpus.jsonl
      │
      ▼
SLM Fine-tuning / Evaluation

Directory Structure

All data files live under the data/ directory at the project root. Subdirectories map to pipeline stages.

data/
├── raw/
│   └── <dataset_id>/
│       ├── metadata.json          # Ingestion metadata (source, upload timestamp, uploader)
│       └── events.csv             # One row per forecast event (primary input format)
│           OR events.json         # JSON array alternative
│
├── annotated/
│   └── <dataset_id>/
│       ├── corpus.jsonl           # Annotation-ready records, one JSON object per line
│       └── annotation_log.json    # Audit trail: who annotated what and when
│
└── evaluated/
    └── <run_id>/
        ├── predictions.jsonl      # Model outputs aligned to corpus records
        └── scores.json            # Aggregated evaluation metrics

<dataset_id> matches the UUID primary key in the datasets PostgreSQL table.

Data Sources and Accepted Formats

Supported Sources

Source Type Description Typical Provider
Numerical Weather Prediction (NWP) alerts Gridded probabilistic forecasts converted to event-level records National meteorological agencies
Impact assessment reports Post-event or forecast-triggered impact summaries Humanitarian response teams, Red Cross/Red Crescent branches
Historical event logs Catalogued past extreme weather events with known impacts DESINVENTAR, EM-DAT, national disaster databases
Trigger activation logs Records of IBF system threshold crossings IBF platform exports

File Formats

The ingestion endpoint accepts two formats. Both must conform to the schema described in the next section.

CSV (events.csv) — preferred for bulk uploads from tabular data pipelines:

event_id,source_system,country_code,admin1_pcode,admin2_pcode,hazard_type,forecast_date,lead_time_hours,trigger_threshold,trigger_probability,affected_population,affected_area_km2,economic_loss_usd,fatalities_estimate,displacement_estimate,infrastructure_damage_score,event_description
EVT-2024-001,IBF-NG,NGA,NG015,NG015003,FLOOD,2024-08-10,72,0.5,0.83,142000,380.5,2400000,12,4500,3,"Significant riverine flooding expected along Benue River corridor following persistent rainfall upstream."
EVT-2024-002,IBF-NG,NGA,NG011,NG011007,FLOOD,2024-08-10,72,0.5,0.61,58000,145.2,870000,4,1200,2,"Moderate flash flood risk in low-lying urban areas of Kogi State."

JSON array (events.json) — preferred when source systems emit structured objects:

[
  {
    "event_id": "EVT-2024-001",
    "source_system": "IBF-NG",
    "country_code": "NGA",
    "admin1_pcode": "NG015",
    "admin2_pcode": "NG015003",
    "hazard_type": "FLOOD",
    "forecast_date": "2024-08-10",
    "lead_time_hours": 72,
    "trigger_threshold": 0.5,
    "trigger_probability": 0.83,
    "affected_population": 142000,
    "affected_area_km2": 380.5,
    "economic_loss_usd": 2400000,
    "fatalities_estimate": 12,
    "displacement_estimate": 4500,
    "infrastructure_damage_score": 3,
    "event_description": "Significant riverine flooding expected along Benue River corridor following persistent rainfall upstream."
  }
]

Raw Input Schema

Every record — regardless of format — must contain the following fields after ingestion normalisation.

Required Fields

Field Type Constraints Description
event_id string Non-empty, unique within dataset Source-system identifier for the event
source_system string Non-empty Originating IBF platform or data feed identifier
country_code string ISO 3166-1 alpha-3 Country the event affects
admin1_pcode string Non-empty Admin level-1 p-code (province / state)
hazard_type string One of: FLOOD, DROUGHT, CYCLONE, HEATWAVE, COLDWAVE, LANDSLIDE Type of hazard being forecast
forecast_date string ISO 8601 date (YYYY-MM-DD) Date the forecast was issued
lead_time_hours integer > 0, <= 240 Hours between forecast issuance and event onset
trigger_probability float 0.0 – 1.0 Model-estimated probability of threshold exceedance
event_description string >= 20 characters Free-text narrative describing the expected event

Optional Fields

Field Type Constraints Description
admin2_pcode string — Admin level-2 p-code (district / county)
trigger_threshold float 0.0 – 1.0 Activation threshold configured for this hazard–country combination
affected_population integer >= 0 Estimated number of people in the impact zone
affected_area_km2 float >= 0 Area of predicted impact in km²
economic_loss_usd float >= 0 Modelled economic loss estimate in USD
fatalities_estimate integer >= 0 Forecast or observed fatality count
displacement_estimate integer >= 0 Forecast or observed number of displaced persons
infrastructure_damage_score integer 1 – 5 Raw damage severity from source system (distinct from annotation scores below)

Validation Rules

Validation runs automatically when a file is uploaded via POST /data/upload. Records failing any rule are rejected and logged; the remainder proceed to storage. A validation report is written to metadata.json.

Schema Validation

  • All required fields must be present and non-null.
  • Field types must match the schema exactly; coercion is applied only for numeric strings in CSV input (e.g. "0.83" → 0.83).
  • hazard_type values not in the allowed enumeration are rejected.
  • country_code must match a value in the internal p-code reference table.

Range Checks

Field Rule
trigger_probability Must satisfy 0.0 <= value <= 1.0
trigger_threshold Must satisfy 0.0 <= value <= 1.0
lead_time_hours Must satisfy 1 <= value <= 240
affected_population Must be non-negative if present
economic_loss_usd Must be non-negative if present
infrastructure_damage_score Must be in {1, 2, 3, 4, 5} if present
forecast_date Must not be in the future by more than 10 days, and not more than 5 years in the past

Deduplication

A record is considered a duplicate if another record in the same dataset shares the same (event_id, forecast_date, admin1_pcode, hazard_type) composite key. Duplicates are dropped and counted in the validation report.

Minimum Dataset Size

Datasets with fewer than 10 valid records after validation are rejected entirely; the upload is rolled back and the user is notified.

Annotation Workflow

Annotation transforms validated raw records into reasoning-ready examples. Annotators are registered users with the annotator role.

Steps

  1. Assignment — A dataset is assigned to one or more annotators via the /corpus interface. Each annotator sees only records assigned to them.
  2. Review — The annotator reads the event_description and optional context fields (impact estimates, probability, lead time).
  3. Scoring — The annotator assigns a score of 1 – 5 on each of the five impact dimensions (described below).
  4. Reasoning text — The annotator writes or edits a reasoning field: a concise paragraph justifying the scores in relation to the event context.
  5. Submission — On submission the record is written to data/annotated/<dataset_id>/corpus.jsonl and a row is inserted into the annotations table.
  6. Review / arbitration — Where two annotators disagree by more than 1 point on any dimension, a third annotator or a senior reviewer resolves the conflict. The resolved scores replace the originals; the disagreement is logged in annotation_log.json.

Inter-Annotator Agreement

Cohen's Kappa is computed per dimension across the dataset after all annotations are complete. A dataset is considered annotation-complete only when Kappa >= 0.6 for all five dimensions. Datasets below this threshold are flagged for additional review rounds.

5-Score Classification

Each annotated record is scored on five impact dimensions. Scores are integers from 1 (lowest severity or confidence) to 5 (highest).

Dimensions

Dimension Key Description
Population Impact score_population Severity of impact on human life: exposure, casualties, displacement
Infrastructure Damage score_infrastructure Degree of damage to physical infrastructure: roads, buildings, utilities
Economic Loss score_economic Estimated financial impact relative to local economic context
Response Urgency score_urgency How quickly humanitarian or government response is required
Forecast Confidence score_confidence Reliability of the underlying forecast: data quality, model certainty, lead time

Score Definitions

Scores are applied consistently using the following rubric:

Score Label Interpretation
1 Negligible Little to no impact expected; uncertainty is very high or event is minor
2 Low Limited localised impact; manageable with routine response capacity
3 Moderate Significant impact affecting a meaningful portion of the population or area; elevated response required
4 High Severe impact; large-scale response, resources, or pre-positioning needed
5 Extreme Catastrophic impact; national or international response level; life-safety critical

Scoring Example

For record EVT-2024-001 (riverine flooding, 142 000 exposed, 83% trigger probability, 72-hour lead time):

score_population:      4   (large exposed population, fatality and displacement estimates significant)
score_infrastructure:  3   (road and bridge damage plausible given extent; no critical national infrastructure)
score_economic:        4   (modelled USD 2.4 M loss, substantial relative to local economy)
score_urgency:         4   (72-hour lead time is actionable; pre-positioning strongly warranted)
score_confidence:      4   (83% trigger probability, sourced from operational NWP system with 72 h lead)

Output JSONL Format

After annotation, each record is serialised as a single JSON object on a single line in corpus.jsonl. The file is UTF-8 encoded with Unix line endings.

Schema

Field Type Source
id string UUID generated at annotation time
dataset_id string UUID of the parent dataset
event_id string From raw input
source_system string From raw input
country_code string From raw input
admin1_pcode string From raw input
admin2_pcode string or null From raw input
hazard_type string From raw input
forecast_date string From raw input
lead_time_hours integer From raw input
trigger_probability float From raw input
affected_population integer or null From raw input
affected_area_km2 float or null From raw input
economic_loss_usd float or null From raw input
fatalities_estimate integer or null From raw input
displacement_estimate integer or null From raw input
event_description string From raw input
reasoning string Written by annotator
score_population integer Annotator score (1–5)
score_infrastructure integer Annotator score (1–5)
score_economic integer Annotator score (1–5)
score_urgency integer Annotator score (1–5)
score_confidence integer Annotator score (1–5)
annotator_id string UUID of the annotator user
annotated_at string ISO 8601 datetime
split string One of: train, validation, test

Example Record

{
  "id": "a3f1c2d4-88b0-4e6a-9c3d-1f2e3a4b5c6d",
  "dataset_id": "d7e8f9a0-1b2c-3d4e-5f6a-7b8c9d0e1f2a",
  "event_id": "EVT-2024-001",
  "source_system": "IBF-NG",
  "country_code": "NGA",
  "admin1_pcode": "NG015",
  "admin2_pcode": "NG015003",
  "hazard_type": "FLOOD",
  "forecast_date": "2024-08-10",
  "lead_time_hours": 72,
  "trigger_probability": 0.83,
  "affected_population": 142000,
  "affected_area_km2": 380.5,
  "economic_loss_usd": 2400000.0,
  "fatalities_estimate": 12,
  "displacement_estimate": 4500,
  "event_description": "Significant riverine flooding expected along Benue River corridor following persistent rainfall upstream.",
  "reasoning": "With 142 000 people in the projected impact zone, double-digit fatality and large displacement estimates, and an economic loss of USD 2.4 M, this event warrants a high population and economic score. Infrastructure damage is significant but limited to secondary roads and local bridges, placing infrastructure impact at moderate-to-high. A 72-hour lead time with an 83% trigger probability from an operational NWP system gives responders meaningful preparation time and supports a high urgency and confidence rating. Pre-positioning of relief supplies and early activation of evacuation protocols along the Benue River corridor is strongly recommended.",
  "score_population": 4,
  "score_infrastructure": 3,
  "score_economic": 4,
  "score_urgency": 4,
  "score_confidence": 4,
  "annotator_id": "u9b1c2d3-4e5f-6a7b-8c9d-0e1f2a3b4c5d",
  "annotated_at": "2024-08-11T09:14:32Z",
  "split": "train"
}

Train / Validation / Test Split

Records are assigned to splits at dataset export time using a stratified split by hazard_type and country_code. Default ratios: 70% train / 15% validation / 15% test. Split assignments are deterministic given a fixed random seed stored in metadata.json.

PostgreSQL Metadata

While files are the authoritative store for record content, the datasets and annotations tables track operational metadata.

datasets table (key columns):

Column Type Description
id UUID Primary key, matches directory name
name VARCHAR Human-readable dataset name
uploaded_by UUID FK → users Uploader
uploaded_at TIMESTAMPTZ Upload timestamp
hazard_types VARCHAR[] Hazard types present in the dataset
total_records INTEGER Count of records after validation
valid_records INTEGER Count of records passing all validation rules
annotation_complete BOOLEAN True when Kappa >= 0.6 on all dimensions

annotations table (key columns):

Column Type Description
id UUID Primary key
dataset_id UUID FK → datasets Parent dataset
event_id VARCHAR Source event identifier
annotator_id UUID FK → users Annotator
scores JSONB All five dimension scores
reasoning TEXT Annotator reasoning text
annotated_at TIMESTAMPTZ Submission timestamp

Ingestion API Reference

Endpoint Method Description
/data/upload POST Upload a raw CSV or JSON file; triggers validation
/data/{dataset_id} GET Retrieve dataset metadata and validation report
/data/{dataset_id}/records GET List validated records (paginated)
/corpus/{dataset_id}/annotate POST Submit annotations for a record
/corpus/{dataset_id}/export GET Download the final corpus.jsonl

Uploads use multipart/form-data. The file part carries the data file; an optional name part provides a human-readable dataset name.

curl -X POST https://<host>/data/upload \
  -H "Authorization: Bearer <token>" \
  -F "[email protected];type=text/csv" \
  -F "name=Nigeria Flood Season 2024"

Clone this wiki locally