Automated end-to-end pipeline that matches historical MTBS burn perimeters to USFS ignition points, downloads all required geospatial inputs, runs wind and fuel-moisture models, executes ELMFIRE and FARSITE, and produces a multi-page validation PDF comparing simulated vs. observed burn areas.
Data/
├── pipelineConfig.py ← single source of truth for all settings
├── parallel_api.py ← shared utilities (Tee, logging, retries)
├── case_metadata.py ← read/write per-case JSON metadata
├── cleanPipelineOutputs.py ← clean per-case simulation outputs
# Master entry point (replaces run_validation.py)
├── runWildfireAV ← single CLI: setup / run / pdf
# Setup scripts (run once, locally)
├── setup/
│ ├── setupPipeline.py ← orchestrates setup steps 1-5
│ ├── processScarsAndPoints.py ← step 1: match perimeters → ignition points
│ ├── separateScarsAndPointsToCases.py ← step 2: create numbered case folders
│ ├── getSatelliteEndTimes.py ← step 3: compute satellite start/end times
│ └── eraseInvalidCases.py ← step 4: remove short/invalid cases
# Per-case simulation pipeline (run in parallel)
├── pipeline/
│ ├── runPipelineParallel.py ← orchestrates per-case steps (see below)
│ ├── runBatch.py ← parallel batch runner
│ ├── getLandfireProductsForFireSim.py ← step 1: download LANDFIRE from LFPS
│ ├── splitLandfireTifBands.py ← step 2: split multi-band LANDFIRE.tif
│ ├── makePhiAndAdjFiles.py ← step 3: create adj/phi rasters
│ ├── downloadWeatherData.py ← step 4: fetch ERA5 weather (OpenMeteo)
│ ├── downloadAndRunWindninja_WXS.py ← step 5a: WindNinja (WINDNINJA_SOURCE=install)
│ ├── downloadAndRunWindninja_wxModel.py ← step 5b: WindNinja wx-model variant
│ ├── downloadAndRunWindninja.py ← step 5c: WindNinja (legacy)
│ ├── wn_to_geotiff.py ← helper: convert WindNinja ASCII → GeoTIFF
│ ├── applyNelsonModel.py ← step 6: compute dead-fuel moisture
│ ├── getBarrierFile.py ← step 7: rasterise road/water barriers
│ ├── createElmfireInputFiles.py ← step 8: write ELMFIRE namelist (.data)
│ ├── prepareFarsite.py ← step 9: create FARSITE inputs (LCP, etc.)
│ ├── runElmfireCase.py ← step 10: execute ELMFIRE
│ ├── runFarsiteCase.py ← step 11: execute FARSITE (Linux native)
│ └── farsiteWindToGeotiff.py ← step 12: extract ws/wd from FARSITE winds
│ (only when WINDNINJA_SOURCE=farsite)
# Validation report
├── validation/
│ └── getValidationPDF.py ← generate multi-page validation PDF
# Developer / debug utilities
├── tools/
│ ├── monitorBatch.py ← real-time batch progress monitor
│ ├── prefetchLandfire.py ← pre-fetch LANDFIRE for all cases
│ ├── debugBandCounts.py ← debug raster band counts per case
│ ├── debugSatelliteEndTimes.py ← inspect satellite coverage curves
│ ├── compareOutputs.py ← multi-model PDF comparison (fast)
│ ├── visualiseSingleCase.py ← plot observed vs simulated for one case
│ └── getWeatherHerbie.py ← fetch weather via Herbie (dev use)
# Data directories
# Input data (not version-controlled — see Input data requirements below)
├── inputs/
│ ├── mtbs_perimeters.gpkg ← MTBS burn perimeters
│ ├── NIFC_FOD.gpkg ← NIFC fire occurrence points
│ ├── nasa_lance_allSatellites.gpkg ← VIIRS/MODIS/Landsat hotspot detections
│ ├── osm_conus_roads.gpkg ← OSM road network
│ ├── grwl.gpkg ← Global River Widths from Landsat waterways
│ └── osm_conus_rivers.gpkg ← Backup river polygons
└── nelson_csharp/ ← Nelson dead-fuel model (C# source + binary)
| Tool / Library | Where configured |
|---|---|
| Python ≥ 3.11 | – |
| geopandas, rasterio, fiona, pyproj, shapely | conda/pip |
| pandas, numpy, requests, matplotlib | conda/pip |
GDAL CLI (gdal_translate, gdalbuildvrt, ogr2ogr) |
on $PATH |
WindNinja CLI (WindNinja_cli) |
conda env (see WINDNINJA_CONDA_ENV) |
ELMFIRE executable (elmfire) |
on $PATH |
FARSITE Linux binary (TestFARSITE) |
pipelineConfig.FARSITE_FB_DIR |
| Nelson C# model | pipelineConfig.NELSON_EXE |
| LFPS API access (USGS email) | LFPS_EMAIL environment variable |
| OpenMeteo ERA5 (free, no key) | pipelineConfig.OPENMETEO_URL |
This pipeline is designed to run inside Windows Subsystem for Linux (WSL2) with Ubuntu 24.04. FARSITE runs as a natively compiled Linux executable so Wine is not required.
In an elevated PowerShell:
wsl --install -d Ubuntu-24.04
wsl --set-default-version 2Restart when prompted, then open the Ubuntu 24.04 terminal and create your Linux user account.
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# Follow the prompts; let the installer run conda init
source ~/.bashrcconda create -n elmfire python=3.11 -y
conda activate elmfire
# Geospatial stack
conda install -c conda-forge \
geopandas rasterio fiona pyproj shapely \
pandas numpy requests matplotlib \
gdal -yAll GDAL CLI tools (gdal_translate, gdalbuildvrt, ogr2ogr) come with the
gdal conda package and are available on PATH inside the environment.
Follow the ELMFIRE build instructions.
The compiled elmfire executable must be on PATH inside the elmfire conda
environment. Set ELMFIRE_PATH_TO_GDAL in pipelineConfig.py to the bin/
directory of the same conda environment, e.g.:
ELMFIRE_PATH_TO_GDAL = "/home/<user>/miniconda3/envs/elmfire/bin/"Follow the instructions in the WindNinja Github. Note that the pipeline was setup in Ubuntu-24.04, when asked to run scripts for 22.04, run the provided files for 24.04 instead. No need to install the GUI version.
Set WINDNINJA_CONDA_ENV in pipelineConfig.py to the environment name that
has WindNinja_cli on its PATH (default "base").
The pipeline now runs FARSITE as a natively compiled Linux executable — Wine
is no longer required. Obtain or build the TestFARSITE Linux binary and
place it in any directory on the machine (e.g. /home/<user>/farsite/src/).
Set FARSITE_FB_DIR in pipelineConfig.py to that directory and
FARSITE_EXE_NAME to the executable filename:
FARSITE_FB_DIR = Path("/home/<user>/farsite/src")
FARSITE_EXE_NAME = "TestFARSITE"Verify:
/home/<user>/farsite/src/TestFARSITE --helpcd /home/<user>/elmfire_validation
git clone https://github.com/nick-cloudfire/autoValidate.git Data
cd Datarun the following command to clone the Nelson Dead Fuel Moisture model made for this pipeline, based on WUINITY-PREACT Copyright (C) 2025 Jonathan Wahlqvist.
git clone https://github.com/nick-cloudfire/Nelson-Dead-Fuel-Moisture.git <nelson_csharp>
# Install .NET 8 SDK
sudo apt install -y dotnet-sdk-8.0
# Build the release binary
cd /home/<user>/elmfire_validation/Data/nelson_csharp
dotnet publish -c ReleaseThe compiled binary path is set automatically by pipelineConfig.NELSON_EXE.
The FARSITE Linux binary is not included in this repository. Place the
compiled TestFARSITE executable in any directory (e.g.
/home/<user>/farsite/src/). No Wine SDK layout is required — only the
single binary needs to be present at the path pointed to by FARSITE_FB_DIR.
Edit pipelineConfig.py — at minimum update:
BASE_VALIDATION = Path("/home/<user>/elmfire/elmfire_validation/")
FARSITE_FB_DIR = Path("/home/<user>/farsite/src") # dir containing TestFARSITE
WINDNINJA_SOURCE = "install" # or "farsite"LANDFIRE_EMAIL is read from the LFPS_EMAIL environment variable (not stored
in the config file). Add the following to your ~/.bashrc (or SLURM job script):
export [email protected] # must be registered at https://lfps.usgs.govThen reload your shell: source ~/.bashrc
All input data lives under inputs/ and is not version-controlled (listed in
.gitignore). Download these datasets once and place them at the paths below.
- Source: MTBS Data Access
- Place at:
inputs/mtbs_perimeters.gpkg
- Source: NIFC Fire Occurrence Database (FOD) — download as GeoPackage
- Place at:
inputs/NIFC_FOD.gpkg - Required fields:
IncidentName,FireDiscoveryDateTime,FireOutDateTime
- Source: NASA FIRMS archive (VIIRS/MODIS active fire detections for the US) Login with your email to download the CONUS dataset.
- Merge all downloaded shapefiles into a single GeoPackage with layer
output. Required columns:ACQ_DATE(date),ACQ_TIME(HHMM string), plus geometry. - Place at:
inputs/nasa_lance_allSatellites.gpkg
| File | Contents | Config key |
|---|---|---|
inputs/osm_conus_roads.gpkg |
US road network (layer lines, field highway) |
ROADS_GPKG |
inputs/grwl.gpkg |
Global River Widths from Landsat (layer lines, field waterway) |
WATER_GPKG |
inputs/osm_conus_rivers.gpkg |
Backup river polygons for areas with poor GRWL coverage | BACKUP_WATER_GPKG |
Road GeoPackages can be extracted from a US OSM .pbf file using
osmium + ogr2ogr, or downloaded from GeoFabrik.
LANDFIRE terrain and fuel rasters are fetched automatically per case via the
USGS LFPS API (step 1 of the simulation pipeline). No manual download needed —
only a valid email registered at USGS LFPS, set via the
LFPS_EMAIL environment variable (see WSL setup step 9).
Data/
├── inputs/
│ ├── mtbs_perimeters.gpkg
│ ├── NIFC_FOD.gpkg
│ ├── nasa_lance_allSatellites.gpkg
│ ├── osm_conus_roads.gpkg
│ ├── grwl.gpkg
│ └── osm_conus_rivers.gpkg
└── nelson_csharp/ (built from source — see WSL setup above)
# FARSITE binary lives outside this repo (see WSL setup step 6 / 9):
/home/<user>/farsite/src/TestFARSITE
Edit the User-modifiable parameters and Paths sections:
# Paths
BASE_VALIDATION = Path("/home/<user>/elmfire/elmfire_validation/")
FIRE_ROOT = Path("/scratch/yourname/FirePairs")
FARSITE_FB_DIR = Path("/home/<user>/farsite/src") # dir with TestFARSITE binary
# User parameters
MIN_FIRE_YEAR = 2024
MAX_FIRE_YEAR = 2025
MAX_PARALLEL_CASES = 14
# Wind source: "install" = WindNinja, "farsite" = derive winds from FARSITE output
WINDNINJA_SOURCE = "install"
# ELMFIRE
ELMFIRE_PATH_TO_GDAL = "/path/to/your/conda/envs/elmfire/bin/"runWildfireAV is the single entry point. The --start flag selects the
phase to begin from; all later phases also run automatically.
# Full pipeline from scratch (clean cases → setup → run → pdf):
./runWildfireAV --start setup
# Clean simulation outputs, re-run all cases, regenerate pdf:
./runWildfireAV --start run
# Regenerate PDF only (no simulation):
./runWildfireAV --start pdf
# Parallel run with 8 workers, skip completed cases:
./runWildfireAV --start run --workers 8 --skip-done
# Process specific cases only:
./runWildfireAV --start run --cases 00001 00005 00008See ./runWildfireAV --help for all options.
Runs the one-time case-preparation pipeline. Deletes all existing numbered case folders first (full reset), then rebuilds them from the raw MTBS and USFS input data.
| Step | Script | Output |
|---|---|---|
| 1 | setup/processScarsAndPoints.py |
perimeters_ignitions.gpkg, all_ignitions.gpkg |
| 2 | setup/separateScarsAndPointsToCases.py |
00001/, 00002/, …, fire_pairs_summary.csv |
| 3 | setup/getSatelliteEndTimes.py |
fire_pairs_summary_with_satellite.csv |
| 4 | setup/eraseInvalidCases.py |
removes cases shorter than MIN_HOURS_DURATION h |
| 5 | write_metadata_from_summary |
case_metadata.json in each folder |
Cleans per-case simulation outputs (from landfire_bands onwards), then
executes the per-case simulation pipeline in parallel using
ProcessPoolExecutor. Step order depends on WINDNINJA_SOURCE:
Step 1 getLandfireProductsForFireSim → LANDFIRE.tif
Step 2 splitLandfireTifBands → inputs/{dem,slp,asp,fbfm40,cc,ch,cbh,cbd}.tif
Step 3 makePhiAndAdjFiles → inputs/{adj,phi}.tif
Step 4 downloadWeatherData → inputs/weather.wxs
Step 5 downloadAndRunWindninja → inputs/{ws,wd}.tif (WindNinja)
Step 6 applyNelsonModel → inputs/{m1,m10,m100}.tif
Step 7 getBarrierFile → inputs/barrier.tif
Step 8 createElmfireInputFiles → <case>.data
Step 9 prepareFarsite → farsite/{landscape.lcp, farsite.input, …}
Step 10 runElmfireCase → outputs/time_of_arrival_*.tif
Step 11 runFarsiteCase → farsite/outputs/farsite_ArrivalTime.asc
Step 1 getLandfireProductsForFireSim → LANDFIRE.tif
Step 2 splitLandfireTifBands → inputs/{dem,slp,asp,fbfm40,cc,ch,cbh,cbd}.tif
Step 3 makePhiAndAdjFiles → inputs/{adj,phi}.tif
Step 4 downloadWeatherData → inputs/weather.wxs
Step 5 (WindNinja skipped)
Step 6 applyNelsonModel → inputs/{m1,m10,m100}.tif
Step 7 getBarrierFile → inputs/barrier.tif
Step 8 createElmfireInputFiles → <case>.data
Step 9 prepareFarsite → farsite/{landscape.lcp, farsite.input, …}
Step 10 runFarsiteCase → farsite/outputs/farsite_ArrivalTime.asc
farsite/outputs/farsite_WindGrids.tif
Step 11 farsiteWindToGeotiff → inputs/{ws,wd}.tif (from FARSITE wind grids)
deletes farsite_WindGrids.tif after extraction
Step 12 runElmfireCase → outputs/time_of_arrival_*.tif
Generates validation.pdf in FIRE_ROOT. The report contains one page per
case plus four summary pages:
| Page | Content |
|---|---|
| Per-case | Burn-mask map, area-evolution curves, metrics table |
| Summary 1 | Similarity scores vs. average wind speed |
| Summary 2 | Similarity scores vs. burn area and fire duration (log x) |
| Summary 3 | Area estimation bias per case (asymmetric log₂ bar chart) |
| Summary 4 | Similarity score distribution histograms |
Cleaning is integrated into the entry point:
--start setupdeletes all numbered case folders before re-running setup.--start rundeletes per-case simulation outputs (fromlandfire_bandsonwards, includingfarsite/directories) before re-running simulations.LANDFIRE.tif,case_metadata.json, and fire geometry files are preserved.
For fine-grained control use cleanPipelineOutputs.py directly:
python cleanPipelineOutputs.py # dry-run by default
python cleanPipelineOutputs.py --execute # actually delete
python cleanPipelineOutputs.py --from elmfire_outputs # only ELMFIRE outputs
python cleanPipelineOutputs.py --include-wind-tifs # also delete ws/wd.tifFARSITE runs as a natively compiled Linux executable (TestFARSITE).
Wine is not required. runFarsiteCase.py writes a plain-text command file
(farsite_linux.txt) with absolute Linux paths and invokes the binary directly.
The completion sentinel is farsite_ArrivalTime.asc; the script verifies this
file exists after the run.
After a successful FARSITE run, outputs are cleaned automatically:
WINDNINJA_SOURCE = "install": keeps onlyfarsite_ArrivalTime.asc; deletes all other FARSITE outputs includingfarsite_WindGrids.tif.WINDNINJA_SOURCE = "farsite": keepsfarsite_ArrivalTime.asc+farsite_WindGrids.tifuntilfarsiteWindToGeotiff.pyhas extractedws.tif/wd.tif, then deletesfarsite_WindGrids.tiftoo.
<case_dir>/ e.g. FirePairs/00001/
├── case_metadata.json all case attributes (times, name, area, …)
├── firescar.gpkg MTBS burn polygon
├── ignition_point.gpkg USFS ignition point
├── satellite_points.gpkg hotspot points inside the burn polygon
├── LANDFIRE.tif raw multi-band download
├── 00001.data ELMFIRE namelist
├── pipeline.log stdout/stderr from the pipeline run
├── inputs/
│ ├── dem.tif elevation (m)
│ ├── slp.tif slope (degrees)
│ ├── asp.tif aspect (degrees)
│ ├── fbfm40.tif 40-class fuel model
│ ├── cc.tif canopy cover (%)
│ ├── ch.tif canopy height (m)
│ ├── cbh.tif canopy base height (m)
│ ├── cbd.tif canopy bulk density (kg/m³)
│ ├── adj.tif adjacency (all 1s)
│ ├── phi.tif phi (all 1s)
│ ├── barrier.tif road/water width raster (m)
│ ├── weather.wxs RAWS-format weather file
│ ├── ws.tif wind speed (N bands, one per hour)
│ ├── wd.tif wind direction (N bands, one per hour)
│ ├── m1.tif 1-hr fuel moisture
│ ├── m10.tif 10-hr fuel moisture
│ ├── m100.tif 100-hr fuel moisture
│ └── windninja/ WindNinja workspace (deleted after ws/wd built)
├── farsite/
│ ├── landscape.lcp FARSITE landscape file (converted from LANDFIRE)
│ ├── farsite.input FARSITE control file
│ ├── farsite_linux.txt command-line args file (auto-generated)
│ ├── ignition.shp reprojected ignition point
│ ├── barrier.shp merged barrier polygons (optional)
│ └── outputs/
│ └── farsite_ArrivalTime.asc FARSITE time-of-arrival output (sentinel)
├── outputs/
│ └── time_of_arrival_<HH>.tif ELMFIRE TOA raster
└── scratch/ ELMFIRE working directory
All settings are documented inline in pipelineConfig.py.
| Section | Key settings |
|---|---|
| 1. User-modifiable | MIN/MAX_FIRE_YEAR, MTBS_AREA_THRESHOLD_ACRES, MAX_PARALLEL_CASES, WINDNINJA_SOURCE |
| 2. Paths | FIRE_ROOT, BASE_VALIDATION, FARSITE_FB_DIR |
| 3. File/dir names | CSV names, shapefile names, subfolder names |
| 4. Column names | master CSV column keys shared across scripts |
| 5. MTBS / USFS | field names in raw input datasets |
| 6. LANDFIRE | band order, product names, raster naming |
| 7. Satellite | coverage fraction, gap tolerance, buffer size |
| 8. Weather | OpenMeteo URL and model |
| 9. WindNinja | exe, model type, chunk size, output height, mesh resolution factor |
| 10. ELMFIRE | exe, GDAL path, simulation parameters, moisture content |
| 11. FARSITE | FARSITE_FB_DIR (dir with Linux binary), FARSITE_EXE_NAME |
| 12. Barrier | road/water widths, OSM field names |
| 13. LFPS API | base URL, product list, LFPS_POLL_SLEEP_S, LFPS_POLL_HEARTBEAT_S |
| 14. Nelson | path to C# executable |
- The LFPS queue can be very long when many users are submitting jobs simultaneously. The pipeline polls indefinitely — it will wait however long the queue takes.
- A "still waiting…" heartbeat is logged every
LFPS_POLL_HEARTBEAT_Sseconds (default 1800 s / 30 min) so you can confirm the job is still live. - Do not kill and resubmit the job — that adds another entry to the queue and makes the wait longer for everyone.
- Retries (up to
MAX_RETRIES = 3) are reserved for genuine failures (job statusFailedor network errors), not queue waits. - Check that
LFPS_EMAILis set in your shell (echo $LFPS_EMAIL) and that the address is registered at https://lfps.usgs.gov.
- These are transient gateway errors from the OpenMeteo server. The pipeline automatically retries up to 6 times with exponential back-off (5 s → 10 s → 20 s → 40 s → 60 s → 60 s). No action needed unless all 6 attempts fail.
- A permanent failure (any other 4xx status) is not retried and will appear in
pipeline.logwith the URL and response body.
- Confirm
WINDNINJA_CONDA_ENVmatches the conda environment withWindNinja_clion PATH. - Check
inputs/windninja/chunk_000/windninja_cli.logfor the error message. - Ensure the DEM covers the full fire domain.
- Confirm
FARSITE_FB_DIRpoints to the directory containingTestFARSITEand that the binary is executable:ls -l $FARSITE_FB_DIR/TestFARSITE. - Check
farsite/pipeline.logfor error output. - Run the binary interactively with the generated
farsite_linux.txtto test:<FARSITE_FB_DIR>/TestFARSITE farsite/farsite_linux.txt.
- Confirm
ELMFIRE_EXEis on PATH:which elmfire. - Check
ELMFIRE_PATH_TO_GDALpoints to a directory containinggdal_translate. - The
outputs/andscratch/directories are created automatically; existing contents are overwritten on re-run.
- The four filter stages (area, year, name+date match, spatial) each raise a
RuntimeErrorwith a specific message and the relevantpipelineConfig.pyknobs to relax:- No perimeters: lower
MTBS_AREA_THRESHOLD_ACRESor widenMIN/MAX_FIRE_YEAR. - No points: widen
MIN/MAX_FIRE_YEAR. - No name+date matches: increase
DAY_TOLERANCE_DAYS. - No spatial matches: increase
DAY_TOLERANCE_DAYSor lowerMTBS_AREA_THRESHOLD_ACRES.
- No perimeters: lower
- Some fires have sparse satellite coverage. Cases where coverage never reaches the threshold are skipped gracefully.
- Adjust
COVERAGE_FRACTIONorSAT_CHAIN_MAX_GAP_DAYSinpipelineConfigto be more lenient.
- Raised as a
RuntimeErrorif no valid FBFM40 pixel (≥ 101) exists within 2000 cells of the ignition point. Usually means the burn perimeter is outside the LANDFIRE domain or the fuel download failed.
- WindNinja creates many
step_NNN/subdirectories. These are deleted automatically bywn_to_geotiff.clean_windninja_outputs()afterws.tifandwd.tifare built. If a run was interrupted mid-step, re-run from thewindninjastep:python cleanPipelineOutputs.py --from windninja --executethen./runWildfireAV --start run. - All GeoTIFF outputs use LZW compression + tiling.
-
pipelineConfig.pyis the single source of truth. Never hard-code paths, filenames, or tunable numbers inside individual scripts. -
Each step script is independently runnable. Pass a
case_dirargument tomain()for single-case testing, or omit it to process all cases underFIRE_ROOT. -
parallel_api.pyprovidesTee,make_logger,retry_call, andrun_subprocess. Import from here instead of reimplementing locally. -
case_metadata.pyhandles all JSON serialisation of per-case metadata, including datetime normalisation. -
Scripts in
tools/are standalone dev/debug utilities and do not participate in the main pipeline.