orthophoto merge: parallelize the block loop (in-order writer)

[Chouffe]

Summary

Parallelizes the per-block blend loop in merge(), behind a max_workers parameter (default 1 = unchanged serial behavior), wired from stages/splitmerge.py as args.max_concurrency.

Stacked on #2036 (gated reads). This PR currently shows both commits; once #2036 merges it reduces to just the parallelization. The gating is a prerequisite — see below.

Why it needs the gated reads (#2036)

The block loop is embarrassingly parallel, but a naive thread pool stalls: rasterio's boundless=True reads serialize a VRT (_serialize_xml) on every read, which under concurrency dominates and effectively hangs the merge (workers park in _serialize_xml). #2036 replaces those with gated reads (no VRT for in-bounds windows), which makes a parallel loop viable.

Parallelization

• Parallel compute + single in-order writer. Blocks are computed in a ThreadPoolExecutor; one thread writes them in strict block order with a bounded look-ahead (cap = 2 * max_workers). A naive "write each block as it finishes" version can't flush incrementally — GDAL needs writes in row-major (block) order — so dirty blocks accumulate in RAM until it OOMs; the in-order writer keeps writes sequential/flushable and memory small.
• Bounded GDAL block cache during the merge (restored on exit) so dirty-tile eviction/flush under GDAL's global lock stays prompt.
• Per-thread source handles — GDAL/rasterio datasets are not thread-safe.
• Preserves --merge-skip-blending.

max_workers <= 1 is a plain serial compute-then-write path, byte-for-byte identical to the original loop.

Correctness

• Deterministic across worker counts. On a real 15.9 Gpx survey (3 submodels, ~61k blocks), the merged orthophoto is byte-for-byte identical at max_workers=1 (serial, ~28 min) and max_workers=16 (~8 min) — the in-order writer makes the result independent of worker count, and the serial/default path completes cleanly with no read↔write stall.
• End-to-end. Verified on the same survey: the merge completes with zero stalled blocks and produces a valid orthophoto, pixel-identical to the serial baseline (confirmed for both LZW-compressed and uncompressed output).

Notes

• No new dependencies.
• Default behavior unchanged (max_workers=1).

[spwoodcock]

Thanks @Chouffe - this could be a really nice change! 😁

Before I do a full review, could you help me with two things please:

1. You say the outputs are unchanged from before. Which test dataset did you use to verify? Could you provide any logs of the successful run?

2. I don't think ODM has a policy set for LLM-assisted PRs needing to be declared (yet). For full transparency, was every line written by you, or was this assisted?

Thanks a lot for the contribution! 🙌

[Saijin-Naib]

Our policy, as it stands currently, id here:
https://github.com/OpenDroneMap/documents/blob/master/CONTRIBUTING.md

[spwoodcock]

Our policy, as it stands currently, id here:
https://github.com/OpenDroneMap/documents/blob/master/CONTRIBUTING.md

Thank you @Saijin-Naib! 🙏

I might make a little PR to ensure thats linked in the pull request template, so devs are definitely made aware when they create a new PR =)

[Chouffe]

@spwoodcock thanks for taking the time to review this PR!

For context: we’re using ODM on large drone surveys and letting users upload flight data to an online platform. We’ve already tuned ODM for our bird-monitoring domain so it runs reliably on AWS ECS Fargate. Even so, we hit one remaining issue: a large stress-test survey would deadlock after ~7 hours of compute.

To address this, I ran ODM from a pinned base Docker image (v3.6.0) with the relevant runtime patches baked in. With these changes, our very large surveys complete successfully without the earlier hiccups.

After extended investigation, the deadlock turned out to be caused by a combination of compounding factors:

1. rasterio “boundless” mode becoming pathologically slow as the GDAL cache fills (including occasional read stalls). The fix is in a companion ODM PR: orthophoto merge: avoid boundless-read VRT serialization (gated reads) #2036

2. The merge step running single-threaded, which is a significant bottleneck when multiple cores are available. We can speed it up with parallelism, while still ensuring GDAL writes blocks in the correct order to avoid write stalls.

To validate the merge changes, I reproduced the merge path in isolation by generating three orthophotos (via ODM’s split phase) and then running the merge step across those three outputs. I also confirmed the merged results are byte-identical whether run serially or with a high worker count.

The sub-orthophotos are large (≈5–10GB each), so if you’d like to verify that the read stalls occur in the problematic scenario, tell me what’s the best way to share artifacts with you.

Finally, this work was supported by an LLM, and the investigation itself took about a week of iterative runs to reproduce the exact prod behavior.

[spwoodcock]

This is really nice - I only left two comments, for the same line of code - hopefully useful 😄

[Chouffe]

Thanks for the review @spwoodcock, both are good catches. Addressed in the latest push:

1. File-handle bound. Agreed on the EMFILE risk now that split-merge passes max_concurrency by default. I went with your hard-cap suggestion rather than an RLIMIT_NOFILE probe, for the cross-platform reason you gave:

MERGE_WORKER_CAP = 16

if max_workers > MERGE_WORKER_CAP:
    log.ODM_INFO("Capping orthophoto merge workers %d -> %d to limit open file handles"
                 % (max_workers, MERGE_WORKER_CAP))
    max_workers = MERGE_WORKER_CAP

The cap is applied once inside merge(), so it protects every caller. This is a pure concurrency bound, so it doesn't change the output (blocks are computed independently and written in strict order regardless of worker count) and the byte-identical guarantee still holds.

2. Skip cut opens in skip-blending mode. Done. The cut rasters are only read in the third (blend) pass, which is skipped in that mode, so there's no reason to open them:

srcs = [(rasterio.open(o), None if merge_skip_blending else rasterio.open(c))
        for o, c in inputs]

The cleanup loop now guards the None handle.

[Chouffe]

@spwoodcock let me know if there is anything else I can do for this PR to get merged.

[spwoodcock]

Good to merge imo! Thanks again 🎉

[Chouffe]

Great to hear! I can't merge myself as I don't have the permissions on this repo. Looking forward to seeing this change land in ODM 🎉