Skip redundant texture decodes during WMO export - #591
Conversation
Exporting a map or a large WMO could spend a very long time in the texture phase because each material reference re-fetched and re-decoded its BLP, even when the same texture had already been exported moments earlier. A map that reuses a handful of WMOs across many tiles references the same textures thousands of times - a 12.1 raid export logged ~28,000 texture writes for only 436 distinct textures, and the texture phase ran over an hour. Track the fileDataIDs already fetched, decoded and written this export in a set (cleared by clearCache() alongside doodadCache) and skip the redundant work on repeat references. The MTL material mapping is still recorded for every reference, so output is unchanged - only the duplicate fetch/decode/write is skipped. Also guard the GLB buffering path against re-decoding into its buffer. For the raid case this cuts ~28,000 decode operations to 436.
|
Closing alongside #590 to re-verify end-to-end before review. Will reopen once confirmed. |
|
Reopening — verified on the same raid export: the WMO texture phase dropped from ~62 minutes / 28,000+ decodes to ~90 seconds / 423 decodes (once per distinct texture), with 15,000+ redundant decodes skipped. Output unchanged. |
The cache was keyed on fileDataID alone, but with shared textures disabled the same texture legitimately lands in a different folder per WMO. The first write cached the id and every later WMO's copy was skipped, leaving its MTL pointing at a file that was never written. Keying on id plus destination keeps the dedupe for repeat references to the same path and writes each distinct destination once.
|
Pushed a fix found while re-reviewing: the cache was keyed on fileDataID alone, which with shared textures disabled skipped writing the same texture into a second WMO's folder and left that MTL referencing a file that was never written. The key now includes the destination path, so repeat references to the same path still dedupe and each distinct destination is written once. The measured speedup is unaffected — the redundant decodes it eliminates are same-path references. |
2 participants
Problem
Exporting a map (or a large WMO) can spend an enormous amount of time in the texture phase because the same texture is fetched from CASC and decoded from BLP once per material reference, with no memory of what was already exported.
Maps reuse a small set of WMOs across many tiles, and each WMO references its textures across many materials, so shared textures get decoded over and over. A 12.1 raid map export logged ~28,000 texture writes for only 436 distinct textures — the texture phase ran for over an hour, dominated almost entirely by redundant decodes.
The existing
file_existedcheck doesn't help here: withoverwriteFilesenabled it re-decodes regardless, and even the decode+saveToPNGruns before the write is skipped.Fix
Track the
fileDataIDs already fetched, decoded and written during the current export in aSet, and skip the fetch/decode/write on repeat references. The set is cleared byclearCache()alongside the existingdoodadCache, so it has the same per-export lifetime and clears at the same points.The MTL material mapping (
mtl.addMaterial/textureMap.set) still runs for every reference, so the generated.mtlis unchanged — only the duplicate fetch/decode/write is skipped. The GLB buffering path is likewise guarded so it doesn't re-decode a texture already in its buffer.Impact
For the raid case above this reduces ~28,000 decode operations to 436 — a ~65× reduction in texture work, turning an hour-long texture phase into seconds. Smaller exports see proportional savings wherever textures are shared.
Scope
One file (
WMOExporter.js), no new dependencies, no build-pipeline changes. Output is byte-identical. Follows the existingdoodadCachepattern for consistency.