Remove processing-training throughput bottlenecks

[iskandr]

Problem

Release processing training on a 4x A100-40GB Brev instance was unexpectedly slow: each 512-model processing variant took about six hours with low GPU utilization.

Root causes

1. The affected release was launched with PROCESSING_MAX_WORKERS_PER_GPU=1, overriding the checked-in auto default. The processing workload planner estimates 4 GB per worker and resolves this machine to four workers per GPU.
2. Class1ProcessingNeuralNetwork.fit() forces a full Python gc.collect() after every epoch. This cleanup was added to the legacy Keras/TensorFlow trainer and survived the PyTorch port even though the CLI now performs cleanup at model boundaries.
3. Processing validation reuses the training minibatch size (1,024 in the release), while the shared PyTorch validation heuristic uses a deterministic 4,096-row CUDA minimum for forward-only validation.

Evidence

• Mean GPU utilization was 11–15%; GPUs were idle in 60–68% of 30-second samples, with roughly 1 GB steady-state memory per GPU.
• Four processing workers were CPU-saturated, one per GPU.
• Mean model fit time was 165.6 seconds over about 66.6 epochs.
• The logged training loop averaged 0.765 seconds/epoch, while total fit time averaged 2.508 seconds/epoch.
• Repeated live stack snapshots found gc.collect() in 31 of 56 observable worker samples (55%).

Fix

• Do not pin processing workers in release launches; retain VRAM-aware auto packing.
• Remove forced per-epoch collection and retain the existing model-boundary CUDA/cache/GC cleanup.
• Apply the shared deterministic validation-batch heuristic to processing validation, while retaining an explicit validation_batch_size cap for constrained hardware.

The next full release run must regenerate processing ensembles and compare held-out metrics because validation reduction grouping can introduce floating-point-noise-level differences in early stopping.