Per-iteration scalars accumulate unbounded in a single ES index

[michelkok]

Every Logger.report_scalar(...) call becomes a doc in events-training_stats_scalar-<co>. That index is single-shard and has no rolling/retention. After enough experiments × enough scalars × enough iterations, it gets big enough that:

• Bulk writes take 30–1500 s; ES write thread pool stays saturated
• apiserver hits its 60 s ES read timeout → uploads "freeze"
• Because all writes share one thread pool, slow writes here also stall events-log, serving_stats_*, worker_stats_* etc. — server appears unresponsive even when no training is running
• Cleanup via _delete_by_query against the giant shard is itself slow and memory-hungry

We hit this at 207 GB / 1.5 B docs in one shard (top segment 82 GB / 582 M docs) on a self-hosted server with a few thousand experiments accumulated over ~2 years. Hardly an extreme scale.

As far as I understand (but I'm far from an expert) every per-iter scalar is fully indexed in Elasticsearch, even though for old/finished experiments the search/aggregation capabilities aren't really used — people just want to look at the curve. Summary stats are already in MongoDB (last_metrics); full curves could live on the fileserver and be loaded on demand. ES feels like the wrong tier for the bulk of this data.

I could see a workaround: something like the cleanup script that, for old tasks, replaces per-iteration scalar series with a rendered plot in events-plot and drops the per-iter docs — keeping summary scalars (count==1 per metric/variant) untouched.
But it's really a workaround and a limitation of the framework.

Any help would be appreciated.

References

• Manage disk space usage #32, elasticsearch performance issue #85, Large experiments cause instability #87, Elasticsearch Error Trying to create too many buckets. #89 — same sort of failure mode, all closed without an architectural fix

[m-matthias]

@michelkok You mention that the top segment inside your index reached 82 GB in size. Do you know how this is actually possible, since I thought that ES caps the segment size at 5 GB by default. Does ClearML change the default limits? We also experience similar issues, especially when deleting old tasks with lots of logged metrics.

[michelkok]

@m-matthias I wouldn't know to be honest. As said, I'm far from an expert. I could do some more digging, but I'd prefer if someone with actual knowledge about this could look into it or someone from the team.

[m-matthias]

Ok, thank you for raising the issue, hopefully someone from ClearML can look into it. Our events-training_stats_scalar index also has segments much larger than the recommended 5GB, which might be another reason for the performance issues.