Memory leak in service-worker

[mindplay-dk]

hi,

I plan on using this script to download very large files, so I set up a quick test-project to try it out.

https://github.com/mindplay-dk/downzip/tree/original-downzip

in the files folder, I have 10 files of about 0.46 GB each, and as you can see, I'm zipping and downloading them all - the idea here was just to see if this would work for over 4 GB of files, which it actually does. 🙂👍

it appears there's a memory issue though - while I'm downloading 4.6GB of files, the service worker appears to eat up 4.6GB of RAM, as you can see in this screen recording:

(for the record, the progress events you see in this recording is from this fork, which has a few extra features - I did test with your original package as well, which is what I'm using in the repo I linked to above, and it does have the same issue, I just didn't bother with a second screen recording.)

also, as you can see, it does not appear to really be streaming (?) so much as buffering the whole thing in the service worker - it appears to work up to the full 4.6GB of memory usage before the download really begins. this is probably a related issue. (and this is true of both your version and that fork as well.)

what's worse, memory appears to remain allocated after the download finishes - sometimes all of it, sometimes about half, sometimes for several minutes after closing the window. I suppose this could just be how Chrome operates service-workers? no idea.

I'm using latest Chrome on Win 11 Pro: Version 120.0.6099.71 (Official Build) (64-bit).

this all seemed rather odd, so I decided to give Firefox a go as well.

in Firefox, the downloads begin immediately - there's no delay, like there is in Chrome.

however, the downloads do not complete - they appear to randomly hang after 5-10 seconds of downloading:

similarly, memory usage does appear to climb while downloading, to about the same as amount of memory as the amount of the download that completes before it hangs - so the likely explanation is Firefox hitting the service-worker with some sort of memory usage limit designed to keep service-workers from eating all memory? however, there's no indication of any error in the debugger - and the downloads stall, forever, they're never canceled.

again, this is with Firefox on Win 11 Pro: 120.0.1 (64-bit).

just for completeness, I also gave Edge a shot - as you might expect, the experience is the same as in Chrome.

returning to Chrome, I also attempted to profile memory usage - oddly, if the memory profiler is running, memory will get freed immediately after the download, much sooner than without devtools... as you can see though, it is clearly allocating all the memory while the service-worker is downloading, before the host browser actually begins downloading the zip stream:

that's all I have right now - I will get back to you if I uncover anymore clues.

[mindplay-dk]

I did some more testing, and in Firefox Nightly, downloads do complete - so that part appears to be a bug in Firefox, which will hopefully be resolved in a future release.

It's still using gigabytes of memory though - that part appears to be a bug? I suspect something to do with "backpressure" - I'm not an expert at W3G streams, but in Chrome it looks like we're pushing data too fast? as explained, there is a long pause where almost nothing is downloaded and memory climbs to 4.6GB (and the download time estimate is way off) and then, as soon as the service-worker is done downloading everything, the actual zip download speeds up... which tells me the browser is being forced to buffer everything initially. The long delay before garbage collection may be due to the fact that so many buffers have been allocated, it just takes that long to collect and free them in the background?

EDIT: note that, in Firefox Nightly, downloads do start immediately, even though memory usage climbs to several gigabytes - I'm not sure if this tells us anything. It could just be that Firefox is better at prioritizing between the service-worker and the page, whereas Chrome seems to get bogged down by the service-worker, leaving no time for the page to download initially - as well as Firefox being potentially better at freeing buffers, e.g. being able to free them head-to-tail as it goes...

[robbederks]

Interesting report, I don't recall seeing this behavior when developing. Thanks for looking into it as deep as you already have!

To be honest, it's been a few years since I've worked with web based technologies too, so I don't have any good clues as to what might be causing this either without spending a few hours looking into it. I suspect that there shouldn't be any fundamental issues that would cause it to use so much RAM, so it's likely a relatively simple fix.

I'll leave this issue open until I find some time to debug, or until somebody else figures it out!
Would also be great to add some CI tests for this, since having low memory requirements is kind of the main point of this library

[mindplay-dk]

tbh I suspect a Chrome bug (or design issue) with regards to workers, streams, memory management, garbage collection, something along those lines - both Chrome an Edge suffer from this issue, but Firefox does not.

I've tried at least 5 different client-side solutions btw - the ones I could get to work suffer from similar issues, at least 3 other similar libraries/demos showed similar symptoms.

I don't have enough to go on, to file a meaningful Chrome issue, so I'm not going to - we'd need a minimal repro to prove an issue, and at this time (at work) we've resolved to just use a server-side streaming zip-service for our product for now.