DTS staging requests for JDP return 404 for some large transfers

[jeff-cohere]

Right now, as a matter of practice, DTS asks for all requested files to be staged from a database/provider. This is fine, but the JGI data portal has been returning a 404 error in some case involving large transfers, likely where not all files are actually archived. Because the response with the 404 code is essentially a small HTML page, it's hard to tell exactly what produces this error. It could be:

• the request is too large (too many files)
• some of the files are not archived, and so are not found in the set of "archived files".

I suspect that the problem is that there are too many files in the staging request, since I've never seen this error mode for smaller transfers, and I think it very unlikely that the DTS has never requested staging for a set of files for which at least one was readily available. But "unlikely" is not "impossible."

[jeff-cohere]

So far it does indeed look like smaller numbers of files (e.g. 1000 instead of 5000) works better, so batching these staging requests seems like a good approach. The only complication is that one DTS staging request then corresponds to a number of JDP archival requests. But that's a reasonable price to pay for the flexibility.