One problem one could encounter is that if the docs file is not in the same local server (say, on a network storage server), network latency is considerable and, even indexed, reading a block of documents can be too slow.
While PyTorch's DataLoader already does some preemptive loading, it is too conservative to make a considerable difference, even with pinning memory and multiple workers.
One way to work around this is to have a separate process(es?) that will cache documents preemptively more aggressively than DataLoader.
This can be implemented by passing to the Indxr initializer a list with the order of the documents the DataLoader will fetch, so that it can keep a local cache of size K, adding more documents as needed. If Kis large enough, it should be fetching documents way ahead of when they are needed.
Possible issues:
- Eventually, if the process is fast enough and the latency is too high, the
DataLoader will catch up with the cache, and subsequent requests of documents will result in cache misses.
- Shuffling the
DataLoader can break the cache. (I think DataLoader is eager when shuffling the dataset. If so, It should be possible to extract the list of samples it would fetch and extract the ids from there. But this is something that the user should deal with, not indxr)
- How to deal with multiple epochs? The cache should fetch documents from the list of samples in a circular fashion
One problem one could encounter is that if the docs file is not in the same local server (say, on a network storage server), network latency is considerable and, even indexed, reading a block of documents can be too slow.
While PyTorch's DataLoader already does some preemptive loading, it is too conservative to make a considerable difference, even with pinning memory and multiple workers.
One way to work around this is to have a separate process(es?) that will cache documents preemptively more aggressively than
DataLoader.This can be implemented by passing to the Indxr initializer a list with the order of the documents the
DataLoaderwill fetch, so that it can keep a local cache of sizeK, adding more documents as needed. IfKis large enough, it should be fetching documents way ahead of when they are needed.Possible issues:
DataLoaderwill catch up with the cache, and subsequent requests of documents will result in cache misses.DataLoadercan break the cache. (I thinkDataLoaderis eager when shuffling the dataset. If so, It should be possible to extract the list of samples it would fetch and extract the ids from there. But this is something that the user should deal with, not indxr)