Index thinning#226
Conversation
|
sjfleming
left a comment
sjfleming
left a comment
There was a problem hiding this comment.
Some comments about the "carryover" concept and whether it can be eliminated. If it's necessary, that's okay. Just looking for a potential simplification
| need = training_sample_size - total_training_rows | ||
| train_dfs.append(df.iloc[:need]) | ||
| carry_df = df.iloc[need:] | ||
| total_training_rows += need |
There was a problem hiding this comment.
is need right, or is it len(df) - need ?
|
|
||
|
|
||
| @_tiledb_retry | ||
| def _write_carryover(index: vs.IVFFlatIndex, ids: np.ndarray, emb: np.ndarray, normalize: bool) -> int: |
There was a problem hiding this comment.
The "carryover" concept introduced in this PR is a little complex. I gather it's the "extra" cells opened in a file but not used to fill the initial index. And we want to use them later as updates.
I wonder: is there a way to eliminate the need for the carryover concept entirely? Something like:
- just allow the size of the initially-created index to be a bit flexible, so no carryover is needed
There was a problem hiding this comment.
or:
- have a first phase where you load embedding files, keep only allowed cells in memory, and then write them out to a new file every time you get enough cells. in the new files, n_cells is controlled and you know all cells are allowed. then do what you used to do
There was a problem hiding this comment.
I think I like this second idea, it will make all the subsequent phases to be identical no matter if we need thinning or not. Good one, thanks!
2 participants
No description provided.