Skip to content

Perf: Add a parallel FindMany method to speed up queries for a key #21

Description

@meling

Right now lookup using the Find(key) method for a single key isn't necessarily a significant performance bottleneck. However, when checking all keys, when len(keys) is large, it can be quite slow.

We should add a parallel FindMany method; still need to think more about the signature of the method to make it flexible for several uses, but here is a first sketch:

func (bb BBHash) FindMany(keys []uint64) []uint64 {
	var wg sync.WaitGroup
	keyChunks := slices.Chunk(keys, 1000)
	a := make([][]uint64, 0, len(keys)/1000)
	for chunk := range keyChunks {
		wg.Add(1)
		idx := make([]uint64, len(chunk))
		a = append(a, idx)
		go func() {
			defer wg.Done()
			for i, key := range chunk {
				idx[i] = bb.Find(key)
			}
		}()
	}
	wg.Wait()
	return slices.Flatten(a) // Flatten does not exist
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceCan possibly improve performance

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions