perf(core): Add sharded map to speed up post query aggregations

[ghost]

No description provided.

[trunk-io]

Running Code Quality on PRs by uploading data to Trunk will soon be removed. You can still run checks on your PRs using trunk-action - see the migration guide for more information.

[trunk-io]

     

Failed Test	Failure Summary	Logs
TestVectorGraphQlEuclideanIndexMutationAndQuery		Logs ↗︎
TestACLSuite/TestAddNewPredicate	The ACL group count is not 1, which is unexpected.	Logs ↗︎
TestUniqueUpsertSingleMutationTwoBlankNode		Logs ↗︎
TestACLSuite		Logs ↗︎

View Full Report ↗︎ ⋅ Docs

[rahst12]

I'm always interested in the performance updates. Any early thoughts on what the community can expect with these improvements?

[ghost]

@rahst12 I am going to close this in favor of other smaller diffs. There are 3 dfferent optimizations here: (Most of them are for improving latency of a query, throughput might not be that affected)

1. We have a function called MergeSorted(). It basically takes multiple different sorted list of uids and merge them. It's basically an heap sort. Basically where ever we get multiple different keys, and then merge their data it's used. Almost all the complicated queries will use it. I have parallelized it now, because what would happen is that we might have a lot of different lists, which are small in size. This increases the size of the heap, making it ineffective.
2. Currently while caching, we cache the posting list object. This object has stored data in certain format, so that it's easier to consume for various different tasks. One of the tasks is to get the list of uids in that key. (Which then goes into the merge sorted function). I have now added that the uid list itself would get cached, instead of us deriving it again and again after getting from the cache.
3. We have something called variable propogation. Basically variables in a query are transformed, worked up, and then sent to other parts of the query. Currently this was done in a single thread. I have changed it such that it can be now done in multiple threads.
   The query I am working to optimize, shows significant optimization. I will run it a full suite of queries to see the performance across the board.