Db v4 schema layer#2658
Draft
arienandalibi wants to merge 29 commits into
Draft
Conversation
…global property id i corresponding to the segment. Persisted to disk. In-memory part grows incrementally.
…ow) of properties present in each (segment, layer) to efficiently retrieve the lists of relevant properties. We union over all segments in the storage. We also filter out props/metadata that aren't present now instead of all globally registered properties.
… for properties and one for metadata in Meta. Wired it in so that layers where we know the property isn't present are skipped.
…und in this layer. Added these for the node storage and edge storage. Skip layers in tprop_iter_layers, which is used in time semantics.
…es. These were used to track property presence per-(segment, layer) on disk and in-memory. Now, we only keep track of property presence per layer. We skip layers where we know the property was added, but that's it.
… previous layer property schemas (property presence bitsets) to avoid work. All of it was redundant and didn't save anything. Removed functions that pass these `LayerPropSchema`s around.
…pe, dst_node_type). Introduce limits for max number of edges/values in properties and edges.
# Conflicts: # raphtory-storage/src/graph/nodes/node_storage_ops.rs # raphtory/src/db/graph/views/filter/model/degree_filter.rs
…g all properties. As soon as a property key has more than ENUM_BOUNDARY values, we skip property value collection.
…ies and metadata. This is held by GraphWithVectors(Inner). It gets passed to GqlGraph for base/unfiltered (e.g. materialized) graphs, which passes them to EdgeSchema by Arc.
…String, String) type alias. This allows us to pass references to the key easily. Faster on cache lookup
# Conflicts: # raphtory-storage/src/graph/nodes/node_storage_ops.rs
# Conflicts: # raphtory-graphql/src/data.rs # raphtory-graphql/src/graph.rs # raphtory-storage/src/graph/nodes/node_storage_ops.rs
…es and metadata per layer. We use the maintained property presence bitsets to do this
… rid of NodeSchemaKey. Wire the cache through in the schemas.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: bb5c4dc | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
4 ns/iter (± 0) |
1 ns/iter (± 0) |
4 |
lotr_graph/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
8429761 ns/iter (± 103720) |
1564816 ns/iter (± 35303) |
5.39 |
lotr_graph_window_100/num_nodes |
14 ns/iter (± 0) |
5 ns/iter (± 0) |
2.80 |
lotr_graph_window_100_materialise/materialize |
8315151 ns/iter (± 48887) |
1669150 ns/iter (± 10700) |
4.98 |
lotr_graph_window_10/has_node_existing |
143 ns/iter (± 12) |
62 ns/iter (± 11) |
2.31 |
lotr_graph_window_10_materialise/materialize |
3364662 ns/iter (± 10300) |
971980 ns/iter (± 4278) |
3.46 |
lotr_graph_subgraph_10pc/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
2039711 ns/iter (± 13488) |
334634 ns/iter (± 1287) |
6.10 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
150 ns/iter (± 13) |
62 ns/iter (± 14) |
2.42 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1216398 ns/iter (± 9165) |
230399 ns/iter (± 2617) |
5.28 |
lotr_graph_window_50_layered/num_edges_temporal |
154738 ns/iter (± 4448) |
70121 ns/iter (± 7586) |
2.21 |
lotr_graph_window_50_layered/has_node_existing |
391 ns/iter (± 23) |
129 ns/iter (± 12) |
3.03 |
lotr_graph_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_window_50_layered/graph_latest |
86183 ns/iter (± 1540) |
36649 ns/iter (± 916) |
2.35 |
lotr_graph_window_50_layered_materialise/materialize |
30622131 ns/iter (± 111615) |
3488825 ns/iter (± 24948) |
8.78 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
646869 ns/iter (± 10301) |
192686 ns/iter (± 1569) |
3.36 |
lotr_graph_persistent_window_50_layered/has_node_existing |
434 ns/iter (± 415) |
174 ns/iter (± 83) |
2.49 |
lotr_graph_persistent_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
3555447 ns/iter (± 26223) |
1659940 ns/iter (± 19402) |
2.14 |
lotr_graph_persistent_window_50_layered/graph_latest |
138817 ns/iter (± 6898) |
57549 ns/iter (± 4809) |
2.41 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
54842058 ns/iter (± 396929) |
5298035 ns/iter (± 147912) |
10.35 |
This comment was automatically generated by workflow using github-action-benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adding a bitset (Vec) to keep track of which properties exist in which layers. Use this to optimize iterators (especially relevant when using disk storage). Also optimized the generation of schemas (node schema, edge schema, ...) by iterating edges only once and implementing a cache for already generated schemas.
Why are the changes needed?
Optimizations can be made to improve UI experience and disk accesses.
Does this PR introduce any user-facing change? If yes is this documented?
It shouldn't
How was this patch tested?
Will need to be tested using the UI
Are there any further changes required?
There shouldn't be