perf(profiles): ⚡️ speed up SPARQL target queries on medium/large crates - #191
perf(profiles): ⚡️ speed up SPARQL target queries on medium/large crates#191kikkomep wants to merge 2 commits into
Conversation
The `sh:select` queries of several `sh:SPARQLTarget` shapes combined two high-cardinality patterns sharing no variables — `?this a <Class>` and `?metadatafile schema:about ?root` — related only by the FILTERs applied afterwards. rdflib materialises the full cartesian product before filtering. Pre-compute the metadata descriptor in a `SELECT DISTINCT` subquery, which SPARQL evaluates and materialises first, so the outer pattern joins against an already-known `?root` instead of building the product.
The `ro-crate:ContextualEntityDefinition` target matched every triple with `?this ?p ?o` and then applied 6 `FILTER NOT EXISTS` plus 17 `STRSTARTS` exclusions to each match. The `SELECT DISTINCT` collapsed the duplicates only afterwards, so the filters ran once per triple rather than once per entity.
|
Can confirm it is much faster at validating an RO-Crate with 3000+ entities following this change. Before: After: |
There was a problem hiding this comment.
Thanks @kikkomep !
For info, another way we have approached this (no idea if it's better or worse performance-wise) is to make use of sh:order to enable classes like ro-crate:RootDataEntity to be used in SPARQL targets directly. Example: https://github.com/eScienceLab/rocrate-validator/blob/ddf10285cb1df2effde1032be0939eb11870db5d/rocrate_validator/profiles/five-safes-crate/0_workflow_run_inference.ttl#L22-L47
| { SELECT DISTINCT ?root WHERE { | ||
| ?metadatafile schema:about ?root . | ||
| FILTER(contains(str(?metadatafile), "ro-crate-metadata.json")) | ||
| } } |
There was a problem hiding this comment.
I don't see ?metadatafile or ?root referenced elsewhere in this target - is this part necessary?
| { SELECT DISTINCT ?root WHERE { | ||
| ?metadatafile schema:about ?root . | ||
| FILTER(contains(str(?metadatafile), "ro-crate-metadata.json")) | ||
| } } |
There was a problem hiding this comment.
same as above - I don't see ?metadatafile or ?root referenced elsewhere in this target - is this part necessary?
| { SELECT DISTINCT ?root WHERE { | ||
| ?metadatafile schema:about ?root . | ||
| FILTER(contains(str(?metadatafile), "ro-crate-metadata.json")) | ||
| } } |
There was a problem hiding this comment.
same as above - I don't see ?metadatafile or ?root referenced elsewhere in this target - is this part necessary?
| { SELECT DISTINCT ?root WHERE { | ||
| ?metadatafile schema:about ?root . | ||
| FILTER(contains(str(?metadatafile), "ro-crate-metadata.json")) | ||
| } } |
There was a problem hiding this comment.
It would be really useful to document this as a recommended pattern for profile developers when they need to fetch the root in a query
There was a problem hiding this comment.
Also: the wording of this PR is very jargon-heavy, and I found it difficult to parse until I read the code myself ("Cartesian products in target queries" in particular meant nothing to me). It would help in future if you could describe the solution in simpler language, to make it easier for others to review the code, understand why it works, and (in my case) understand what further changes might be needed downstream.
|
Another thought - it'd be good to add a regression test for this fix, e.g. validate a large crate and ensure it doesn't take too long |
|
In the ISA profile, we tried to avoid SPARQL targets completely, here's an example how we did this: https://github.com/crs4/rocrate-validator/blob/develop/rocrate_validator/profiles/isa-ro-crate/1_study.ttl#L26 |
4 participants
On medium and large RO-Crates — anything with a few hundred data entities or more — validation time grew far faster than the size of the crate. The cost was concentrated in a handful of
sh:selectqueries in the RO-Crate profile shapes, whose cost is quadratic in the number of entities.Two distinct pathologies, both of which only become visible once a crate is large enough:
Cartesian products in target queries. Several
sh:SPARQLTargetshapes combined two high-cardinality patterns sharing no variables —?this a <Class>and?metadatafile schema:about ?root— related only by the FILTERs applied afterwards.rdflibmaterialises the full product before filtering, so the number of intermediate rows grows as (entities of that class) × (schema:about triples).Filters applied per triple instead of per entity. The
ro-crate:ContextualEntityDefinitiontarget matched every triple with?this ?p ?o, then ran 6FILTER NOT EXISTSand 17STRSTARTSexclusions on each match.SELECT DISTINCTcollapsed the duplicates only afterwards — too late to save any work, so the filters ran once per triple rather than once per entity.Both scale with crate size, so small crates and the test fixtures were never slow enough for this to surface.
What changes
Both are fixed with the same idea: materialise the small side first in a
SELECT DISTINCTsubquery, which SPARQL evaluates before the rest of the query.?root(and?metadatafilewhere the outer FILTERs need it in scope) instead of joining against everyschema:abouttriple.No changes to the validator code, the ontologies, or the inference settings — this is entirely in the profile shapes.