Cache all handler-reachable queries (v1.19.0)#45
Merged
Conversation
Complete the half-finished migration that left most query functions recomputing against Neo4j/Owlery on every call. Tracing from the ha_api handlers (not just cached_functions.py) showed 12 handler-reachable functions with no @with_solr_cache at either layer, plus the cross-dataset query_connectivity persisted only in the per-container in-memory cache. - Decorate the missing cacheable *_cached wrappers (cluster_expression, expression_cluster, scrnaseq_dataset_data, individual_neuron_inputs, the similar_morphology family, dataset_images, all_aligned_images, all_datasets, transgene_expression_here, related_anatomy). Leave similar_morphology_userdata uncached (per-session upload id). - Register the five new buckets in expensive_query_types and dataframe_query_types so limited requests slice from the cached full result, and add an all_datasets fixed-key branch for the no-id case. - Persist query_connectivity behind the in-memory cache using a hashed composite key over all five parameters; keep graph post_fn out of the cached payload; wire force_refresh through the handler and worker. - Bump to 1.19.0 so the major.minor cache key rotates on deploy. - Default SolrResultCache to http://vfbquerycache.virtualflybrain.org:80/solr/vfb_json (the dedicated query-cache Solr; port 80 because the Solr native port is firewalled externally) instead of solr.virtualflybrain.org. Add a VFBQUERY_SOLR_URL env override. Data reads in vfb_queries.py are unchanged. No bucket renames (cache keys are stable); force_refresh still bypasses and invalidates. Coverage sweep: 12 uncached handler entry points -> 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the half-finished SOLR result-cache migration so every query-result
function reachable from the HA API handlers is served from the persistent cache
instead of recomputing against Neo4j/Owlery on each call.
What changed
*_cachedwrappers (transcriptomics, thesimilar_morphology sub-types, dataset/image and transgene queries,
individual_neuron_inputs).
get_similar_morphology_userdataleft uncached(per-session upload id).
expensive_query_typesanddataframe_query_typesso limited requests slice from the cached fullresult; add an
all_datasetsfixed-key branch for the no-id case.query_connectivitybehind the in-memory cache via a hashedcomposite key over all five parameters; graph post_fn stays out of the
cached payload;
force_refreshwired through handler and worker.http://vfbquerycache.virtualflybrain.org:80/solr/vfb_json(port 80; thenative Solr port is firewalled externally), with a
VFBQUERY_SOLR_URLoverride. Data reads in
vfb_queries.pyare unchanged.No bucket renames (cache keys stay stable);
force_refresh=truestill bypassesand invalidates.
Verification
prod writes): cold->warm speed-ups 46x-2790x across five query types incl.
query_connectivity; force_refresh recompute confirmed.
failover test failure pre-exists on main.
Note for reviewer
The new cache host was read-verified only; a cache write to it was not tested
(production core, not staging). Confirm a
vfb_query_*doc lands there afterdeploy, or test against a staging core via
VFBQUERY_SOLR_URLfirst. The olddefault
solr.virtualflybrain.orgcore was returning HTTP 500 on writes(
write.lock: Input/output error), which moving the cache off it should fix.