External ghost metadata provider (import/export + periodic refresh)#88
Open
tomjn wants to merge 9 commits into
Open
External ghost metadata provider (import/export + periodic refresh)#88tomjn wants to merge 9 commits into
tomjn wants to merge 9 commits into
Conversation
Mods previously kept their hash, options and sides only in the in-memory %cachedMods hash, lost on restart - unlike maps which persist full info via mapInfoCache.dat and mapHashes.dat. Mirror the map machinery for mods: - modHashes.dat: version-keyed mod checksums (fast table), dumped in dumpDynamicData alongside mapHashes - modInfoCache.dat: mod options and sides (Storable), version-independent - accessors getCachedModInfo/cacheModInfo/getModHash/saveModHash - persist mod hash and info on first archive load in loadArchivesPostActions Foundation for ghost mods (hosting a game whose archive is absent).
getModHash/getModOptions/getModSides only consulted the in-memory %cachedMods,
so a mod whose archive is absent returned no hash/options/sides. Mirror the map
accessors (getMapHash/getMapOptions): prefer live in-memory data, then fall back
to the durable mod cache (getModHash/getCachedModInfo) keyed by spring version.
No behavior change on existing installs until the cache is populated (empty
modHashes.dat/modInfoCache.dat still yield 0/{}/[]).
A dedicated host can now host a configured game whose archive is absent, using the durable mod cache, mirroring allowGhostMaps for maps. - new allowGhostMods preset setting (etc/spads.conf template + spadsSectionParameters) - canUseGhostMod(): gated on allowGhostMods + dedicated server, and requires BOTH a version-matching cached hash AND cached info (options+sides) so a battle is never opened with incomplete mod metadata (fail-loud) - updateTargetMod() falls back to the ghost mod when the configured mod is not found among locally available mods - loadArchivesPostActions() resolves to the ghost mod on initial load when the configured mod archive is absent Existing configs must add allowGhostMods (handled by the normal config update flow, as with any new setting). Setting docs (doc/*.html) are generated and not updated here.
Foundation for Part C (external metadata provider). Adds to SpadsConf: - importedMetadata registry (importedMetadata.dat) tracking which cached entries came from import; absence means local/archive-derived and authoritative - getMetadataForExport(): flat version-keyed structure for JSON serialization - importMetadataStructure(): gap-fill merge - never overwrites local entries, refreshes previously-imported ones; only imports hashes for the host spring major version (map/mod info is version-independent) - local saves (saveMapHash/saveModHash/cacheModInfo/cacheMapsInfo) clear the imported flag so a real archive load reclaims authority (self-heal) Map info import is skipped when mapInfoCache is shared across instances (logged). Methods are unused until wired to config/command in following commits.
- new global settings ghostMetadataSources (semicolon-separated list of local file paths and/or http(s) URLs) and ghostMetadataRefreshDelay (registered in globalParameters + spads.conf template, default empty/0 = disabled) - fetchMetadataDump(): reads a dump from a file or URL (HTTP::Tiny) and decodes JSON, mutating no state (safe to fork) - importGhostMetadataFromSources(): imports all sources via importMetadataStructure() then rebuilds the ghost map list - called at startup after archive load, gated on ghostMetadataSources Existing configs must add the two settings (normal config update flow).
- RefreshGhostMetadata SimpleEvent timer (interval = ghostMetadataRefreshDelay minutes), registered when the delay is set and sources are configured, torn down on exit alongside the other timers - refreshGhostMetadata(): fetches all sources in a forked child (network I/O off the main loop) and applies results in the parent via importMetadataStructure, refreshing imported/previously-imported entries while leaving local entries untouched; rebuilds the ghost map list when anything changed
Admin command (level 120, like reloadArchives) that writes the cached map and mod metadata (version-keyed hashes + options/sides/start positions) as pretty, canonical JSON to a file in the instance directory, so a well-stocked host can seed others via ghostMetadataSources. - hExportMetadata handler + registration + commands.conf entry + help.dat entry - optional filename argument (basename only, defaults to metadataExport.json)
…ttings.dat Adds settings-help entries for the two new global settings introduced by the external metadata provider. The exportMetadata command is already documented in help.dat. The generated doc/*.html is produced by 'spads.pl --doc' at release time and is not regenerated here.
The metadata import path pulls data from operator-configured URLs into the host, which widens the trust boundary. Two hardening changes: - fetchMetadataDump now sets HTTP::Tiny verify_SSL => 1, so https sources are certificate-verified (HTTP::Tiny's default varies by version; rely-on-default was fragile and could allow a MITM to serve forged metadata). http sources and local files are unaffected; if TLS support is missing the fetch fails closed. - importMetadataStructure now validates every entry before storing it. Hashes must be plain signed-integer scalars. Map/mod info blobs are recursively checked to reject control characters (which could inject lines into the generated Spring start script via option keys/values), unexpected reference types, and pathologically deep nesting. Rejected entries are counted and logged rather than stored. Provenance rules are unchanged (local data still wins). Validator verified with a direct unit test covering newline injection in keys and values, code refs, and deep nesting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on top of #87 (durable mod metadata cache + ghost mods). This branch is
stacked on that one, so it includes the commits from #87 as well as the Part
C work below; if #87 is merged first, only the Part C commits remain.
Part C makes the metadata cache shareable between hosts, so a dedicated relay can
be seeded with map/mod metadata it never loaded locally, then kept current.
Changes (Part C)
!exportMetadata [<file>], admin level 120): writes the cachedmap/mod metadata (version-keyed hashes + options/sides/start positions) as
pretty, canonical JSON to the instance directory.
ghostMetadataSourcessetting (semicolon-separated list oflocal file paths and/or
http(s)://URLs, fetched withHTTP::Tiny). Importedat startup.
ghostMetadataRefreshDelaysetting (minutes). ASimpleEvent timer re-fetches the sources via
forkCall(network I/O off themain loop) and applies the results in the parent process.
importedMetadata.datregistry recordswhich cached entries came from import. Local (archive-derived) entries are never
overwritten; previously-imported entries are refreshed. A real archive load
reclaims authority (self-heal). Only hashes matching the host's spring major
version are imported; map/mod info is version-independent.
All new settings default to disabled, so existing hosts are unaffected until an
operator opts in. JSON via core
JSON::PP; no new dependencies. Map info importis skipped when
mapInfoCacheis shared across instances (logged).Testing status (please read)
Like #87, this is compile-checked only (
perl -c), not runtime-tested - therepo has no automated tests and I could not run SPADS in my environment. Someone
has offered to test this on a real host; I wanted the branch available for that.
Review and correction very welcome, particularly on the import/refresh paths and
the start-up ordering relative to the ghost map list rebuild.