feat: add Tsundoku release-source plugin#34
Merged
Merged
Conversation
Add release-tsundoku, an api-feed release source that will consume a Tsundoku instance's incremental series feed and announce new volume/chapter coverage for tracked series, matched by exact external IDs (no fuzzy matching). This change lays the foundation: package scaffolding, the manifest (api-feed capability declaring all supported Tsundoku external-ID providers, plus a config schema with a required baseUrl and optional defaultLanguage/pageLimit/requestTimeoutMs), config-driven initialization that captures the host RPC and KV-store clients, and single-source registration with a METHOD_NOT_FOUND retry. The poll handler is a no-candidate stub for now; the feed walk, cursor persistence, and matching land in follow-up changes. Tests cover base URL normalization and source registration. Wire the plugin into the build and CI matrices, docker-compose (backend and worker dist mounts, dev-watch volume, build and dev commands), and the official plugin gallery. No Makefile change is needed since it auto-discovers plugins via a glob.
Add the building blocks for walking the Tsundoku series feed: wire types for the feed response, a feedUrl builder, and fetchFeedPage, which wraps fetch with a hard timeout and JSON parsing and returns a discriminated ok/error result (passing upstream status through for host-side backoff, and rejecting malformed bodies). Add loadCursor/saveCursor over the plugin KV store so the feed position survives between polls. Both are tolerant of a missing key and storage errors: a failed read restarts the walk from the beginning and a failed write is logged without aborting the poll, which is safe given keyset pagination plus host-side dedup. Includes tests for the fetcher and cursor helpers.
Add the matcher that resolves Tsundoku feed items to tracked Codex series by exact external ID. buildIndex turns the host's tracked-series rows into a "provider:id" -> series reverse index, and matchItem sweeps an item's provider IDs in priority order (mangabaka first) for a stable, fuzzy-free match. Add the candidate mapping that turns a matched feed item into a release candidate: coverage spans pass through onto the volume/chapter axes, the external release id is keyed on the coverage high-water mark so a new row fires only when the frontier advances, and confidence is fixed at 1.0 since the match is exact. Includes tests for both modules.
Wire the Tsundoku feed walk end to end. Each poll builds an exact-match index from the host's tracked series, loads the saved feed cursor, then walks the feed page by page: matching items by external ID, recording hits as release candidates, and persisting the cursor after every page so an interrupted walk resumes cleanly. Per-candidate record failures are logged and skipped, the walk stops on a fetch error with the cursor preserved, and the poll returns parsed/matched/recorded/deduped counters plus the worst upstream status for host backoff. Add a README covering setup, configuration, the exact-match model, cursor behavior, and known limitations (default language, incremental backfill gap, high-water dedup). Includes a poll test suite covering multi-page walks, cursor persistence, host dedup, no-match skips, record-error tolerance, and fetch-error handling.
…n cursor storage Two related fixes for release-source plugins, which run as system plugins with no per-user context. Host: add a readiness barrier to reverse-RPC dispatch. A plugin can call a reverse method (e.g. releases/register_sources) from onInitialize the moment it returns its manifest — before the host finishes installing the post-initialize context — and the dispatcher bounced it with METHOD_NOT_FOUND, leaving registration to flaky plugin-side retries. The context now carries a readiness notification that set_capabilities fires; an early reverse-RPC parks until capabilities are installed (bounded by a timeout) and then dispatches normally, falling back to the previous METHOD_NOT_FOUND only if initialization never completes. This fixes source registration for every release-source plugin, not just one. Plugin (tsundoku): persist the feed cursor in the source's etag slot via releases/source_state instead of the plugin KV store. The KV store is scoped per user, but a release source has no user context, so storage/* is rejected with "Storage is not available". The host hands the saved cursor back as the poll request's etag and persists the etag returned on the poll response, so the per-source state slot is the correct single-row bookmark. Also surface a hard error when the first feed page can't be fetched so the source reports last_error rather than a silent empty poll. Includes host tests for the readiness barrier (early call succeeds once capabilities land) and updates the plugin tests for the new cursor source.
…sundoku cursor The plugin KV store (storage/*) was keyed by user_plugin_id, so system plugins — which have no user context — were rejected with "Storage is not available". Add a parallel per-plugin store so system plugins (e.g. release sources) get durable key-value storage. - DB: new plugin_data table keyed by plugin_id (FK to plugins, cascade, unique (plugin_id, key), TTL + indexes), a matching entity, and PluginDataRepository mirroring the per-user repository. - Services: StorageRequestHandler is now scope-aware (user vs system) behind a normalized entry type, with a system constructor; the system spawn path installs a system-scoped storage handler keyed by the plugin row. The per-user storage path is unchanged. - Tasks: the expired-data cleanup now sweeps both the user and system tables. Migrate the Tsundoku release plugin's feed cursor from the source-state etag slot to this KV store (feed_cursor), removing the bespoke source_state round-trips. The source-state etag slot remains for its intended use (HTTP ETag conditional GET); only the cursor moved. Also drop the registerSources retry loop, now redundant thanks to the host's reverse-RPC readiness barrier. Includes repository, handler, and plugin tests.
In distributed mode the worker records the entity events a task emits and the web TaskListener replays them to SSE subscribers — but only for tasks that completed. Failed tasks dropped their events, so a release poll that errored never delivered its `release_source_polled` event and the Release tracking UI didn't refresh until a manual reload. Carry recorded events through the failure path: `mark_failed` now accepts and persists result data, `fail_task` passes the recorded events into it, and the TaskListener replays recorded events on the Failed terminal state as well as Completed. Adds repository tests for the new result-data persistence.
Matching on the first shared provider id was unsafe: non-MangaBaka providers (MAL, MangaUpdates, …) sometimes share or merge ids across distinct series, so a single matching low-trust id could attribute one series' coverage to another. Match by weighted voting instead. For each provider both the feed item and a candidate series carry, an agreeing id adds its weight (MangaBaka 3, AniList 2, rest 1) and a disagreeing one subtracts it; a series matches only on a net-positive score, so a trusted disagreement (different MangaBaka ids) overrides a sloppy agreement (a shared MAL id). Confidence is derived from the score, and a post-walk cross-item pass keeps at most one feed entry per Codex series — best score wins, and genuinely ambiguous ties between different series are skipped rather than mis-attributed. Candidate confidence/reason now reflect the vote instead of a hardcoded 1.0. Tests cover the voting, conflict rejection, ambiguity, and cross-item resolution.
Replace the catalog-wide GET + persisted cursor with Tsundoku's filtered POST /series/feed, posting the tracked series' provider:externalId set so the response contains only those series. There's no persisted cursor: each poll re-walks the tracked set's current coverage and relies on host-side dedup to suppress unchanged releases. This makes newly tracked series backfill automatically and untracked ones drop out, with no cursor bookkeeping — closing the new-series backfill gap by construction. When the tracked set is empty the poll skips entirely (an empty filter would mean "whole catalog" upstream). The system KV store and its cursor helpers are no longer needed and are removed from the plugin. Also fix a provider-name mismatch: Codex stores some sources under different names than the feed uses (e.g. myanimelist vs mal, animeplanet vs anime_planet, animenewsnetwork vs anime_news_network). Add a Codex → Tsundoku name map; requiresExternalIds now declares the Codex names the host filters on, and the matcher translates to the feed's names so the filter and the vote line up across all eight providers. Tests and the README are updated for the POST flow and the name mapping.
Deploying codex with
|
| Latest commit: |
5cb9152
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://a8a49cb9.codex-asm.pages.dev |
| Branch Preview URL: | https://tsundoku-plugin.codex-asm.pages.dev |
The `mark_failed` signature gained a fourth `result_data: Option<JsonValue>` parameter when failed-task event replay was added, but the task queue tests were not updated and failed to compile. Pass `None` at each call site since these tests don't exercise event replay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new built-in release source that ingests Tsundoku's series feed and announces new volume and chapter coverage for the series a user tracks. Series are matched to the Codex catalog by their external provider IDs, with no fuzzy title matching, and each poll re-evaluates the full tracked set so newly tracked series are picked up automatically. Ships alongside host improvements that make system plugins register reliably and give them durable storage.
Motivation
Tsundoku aggregates per-series volume and chapter coverage across many upstream providers and exposes it as a single catalog-wide feed carrying provider IDs and merged coverage spans. Codex had no way to consume it. Compared to the per-series RSS approach of the existing MangaUpdates source, one filtered request covers the whole tracked set and the provider IDs allow deterministic, fuzzy-free matching that maps directly onto Codex's release candidate model.
Changes
baseUrl(required),defaultLanguage(defaulten),pageLimit(default 100), andrequestTimeoutMs(default 10000). The feed is public, so no credentials are required.Notes
en) since the feed carries none; per-series language gating still applies host-side.