Skip to content

fix: always write root index in v4 format#56

Merged
ddvk merged 2 commits into
ddvk:masterfrom
pmleveque:fix/root-schema-v4-write-regression
Apr 29, 2026
Merged

fix: always write root index in v4 format#56
ddvk merged 2 commits into
ddvk:masterfrom
pmleveque:fix/root-schema-v4-write-regression

Conversation

@pmleveque

@pmleveque pmleveque commented Apr 27, 2026

Copy link
Copy Markdown

Problem

After f5c7b9c, all write operations (mkdir, put, mv) fail with:

400 {"message":"invalid hash"}

on the root docSchema PUT.

Root cause

f5c7b9c correctly reads the server's current schema version in Mirror(), but then propagates that version into IndexReader() and Rehash() for writes. For accounts whose root index was still v3 format, this causes rmapi to write a v3-format root blob — which the server rejects.

In testing, the server still accepted existing v3 per-document docSchemas, but rejected new v3-format root docSchema writes. New root writes appear to require v4 format.

Fix

IndexReader() always produces v4 output. Rehash() is simplified to always SHA-256 the IndexReader() content. SchemaVersion on HashTree is still populated by Mirror() for informational logging but no longer drives write behaviour.

The root self-migrates: after the first successful v4 write, the server's root becomes v4 permanently and future Mirror() calls detect v4 going forward.

RMAPI_FORCE_SCHEMA_VERSION still overrides the serialized schema version for debugging. Note that Rehash() always uses SHA-256 regardless, so forcing v3 is only useful for inspecting failure modes — it does not restore the old v3 write path.

Testing

Added TestRootIndexWritesV4WhenMirroredV3: creates a HashTree with SchemaVersion: SchemaVersionV3 (as mirrored from an old server root), calls IndexReader(), and asserts the output is v4. Also asserts Rehash() produces sha256(IndexReader bytes).

Verified mkdir and put succeed on a live account that previously had a v3 root (was broken after f5c7b9c).

pmleveque and others added 2 commits April 26, 2026 22:01
The server now rejects root docSchema writes that use v3 format,
returning 400 {"message":"invalid hash"} regardless of whether the
hash algorithm matches. Only v4 (SHA-256 of the text content) is
accepted for new writes.

The previous commit (f5c7b9c) correctly detected the server's schema
version from Mirror(), but then applied that same version to writes,
reintroducing v3 writes for accounts whose root was still v3.

Fix: IndexReader() always produces v4 output. Rehash() is simplified
to always SHA-256 the IndexReader() output (v4). SchemaVersion on
HashTree is still populated by Mirror() for informational purposes
but no longer drives write behaviour.

The RMAPI_FORCE_SCHEMA_VERSION env var still overrides both, useful
for debugging.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
TestRootIndexWritesV4WhenMirroredV3 directly captures the regression:
a HashTree with SchemaVersion=v3 (as mirrored from an old root) must
still emit a v4-format root docSchema from IndexReader(), and Rehash()
must produce SHA-256 of that v4 body.

Also expand the IndexReader() comment to clarify that
RMAPI_FORCE_SCHEMA_VERSION only overrides the serialized schema format,
not the hash algorithm — forcing v3 is for inspecting failure modes,
not a supported write path.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@pmleveque

Copy link
Copy Markdown
Author

Additional verification on the root cause: I tested both sides of the earlier hypothesis.

For a v3 root body, HashEntries(parseIndex(body)) matched the URL hash — so the v3 hash algorithm was correct. But the server still rejected the root PUT with 400 {"message":"invalid hash"}. After serializing the root as v4 and hashing the v4 body with SHA-256, the same account successfully completed mkdir and put.

The failure is not a hash mismatch in the v3 sense — it's that the server no longer accepts v3-format bodies for new root docSchema writes at all.

@pmleveque pmleveque mentioned this pull request Apr 27, 2026
@JeeWeetje

Copy link
Copy Markdown

Independent reproducer — same symptom, different account.

Context

I have a Cloude Cowork daily skill that rolls over my last todo (usually yesterday's) to today and only rolls over itms not checked off. The older copy gets back upped in 1.Done folder.

What I did

Hit this on April 26–27 2026 across multiple attempts. All put calls failed with 400 {"message":"invalid hash"} on the root docSchema PUT. Notably, mv between existing folders (e.g. rmapi mv "/To Do" "/1. Done/To Do 2026-04-24") worked — only put was blocked. Read ops (ls, get, geta, refresh, account) all normal. That mv-vs-put asymmetry might narrow down which write paths actually go through the v3 root-rewrite codepath.

Things that did not help, in case it saves anyone time:

  • Clearing ~/Library/Caches/rmapi/tree.cache
  • rmapi refresh immediately before put (no other ops between)
  • Re-authenticating from scratch (rm rmapi.conf, fresh device pairing, fresh JWT)
  • Rebuilding from master HEAD
  • --force, --coverpage=1, --content-only
  • Different upload paths (/, /8. Temp/), different filenames, varied content (1 KB hello-world PDF, the original .pdf extracted from a working in-cloud notebook, full .rmdoc bundle)

Trace excerpt with RMAPI_TRACE=1:

PUT /sync/v3/files/<sha256>
rm-filename: root.docSchema
content-type: text/plain; charset=UTF-8
x-goog-hash: crc32c=...
→ HTTP 400 {"message":"invalid hash"}

Account: SyncVersion: 1.5, root presumably v3 (account created earlier this month, never explicitly migrated). Will build the fix branch and report back.

@JeeWeetje

Copy link
Copy Markdown

Confirmed — fix branch resolves it.

Built pmleveque:fix/root-schema-v4-write-regression and re-ran the same put that was 400-ing minutes earlier on master:

$ rmapi put hello.pdf /8. Temp/
uploading: [hello.pdf]...OK

Followed by a real-world mv + put round-trip on the same account that was previously broken — both succeeded, root self-migrated to v4 as predicted. Ran another put afterwards: still works, so the migration is sticky.

LGTM. Thanks for the fast diagnosis and clean fix.

@charlieellington

Copy link
Copy Markdown

Hitting this exact bug today on v0.0.32 / macOS. ls works fine, but every put / mkdir / mv fails on the root docSchema PUT with:

PUT /sync/v3/files/<hash> HTTP/1.1
→ HTTP/2.0 400 Bad Request
{"message":"invalid hash"}

Confirms the v3-format root index is no longer accepted server-side, matching your root-cause analysis. Thanks for the patch — would be great to see this merged so a v0.0.33 can ship.

@rmitchellscott

Copy link
Copy Markdown

@ddvk can you take a look at this one and cut a release if good to go?

@ddvk ddvk merged commit a28ac74 into ddvk:master Apr 29, 2026
1 check passed
madhavsuresh added a commit to madhavsuresh/homebrew-rmsync that referenced this pull request Apr 30, 2026
Bump from v0.0.32 → v0.0.33. Released 2026-04-30.

This release ships ddvk/rmapi#56 ("fix: always write root index in
v4 format"), which fixes the HTTP 400 on every put/mkdir/mv that
v0.0.32 had against post-2026-04-rollout cloud accounts. v0.0.32's
schema-v4 implementation correctly *read* the server's current
schema version, but then propagated that into IndexReader() and
Rehash() — so for accounts whose root was still v3, rmapi tried
to write a v3-format root blob and the cloud rejected it.

v0.0.33 always writes v4-format root blobs regardless of what
Mirror() detected. The root self-migrates after the first
successful v4 write.

Sha256s computed via:
  curl -fsSL <release-url> | shasum -a 256
on each of the four prebuilt archives (macOS arm64/intel, Linux
arm64/amd64).

Done manually rather than via the auto-bump workflow because the
workflow_dispatch trigger hadn't yet been indexed by GitHub
Actions (workflow added in last push; async parser hadn't run).
The cron will pick up future releases.

Upstream changelog: https://github.com/ddvk/rmapi/releases/tag/v0.0.33

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
nokry56 pushed a commit to nokry56/readwise-to-remarkable that referenced this pull request May 26, 2026
reMarkable cloud bumped the document-index schema in May 2026 to require
a .docSchema suffix on the index fetch. v0.0.33 doesn't send it, so all
tree-building operations (ls, mkdir, put, rm) fail with:
  failed to mirror was not ok: request failed with status 400
Fixed in ddvk/rmapi#56 -> v0.0.34 (2026-05-22). See issue #58.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants