read_vortex returns rows from the wrong chunk after a same-session COPY ... (FORMAT vortex)

[moshap-firebolt]

Summary

After writing a Vortex file with COPY (...) TO 'f.vortex' (FORMAT vortex), reading it back with read_vortex('f.vortex') in the same DuckDB session returns rows in a scrambled physical order, offset by whole 65536-row chunks. The set of rows is correct (count/sum/filter all exact); only each row's physical position is wrong. Reading the identical file from a fresh process is correct.

This silently corrupts merge-on-read positional deletes on Vortex-backed tables (e.g. DuckLake with data_file_format='vortex'): the writer computes delete positions from a same-session read-back, so the recorded positions point at the wrong rows. No error is raised.

Environment

• duckdb-vortex main @ 0867246, extension version 0.1.0, statically linked.
• DuckDB 1.5.x (source_id d8cdaa33).
• Local filesystem, single node — no S3, no DuckLake needed to reproduce.
• Reproduces at threads=1 (smaller offset); magnitude grows with thread count.

Minimal repro

COPY (SELECT range AS id FROM range(1000000)) TO 't.vortex' (FORMAT vortex);
SELECT id FROM read_vortex('t.vortex') LIMIT 1;   -- expected 0; observed 262144 / 196608 / 327680 (varies per run)

Observed LIMIT 1 across three runs: 262144, 196608, 327680 — all exact multiples of 65536 (chunks 4, 3, 5). Nondeterministic.

Unarguable form (no ORDER BY semantics involved)

Two reads of the same immutable file in one session disagree on which row is at position 100000:

COPY (SELECT range AS id FROM range(1000000)) TO 't.vortex' (FORMAT vortex);
SELECT (SELECT id FROM read_vortex('t.vortex') LIMIT 1 OFFSET 100000)                                       AS via_offset,  -- 100000 (correct)
       (SELECT id FROM (SELECT id, row_number() OVER () - 1 p FROM read_vortex('t.vortex')) WHERE p=100000) AS via_rownum;  -- 689824 (WRONG)

Control — identical queries in a fresh process are correct

-- new duckdb process, file already on disk:
SELECT (SELECT id FROM read_vortex('t.vortex') LIMIT 1 OFFSET 100000)                                       AS via_offset,  -- 100000
       (SELECT id FROM (SELECT id, row_number() OVER () - 1 p FROM read_vortex('t.vortex')) WHERE p=100000) AS via_rownum;  -- 100000 (correct)

So the defect is in in-session state after a write, not the bytes on disk.

What is / isn't affected

• Wrong: physical row position on a same-session read-after-write (positional deletes, row_number(), LIMIT/OFFSET-pushdown consistency).
• Correct: all set-level results (count(*)=1000000, sum(id)=499999500000, count(*) FILTER (WHERE id<100000)=100000) and any read from a fresh process.

Likely cause

The read path appears to serve a same-session scan from writer-side in-memory chunk state whose ordering doesn't match the on-disk physical layout (offset is always a whole 65536-row chunk; magnitude scales with thread count). A fresh process, reading purely from disk, orders chunks correctly.

Self-contained repro script

#!/usr/bin/env bash
# DUCKDB=/path/to/duckdb ./repro.sh
set -euo pipefail
DUCKDB="${DUCKDB:-duckdb}"; TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT; F="$TMP/t.vortex"
for i in 1 2 3; do
  "$DUCKDB" -noheader -list -c "COPY (SELECT range AS id FROM range(1000000)) TO '$F' (FORMAT vortex);
    SELECT 'run $i LIMIT1='||(SELECT id FROM read_vortex('$F') LIMIT 1);"; rm -f "$F"
done
"$DUCKDB" -c "COPY (SELECT range AS id FROM range(1000000)) TO '$F' (FORMAT vortex);
  SELECT (SELECT id FROM read_vortex('$F') LIMIT 1 OFFSET 100000) via_offset,
         (SELECT id FROM (SELECT id,row_number() OVER ()-1 p FROM read_vortex('$F')) WHERE p=100000) via_rownum;"

[moshap-firebolt]

cc @myrrc — flagging for your attention (Vortex DuckDB read-after-write positional ordering).