Skip to content

fix(rag)!: preserve string boundaries when indexing a string-under-axis Ragged - #72

Merged
d-laub merged 5 commits into
mainfrom
worktree-issue-71-string-under-axis-getitem
Jul 27, 2026
Merged

fix(rag)!: preserve string boundaries when indexing a string-under-axis Ragged#72
d-laub merged 5 commits into
mainfrom
worktree-issue-71-string-under-axis-getitem

Conversation

@d-laub

@d-laub d-laub commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Closes #71.

Integer indexing on a string-under-axis Ragged concatenated a whole group into one blob, dropping the per-string boundaries already sitting in str_offsets. s.lengths[0] == 2 but len(s[0]) == 3.

Two sites, not one

  • Ragged.__getitem__ integer branch — the one reported.
  • _getitem_record_rows integer branch — worse, and the one GenVarLoader actually hits. RaggedVariants is a record Ragged whose alt/ref are opaque-string fields sharing offsets with numeric start/ilen, so a peeled row returned a concatenated S1 char array (not even bytes) sitting next to a per-variant numeric array in the same dict:
rec[0] -> {'alt':   array([b'A', b'G', b'G'], dtype='|S1'),   # 3 chars
           'start': array([1, 2], dtype=int32)}                # 2 variants

_getitem_record_rows_r2 delegates to the first site and inherits the fix; it's covered by a test.

Fix

Both now peel to the standalone opaque-string layout (offsets == [], str_offsets set, shape == (k,)), which Spec C already defines as the zero-real-level special case of string-under-axis — so this is what "peel one real level" already means in the layout algebra, not a new return type. Shared helper _peel_string_row, mirroring the narrowing _slice_contig_string already does for slices. Data stays zero-copy; only the small (k+1,) offsets slice is copied, rebased to zero as is_contiguous requires.

Result:

lengths     : [2 1]
numeric[0]  : [10 20]        len 2
string [0]  : [b'A', b'GG']  len 2
zip aligned : [(10, b'A'), (20, b'GG')]

This makes zip(rv.start[0][h], rv.alt[0][h]) correct in GenVarLoader (mcvickerlab/GenVarLoader#330).

Alternatives considered and rejected in the spec: an object array of bytes (same ergonomics but allocates k Python objects per row access, against the repo's hot-path rule), to_chars()[i] (forces consumers to re-assemble strings), and raising NotImplementedError (indexing is far more central than the to_numpy()/to_packed() that set that precedent).

Breaking

The integer index returns a Ragged of strings rather than one concatenated bytes. b"".join(s[i]) recovers the old value. major_version_zero is set, so this bumps 0.21.2 → 0.22.0.

test_string_under_axis_integer_index pinned the old behavior but used exactly one string per group, so it could never distinguish the two interpretations — that's why the bug survived. It has been rewritten.

Testing

16 new/rewritten tests: the issue repro, numeric/string length parity, multi-dim (batch, ploidy, ~variants), record rows with a multi-byte (indel) entry so 1-byte coincidence can't mask a regression, zero-copy, empty group, to_chars() agreement, negative/OOB indexing, and a regression guard that the standalone string case still returns bytes.

25 of 26 test files pass. tests/test_coords.py fails to collect with OSError: Could not find/load shared object file from llvmlite — pre-existing environment breakage, verified identical on unmodified main.

Spec: docs/superpowers/specs/2026-07-27-ragged-string-getitem-design.md
Plan: docs/superpowers/plans/2026-07-27-ragged-string-getitem.md

🤖 Generated with Claude Code

d-laub added 5 commits July 27, 2026 10:45
Integer indexing on a string-under-axis Ragged concatenated the whole
group into one bytes, dropping the per-string boundaries already held in
str_offsets. It now peels to the standalone opaque-string layout (k,),
so len(s[i]) == s.lengths[i] and s[i][j] is one string.

BREAKING CHANGE: Ragged.__getitem__ with an integer on a string-under-axis
array returns a Ragged of strings, not one concatenated bytes. Use
b"".join(s[i]) for the old value.

Refs #71
_getitem_record_rows returned a string field as the raw concatenated S1
buffer, so a peeled row mixed a 3-char array with a 2-element numeric
array. String fields now peel to the standalone opaque-string layout,
giving every field of the row the same length.

BREAKING CHANGE: peeling a record row returns opaque-string fields as a
Ragged of strings, not a concatenated S1 array.

Refs #71
@codspeed-hq

codspeed-hq Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will regress 1 benchmark

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 10 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
test_bench_ragged_cres 978.1 µs 1,465.7 µs -33.27%
test_bench_baseline_ragged_short_alleles 4.2 ms 2.7 ms +57.94%
test_bench_baseline_ragged_cres 8.2 ms 6.2 ms +33.01%
test_bench_dense_batch 3.4 ms 2.9 ms +17.24%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing worktree-issue-71-string-under-axis-getitem (e970469) with main (c7f4a5a)

Open in CodSpeed

@d-laub
d-laub marked this pull request as ready for review July 27, 2026 18:18
@d-laub
d-laub merged commit 25eb570 into main Jul 27, 2026
8 of 9 checks passed
@d-laub
d-laub deleted the worktree-issue-71-string-under-axis-getitem branch July 27, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ragged.__getitem__: integer index on a string-under-axis Ragged concatenates the whole group into one bytes, dropping per-string boundaries

1 participant