PRD: sheet-compressor — multi-language SheetCompressor library

[mythopoeic]

Problem Statement

People want to feed spreadsheets to LLMs, but raw sheets are enormous in tokens — mostly
homogeneous filler, repeated values, and sparse layout that the model doesn't need spelled out
cell by cell. The SpreadsheetLLM paper's SheetCompressor solves this, but there's no clean,
reusable implementation a developer can drop into their own project. The author has rebuilt it
several times in different languages, and those copies have drifted: some hard-code invoice
business logic (a "Qty" column, "Total"/"Subtotal" row splitting), one had a live API key
committed in plaintext, only the Office Script version handles charts, and none expose a stable,
consistent library API. A developer who finds one of these can't simply pip install /
npm install it and compress a sheet — and if they use two of them in different languages, they
get different output for the same sheet.

Solution

A single repository, sheet-compressor, providing pluggable implementations of the paper's
SheetCompressor encoding in six languages — TypeScript/Node (reference), Python, C#, Go, VBA, and
Office Script — that all produce identical output for the same input, verified against one shared
golden-fixture corpus. Each implementation is a pure compression core (a function over an
in-memory grid) plus an optional thin adapter for that ecosystem's common spreadsheet library, so
a developer can pass either their own cell data or an .xlsx file. The library makes no LLM calls
and pulls in no provider SDKs; instead it ships prompt templates that teach an LLM how to read the
compressed output. Charts and graphs are represented portably as text descriptors in every
language. The invoice-specific logic is dropped; the leaked key never enters history.

User Stories

1. As a developer with a 50,000-token spreadsheet, I want to compress it into a compact text
   form, so that I can fit it into my LLM's context window affordably.
2. As a developer, I want to npm install / pip install / go get / add a NuGet package and
   call one function, so that I can compress a sheet without copying source code.
3. As a developer who already has cell values in memory, I want to pass a plain grid to the
   compressor, so that I don't have to round-trip through a file.
4. As a developer with an .xlsx file, I want an optional adapter that reads it into the grid for
   me, so that I don't have to wire up a spreadsheet parser myself.
5. As a developer who doesn't want heavy dependencies, I want the core to work with zero required
   third-party packages, so that it drops cleanly into a constrained project.
6. As a developer, I want all three encodings (structural-anchor skeleton, inverted index, format
   aggregation) returned, so that I can choose the best compression for my sheet's shape.
7. As a developer, I want each encoding as both a raw string and a JSON object, so that I can pick
   the form that fits my prompt or my downstream parsing.
8. As a developer, I want a token-count estimate for the raw sheet and for each encoding, so that
   I can see and report the compression ratio.
9. As a developer targeting a specific model, I want the token count to use a real tokenizer when
   my ecosystem has one (tiktoken, gpt-tokenizer, SharpToken, tiktoken-go), so that the number is
   accurate rather than a guess.
10. As a developer in an environment with no tokenizer (Office Script, VBA), I want a documented
    heuristic fallback, so that I still get a reasonable estimate.
11. As a developer, I want to inject my own token-counter function, so that I can match whatever
    model/encoding I actually use.
12. As a developer, I want documentation that recommends which encoding to use when, so that I
    don't have to reverse-engineer the trade-offs.
13. As a developer with charts in my sheet, I want each chart represented as a text descriptor
    (type, anchor range, title, data ranges, series/axis names), so that the LLM knows the chart
    exists, where it sits, and what it plots.
14. As a developer, I want chart descriptors to appear in the compressed output in a consistent
    syntax across all six languages, so that my prompts work regardless of language.
15. As a developer on a rendering-capable host (Office Script, desktop Excel via VBA), I want the
    option to also attach a base64 image of a chart, so that I can use a multimodal model.
16. As a developer using a text-only model, I want the descriptor (not an image) by default, so
    that I get useful chart information without multimodal overhead.
17. As a developer integrating the LLM step, I want a "reader" prompt for each encoding that
    teaches the model how to decode it, so that the model interprets the compressed text correctly.
18. As a developer, I want ready-made task prompts (table/region detection, cell-value lookup,
    sheet Q&A), so that I have working starting points instead of writing prompts from scratch.
19. As a developer, I want a prompt snippet explaining how to read CHART(...) descriptors, so that
    the model can reason about charts.
20. As a maintainer, I want all six implementations verified against one language-neutral golden
    corpus, so that they can't silently diverge.
21. As a maintainer, I want a single command per language to run the conformance corpus and diff
    against the goldens, so that CI catches drift on every change.
22. As a maintainer, I want to regenerate the golden fixtures from the reference implementation in
    one step, so that an intentional algorithm change propagates to all languages' tests at once.
23. As a maintainer, I want anchor detection behind a swappable strategy interface, so that I can
    upgrade it later (toward fuller paper fidelity) without changing the public output contract.
24. As a maintainer, I want the input contract to allow optional per-cell metadata (data type now,
    style flags later), so that the Phase 2 styling-aware anchor detection is a drop-in, not a
    rewrite.
25. As a contributor, I want a written spec (SPEC.md) describing the algorithm precisely, so that I
    can port it to a new language without reading another language's source.
26. As a user evaluating the project, I want a README that credits the SpreadsheetLLM paper and
    states this is an independent implementation, so that provenance is clear.
27. As a security-conscious user, I want assurance that no secrets are committed and that the old
    reference sources are excluded from history, so that I can trust the repo.
28. As a developer, I want the package named after what it is (the SheetCompressor component), so
    that I'm not misled into thinking it's Microsoft's official release.
29. As a developer processing a multi-sheet workbook, I want to compress each sheet individually,
    so that I can assemble results however my application needs.
30. As a developer, I want consistent A1-style cell addressing in the output that respects the
    sheet's origin offset, so that addresses in the compressed form map back to real cells.
31. As a developer, I want empty rows/columns dropped from the kept region appropriately, so that
    the compressed output isn't padded with blanks.
32. As a developer reading the inverted index, I want repeated values merged into minimal cell
    ranges, so that the encoding is as compact as possible.
33. As a developer reading the format aggregation, I want adjacent same-type cells merged into
    ranges with a type label, so that large numeric blocks collapse.

Implementation Decisions

• Scope: compression core + prompt templates only. No LLM/HTTP calls, no provider SDKs, no
  invoice/Qty domain logic. The drifted Python and VBA business logic (RemoveZeroQtyRows,
  SplitByTotal/SplitSheet) is intentionally removed.
• Reference behavior is the paper pipeline as embodied in the existing Office Script
  SheetCompressor.osts (structural-anchor extraction → inverted index → format aggregation).
• Six implementations: TypeScript/Node, Python, C#, Go, VBA, Office Script.
• Architecture: pure core + optional adapters. The compression core is a pure function over an
  in-memory grid and has zero required dependencies. Each package additionally ships a thin,
  optional adapter for its ecosystem's common spreadsheet library (SheetJS, openpyxl,
  ClosedXML/EPPlus, excelize, Excel COM, ExcelScript) that builds the grid + chart descriptors from
  an .xlsx.
• Input contract: rows of cell text + the sheet's origin offset (top-left row/col) + OPTIONAL
  per-cell metadata (data type now; reserved for style flags later) + an OPTIONAL list of
  ChartDescriptors.
• ChartDescriptor: a portable structure carrying name, type (bar/line/pie/…), anchorRange
  (A1), title, dataRanges, and series/axis names. Rendered into the encodings as a compact
  text token, e.g. CHART(bar)@B5:F20 title="Sales" data=A1:D10 series=[Q1,Q2]. Base64 image
  rendering is an optional, host-only extra (Office Script; VBA via COM) and is never required.
• Output contract (identical across languages): a result object exposing all three encodings —
  anchor skeleton, inverted index, format aggregation — each as a raw string and a JSON form, plus
  a token-estimate per stage and a raw-sheet baseline.
• Anchor detection is a swappable strategy behind a stable interface. Phase 1 strategy uses
  grid-only cues: value heterogeneity (unique ÷ non-empty per row/col) combined with data-type
  transitions between adjacent rows/columns, plus a k-neighborhood keep window around anchors.
  Phase 2 (roadmap, out of scope for the first release) adds styling-aware detection
  (borders/bold/merges/number-formats) fed through the optional per-cell metadata.
• Token counting is performed by an injectable counter. Each package's default wiring uses a
  real tokenizer when the ecosystem has one (tiktoken / gpt-tokenizer or js-tiktoken / SharpToken /
  tiktoken-go), declared as an optional/peer dependency so it is accurate but not forced; Office
  Script and VBA fall back to one shared heuristic defined in the spec. Default encoding is
  o200k_base (GPT-4o family), configurable.
• Prompt templates ship as plain strings/constants: a per-encoding "reader" explainer (anchor /
  inverted-index / format-aggregation), a small set of task templates (table/region detection,
  cell-value lookup, sheet Q&A), and a snippet for interpreting CHART(...) descriptors. Authored
  once in a shared prompts/ source and mirrored into each package so they stay consistent.
• Per-sheet scope. The core compresses one sheet; workbook handling is "iterate sheets," not a
  merged mega-encoding.
• Repository layout (monorepo): spec/ (SPEC.md), fixtures/ (golden corpus), prompts/
  (shared prompt source), packages/<language>/, docs/agents/ (skills config). MIT licensed;
  README credits the paper. The local reference sources stay gitignored and out of history.
• Build order: write SPEC.md + the TypeScript reference core (which generates the golden
  corpus), then port Python → C# → Go → VBA → Office Script against the goldens.

Testing Decisions

• What makes a good test here: assert external behavior — the encodings, token estimates, and
  chart descriptors a given input produces — never internal helpers or intermediate data
  structures. Tests are driven by inputs and expected outputs, not by inspecting how the
  compressor reaches them.
• Seam 1 — compress() core (primary). The pure compress(grid, options) → result function is
  the highest, most language-neutral seam. The shared golden-fixture corpus (language-neutral input
  grids + chart descriptors, with expected compressed outputs) drives it directly in every
  language; each language's conformance test diffs its output against the goldens. This is where the
  bulk of testing lives and is what guarantees cross-language consistency.
• Seam 2 — adapter readSheet(file) → { grid, origin, cellMeta?, charts[] }. Host-specific, so
  tested per language with a few tiny sample .xlsx files, asserting only the produced grid + chart
  descriptors — NOT the compression. This isolates host coupling from the core tests.
• Seam 3 — injectable token counter. Assert the shared heuristic is deterministic and that an
  injected real tokenizer is actually used in place of the heuristic.
• Deliberately not a seam: the anchor-detection strategy is exercised through compress() by
  selecting a strategy in options, not tested directly — so the Phase 2 upgrade is validated by the
  same golden corpus rather than brittle internal tests.
• Prior art: the existing Office Script SheetCompressor.osts already computes per-stage token
  counts and emits the three encodings; its behavior is the seed for the reference implementation
  and the first golden fixtures.
• CI: each language has a single command to run the corpus; CI runs all language jobs on every
  change. Regenerating goldens from the reference implementation is a one-step operation.

Out of Scope

• Any LLM/HTTP integration, provider SDKs, API-key handling, or a chatbot/query UI.
• The invoice/Qty/Total domain logic from the old Python and VBA versions.
• Phase 2 styling-aware anchor detection (borders/bold/merges/number-formats) — designed-for via the
  optional metadata contract, but not implemented in the first release.
• Server-side chart image rendering in non-host languages (Python/Go/C#/TS) — descriptors only there.
• Multi-sheet "mega-encoding" that merges sheets into one representation.
• Publishing the packages to npm/PyPI/NuGet/etc. (the libraries are built and tested in-repo first).

Further Notes

• The reference SpreadsheetLLM paper PDF and the prior implementations live locally under
  sources/ and are gitignored; they are reference material, not shipped code.
• A live OpenAI API key was previously hardcoded in the old sources/python/SpreadsheetLLM.py; it
  must be revoked/rotated regardless of repo changes, and the gitignore ensures it never enters
  history.
• This issue is the overarching PRD; it is expected to be broken into tracer-bullet vertical slices
  (e.g. via the to-issues skill) — starting with SPEC.md + the TypeScript core + initial golden
  fixtures — before implementation.