perf: reduce S3 read latency (connection pooling, prewarm, GridIndex)

[guidocerqueira]

Contexto

A latência de leitura no S3 é dominada por round-trips por requisição (~127 ms RTT Brasil→us-east-1). Estas mudanças reduzem o custo por round-trip e permitem evitar os recorrentes. Implementa o plano em docs/plano-reducao-latencia-s3.md.

Mudanças

Item 1 — Pooling de conexão (S3Store)

• S3StoreOptions: maxSockets (default 128, keep-alive on), keepAlive, connectionTimeoutMs, requestHandler (escape hatch).
• createClient monta NodeHttpHandler sobre um https.Agent keep-alive — reads de muitos chunks deixam de ser limitados ao default de ~50 sockets do SDK. Degrada graciosamente se @smithy/node-http-handler faltar.
• Default de leitura (concurrency: 50) inalterado — mudança puramente aditiva.

Item 3 — Prewarming (S3Store.prewarm())

• Abre uma conexão TLS no pool antes da 1ª leitura (best-effort). warmOnCreate dispara na construção.

Item 2 — GridIndex (@i4sea/zarr-node/spatial)

• Nearest (lat,lon)→(i,j) numa grade 2D curvilínea. fromCoordinates/fromGroup montam 1× (queries viram CPU puro).
• loadCached(group, { cache }) com L1 (processo) + L2 (Redis) — só o primeiro pod paga o fetch das coordenadas. Chave por domínio (source_model/experiment/grid_id + shape), com override gridKey e verifyGrid opcional. toBytes/fromBytes para snapshot binário.

Validação (S3 real, padrão de serviço)

Momento	Custo
POD 1 cold — setup do grid (metadata + lat/lon 1×)	~1.34 s
Query 2ª+ no mesmo pod (só o chunk do ponto)	~230 ms
Query em ponto repetido (MemoryCache)	~0.6 ms
POD 2 cold com Redis quente — grid rehidratado	~10 ms (100× vs S3)
Metadata com prewarm + keep-alive	2.3–2.5× vs legado

Ganhos de pooling/concurrency são maximizados in-region — recomenda-se re-rodar MODE=s3-ab de um pod em us-east-1 antes de afinar o concurrency default.

Testes

• npm test (326 passam, +17 novos: grid-index, s3-store), lint, format, typecheck, build CJS+ESM, validate-exports (/spatial), interop ESM/CJS — todos verdes.
• Benchmark: examples/benchmark-local-flow.ts (modos local/s3/s3-ab/s3-serving).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR reduces S3 read latency by (1) enabling connection pooling/keep-alive tuning in S3Store, (2) adding best-effort connection prewarming, and (3) introducing a new @i4sea/zarr-node/spatial subpath with GridIndex for fast (lat,lon)→(i,j) lookups with optional Redis-backed caching to avoid repeatedly fetching static grid coordinates.

Changes:

• Add configurable S3 HTTP connection pooling (maxSockets, keep-alive, connection timeout) plus an escape-hatch requestHandler.
• Add S3Store.prewarm() and warmOnCreate for eager TLS/socket warm-up.
• Add GridIndex (new /spatial export) with binary snapshotting and layered caching (L1 in-process + optional L2 Cache/Redis), plus documentation/tests/benchmark updates.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/s3-store.test.ts	Adds unit tests for request handler “escape hatch” and prewarm() never rejecting.
tests/unit/grid-index.test.ts	Adds unit tests for GridIndex queries, validation, snapshots, and cached loading behavior.
src/store/store.ts	Extends S3StoreOptions with pooling/prewarm/request handler options.
src/store/s3.ts	Implements pooling handler construction, prewarm(), and warmOnCreate.
src/spatial/index.ts	New /spatial entrypoint exporting GridIndex types and class.
src/spatial/grid-index.ts	Implements GridIndex construction, nearest-neighbor queries, snapshotting, and cache-key derivation.
README.md	Documents S3 pooling/prewarm usage and introduces GridIndex usage patterns.
package.json	Adds ./spatial export and makes @smithy/node-http-handler an optional dependency.
package-lock.json	Reflects version bump and optional dependency resolution changes.
examples/benchmark-local-flow.ts	Adds an end-to-end benchmark example covering caches and S3 before/after modes.
docs/plano-reducao-latencia-s3.md	Adds the detailed latency-reduction plan/design rationale.
CHANGELOG.md	Adds Unreleased notes describing the new pooling/prewarm/GridIndex features.
.gitignore	Ignores .bench/ benchmark dataset directory.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[guidocerqueira]

Os 3 apontamentos do Copilot foram válidos e tratados em 0891b92:

1. grid key omitia lon.chunks/lon.dtype → a chave canônica agora inclui chunks e dtype de lon também; domínios que diferem só no lon não colidem mais.
2. gridKey sanitizado vazio (só não-ASCII/espaços) → agora cai num hash sha256 do valor cru via safeKeyComponent, em vez de colapsar para "gridindex:". Coberto por teste novo.
3. http:// endpoints (MinIO/LocalStack) → buildRequestHandler agora também configura um http.Agent, então maxSockets/keep-alive valem para http e https.

Suíte completa verde (327 testes, lint, typecheck, build, validate-exports).