Proposal: first-class multivariate "state change" detection (a deterministic, multiscale change ledger)

[AdamBraun]

Hi

Disclosure up front: I'm the author of metricchrono (Apache-2.0). I'm opening this because I think a primitive from it fits a real gap in Pecos - but I want to be transparent about that interest and let you decide whether, and in what form, it belongs here. I'm equally happy to land this with no external dependency at all
The goal is the capability, not the package.

The gap. Every built-in Pecos check is univariate / per-column (check_range, check_increment, check_delta, check_outlier). There's no first-class way to ask "did this group of signals move together as a system, and by how much?"
The check_custom_streaming example in the docs hand-rolls a multivariate nearest-neighbour detector with scipy.spatial.distance.cdist, citing CANARY. That works, but it's a per-user recipe, boolean-only, and not reproducible across runs or machines.

What I'm proposing A single metric that treats the selected columns as one system state, measures a causal normalized joint distance (each column standardized against history excluding the current point, then d_t = sqrt(Σ z_it²)), and encodes that distance at multiple scales into a compact ledger of salient changes - quiet periods skipped:

ledger = pecos.metrics.state_change_ledger(df, window=1800)
#                 Time   Scale   Distance   Contributors
#  2020-01-01 02:00:00       4       44.2   [(t1, 27.7), (t3, 27.1), (t2, 21.4)]

Scale answers "how coarse was the move" (0 = below an explicit threshold; larger = bigger). Contributors names the columns that drove it. It composes with your existing tools - flag Scale >= k with check_range so joint events flow into QCI/reports like any other test. It's a complement to check_custom_streaming, not a replacement.

Evidence (real numbers, happy to share a runnable gist). On three co-varying sensors with an injected joint step: the quiet majority of timestamps is skipped and only salient changes are recorded; the coarse-scale view isolates exactly the injected event (Scale 4 above, with the three sensors surfaced as contributors at z ≈ 27/27/21); and re-runs are byte-identical. I have a working prototype with unit tests + a docs example ready to share or open as a scoped PR.

Why a "ledger," not just a threshold. The encoding is deterministic and schema-serializable, so the same change codes can be recomputed and verified on another machine or runtime (e.g. an edge collector + a central re-analysis) and stored as a reproducible provenance record. That cross-runtime property is metricchrono's actual differentiator. If it isn't valuable to your users, the plain in-process version is just as good for detection - which leads to the honest part:

Dependency options - genuinely your call.

1. metricchrono as an optional, lazily-imported extra (pip install pecos[metricchrono]): base install unchanged; an actionable error only if used. It ships a small native (Rust) wheel today; I can also provide a pure-Python build so it matches how you already treat plotly/pvlib.
2. Vendor a small (~80-line) pure-Python encoder inside Pecos - zero external dependency, fully owned by you.
3. Anything in between. I'll implement whichever you prefer.

Suggested scope - start small. Just state_change_ledger (a metric), with tests and a short docs example - no new top-level QC category and no new check_* method initially.

Questions for you. Is joint-state change detection something you'd want in Pecos, or is check_custom_streaming the intended home? If you're open to it, which dependency option fits your maintenance preferences? Would a small metric-only PR be welcome, or settle the shape here first?

Adam Braun github.com/chrono-metrics/metricchrono

[kaklise]

@AdamBraun This is indeed a gap in Pecos. Thanks for opening the issue and providing a solution. Given the vast options to combine raw data into a system state, we generally rely on users to create a custom composite signal and add that column of data to the dataframe. The user can then run any univariate analysis on the composite signal (not just using the check_custom_streaming method).

As you point out, Pecos could include a metric to create this specific type composite signal, the state change ledger. It looks like this method also has the advantage of storing the contributing raw data columns.

I have not explored metricchrono in depth, so it's hard to say if Pecos users will make use of other methods in that package. I'm inclined to say that a pure python implementation is preferable to keep the method simple, documented within Pecos, and keep the installation easy.

I look forward to seeing an example. I'm happy to discuss or collaborate on updates to the code.