Improve visualization of "raw" datasets with array shapes

[axelboc]

This PR improves handling of array datasets by the Raw visualization. The idea is to visualize only one scalar slice at a time instead of the entire array, notably to support stacks of raw JPEG/PNG images.

It's the first of three steps. I'm planning to then:

• remove Scalar vis and let Raw vis handle all scalars;
• rename Raw back to Scalar (and everything else called "raw").

This way we'll end up with a single Scalar vis fallback to visualize all scalar values (whether from a scalar dataset or from a scalar slice of an array dataset).

---

The main challenge was dealing with "scalar selections". I'm talking about selection strings of the form [0-9](,[0-9])* (e.g. "2", "0,1", "2,0,4", etc.) This was previously discussed here.

The first problem came from the providers. h5grove handles these scalar selections as expected — i.e. it returns a scalar value — but h5wasm and HSDS have quirks:

• h5wasm inconsistently unwraps the scalar depending on the dtype of the dataset — I'll endeavour to contribute a fix to the h5wasm repo at some point.
• HSDS never unwraps the scalar — a version 1.0.0 should be coming soon that seems to be a major overhaul, so I'll make sure to adapt the provider and test it thoroughly once it's out.

The second problem came from our Value<D extends HasShape & HasType> type, which is not aware of "scalar selections" — i.e. if you give it a dataset with an array shape as parameter, it returns an ArrayValue type, full stop. So the ValueFetcher component, useValue hook and assertValue guard would either lie or fail when encountering scalar slices taken from array datasets.

For a while, I considered making Value<D>, ValueFetcher, useValue and assertValue aware of selections but this made everything way more complicated and messy. Instead, I ended up creating a ScalarFetcher component that handles this case specifically in RawVisContainer without affecting any of the types, hooks and guards.

---

I've updated sample.h5 (demos: h5grove, h5wasm) by merging compound_nested_scalar and compound_array_vlen_1D into a more comprehensive compound_mixed_1D dataset that demonstrates everything at once (vlen/array dtypes + nested compound with int64 and complex fields).

PROVIDER	BEFORE	AFTER
h5grove
h5wasm

Note that the following datasets in sample.h5 also show the dimension mapper now: byte_string_1D, vlen_*_1D.

I've updated the mock file with a similar oneD_compound_mixed dataset, as well as with multiple opaque datasets: oneD_opaque, oneD_opaque_png, and twoD_opaque.

The mock oneD_opaque_png dataset demonstrates how the new Array vis handles stacks of PNG images, but you can also try it with frogs.h5:

ZIP with h5py script, images and frogs.h5: frogs.zip

[loichuder]

I struggle to understand what you mean by "scalar selection".

Is it a selection that will produce a scalar value ? If so, the test should check also the dataset shape ?

Is it a selection that only has scalar values inside ? In that case, perhaps "selected indices" would be a better name ?

[axelboc]

It's a selection that produces a scalar value, like 2,0,1 or 3, as opposed to 2,0,: or :.

The dataset shape doesn't matter here: when we have a selection string, the dataset is expected to have an array shape (cf. assertDatasetValue). The selection string is also expected to have the same number of elements as the number of dimensions in the dataset.

---

The problem I'm facing (which is why I've switched the PR to draft) is that the Value type has to be aware of the selection so it can conditionally return ScalarValue for "scalar selections".

I've managed to make it work with:

type ValueSelection = (number | null)[];
type ScalarSelection = number[];

export type Value<
  D extends Dataset,
  V extends ValueSelection | undefined = undefined,
> =
  D extends Dataset<infer S, infer T>
    ? S extends ScalarShape
      ? ScalarValue<T>
      : S extends ArrayShape
        ? V extends ScalarSelection
          ? ScalarValue<T>
          : ArrayValue<T>
        : never
    : never;

It requires keeping the selection as an array for longer, which changes quite a few things.

I'm now trying to use the dimension mapping array directly instead of introducing a new structure. I'll keep you updated.

[loichuder]

It's a selection that produces a scalar value, like 2,0,1 or 3, as opposed to 2,0,: or :.

I see. The issue I had is that I am used to Python where you can do stuff like threeD[3] and get a 2D array. In essence, the 3 is a short hand for 3, :, :.

But since we enforce the same number of elements as the number of dimensions in the dataset, we should never have this.

[axelboc]

/approve

[axelboc]

I'm finally bringing this PR back to life, as I found a solution I like to the scalar selection problem. Sorry for the spam, it took a while to finalise... I've updated the PR description and deleted or hid all the outdated comments. Here's the deploy preview https://deploy-preview-1768--h5web.netlify.app/

[loichuder]

Yes, I think adding a new fetcher is the way to go.

We know ahead of time whether we want to fetch a scalar since we already have the selection and the number of dims of the dataset.

[loichuder]

Is this intentional?

[axelboc]

Yes, the return type gets in the way a bit and DataProvider#getValue has return type unknown, so it "swallows" whichever return type we specify here. We assert the value after it's returned by the provider, so we don't need to assert it more than strictly necessary inside this function.

[loichuder]

Remind me what is this for?

[axelboc]

It says that the visualization doesn't have any "axes" and therefore that every dimension needs to be "sliced". The returned dimMapping is either an empty array if dims is empty (i.e. scalar dataset), or an array with as many zeros as there are dimensions in the array dataset.

This is why we are guaranteed to end up with either an undefined selection for scalar datasets, or a "scalar selection" for array datasets.

As a reminder, when axesCount is 1, dimMapping includes one 'x', and when axesCount is 2, dimMapping includes one 'x' and one 'y'. You can check out the unit tests of initDimMapping for some examples.