Allow extending a series backwards?

[HomoCodens]

Currently timeseriesdb only allows storing vintages newer than any stored before. This is to prevent manipulation of the past by sandwiching new Vintages into the history.

It might however be useful in some (admittedly rare) cases to allow storing vintages that extend a series into the past i.e. if the vintage is older than any existing ones.

What are your thoughts?

[mbannert]

This is on hold for now b/c it is unclear how useful this is and how often the use case comes up. Messing with the past can also have negative effects (see also: https://en.wikipedia.org/wiki/Back_to_the_Future_Part_II) and ask for a whole battery of functions and functionality, e.g., deleting from the other end when series were mistakenly added to the past.

[omuelle]

This would be really useful for us, because it happens again and again and again that we have to store old vintages, unfortunately. I suggest we make a store_past_vintage parameter (or similar) that one explicitly has to set to TRUE to override the default behavior, which is to not allow it.

[mbannert]

I am really skeptical a flag is enough. Screwing up the past is really the worst case scenario for a time series archive. Particularly when the original source gets revised and claims the did not change anything or isn't around anymore.

To me, the need to make sure that the entire series is around in memory, i.e., the current series plus whatever is ahead is the best safety, also because it can be tested.

But then I don't have the pain @omuelle describes on a regular basis. Tough call.

[omuelle]

Then what about a separate function? We need the functionality anyway, it's just a matter of where to put it.

[HomoCodens]

@omuelle Are you talking about simply appending "in the other direction" or also inserting vintages at arbitrary points?
The former is probably of less concern in terms of "changing history".

I think such functionality should certainly only be available to timeseries admins.
Maybe you could rework the check at https://github.com/mbannert/timeseriesdb/blob/main/inst/sql/create_functions_ts.sql#L33 to be skipped if 1) the current user is an admin and 2) they specifically opt in with a parameter

[omuelle]

At arbitrary points. Basically whenever a user/process fails to write vintages, but we still have them, e.g. in another schema, on the disc or at the data provider.

Admins only sounds like a good idea.