Skip to content
This repository was archived by the owner on Sep 23, 2024. It is now read-only.
This repository was archived by the owner on Sep 23, 2024. It is now read-only.

Load fresh data into database tables and overwrite any existing data from previous runs #109

Description

@az-hienas

Is your feature request related to a problem? Please describe.
I am having problems using target-postgres because I could not find a way to do a fresh load of the data, i.e. truncate / delete whatever is already in the DB and insert incoming data.
I want to be able to only load the rows that are coming from the extractors and overwrite whatever was in the tables from a previous load.
Am I missing something?
Note: I am using target-postgres in Meltano

Describe the solution you'd like
I would like there to be a flag similar to https://hub.meltano.com/loaders/target-bigquery/#replication_method-setting
in which one can specify replication_method: truncate and have the tables truncated before loading in the incoming data

Describe alternatives you've considered
The only alternative I can actually use (given I am working with multiple types extractors) is to write my own code that truncates a table if the table is available.

Additional context
I understand the point of having incremental loads and merging strategies , but there should also be the option to start fresh.
Perhaps the feature is there and I don't know how to use it?

As a simple example of working with a CSV file with 10 rows:

  • run extract and load for the first time on the CSV file with 10 rows => 10 rows in the DB table , all good
  • delete 3 rows from the CSV file and run extract and load again => 10 rows in the DB table... there needs to be the option to end up with 7 rows because the file has changed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions