Duplicated records in Clickhouse when using LOG_BASED

[gagarinfan]

Hi,

when using tap-postgres with a LOG_BASED replication to CLickhouse I can observe duplicated records with the same _sdc_lsn. Example:

Here is what I have in CH

in fact, current state in Postgres is just:

Now, when I run Meltano, WITHOUT ANY UPDATES TO THE TABLE IN PG, I got an extra record:

We can observe three things:

• url column for a record with pkey id=1 is empty
• it has the same _sdc_lsn as the previous record (the one below)
• if I run Meltano again it creates exactly the same record in CH as in the example above

Why is this record created, why duplicated records are being created? Is this a target-clickhouse or rather tap-postgres problem?

Here is the meltano.yaml

version: 1
default_environment: dev
project_id: 4bd37b1b-3911-4cce-b850-178e28761e3a
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
    config:
      database: REDACTED
      default_replication_method: LOG_BASED
      host: REDACTED
      port: 5432
      user: meltano
      filter_schemas:
      - erp_dim
    select:
      - erp_dim-mgtest
    metadata:
      '*':
        replication_method: LOG_BASED
        replication_key: _sdc_lsn

  loaders:
  - name: target-clickhouse
    variant: shaped-ai
    pip_url: git+https://github.com/shaped-ai/target-clickhouse.git
#     capabilities:
#      - hard-delete
    config:
      database: erp_dim
      host: 127.0.0.1
      port: 8123
      username: meltano