Description:
Certain airbyte taps are incompatible with incremental extraction with tap-airbyte-wrapper
For incremental to work, airbyte is expecting a state message in this form, a list of AirbyteStateMessage
[
{
"type": "STREAM",
"stream": {
"stream_descriptor": {
"name": "events",
"namespace": null
},
"stream_state": {
"server_upload_time": "2024-05-13 19:35:13.833000"
}
}
}
]
The internal dict above is actually what gets sent by airbyte in a state message. But the tap-airbyte-wrapper unpack this and saves the unpacked version in the state table, which then gets feds back without modifications to airbyte binary as state.json. This prevents certain airbyte extractors from not working. (For example intercom’s one works totally fine, while amplitude does not)
Description:
Certain airbyte taps are incompatible with incremental extraction with tap-airbyte-wrapper
For incremental to work, airbyte is expecting a state message in this form, a list of AirbyteStateMessage
The internal dict above is actually what gets sent by airbyte in a state message. But the tap-airbyte-wrapper unpack this and saves the unpacked version in the state table, which then gets feds back without modifications to airbyte binary as state.json. This prevents certain airbyte extractors from not working. (For example intercom’s one works totally fine, while amplitude does not)