Fix assume role issue - #30
Merged
Merged
Conversation
Robert Parton (rjparton)
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What we fixed
Long Iceberg syncs failing on S3 — After ~1 hour, temporary AWS credentials rotated, but PyArrow’s
S3FileSystemkept old key/session strings, so reads broke even though Glue/catalog calls could still work.Fragile / incomplete AWS catalog setup — Credential wiring for IRSA + chained
AssumeRolewasn’t centralized or robust enough for real Glue deployments.Incremental sync skipping “live” data — Fixed-size windows only, with bookmark always
start + window hours, could move the bookmark past now so new rows never matched the next filter.Magnus metadata ignored for windows —
window-size-hours/start-replication-key-valueinTAP_ICEBERG__METADATAweren’t visible on Singer’s parsed catalog metadata, so the tap couldn’t read them frommetadataalone.Timestamp filter crashes — ISO strings with offsets vs Iceberg
timestamp(no TZ) caused PyIceberg errors like zone offset not expected.How we fixed it
RefreshingPyArrowFileIO— RebuildsS3FileSystembefore creds expire, tied to the same refreshable botocore session used for the catalog.tap_iceberg/aws_session.py— Single place that attaches refreshable credentials (legacy keys, IRSA, optional two-hop assume-role) intocatalog_propertiesbeforeload_catalog; tap logs redact secrets.Two-phase incremental in
streams.py— Backfill: closed window whilestart + W ≤ now, bookmarkstart + W. Tail: when that window would go past now, scanrk ≥ start, bookmarkmax(rk)after a full successful run;_finalize_stateonly.Parse
TAP_ICEBERG__METADATAfrom the environment — Merge"*"+ per-stream overrides; cache safely on the tap (__dict__check soMagicMocktests behave).Schema-aware timestamps —
timestamptzvstimestamp: normalize bookmark/filter literals so operators can pass naive or offset ISO strings without blowing up filters.Tests added for incremental phases, AWS session paths, and refreshing IO.