Skip to content

Fix assume role issue - #30

Merged
kibria-shaped merged 13 commits into
mainfrom
fix-assume-role-timeout-issue
May 19, 2026
Merged

Fix assume role issue#30
kibria-shaped merged 13 commits into
mainfrom
fix-assume-role-timeout-issue

Conversation

@kibria-shaped

@kibria-shaped kibria-shaped commented May 12, 2026

Copy link
Copy Markdown
Contributor

What we fixed

  1. Long Iceberg syncs failing on S3 — After ~1 hour, temporary AWS credentials rotated, but PyArrow’s S3FileSystem kept old key/session strings, so reads broke even though Glue/catalog calls could still work.

  2. Fragile / incomplete AWS catalog setup — Credential wiring for IRSA + chained AssumeRole wasn’t centralized or robust enough for real Glue deployments.

  3. Incremental sync skipping “live” dataFixed-size windows only, with bookmark always start + window hours, could move the bookmark past now so new rows never matched the next filter.

  4. Magnus metadata ignored for windowswindow-size-hours / start-replication-key-value in TAP_ICEBERG__METADATA weren’t visible on Singer’s parsed catalog metadata, so the tap couldn’t read them from metadata alone.

  5. Timestamp filter crashes — ISO strings with offsets vs Iceberg timestamp (no TZ) caused PyIceberg errors like zone offset not expected.


How we fixed it

  1. RefreshingPyArrowFileIO — Rebuilds S3FileSystem before creds expire, tied to the same refreshable botocore session used for the catalog.

  2. tap_iceberg/aws_session.py — Single place that attaches refreshable credentials (legacy keys, IRSA, optional two-hop assume-role) into catalog_properties before load_catalog; tap logs redact secrets.

  3. Two-phase incremental in streams.pyBackfill: closed window while start + W ≤ now, bookmark start + W. Tail: when that window would go past now, scan rk ≥ start, bookmark max(rk) after a full successful run; _finalize_state only.

  4. Parse TAP_ICEBERG__METADATA from the environment — Merge "*" + per-stream overrides; cache safely on the tap (__dict__ check so MagicMock tests behave).

  5. Schema-aware timestampstimestamptz vs timestamp: normalize bookmark/filter literals so operators can pass naive or offset ISO strings without blowing up filters.

  6. Tests added for incremental phases, AWS session paths, and refreshing IO.

@kibria-shaped
kibria-shaped merged commit 6e36cc5 into main May 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants