feat: add S3Checkpoint for serverless/containerized environments#402
feat: add S3Checkpoint for serverless/containerized environments#402evanerwee wants to merge 1 commit into
Conversation
Add S3-backed checkpoint that stores markers as zero-byte S3 objects
instead of local files. Enables checkpointing in ECS Fargate, Lambda,
and EKS environments where local disk is ephemeral.
- S3CheckpointFilter: HEAD request to check if node was processed
- S3CheckpointWriter: PUT zero-byte object as completion marker
- S3Checkpoint: drop-in replacement for Checkpoint with same interface
(add_filter, add_writer, enabled)
The local Checkpoint class uses os.path.exists/os.makedirs/open which
requires persistent local filesystem. S3Checkpoint uses the same
pattern but backed by S3, surviving container restarts and retries.
Usage:
from graphrag_toolkit.lexical_graph.indexing.build import S3Checkpoint
checkpoint = S3Checkpoint(
checkpoint_name='enrichment-run-001',
bucket_name='my-pipeline-bucket',
key_prefix='checkpoints/tenant-a'
)
graph_index.extract(docs, handler=extracted_docs, checkpoint=checkpoint)
mykola-pereyma
left a comment
mykola-pereyma
left a comment
There was a problem hiding this comment.
Thank you for putting this together — the problem is real and we understand the need for checkpointing in Fargate/Lambda environments.
We have #325 on our implementation roadmap which covers this use case as part of a broader checkpoint/state-store redesign. That design introduces a URI-based factory pattern (matching how GraphStoreFactory and VectorStoreFactory already work) with support for multiple backends (S3, DynamoDB) and document-level progress tracking for distributed workers.
A few things we noticed in this PR that the broader design would address differently:
- The parameter is accepted but never used (the S3 client comes from GraphRAGConfig)
- No shared interface between Checkpoint and S3Checkpoint — if we add DynamoDB later it would be a third copy
- Per-node HEAD requests in the filter path would be a bottleneck at scale (list_objects_v2 with prefix can batch-check)
We appreciate the contribution and the detailed issue write-up. Would you be open to collaborating on #325 instead? Your experience running this on Fargate would be valuable input for the design — especially around the failure modes and retry patterns you encountered.
2 participants
Description
Add S3-backed checkpoint (
S3Checkpoint) that stores markers as zero-byte S3 objects instead of local files. Enables checkpointing in ECS Fargate, Lambda, and EKS environments where local disk is ephemeral.Changes
S3Checkpointclass — drop-in replacement forCheckpointwith same interfaceS3CheckpointFilter— usesHEADrequest instead ofos.path.exists()S3CheckpointWriter— usesPUTzero-byte object instead ofopen(path, "a")__init__.pyProblem
Related issue: #401
The existing
Checkpointclass uses local filesystem markers (os.path.exists,os.makedirs,open). This fails in serverless/containerized environments (Fargate, Lambda, EKS) where local disk is ephemeral and does not survive container restarts or SQS message retries.Testing
pytest)Manual testing: verified syntax passes
ast.parse(), imports resolve correctly against the existing module structure.Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.