Sanitize StepMutator attributes with to_pod() in _graph_info - #3339
Conversation
node_to_dict() in graph.py sanitizes classic step decorator attributes with to_pod(), but not StepMutator/config-decorator attributes. A StepMutator constructor argument that is a non-primitive object gets embedded raw in _graph_info as a result. Any consumer that later unpickles _graph_info without that argument's class importable fails with ModuleNotFoundError, breaking the Metaflow UI's DAG tab for any flow using such a StepMutator. Apply the same to_pod() sanitization to both branches of the decorators list in node_to_dict(). Fixes Netflix#3338
Greptile SummaryApplies
Confidence Score: 5/5The PR appears safe to merge, with the serialization fix aligned to the existing classic-decorator behavior and covered by a focused regression test. The changed branch now applies the same portable serialization contract as its sibling decorator branch, preventing raw custom objects from leaking into persisted graph metadata without breaking any established consumer contract.
|
| Filename | Overview |
|---|---|
| metaflow/graph.py | Applies the established portable-object conversion to StepMutator/config-decorator attributes without introducing an actionable defect. |
| test/unit/test_graph_structure.py | Adds focused regression coverage confirming that a non-primitive StepMutator argument is sanitized in output graph metadata. |
Reviews (1): Last reviewed commit: "Sanitize StepMutator attributes with to_..." | Re-trigger Greptile
talsperre
left a comment
There was a problem hiding this comment.
Reviewed the fix and regression coverage. The change is correctly scoped to sanitize StepMutator/config-decorator attributes in _graph_info, and I found no blocking issues.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3339 +/- ##
=========================================
Coverage ? 30.65%
=========================================
Files ? 381
Lines ? 52654
Branches ? 9288
=========================================
Hits ? 16142
Misses ? 35316
Partials ? 1196 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
node_to_dict()ingraph.pysanitizes classic step decorator attributes withto_pod(), but not StepMutator/config-decorator attributes. This fix applies the sameto_pod()sanitization to both.Context / Motivation
A StepMutator whose constructor takes a non-primitive argument gets that raw object embedded in
_graph_info, since the second branch of thedecoratorslist innode_to_dict()skipsto_pod(). Any consumer that later unpickles_graph_infowithout that argument's class importable fails withModuleNotFoundError. This breaks the Metaflow UI's DAG tab for any flow using such a StepMutator, since the UI backend has no reason to have a given decorator's package installed. Fixes #3338.Root cause: the two branches of
node_to_dict()'sdecoratorslist construction are inconsistent.node.decorators(classic StepDecorator) goes throughto_pod();chain(node.wrappers, node.config_decorators)(StepMutator/config decorators) does not.Failure modes considered:
to_pod()passes these through unchanged._args/_kwargsare empty andto_pod()on an empty dict is a no-op.Changes Made
to_pod()innode_to_dict()(metaflow/graph.py).test_step_mutator_non_primitive_attribute_sanitized_in_output_stepstotest/unit/test_graph_structure.py, which fails without the fix and passes with it.Testing
assert isinstance(..., str)fails on the raw object) and passes with it.cd test/unit && python -m pytest -q(local runtime, no cloud/k8s involved since this is a pure DAG-serialization path): 563 passed.black --checkon both changed files: clean.ModuleNotFoundErrorfailure mode this fix addresses.AI disclosure
Parts of this PR (investigation, implementation, and test) were developed with AI assistance (Claude Code). I reviewed and understand every change and can answer questions about it.