Handle deferred Action dependency outputs#53
Merged
Conversation
To account for different execution context between the orchestrator task and the actual Action run, Actions must save+load their outputs. This is fine for a local run host where the tasks are directly executed by the main orchestrator process and thus executed in DAG-dependency order. However in HPC environments tasks are submitted into a queue in DAG order with dependency info set at that time. Execution and thus evaluation of outputs will not occur until the real host runs the job. This delay would cause dependency outputs to run with the old values set at submission time instead of the values after dependencies have finished executing. To remedy this, building off of #52, Actions now reload the dependency outputs inside the `action_launcher.py` Note that dependency outputs will still be incorrect within the main orchestrator process context, but at this stage this information should not be needed as is never stored beyond the pickle file. Should users need the dependency output information per action in this situation (e.g. post_run_actions() for the Host) users can call the internal API function `Action.reload_dependency_outputs()` manually.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ensure that dependency outputs are current at Action start
To account for different execution context between the orchestrator task and the actual Action run, Actions must save+load their outputs.
This is fine for a local run host where the tasks are directly executed by the main orchestrator process and thus executed in DAG-dependency order. However in HPC environments tasks are submitted into a queue in DAG order with dependency info set at that time. Execution and thus evaluation of outputs will not occur until the real host runs the job.
This delay would cause dependency outputs to run with the old values set at submission time instead of the values after dependencies have finished executing. To remedy this, building off of #52, Actions now reload the dependency outputs inside the
action_launcher.pyNote that dependency outputs will still be incorrect within the main orchestrator process context, but at this stage this information should not be needed as is never stored beyond the pickle file. Should users need the dependency output information per action in this situation (e.g. post_run_actions() for the Host) users can call the internal API function
Action.reload_dependency_outputs()manually.