fix: avoid crashes in job bundle submission edge cases - #1290
Conversation
When create_job_from_job_bundle is called with debug_snapshot_dir for a bundle that has no job attachments, the attachments block never runs, so the asset_manager local is never assigned. It was then passed unconditionally to _save_debug_snapshot, raising UnboundLocalError and making snapshotting crash. Initialize asset_manager to None before the conditional and accept Optional[S3AssetManager] in _save_debug_snapshot, which only dereferences it when the create_job args carry attachments. Signed-off-by: Stephen Crowe <[email protected]>
wait_for_create_job_to_complete read job["lifecycleStatusMessage"] with subscript access on both the success and failure branches. A get_job response that omits that field raised KeyError, turning an otherwise successful submission into a crash. Read it with .get(..., "") so a missing message degrades to an empty string instead of raising. Signed-off-by: Stephen Crowe <[email protected]>
_upload_attachments and _snapshot_attachments both accept a config parameter, and _upload_attachments uses it to select the telemetry client for the upload summary. create_job_from_job_bundle never passed its config through, so upload telemetry used the default on-disk config instead of the caller-supplied one. Forward config to both call sites. Signed-off-by: Stephen Crowe <[email protected]>
Adds a unit test for the debug-snapshot-WITH-attachments combination: verifies _snapshot_attachments is used (not _upload_attachments), receives the caller's config, the snapshot's create_job_args.json contains the attachments, and the generated submit_job.sh includes the 'aws s3 cp' upload commands. This locks in the non-None side of the Optional asset_manager change and the config forwarding on the snapshot path. Also tightens the no-attachments snapshot test to assert the generated script omits 'aws s3 cp' while still containing the create-job command. Red-green: the new test fails on the pre-fix parent commit because _snapshot_attachments was called without config. Signed-off-by: Stephen Crowe <[email protected]>
| asset_manifests, | ||
| print_function_callback, | ||
| upload_progress_callback, | ||
| config=config, |
There was a problem hiding this comment.
Unlike _upload_attachments (which forwards config to get_deadline_cloud_library_telemetry_client(config=config) at line 220), _snapshot_attachments never uses its config parameter — its body does not reference it, and its @record_success_fail_telemetry_event decorator calls get_deadline_cloud_library_telemetry_client() with no config. So the config=config added here is effectively dead, and the new test docstring claiming the config is "forwarded ... for telemetry" is inaccurate for the snapshot path. Either wire config into the telemetry client inside _snapshot_attachments (to match upload behavior) or drop the parameter and the corresponding test assertion to avoid implying behavior that does not exist.
Fixes:
What was the problem/requirement? (What/Why)
Three crash/robustness bugs in job-bundle submission (
api/_submit_job_bundle.py):asset_managerwas only assigned inside the attachments block but passed unconditionally to_save_debug_snapshot→UnboundLocalErrorwhen submitting a bundle with no attachments and a debug snapshot.job["lifecycleStatusMessage"]was read without a fallback →KeyErrormade a successful submit look like a crash.configwasn't forwarded to_upload_attachments/_snapshot_attachments, so telemetry used the wrong config.What was the solution? (How)
asset_manager = Nonebefore the conditional and acceptOptional[S3AssetManager]in_save_debug_snapshot(it's only dereferenced inside the attachments branch).job.get("lifecycleStatusMessage", "")in both waiter branches.config=configto both attachment helpers.What is the impact of this change?
Debug-snapshot submissions without attachments no longer crash, successful submits report success, and attachment telemetry uses the correct config.
How was this change tested?
Added red-green unit tests for each: no-attachments debug snapshot, missing
lifecycleStatusMessage, and config forwarded to the attachment helpers.test_job_bundle_submission.pyand adjacent bundle-submit tests (58 passed).Was this change documented?
Does this PR introduce new dependencies?
Is this a breaking change?
No.
Does this change impact security?
No. (Known-path containment (C-known-path) and the stack-trace sanitizer are handled in separate PRs; the session-context race and debug-snapshot command-injection findings are out of scope here.)
Testing
All verification is automated unit tests (no live AWS calls; deadline/boto3 clients mocked), each proven red-green against the pre-fix parent commit:
Fix (a) —
asset_managerUnboundLocalError:test_create_job_from_job_bundle_debug_snapshot_no_attachments: debug snapshot of a bundle with no attachments completes cleanly; snapshot files are written, no job is submitted, and the generatedsubmit_job.shcontainsaws deadline create-jobbut correctly omitsaws s3 cplines. Pre-fix failure:UnboundLocalError: cannot access local variable 'asset_manager'.test_create_job_from_job_bundle_debug_snapshot_with_attachments(new): the other side of theOptional[S3AssetManager]change — a snapshot with attachments uses_snapshot_attachments(not_upload_attachments) with a bound asset manager, writesattachmentsintocreate_job_args.json, and emits theaws s3 cpupload commands in the generated script.Fix (b) — missing
lifecycleStatusMessage:test_wait_for_create_job_to_complete_missing_status_message, parametrized over both waiter branches: aget_jobresponse omittinglifecycleStatusMessagereturns(True, "")forREADYand(False, "")forCREATE_FAILED. Pre-fix failure:KeyError: 'lifecycleStatusMessage'in each branch. (Note: the sharedMockDeadlineBackendused by UI/e2e tests always includeslifecycleStatusMessage, so this unit test is the permanent coverage for the omission case.)Fix (c) —
configforwarding:test_create_job_from_job_bundle_forwards_config_to_upload_attachments: normal submit forwards the caller'sconfigobject to_upload_attachments(verified by identity,is). Pre-fix failure:configabsent from the call kwargs.test_create_job_from_job_bundle_debug_snapshot_with_attachments(new) verifies the same for_snapshot_attachmentson the snapshot path; it fails pre-fix withconfigabsent.Suite runs:
test_job_bundle_submission.py(41 tests) plus the fulltest/unit/deadline_client/api/+test_cli_bundle_submit*.pysuites (578 passed).hatch run fmtandhatch run lint(ruff check, ruff format, mypy) pass; full CI matrix is green.Not covered (manual-only, not automatable): executing a generated
submit_job.sh/submit_job.batagainst a live farm, and observing a realCreateJobwhoseGetJobresponse omitslifecycleStatusMessage. These are optional hardening checks; the mocked flows exercise the exact code paths changed.