fix xcom push not working in `SparkKubernetesOperator` by kesem0811 · Pull Request #52051 · apache/airflow

kesem0811 · 2025-06-23T01:11:34Z

Fixes: #39184 where SparkKubernetesOperator tasks hang indefinitely because no sidecar is injected to read /airflow/xcom/return.json.

Lee-W · 2025-06-24T06:29:34Z

+                )
+                self.template_body["spark"]["spec"]= driver_with_xcom_template
+            except KeyError as e:
+                raise AirflowException("Spec missing in SparkApplication template") from e


let's not use AirflowException, we can create a customized exception or just use KeyError

Why?
In all the file, when something doesn't work as expected, an AirflowException is raised

AirflowException is broad and not infomative

lets switch to KeyError

Lee-W · 2025-06-24T06:29:48Z


 def add_xcom_sidecar(
-    pod: k8s.V1Pod,
+    pod: Union[k8s.V1Pod, dict],


Suggested change

pod: Union[k8s.V1Pod, dict],

pod: k8s.V1Pod | dict,

Lee-W · 2025-06-30T07:49:49Z

+                )
+                self.template_body["spark"]["spec"]= driver_with_xcom_template
+            except KeyError as e:
+                raise AirflowException("Spec missing in SparkApplication template") from e


AirflowException is broad and not infomative

…ure/fix_xcom_push_in_SparkKubernetesOperator

eladkal · 2025-08-06T17:32:58Z

@kesem0811 can you fix the static checks?

Lee-W

I think we'll fix static checks

…ator - Resolved merge conflict in spark_kubernetes.py - Preserved xcom sidecar functionality from local changes - Integrated upstream improvements including deferrable support and better reattach logic - Combined both approaches: upstream refactoring + local xcom features

- Add proper type checking for dict vs V1Pod inputs in add_xcom_sidecar() - Convert dict inputs to V1Pod objects to ensure .spec attribute access - Maintains backward compatibility while fixing type safety - Resolves: Item 'dict[Any, Any]' has no attribute 'spec' [union-attr]

kesem0811 · 2025-09-25T12:38:28Z

the static checks failed with the error:
The image /mnt/ci-image-save-v3-linux_amd64-3.10.tar does not exist.
I love to make the tests run again?
@romsharon98 @eladkal @Lee-W

…ator

Lee-W

in general looks good, but we'll need unit tests for this

Lee-W · 2025-10-08T19:49:17Z

+    VOLUME_MOUNT_NAME = "xcom"
+    VOLUME_MOUNT = k8s.V1VolumeMount(name=VOLUME_MOUNT_NAME, mount_path=XCOM_MOUNT_PATH)
+    XCOM_SIDECAR_COMMAND = ["sh", "-c", XCOM_CMD]
+    VOLUME = k8s.V1Volume(name=VOLUME_MOUNT_NAME, empty_dir=k8s.V1EmptyDirVolumeSource())


Suggested change

VOLUME_MOUNT_NAME = "xcom"

VOLUME_MOUNT = k8s.V1VolumeMount(name=VOLUME_MOUNT_NAME, mount_path=XCOM_MOUNT_PATH)

XCOM_SIDECAR_COMMAND = ["sh", "-c", XCOM_CMD]

VOLUME = k8s.V1Volume(name=VOLUME_MOUNT_NAME, empty_dir=k8s.V1EmptyDirVolumeSource())

XCOM_SIDECAR_COMMAND = ["sh", "-c", XCOM_CMD]

VOLUME_MOUNT_NAME = "xcom"

VOLUME_MOUNT = k8s.V1VolumeMount(name=VOLUME_MOUNT_NAME, mount_path=XCOM_MOUNT_PATH)

VOLUME = k8s.V1Volume(name=VOLUME_MOUNT_NAME, empty_dir=k8s.V1EmptyDirVolumeSource())

let's group similar concepts together

Lee-W · 2025-10-08T19:50:17Z


 def add_xcom_sidecar(
-    pod: k8s.V1Pod,
+    pod: k8s.V1Pod | dict,


Suggested change

pod: k8s.V1Pod | dict,

pod: k8s.V1Pod | dict[str, Any],

Lee-W · 2025-10-08T19:51:25Z

+def add_sidecar_to_spark_operator_pod_spec(
+    spec: dict, sidecar_container_image: str | None = None, sidecar_container_resources: dict | None = None
+):


Suggested change

def add_sidecar_to_spark_operator_pod_spec(

spec: dict, sidecar_container_image: str | None = None, sidecar_container_resources: dict | None = None

):

def add_sidecar_to_spark_operator_pod_spec(

spec: dict[str, Any], sidecar_container_image: str | None = None, sidecar_container_resources: dict[str, Any] | None = None

) -> dict[str, Any]:

please confirm whether these type are correct. Thanks!

kunaljubce · 2026-01-02T06:30:51Z

@kesem0811 Can you also fix the static checks CI error? Should be autofixed by your prek pre-commit hooks, assuming you have them installed.

Lee-W · 2026-01-04T03:53:11Z

will also need your help resolving the conflict. Thanks!

potiuk · 2026-04-02T13:07:15Z

This pull request has had no activity from the author for over 4 weeks. We are converting it to draft to keep the review queue manageable.

@kesem0811, please mark this PR as ready for review when you are ready to continue working on it. Thank you for your contribution!

potiuk · 2026-04-26T17:01:02Z

@kesem0811 This draft PR has had no activity for 3 weeks. Closing to keep the queue clean.

You are welcome to reopen and continue when you're ready. If you'd like to pick it back up, please rebase onto the current main branch first.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

kesem0811 · 2026-06-21T00:00:23Z

Hi everyone - sorry for not responding earlier. That’s on me, and I understand why the PR was closed.

I’ve opened a new PR that addresses the review feedback and completes the fix: #68788

kesem0811 added 4 commits June 19, 2025 01:34

add xcom sidecar

ee4a5d8

this version is working

f63c0d2

final version without test changes

efb2bd7

delete prints

48b5826

kesem0811 requested review from dstandish, hussein-awala and jedcunningham as code owners June 23, 2025 01:11

your commit message

4f172d6

eladkal requested review from Lee-W and romsharon98 June 24, 2025 04:36

Lee-W reviewed Jun 24, 2025

View reviewed changes

kesem0811 added 2 commits June 29, 2025 21:42

cr fix

bb747dd

cr fix

87085c8

Lee-W requested changes Jun 30, 2025

View reviewed changes

kesem0811 added 4 commits July 30, 2025 13:17

your message

c4919ea

Merge branch 'main' of https://github.com/kesem0811/airflow into feat…

4c022ca

…ure/fix_xcom_push_in_SparkKubernetesOperator

resolve conflicts

95054f2

run ruff format

6d892e9

eladkal requested a review from Lee-W July 30, 2025 12:01

eladkal changed the title ~~Feature/fix xcom push in spark kubernetes operator~~ fix xcom push not working in SparkKubernetesOperator Jul 30, 2025

Lee-W reviewed Aug 18, 2025

View reviewed changes

kesem0811 requested a review from Lee-W September 25, 2025 11:20

romsharon98 and others added 4 commits October 3, 2025 10:49

Merge branch 'main' into feature/fix_xcom_push_in_SparkKubernetesOper…

92125a6

…ator

style: apply ruff formatting to spark_kubernetes.py

68a1324

ruff formating

2bc61bc

Merge branch 'main' into feature/fix_xcom_push_in_SparkKubernetesOper…

fe841e3

…ator

Lee-W reviewed Oct 8, 2025

View reviewed changes

potiuk marked this pull request as draft April 2, 2026 13:07

potiuk closed this Apr 26, 2026

Conversation

kesem0811 commented Jun 23, 2025 • edited by eladkal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eladkal commented Aug 6, 2025

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

kesem0811 commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunaljubce commented Jan 2, 2026

Uh oh!

Lee-W commented Jan 4, 2026

Uh oh!

potiuk commented Apr 2, 2026

Uh oh!

potiuk commented Apr 26, 2026

Uh oh!

kesem0811 commented Jun 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kesem0811 commented Jun 23, 2025 •

edited by eladkal

Loading

kesem0811 commented Sep 25, 2025 •

edited

Loading