Skip to content

Commit 91172e5

Browse files
committed
edit apache article to be up-to-date, clean sfi violations and bring acrolinx scorte above 80
1 parent 3a4e1d2 commit 91172e5

1 file changed

Lines changed: 24 additions & 28 deletions

File tree

articles/synapse-analytics/spark/apache-spark-handle-livy-error.md

Lines changed: 24 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -3,73 +3,69 @@ title: Handle Livy Errors on Apache Spark in Synapse
33
description: Learn how to handle and interpret job failures on Apache Spark in Synapse Analytics.
44
author: midesa
55
ms.author: midesa
6-
ms.date: 02/27/2025
6+
ms.date: 03/30/2026
77
ms.service: azure-synapse-analytics
88
ms.subservice: spark
99
ms.topic: error-reference
1010
---
1111

12-
# Interpret error codes in Synapse Analytics
12+
# Interpret error codes in Azure Synapse Analytics
1313

14-
There are many factors that can play into why a spark application fails in Azure Synapse Analytics today. For instance, it can be due to a system error or even a user related error. Previously, all errors corresponding to failing jobs on Synapse Analytics were surfaced with a generic error code displaying *LIVY_JOB_STATE_DEAD*. This error code gave no further insight into why the job failed. It requires significant effort to identify the root cause by digging into the driver, executor, Spark Event, Livy logs, and find a resolution.
14+
Many factors can cause a Spark application to fail in Synapse Analytics. For example, the failure can stem from a system error or a user error. Previously, all errors that correspond to failing jobs on Synapse Analytics surfaced a generic error code that displayed *LIVY_JOB_STATE_DEAD*. This error code gave no further insight into why the job failed. You had to put in significant effort to identify the root cause by digging into the driver, executor, Spark Event, and Livy logs to find a resolution.
1515

16-
:::image type="content" source="media/apache-spark-handle-livy-error/apache-spark-old-error-view.png" border="true" alt-text="Screenshot of Apache Spark error code without detailed message." lightbox="media/apache-spark-handle-livy-error/apache-spark-old-error-view.png":::
17-
18-
We have introduced a more precise list of error codes that replaces the previous generic message. The new message describes the cause of failure. Whenever a job fails on Azure Synapse Analytics, the error handling feature parses and checks the logs on the backend to identify the root cause. It then displays a message to the user on the monitoring pane along with the steps to resolve the issue.
19-
20-
:::image type="content" source="media/apache-spark-handle-livy-error/apache-spark-new-error-view.png" border="true" alt-text="Screenshot of Apache Spark error code with detailed message." lightbox="media/apache-spark-handle-livy-error/apache-spark-new-error-view.png":::
16+
The new error codes provide a more precise list that replaces the previous generic message. The new message describes the cause of failure. Whenever a job fails on Synapse Analytics, the error handling feature parses and checks the logs on the backend to identify the root cause. It then displays a message to the user on the monitoring pane along with the steps to resolve the issue.
2117

2218
## Enable error classification in Synapse
2319

24-
The error classification feature can be enabled or disabled by setting the following Spark configuration to `true` or `false` at the job or pool level:
20+
Set the following Spark configuration to `true` or `false` at the job or pool level to enable or disable the error classification feature:
2521

2622
`livy.rsc.synapse.error-classification.enabled`
2723

28-
The following section lists some error types that are currently supported. We're continuously refining and adding more to these error codes by improving our model.
24+
The following section lists some error types that are currently supported. The product team is continuously refining and adding more error codes by improving the model.
2925

3026
## Error code categories
3127

32-
Each error code falls under one of the following four buckets:
28+
Each error code falls under one of the following four categories:
3329

34-
- **User** - Indicating a user error
35-
- **System** - Indicating a system error
30+
- **User** - Indicates a user error
31+
- **System** - Indicates a system error
3632
- **Ambiguous** - Could be either user or system error
3733
- **Unknown** - No classification yet, most probably because the error type isn't included in the model
3834

3935
## Error code examples for each classification type
4036

4137
### Spark_User_TypeError_TypeNotIterable
4238

43-
In Python, the error `TypeError: argument of type 'insert type' is not iterable` occurs when the membership operator (in, not in) is used to validate the membership of a value in non iterable objects such as list, tuple, dictionary. This is usually due to the search of value in a non-iterable object. Possible solutions:
39+
In Python, the error `TypeError: argument of type 'insert type' is not iterable` occurs when you use the membership operator (`in`, `not in`) to check if a value is in non-iterable objects such as list, tuple, or dictionary. This error usually happens because you try to search for a value in a non-iterable object. To fix this error, try the following solutions:
4440

4541
- Check if the value is present in the iterable object.
46-
- If you want to check one value to another, use logical operator instead of Membership Operator.
47-
- If the membership operator contains "None" value, it won't be able to iterate, and a null check or assigned default must be done.
48-
- Check if the type of the value used can actually be checked and the typing is correct.
42+
- If you want to check one value against another, use a logical operator instead of the membership operator.
43+
- If the membership operator contains a `None` value, it can't iterate. Add a null check or assign a default value.
44+
- Check if the type of the value you're using can actually be checked and if the typing is correct.
4945

5046
### Spark_System_ABFS_OperationFailed
5147

52-
An operation with Azure Data Lake Storage (ADLS) Gen2 has failed.
48+
An operation with Azure Data Lake Storage (ADLS) Gen2 failed.
5349

54-
This error occurs typically due to a permissions issue.
50+
This error typically happens because of a permissions problem.
5551

56-
Ensure that for all ADLS Gen2 resources referenced in the Spark job, has "Storage Blob Data Contributor" RBAC role on the storage accounts the job is expected to read and write from.
57-
Check the logs for this Spark application. Navigate to your Synapse Studio, select the **Monitor** tab from the left pane. From the **Activities** section, select **Apache Spark Applications** and find your Spark job from the list. For the ADLS Gen2 storage account name that is experiencing this issue, inspect the logs available in the **Logs** tab at the bottom part of this page.
52+
Make sure that all ADLS Gen2 resources referenced in the Spark job have the **Storage Blob Data Contributor** RBAC role on the storage accounts that the job needs to read and write from.
53+
Check the logs for this Spark application. Go to your Synapse Studio, select the **Monitor** tab from the left pane. From the **Activities** section, select **Apache Spark Applications** and find your Spark job from the list. For the ADLS Gen2 storage account name that is experiencing this problem, check the logs available in the **Logs** tab at the bottom part of this page.
5854

5955
### Spark_Ambiguous_ClassLoader_NoClassDefFound
6056

61-
A class required by the code couldn't be found when the script was run. For more information, see:
57+
A class required by the code couldn't be found when the script ran. For more information, see:
6258

63-
- For Notebook scenarios: [Apache Spark manage packages for interactive jobs](./apache-spark-manage-scala-packages.md)
59+
- For Notebook scenarios: [Apache Spark manage packages for interactive jobs](./apache-spark-manage-scala-packages.md)
6460
- For Spark batch scenarios (see section 6): [Apache Spark manage packages for batch jobs](./apache-spark-job-definitions.md#create-an-apache-spark-job-definition-for-apache-sparkscala)
6561

66-
Ensure that all the code dependencies are included in the JARs Synapse runs. If you do not or cannot include third party JARs with your own code, ensure that all dependencies are included in the workspace packages for the Spark pool you are executing code on, or they are included in the "Reference files" listing for the Spark batch submission.
62+
Make sure that all the code dependencies are included in the JARs that Synapse runs. If you don't or can't include third party JARs with your own code, make sure that all dependencies are included in the workspace packages for the Spark pool you're executing code on, or they're included in the **Reference files** listing for the Spark batch submission.
6763

6864
### Spark_Unknown_Unknown_java.lang.Exception
6965

70-
An unknown failure, the model wasn't able to classify.
66+
An unknown failure. The model can't classify the error.
7167

72-
The error codes (including and beyond the previous list) along with the troubleshooting instructions on how to resolve the issue will show up on the Synapse Studio application error pane if this feature is enabled.
68+
If you enable this feature, Synapse Studio shows the error codes (including and beyond the previous list) along with troubleshooting instructions in the application error pane.
7369

74-
> [!NOTE]
75-
> If you built any tooling around the Synapse monitoring job that checks for a failing job by filtering the `LIVY_JOB_STATE_DEAD` error code, your app would no longer work. Because the returned error codes are different. Modify any scripts accordingly in order to utilize this feature or disable the feature if it's not needed.
70+
> [!NOTE]
71+
> If you built any tooling around the Synapse monitoring job that checks for a failing job by filtering the `LIVY_JOB_STATE_DEAD` error code, your app no longer works because the returned error codes are different. Modify any scripts accordingly to use this feature or disable the feature if it's not needed.

0 commit comments

Comments
 (0)