Add mechanism for DAG files to report errors without raising exception/loosing all DAGs in a file

[ashb]

Description

With dynamic DAGs (either "old school" of for dag_id in some_list or using something like Dag Factory) it would be nice to be able to report an error to show up in the Airflow UI without loosing all the other dags in the file.

Right now this is not possible as the only way of reporting an DAG error is to raise an exception, but if the exception is raise Airflow doesn't look for any DAGs in the loaded dag file/module anymore.

We should add some mechanism that lets these tools or dynamic dags report a list of exceptions so that they can still parse/load the valid dags but have a more natural experience for reporting errors.

I don't have any concrete ideas on this, but it will likely have to be done by setting some "magic"/special variable at the module level that contains a list of exceptions. (The Dag Parsing process already iterates over all the attributes on the module looking for dags, so looking for this special var, say __airflow_dag_parsing_errors, wouldn't be a big lift.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Shlomit-B]

Hey, I would like to work on this one if that's ok

[ashb]

@Shlomit-B Sure, lets talk about an implementation plan. What is your idea here?

[Shlomit-B]

@Shlomit-B Sure, lets talk about an implementation plan. What is your idea here?

Based on your suggestion about the module-level variable, I thought an implementation could be a safe wrapper API that users apply as a decorator or context manager—decorators for function-based DAG factories and context managers for inline DAG creation or loops—to isolate errors during DAG creation. This wrapper would catch exceptions per DAG, safely register valid DAGs, and collect error information consistently.

Internally, the wrapper would set the special module-level variable (__airflow_dag_parsing_errors) with collected errors, which Airflow would read during DAG parsing to report them in the UI.

Let me know what you think.

[ashb]

Sounds like a plan, let's go for it

[Shlomit-B]

@ashb

After digging more into the code:

Regarding the decorator - I realized it would really only make sense for DAG generator functions. For regular DAGs it doesn’t fit well. The context manager will work in way more cases, so I think it’s better to just stick with that for now.

For the __airflow_dag_parsing_errors part, currently ParseImportError stores one error per file. If multiple DAGs fail in a file, only the last error is saved. With per-DAG error handling, this won’t work. Possible approaches:

1. Single blob of errors per file - store all DAG errors in one row, tracktrace; simple, but harder to query/filter by dag_id.
2. New DagParsingError table - separate table for file vs DAG errors
3. Add dag_id to ParseImportError - add dag_id (nullable) where File-level errors: dag_id is NULL

I think option 3 is the best approach. What’s your thoughts? Would it make sense to open an AIP for this change?

[ashb]

Option3 doesn't make sense to me -- Errors almost by definition can't be associated with a dag id because we weren't able to parse something succesfully.

Looking at the datamodel for ParseImportError there are no uniqueness constraints, so we could store multiple errors per filename in the DB without changing anything I think?

[Shlomit-B]

Sorry about the confusion, you’re right. The code currently assumes (filename, bundle_name) is unique, but we can adjust that and store multiple errors.