Merge pull request #53087 from weslbo/updates-dp-750-jan

DP-750 Renamed Lakeflow Declarative Pipelines to Lakeflow Spark Declarative …

Author: JillGrant615

learn-pr/paths/azure-databricks-data-engineer-deploy-maintain-data-pipelines-workloads/index.yml

@@ -18,7 +18,7 @@ summary: |
 
   By the end of this learning path, you'll be able to:
   
-  - Design and implement robust data pipelines using notebooks and Lakeflow Declarative Pipelines
+  - Design and implement robust data pipelines using notebooks and Lakeflow Spark Declarative Pipelines
   - Create and orchestrate Lakeflow Jobs with triggers, schedules, and error handling
   - Apply version control and deploy pipelines across environments using Git and Databricks Asset Bundles
   - Monitor, troubleshoot, and optimize data workloads for reliability and performance

learn-pr/wwl-databricks/create-and-organize-objects-in-unity-catalog/includes/2-apply-naming-conventions.md

@@ -90,7 +90,7 @@ Name clusters according to their purpose and environment to make resource alloca
 
 Structure job names using the pattern `job_{layer}_{purpose}` to align with your data transformation pipeline. Examples include `job_bronze_orders_ingestion`, `job_silver_orders_transformation`, and `job_gold_sales_aggregation`. This naming pattern makes dependencies between jobs immediately visible and helps you trace data lineage across the medallion architecture.
 
-For Lakeflow Declarative Pipelines pipelines, use the prefix `pipe_` followed by the data domain or purpose: `pipe_orders_processing`, `pipe_customer_data_cleaning`. 
+For Lakeflow Spark Declarative Pipelines pipelines, use the prefix `pipe_` followed by the data domain or purpose: `pipe_orders_processing`, `pipe_customer_data_cleaning`. 
 
 Name streaming pipelines to include both source and target, following patterns like `stream_{source}_to_{target}`. Examples such as `stream_kafka_to_bronze` and `stream_iot_sensor_data` make data flow explicit without requiring pipeline documentation. This convention is especially valuable when managing multiple concurrent streaming workloads.
 

learn-pr/wwl-databricks/design-implement-data-pipelines/3-choose-notebook-lakeflow-pipelines.yml

@@ -3,7 +3,7 @@ uid: learn.wwl.design-implement-data-pipelines.choose-notebook-vs-lakeflow-pipel
 title: Choose notebook vs Lakeflow Pipelines
 metadata:
   title: Choose Notebook vs Lakeflow Pipelines
-  description: Learn how to choose between notebooks and Lakeflow Declarative Pipelines for building data pipelines in Azure Databricks, comparing flexibility, maintainability, and use cases.
+  description: Learn how to choose between notebooks and Lakeflow Spark Declarative Pipelines for building data pipelines in Azure Databricks, comparing flexibility, maintainability, and use cases.
   ms.date: 12/07/2025
   author: weslbo
   ms.author: wedebols

learn-pr/wwl-databricks/design-implement-data-pipelines/7-create-pipeline-lakeflow-declarative.yml

@@ -1,9 +1,9 @@
 ### YamlMime:ModuleUnit
 uid: learn.wwl.design-implement-data-pipelines.create-pipeline-lakeflow-declarative
-title: Create pipeline with Lakeflow Declarative Pipelines
+title: Create pipeline with Lakeflow Spark Declarative Pipelines
 metadata:
-  title: Create Pipeline with Lakeflow Declarative Pipelines
-  description: Learn how to create data pipelines using Lakeflow Declarative Pipelines in Azure Databricks, including streaming tables, materialized views, and data quality expectations.
+  title: Create Pipeline with Lakeflow Spark Declarative Pipelines
+  description: Learn how to create data pipelines using Lakeflow Spark Declarative Pipelines in Azure Databricks, including streaming tables, materialized views, and data quality expectations.
   ms.date: 12/07/2025
   author: weslbo
   ms.author: wedebols

learn-pr/wwl-databricks/design-implement-data-pipelines/8-knowledge-check.yml

@@ -27,14 +27,14 @@ quiz:
     - content: "Bronze layer"
       isCorrect: true
       explanation: "Correct. The bronze layer stores ingested data with minimal transformation, preserving the raw state for auditing and potential reprocessing."
-  - content: "What is the primary advantage of using Lakeflow Declarative Pipelines over notebooks for production data pipelines?"
+  - content: "What is the primary advantage of using Lakeflow Spark Declarative Pipelines over notebooks for production data pipelines?"
     choices:
     - content: "Declarative pipelines allow rapid prototyping and cell-by-cell inspection"
       isCorrect: false
       explanation: "Incorrect. Notebooks are better suited for rapid prototyping and interactive exploration."
     - content: "Declarative pipelines automatically handle orchestration, incremental processing, and error recovery"
       isCorrect: true
-      explanation: "Correct. Lakeflow Declarative Pipelines reduce operational burden by automatically managing orchestration, dependency analysis, incremental processing, and retry logic."
+      explanation: "Correct. Lakeflow Spark Declarative Pipelines reduce operational burden by automatically managing orchestration, dependency analysis, incremental processing, and retry logic."
     - content: "Declarative pipelines support more external library dependencies than notebooks"
       isCorrect: false
       explanation: "Incorrect. Notebooks provide more flexibility for installing and using custom Python or Scala packages."
@@ -71,7 +71,7 @@ quiz:
     - content: "To automatically trigger a retry of the failed notebook task"
       isCorrect: false
       explanation: "Incorrect. Retry policies are configured at the job level, not triggered by dbutils.notebook.exit()."
-  - content: "When should a data engineer choose streaming tables over materialized views in Lakeflow Declarative Pipelines?"
+  - content: "When should a data engineer choose streaming tables over materialized views in Lakeflow Spark Declarative Pipelines?"
     choices:
     - content: "When the transformation requires complex aggregations or joins"
       isCorrect: false

learn-pr/wwl-databricks/design-implement-data-pipelines/includes/1-introduction.md

@@ -1,4 +1,4 @@
-Building reliable data pipelines requires more than connecting data sources to destinations. You need to design workflows that handle failures gracefully, scale with growing data volumes, and remain maintainable as business requirements evolve. Azure Databricks provides multiple approaches for creating data pipelines—from flexible **notebooks** with procedural code to **Lakeflow Declarative Pipelines** that automate orchestration and data quality enforcement.
+Building reliable data pipelines requires more than connecting data sources to destinations. You need to design workflows that handle failures gracefully, scale with growing data volumes, and remain maintainable as business requirements evolve. Azure Databricks provides multiple approaches for creating data pipelines—from flexible **notebooks** with procedural code to **Lakeflow Spark Declarative Pipelines** that automate orchestration and data quality enforcement.
 
 When you design data pipelines, you make decisions that affect every downstream consumer of your data. The order of operations determines whether transformations build on validated, well-structured data. Your choice between notebooks and declarative pipelines influences how much orchestration code you write versus how much the platform manages for you. **Task dependencies** in **Lakeflow Jobs** control execution flow and enable parallel processing that reduces pipeline runtime.
 

learn-pr/wwl-databricks/design-implement-data-pipelines/includes/3-choose-notebook-lakeflow-pipelines.md

@@ -1,14 +1,14 @@
-When you build data pipelines in Azure Databricks, you have two primary approaches: **notebooks** with procedural code and **Lakeflow Declarative Pipelines**. Each approach serves different needs, and understanding when to use each helps you deliver maintainable, efficient data solutions.
+When you build data pipelines in Azure Databricks, you have two primary approaches: **notebooks** with procedural code and **Lakeflow Spark Declarative Pipelines**. Each approach serves different needs, and understanding when to use each helps you deliver maintainable, efficient data solutions.
 
 ## Understand the two approaches
 
 Notebooks execute code **step by step**. You control every aspect of data processing—from reading sources to writing outputs. This **procedural approach** gives you full control over execution flow, error handling, and optimization decisions.
 
-Lakeflow Declarative Pipelines work differently. Instead of specifying **how** to process data, you define **what** you want as the end result. You declare your **streaming tables** and **materialized views**, and the pipeline engine handles **orchestration**, **parallelization**, and **error recovery** automatically.
+Lakeflow Spark Declarative Pipelines work differently. Instead of specifying **how** to process data, you define **what** you want as the end result. You declare your **streaming tables** and **materialized views**, and the pipeline engine handles **orchestration**, **parallelization**, and **error recovery** automatically.
 
-:::image type="content" source="../media/3-understand-notebook-pipeline-approach.png" alt-text="Diagram explaining the two approaches when it comes to choosing notebooks or Lakeflow Declarative Pipelines." border="false" lightbox="../media/3-understand-notebook-pipeline-approach.png":::
+:::image type="content" source="../media/3-understand-notebook-pipeline-approach.png" alt-text="Diagram explaining the two approaches when it comes to choosing notebooks or Lakeflow Spark Declarative Pipelines." border="false" lightbox="../media/3-understand-notebook-pipeline-approach.png":::
 
-Consider a scenario where you need to ingest sales data, join it with product information, and calculate regional aggregates. With a notebook, you write explicit read, join, and aggregation commands in sequence. With Lakeflow Declarative Pipelines, you define the final tables and their relationships—the system determines the most efficient execution plan.
+Consider a scenario where you need to ingest sales data, join it with product information, and calculate regional aggregates. With a notebook, you write explicit read, join, and aggregation commands in sequence. With Lakeflow Spark Declarative Pipelines, you define the final tables and their relationships—the system determines the most efficient execution plan.
 
 ## When notebooks fit best
 
@@ -22,17 +22,17 @@ Notebooks excel in scenarios requiring **flexibility** and **detailed control**.
 
 **Fine-grained performance tuning**. When you need to manually control **partitioning**, **caching strategies**, or specific **Spark configurations**, notebooks give you direct access to these optimizations.
 
-## When Lakeflow Declarative Pipelines fit best
+## When Lakeflow Spark Declarative Pipelines fit best
 
-Lakeflow Declarative Pipelines simplify **production data pipelines** by handling operational complexity automatically. Choose this approach when your pipeline needs:
+Lakeflow Spark Declarative Pipelines simplify **production data pipelines** by handling operational complexity automatically. Choose this approach when your pipeline needs:
 
 **Standardized ETL patterns**. For common ingestion and transformation workflows—reading from cloud storage, applying **schema evolution**, maintaining **slowly changing dimensions**—the declarative approach reduces thousands of lines of code to a few statements.
 
 **Built-in data quality enforcement**. Declarative pipelines include **expectations** that validate data as it flows through. You define **quality rules** directly in your pipeline definition, and the system tracks violations and can halt processing when data quality degrades.
 
 **Automatic dependency management**. The pipeline engine analyzes relationships between your tables and determines the correct **execution order**. When source data updates, the engine refreshes only the **affected downstream tables**.
 
-**Operational visibility**. Lakeflow Declarative Pipelines provide **lineage tracking**, **execution graphs**, and **monitoring dashboards** without additional configuration. Operations teams can trace data from source to target and troubleshoot issues faster.
+**Operational visibility**. Lakeflow Spark Declarative Pipelines provide **lineage tracking**, **execution graphs**, and **monitoring dashboards** without additional configuration. Operations teams can trace data from source to target and troubleshoot issues faster.
 
 ## Compare the approaches
 
@@ -57,8 +57,8 @@ Start by evaluating your specific requirements. Ask these questions:
 - What level of **operational monitoring** does your team need?
 - Who maintains this pipeline—**seasoned developers** or a broader team with varied skills?
 
-For production pipelines with standard ingestion and transformation patterns, Lakeflow Declarative Pipelines **reduce operational burden** and **improve maintainability**. You spend less time writing orchestration code and more time defining business logic.
+For production pipelines with standard ingestion and transformation patterns, Lakeflow Spark Declarative Pipelines **reduce operational burden** and **improve maintainability**. You spend less time writing orchestration code and more time defining business logic.
 
 For exploratory work, complex integrations, or pipelines requiring extensive customization, notebooks provide the **flexibility** you need. You can always refactor successful notebook prototypes into declarative pipelines once the logic stabilizes.
 
-Many teams use **both approaches together**. Notebooks handle custom preprocessing or machine learning model training, while Lakeflow Declarative Pipelines manage the core ETL workflow. This **hybrid approach** lets you use each tool where it performs best.
+Many teams use **both approaches together**. Notebooks handle custom preprocessing or machine learning model training, while Lakeflow Spark Declarative Pipelines manage the core ETL workflow. This **hybrid approach** lets you use each tool where it performs best.

learn-pr/wwl-databricks/design-implement-data-pipelines/includes/4-design-task-logic-lakeflow-job.md

@@ -114,12 +114,12 @@ Azure Databricks provides **dynamic value references** that inject runtime conte
 
 Each task type has recommended compute options that affect both capability and cost. Your task logic design should account for compute requirements:
 
-| Task type                      | Recommended compute         |
-| ------------------------------ | --------------------------- |
-| Notebooks, Python scripts      | Serverless jobs compute     |
-| SQL queries and files          | Serverless SQL warehouse    |
-| Lakeflow Declarative Pipelines | Serverless pipeline compute |
-| JAR and Spark Submit           | Classic jobs compute        |
+| Task type                            | Recommended compute         |
+| ------------------------------------ | --------------------------- |
+| Notebooks, Python scripts            | Serverless jobs compute     |
+| SQL queries and files                | Serverless SQL warehouse    |
+| Lakeflow Spark Declarative Pipelines | Serverless pipeline compute |
+| JAR and Spark Submit                 | Classic jobs compute        |
 
 Tasks within the same job can use different compute resources. A common pattern assigns SQL tasks to a SQL warehouse while notebook-based transformations run on jobs compute.
 

learn-pr/wwl-databricks/design-implement-data-pipelines/includes/5-design-error-handling-pipelines.md

@@ -19,7 +19,7 @@ Each scenario requires a different response. Some errors warrant immediate pipel
 
 ## Define data quality expectations in declarative pipelines
 
-Lakeflow Declarative Pipelines provides built-in data quality constraints called **expectations**. These constraints validate records as data flows through your pipeline, giving you control over how to handle invalid data.
+Lakeflow Spark Declarative Pipelines provides built-in data quality constraints called **expectations**. These constraints validate records as data flows through your pipeline, giving you control over how to handle invalid data.
 
 :::image type="content" source="../media/5-define-data-quality-expectations.png" alt-text="Screenshot of the declarative pipeline editor, highlighting expectations." border="false" lightbox="../media/5-define-data-quality-expectations.png":::
 

learn-pr/wwl-databricks/design-implement-data-pipelines/includes/7-create-pipeline-lakeflow-declarative.md

@@ -1,10 +1,10 @@
-Production data pipelines require reliability, maintainability, and clear data quality enforcement. As a data engineer, you likely spend significant time writing code to handle incremental processing, orchestrate dependencies, and validate data quality. **Lakeflow Declarative Pipelines** in Azure Databricks addresses these challenges by letting you define *what* your data should look like rather than *how* to process it step by step.
+Production data pipelines require reliability, maintainability, and clear data quality enforcement. As a data engineer, you likely spend significant time writing code to handle incremental processing, orchestrate dependencies, and validate data quality. **Lakeflow Spark Declarative Pipelines** in Azure Databricks addresses these challenges by letting you define *what* your data should look like rather than *how* to process it step by step.
 
 In this unit, you learn how to create data pipelines using the declarative approach, define streaming tables and materialized views, and apply data quality expectations to enforce constraints on your data.
 
 ## Understand the declarative approach
 
-Traditional data pipelines require you to write imperative code that specifies every processing step. You handle incremental processing logic, manage checkpoint recovery, and orchestrate dependencies between tables. With Lakeflow Declarative Pipelines, you instead declare the **desired end state**, and the framework handles the execution details.
+Traditional data pipelines require you to write imperative code that specifies every processing step. You handle incremental processing logic, manage checkpoint recovery, and orchestrate dependencies between tables. With Lakeflow Spark Declarative Pipelines, you instead declare the **desired end state**, and the framework handles the execution details.
 
 The declarative approach provides three key benefits for production pipelines: