Merge pull request #52958 from weslbo/images-implement-lakeflow-jobs

Added images for implement-lake flow-jobs

Author: v-ccolin

learn-pr/wwl-databricks/implement-lakeflow-jobs/includes/2-create-lakeflow-job.md

@@ -23,6 +23,8 @@ To create a new job in Azure Databricks:
 3. Enter a descriptive name for your job.
 4. Configure your first task by specifying the **Task name** and selecting the **Type** (such as Notebook, Python script, or SQL).
 
+:::image type="content" source="../media/2-create-lakeflow-job.png" alt-text="Screenshot of the lakeflow editor." lightbox="../media/2-create-lakeflow-job.png":::
+
 The task type determines which configuration options appear. For a notebook task, you specify the notebook path and any parameters. For a SQL task, you select a query and SQL warehouse. The following table summarizes common task types and their configuration requirements:
 
 | Task type     | Key configuration            | Compute options                             |
@@ -44,6 +46,8 @@ Tasks that run code (notebooks, Python scripts, SQL files) need a source locatio
 
 **DBFS/ADLS** (for Python scripts) allows you to reference files stored in volumes or cloud storage. Provide the full URI, such as `abfss://[email protected]/path/script.py`.
 
+:::image type="content" source="../media/2-configure-lakeflow-task.png" alt-text="Screenshot of the lakeflow editor task configuration." lightbox="../media/2-configure-lakeflow-task.png":::
+
 ## Configure compute resources
 
 Each task needs compute resources to execute. Azure Databricks offers several compute options optimized for different workloads.
@@ -94,6 +98,8 @@ To add job parameters:
 1. In the **Job details** panel, locate the **Parameters** section.
 2. Select **Add** and enter a key-value pair.
 
+:::image type="content" source="../media/2-add-parameters.png" alt-text="Screenshot of the lakeflow editor add parameter section." lightbox="../media/2-add-parameters.png":::
+
 Tasks access parameters differently based on their type. In notebooks, use `dbutils.widgets.get("parameter_name")` to retrieve parameter values. Python scripts receive parameters as command-line arguments.
 
 You can also reference dynamic values in parameters. For example, `{{job.trigger.time.iso_date}}` inserts the trigger date, useful for processing data based on when the job runs.

learn-pr/wwl-databricks/implement-lakeflow-jobs/includes/3-configure-job-triggers.md

@@ -6,6 +6,8 @@ In this unit, you learn how to configure event-based triggers that automate your
 
 Lakeflow Jobs supports several trigger types that determine when your jobs run. While scheduled triggers run at fixed times, event-based triggers respond to data changes in your environment.
 
+:::image type="content" source="../media/3-configure-trigger.png" alt-text="Screenshot of the lakeflow job trigger dialog." lightbox="../media/3-configure-trigger.png":::
+
 | Trigger type | Behavior |
 |--------------|----------|
 | **Table update** | Runs when monitored tables receive new data or modifications |
@@ -64,6 +66,8 @@ To configure a file arrival trigger, specify the storage location using the full
 /Volumes/mycatalog/myschema/myvolume/incoming/
 ```
 
+:::image type="content" source="../media/3-configure-file-arrival-trigger.png" alt-text="Screenshot of the lakeflow file arrival trigger dialog." lightbox="../media/3-configure-file-arrival-trigger.png":::
+
 Before creating a file arrival trigger, verify you have:
 
 - A workspace with Unity Catalog enabled
@@ -83,6 +87,8 @@ Continuous triggers keep your job running by starting a new run immediately afte
 
 When you configure a continuous trigger, your job enters a perpetual cycle: complete a run, start the next run. If a run fails, the trigger still starts a new run, making continuous jobs resilient to transient errors.
 
+:::image type="content" source="../media/3-configure-continuous-trigger.png" alt-text="Screenshot of the lakeflow continuous trigger dialog." lightbox="../media/3-configure-continuous-trigger.png":::
+
 Continuous triggers work well for:
 
 - Processing streaming data where low latency matters

learn-pr/wwl-databricks/implement-lakeflow-jobs/includes/5-configure-job-alerts.md

@@ -31,6 +31,8 @@ Supported system destinations include:
 
 Before you can use system destinations, a workspace administrator must configure them. Navigate to **Admin Settings** and select **Notifications** to create destinations. Each destination requires appropriate credentials—webhook URLs for Slack and Teams, integration keys for PagerDuty.
 
+:::image type="content" source="../media/5-create-new-destination.png" alt-text="Screenshot of the admin notification settings." lightbox="../media/5-create-new-destination.png":::
+
 > [!TIP]
 > Use different credentials for each configured destination. If one third-party endpoint is compromised, you can revoke its access without affecting other notification destinations.
 
@@ -48,6 +50,8 @@ To configure job-level notifications, follow these steps:
 6. Select the event types you want to trigger notifications: **Start**, **Success**, **Failure**, **Duration warning**, or **Streaming backlog**.
 7. Select **Save** when you finish adding notifications.
 
+:::image type="content" source="../media/5-create-job-notification.png" alt-text="Screenshot showing job notifications dialog." lightbox="../media/5-create-job-notification.png":::
+
 Job-level notifications apply to the overall job run. However, job-level notifications aren't sent when individual tasks fail and retry. Consider this behavior when designing your notification strategy.
 
 ## Configure task-level notifications
@@ -61,6 +65,8 @@ To add task notifications:
 3. Configure the destination and event types just as you would for job notifications.
 4. Save the task configuration.
 
+:::image type="content" source="../media/5-create-task-notification.png" alt-text="Screenshot showing task-level notifications." lightbox="../media/5-create-task-notification.png":::
+
 Task notifications are particularly valuable when your job contains multiple independent tasks. If Task A fails, you receive an immediate notification rather than waiting for the entire job to complete or fail.
 
 By default, Azure Databricks retries failed tasks three times. If you don't want notifications for every retry attempt, select **Mute notifications until the last retry**. This reduces noise while still alerting you when a task ultimately fails.
@@ -69,6 +75,8 @@ By default, Azure Databricks retries failed tasks three times. If you don't want
 
 Alert fatigue undermines the value of notifications. When teams receive too many alerts, they start ignoring them—including the critical ones. Apply these strategies to keep your alerts meaningful.
 
+:::image type="content" source="../media/5-alert-fatigue.png" alt-text="Diagram explaining alert fatigue." border="false" lightbox="../media/5-alert-fatigue.png":::
+
 **Filter out skipped and canceled runs**: When you cancel a job or a run gets skipped due to concurrent run limits, you might not need a notification. Select **Mute notifications for skipped runs** or **Mute notifications for canceled runs** to suppress these.
 
 **Use duration warnings strategically**: Rather than alerting on every long-running job, set duration thresholds based on historical performance. A job that usually takes 30 minutes might warrant a warning at 45 minutes—not at 35.
@@ -103,6 +111,8 @@ If you require specific formatting for notifications, webhooks let you control t
 
 When your Databricks jobs are orchestrated by Azure Data Factory (ADF), you have additional monitoring options. ADF provides visual monitoring in the Azure portal where you can track pipeline runs, activity status, and execution duration.
 
+:::image type="content" source="../media/5-azure-data-factory-pipeline-runs.png" alt-text="Screenshot of Azure Databricks list view for monitoring pipeline runs." lightbox="../media/5-azure-data-factory-pipeline-runs.png":::
+
 ADF also supports creating alerts on metrics such as:
 
 - Pipeline run status (failed, succeeded, canceled)

learn-pr/wwl-databricks/implement-lakeflow-jobs/includes/6-configure-job-automatic-restarts.md

@@ -24,8 +24,12 @@ To set a retry policy for a task:
 3. In the task configuration panel, select **+ Add** next to **Retries**
 4. Specify the number of retry attempts and the interval between retries
 
+:::image type="content" source="../media/6-configure-task-level-retry-policy.png" alt-text="Screenshot of the retry policy dialog." lightbox="../media/6-configure-task-level-retry-policy.png":::
+
 The **retry interval** is calculated in milliseconds between the start of the failed run and the subsequent retry run. For example, if you set an interval of 60,000 milliseconds (1 minute) and your task took 30 seconds before failing, the next retry starts 30 seconds after the failure.
 
+:::image type="content" source="../media/6-task-execution-flow.png" alt-text="Diagram explaining task execution flow." border="false" lightbox="../media/6-task-execution-flow.png":::
+
 > [!TIP]
 > Set reasonable retry limits—typically 1 to 3 retries for most workloads. Avoid configuring unlimited retries, as a persistent failure will waste resources without resolving the underlying issue.
 
@@ -43,12 +47,16 @@ To configure a job to run in continuous mode:
 4. Optionally, select a **Task retry mode**—choose **On failure** to retry failed tasks within a job, or **Never** to only retry at the job level
 5. Select **Save**
 
+:::image type="content" source="../media/6-configure-continuous-job-retry-mode.png" alt-text="Screenshot of the continuous job task retry mode dialog." lightbox="../media/6-configure-continuous-job-retry-mode.png":::
+
 The **exponential backoff algorithm** works as follows:
 
 1. When consecutive failures exceed a threshold, the job waits before the next retry
 2. Each subsequent failure increases the wait period up to a maximum set by the system
 3. If a run completes successfully or runs without failure for a threshold period, the **backoff sequence resets**
 
+:::image type="content" source="../media/6-exponential-backoff-algorithm.png" alt-text="Diagram explaining the exponential backoff pattern." border="false" lightbox="../media/6-exponential-backoff-algorithm.png":::
+
 > [!NOTE]
 > Continuous jobs don't support task dependencies or task-level retry policies in the traditional sense. Instead, set the **Task retry mode** to control whether failed tasks retry before triggering a new job run.
 
@@ -58,13 +66,17 @@ You can monitor continuous jobs in the exponential backoff state through the **J
 
 A task that hangs indefinitely blocks downstream processing and wastes compute resources. **Timeout thresholds** terminate unresponsive tasks so your job can either retry or fail cleanly.
 
+:::image type="content" source="../media/6-timeout-threshold.png" alt-text="Diagram explaining timout behavior." border="false" lightbox="../media/6-timeout-threshold.png":::
+
 To configure timeout thresholds:
 
 1. In the task configuration panel, select **Metric thresholds**
 2. Select **Run duration** in the **Metric** dropdown
 3. Enter a duration in the **Warning** field to trigger a notification when the task exceeds expected completion time
 4. Enter a duration in the **Timeout** field to terminate the task if it exceeds maximum completion time
 
+:::image type="content" source="../media/6-configure-metric-thresholds.png" alt-text="Screenshot of the metric thresholds dialog." lightbox="../media/6-configure-metric-thresholds.png":::
+
 When a task times out, Azure Databricks sets its status to "Timed Out" and handles it according to your retry policy. If retries remain, the task restarts. If all retries are exhausted, the task fails and any dependent tasks are affected based on your job's dependency configuration.
 
 ## Combine automatic restarts with notifications