Merge pull request #54135 from staleycyn/patch-3

Content drift for the design data integration module

Author: v-dirichards

learn-pr/wwl-azure/design-data-integration/includes/2-solution-azure-data-factory.md

@@ -7,9 +7,12 @@
 There are four major steps to create and implement a data-driven workflow in the Azure Data Factory architecture:
 
 1. **Connect and collect**. First, ingest the data to collect all the data from different sources into a centralized location.
-2. **Transform and enrich**. Next, transform the data by using a compute service like Azure Databricks and Azure HDInsight Hadoop.
-3. **Provide continuous integration and delivery (CI/CD) and publish**. Support CI/CD by using GitHub and Azure Pipelines to deliver the ETL process incrementally before publishing the data to the analytics engine.
-4. **Monitor**. Finally, use the Azure portal to monitor the pipeline for scheduled activities and for any failures.
+   
+1. **Transform and enrich**. Next, transform the data by using a compute service like Azure Databricks and Azure HDInsight Hadoop.
+   
+1. **Provide continuous integration and delivery (CI/CD) and publish**. Support CI/CD by using GitHub and Azure Pipelines to deliver the ETL process incrementally before publishing the data to the analytics engine.
+   
+1. **Monitor**. Finally, use the Azure portal to monitor the pipeline for scheduled activities and for any failures.
 
 The following diagram shows how Azure Data Factory orchestrates the ingestion of data from different data sources. Data is ingested into a Storage blob and stored in Azure Synapse Analytics. Analysis and visualization components are also connected to Azure Data Factory. Azure Data Factory provides a common management interface for all of your data integration needs.
 
@@ -34,16 +37,25 @@ A significant challenge for a fast-growing home improvement retailer like Tailwi
 Let's review how the Azure Data Factory components are involved in a data preparation and movement scenario for Tailwind Traders. They have many different data sources to connect to and that data needs to be ingested and transformed through stored procedures that are run on the data. Finally, the data should be pushed to an analytics platform for analysis.
 
 - In this scenario, the linked service enables Tailwind Traders to ingest data from different sources and it stores connection strings to fire up compute services on demand.
+  
 - You can execute stored procedures for data transformation that happens through the linked service in Azure-SSIS, which is the integration runtime environment for Tailwind Traders.
+  
 - The datasets components are used by the activity object and the activity object contains the transformation logic.
+  
 - You can trigger the pipeline, which is all the activities grouped together.
+  
 - You can use Azure Data Factory to publish the final dataset consumed by technologies, such as Power BI or Machine Learning.
 
 ### Things to consider when using Azure Data Factory
 
 Evaluate Azure Data Factory against the following decision criteria and consider how the service can benefit your data integration solution for Tailwind Traders.
 
 - **Consider requirements for data integration**. Azure Data Factory serves two communities: the big data community and the relational data warehousing community that uses SQL Server Integration Services (SSIS). Depending on your organization's data needs, you can set up pipelines in the cloud by using Azure Data Factory. You can access data from both cloud and on-premises data services.
+  
 - **Consider coding resources**. If you prefer a graphical interface to set up pipelines, then Azure Data Factory authoring and monitoring tool is the right fit for your needs. Azure Data Factory provides a low code/no code process for working with data sources.
-- **Consider support for multiple data sources**. Azure Data Factory supports 90+ connectors to integrate with disparate data sources.
+  
+- **Consider support for multiple data sources**. Azure Data Factory supports 100+ connectors, including Microsoft Fabric Warehouse and Fabric Lakehouse alongside Azure, AWS, Google Cloud, SaaS, and database sources.
+  
 - **Consider serverless infrastructure**. There are advantages to using a fully managed, serverless solution for data integration. There's no need to maintain, configure, or deploy servers, and you gain the ability to scale with fluctuating workloads.
+
+

learn-pr/wwl-azure/design-data-integration/includes/3-solution-azure-data-lake.md

@@ -2,19 +2,25 @@ A data lake is a repository of data stored in its natural format, usually as blo
 
 > [!VIDEO https://learn-video.azurefd.net/vod/player?id=b4c743cb-38b8-4d39-a99a-3e7c39803836]
 
-> [!Note] 
-> The current implementation of the service is Azure Data Lake Storage Gen2.
+> [!Important] 
+> Azure Data Lake Storage Gen1 was retired on February 29, 2024. Existing Gen1 accounts are no longer accessible and new accounts cannot be created. This unit covers Azure Data Lake Storage Gen2 exclusively.
 
 ### Things to know about Azure Data Lake Storage
 
 To better understand Azure Data Lake Storage, let's examine the following characteristics.
 
 - Azure Data Lake Storage can store any type of data by using the data's native format. With support for any data format and massive data sizes, Azure Data Lake Storage can work with structured, semi-structured, and unstructured data.
+  
 - The solution is primarily designed to work with Hadoop and all frameworks that use the Apache Hadoop Distributed File System (HDFS) as their data access layer. Data analysis frameworks that use HDFS as their data access layer can directly access.
+  
 - Azure Data Lake Storage supports high throughput for input and output–intensive analytics and data movement.
+  
 - The Azure Data Lake Storage access control model supports both Azure role-based access control (RBAC) and Portable Operating System Interface for UNIX (POSIX) access control lists (ACLs).
-- Azure Data Lake Storage utilizes Azure Blob replication models. These models provide data redundancy in a single datacenter with locally redundant storage (LRS).
+  
+- Azure Data Lake Storage utilizes Azure Blob replication models. These models support the same redundancy options available for Azure Blob Storage. Microsoft recommends ZRS for Azure Data Lake Storage workloads.
+  
 - Azure Data Lake Storage offers massive storage and accepts numerous data types for analytics.
+  
 - Azure Data Lake Storage is priced at Azure Blob Storage levels.
 
 #### How Azure Data Lake Storage works
@@ -57,4 +63,7 @@ The following table compares storage solution criteria for using Azure Blob Stor
 | **Geographic redundancy** | Must manually configure data replication               | Provides geo-redundant storage by default                    |
 | **Namespaces**            | Supports hierarchical namespaces                       | Supports flat namespaces                                     |
 | **Hadoop compatibility**  | Hadoop services can use data stored in Azure Data Lake | By using Azure Blob Filesystem Driver, applications and frameworks can access data in Azure Blob Storage |
-| **Security**              | Supports granular access                               | Granular access isn't supported                              |
+| **Security**              | Supports granular access                               | Granular access isn't supported                              |
+
+> [!Tip]
+> Learn more with self-paced training, [Introduction to Azure Data Lake Storage Gen2](/training/modules/introduction-to-azure-data-lake-storage/).

learn-pr/wwl-azure/design-data-integration/includes/4-solution-azure-data-brick.md

@@ -6,15 +6,18 @@
 
 Azure Databricks is entirely based on Apache Spark, and it's a great tool for users who are already familiar with the open-source cluster-computing framework. Databricks is designed specifically for big data processing. Data scientists can take advantage of the built-in core API for core languages like SQL, Java, Python, R, and Scala.
 
-Azure Databricks has a Control plane and a Data plane:
+Azure Databricks has a Control plane and a Compute plane:
 
-- **Control Plane**: Hosts Databricks jobs, notebooks with query results, and the cluster manager. The Control plane also has the web application, hive metastore, and security access control lists (ACLs), and user sessions. Microsoft manages these components in collaboration with Azure Databricks.
-- **Data Plane**: Contains all the Azure Databricks runtime clusters that are hosted within the workspace. All data processing and storage exists within the client subscription. No data processing ever takes place within the Microsoft/Databricks-managed subscription.
+- **Control Plane**: Hosts Databricks jobs, notebooks with query results, and the cluster manager. The Control plane also has the web application, security access control lists (ACLs), and user sessions. Microsoft manages these components in collaboration with Azure Databricks.
+  
+- **Compute Plane**: Contains all the Azure Databricks runtime clusters that are hosted within the workspace. All data processing and storage exists within the client subscription. 
 
 Azure Databricks offers three environments for developing data intensive applications.
 
 - **Databricks SQL**: Azure Databricks SQL provides an easy-to-use platform for analysts who want to run SQL queries on their data lake. You can create multiple visualization types to explore query results from different perspectives, and build and share dashboards.
-- **Databricks Data Science & Engineering**: Azure Databricks Data Science & Engineering is an interactive *workspace* that enables collaboration between data engineers, data scientists, and machine learning engineers. For a big data pipeline, the data (raw or structured) is ingested into Azure through Azure Data Factory in batches, or streamed near real-time by using Apache Kafka, Azure Event Hubs, or Azure IoT Hub. The data lands in a data lake for long term persisted storage within Azure Blob Storage or Azure Data Lake Storage. As part of your analytics workflow, use Azure Databricks to read data from multiple data sources and turn it into breakthrough insights by using Spark.
+  
+- **Databricks Data Science & Engineering**: Azure Databricks Data Science & Engineering lets data teams work together in an interactive workspace. Data is brought into Azure through batch or real-time tools like Azure Data Factory, Kafka, Event Hubs, or IoT Hub. Data is stored in Azure Blob Storage or Data Lake Storage. Databricks reads data from these sources and uses Spark to generate insights.
+  
 - **Databricks Machine Learning**: Azure Databricks Machine Learning is an integrated end-to-end machine learning environment. It incorporates managed services for experiment tracking, model training, feature development and management, and feature and model serving.
 
 #### Business scenario
@@ -23,21 +26,34 @@ Let's analyze a scenario for Tailwind Traders in the heavy machinery manufacturi
 
 Let's review why Azure Databricks can be the right choice to meet these requirements.
 
-- Azure Databricks provides an integrated Analytics *workspace* based on Apache Spark that allows collaboration between different users.
-- By using Spark components like Spark SQL and Dataframes, Azure Databricks can handle structured data. It integrates with real-time data ingestion tools like Kafka and Flume for processing streaming data.
-- Secure data integration capabilities built on top of Spark enable you to unify your data without centralization. Data scientists can visualize data in a few steps, and use familiar tools like Matplotlib, ggplot, or d3.
-- The Azure Databricks runtime abstracts out the infrastructure complexity and the need for specialized expertise to set up and configure your data infrastructure. Users can use existing languages skills for Python, Scala, and R, and explore the data.
-- Azure Databricks integrates deeply with Azure databases and stores like Azure Synapse Analytics, Azure Cosmos DB, Azure Data Lake Storage, and Azure Blob Storage. It supports diverse data store platforms, which satisfies the Tailwind Traders big data storage needs.
-- Integration with Power BI allows for quick and meaningful insights, which is a requirement for Tailwind Traders.
-- Azure Databricks SQL isn't the right choice because it can't handle unstructured data.
-- Azure Databricks Machine Learning is also not the right environment choice because machine learning isn't a requirement in this scenario.
+-	Azure Databricks is an analytics workspace built on Apache Spark.
+
+-	Supports collaboration and handles both structured and streaming data.
+  
+-	Integrates with real-time tools like Kafka and Flume.
+  
+-	Lets users work with Python, Scala, or R.
+  
+-	Connects to Azure databases and storage solutions, meeting big data needs.
+  
+-	Works with Power BI for fast insights.
+  
+-	Databricks SQL and Machine Learning aren't suitable here, as unstructured data and machine learning aren't required.
+
 
 ### Things to consider when using Azure Databricks
 
 You can use Azure Databricks as a solution for multiple scenarios. Consider how the service can benefit your data integration solution for Tailwind Traders.
 
 - **Consider data science preparation of data**. Create, clone, and edit clusters of complex, unstructured data. Turn the data clusters into specific jobs. Deliver the results to data scientists and data analysts for review.
+  
 - **Consider insights in the data**. Implement Azure Databricks to build recommendation engines, churn analysis, and intrusion detection.
+  
 - **Consider productivity across data and analytics teams**. Create a collaborative environment and shared workspaces for data engineers, analysts, and scientists. Teams can work together across the data science lifecycle with shared workspaces, which helps to save valuable time and resources.
-- **Consider big data workloads**. Exercise Azure Data Lake and the engine to get the best performance and reliability for your big data workloads. Create no-fuss multi-step data pipelines.
-- **Consider machine learning programs**. Take advantage of the integrated end-to-end machine learning environment. It incorporates managed services for experiment tracking, model training, feature development and management, and feature and model serving.
+  
+- **Consider big data workloads**. Use Azure Data Lake and the engine to get the best performance and reliability for your big data workloads. Create no-fuss multi-step data pipelines.
+  
+- **Consider machine learning programs**. Take advantage of the integrated end-to-end machine learning environment. It incorporates managed services for experiment tracking, model training, feature development and management, and feature and model serving.
+
+> [!Tip]
+> Learn more with self-paced training, [Explore Azure Databricks](/training/wwl-databricks/explore-azure-databricks/).

learn-pr/wwl-azure/design-data-integration/includes/5-solution-azure-synapse-analytics.md

@@ -22,11 +22,15 @@ Azure Synapse Analytics is composed of the five elements:
 
 :::image type="content" source="../media/azure-synapse-analytics-overview.png" alt-text="Diagram that shows an overview of Azure Synapse Analytics capabilities." border="false":::
 
-- **Azure Synapse SQL pool**: Synapse SQL offers both serverless and dedicated resource models to work with a node-based architecture. For predictable performance and cost, you can create dedicated SQL pools. For irregular or unplanned workloads, you can use the always-available, serverless SQL endpoint.
-- **Azure Synapse Spark pool**: This pool is a cluster of servers that run Apache Spark to process data. You write your data processing logic by using one of the four supported languages: Python, Scala, SQL, and C# (via .NET for Apache Spark). Apache Spark for Azure Synapse integrates Apache Spark (the open source big data engine used for data preparation, data engineering, ETL, and machine learning).
-- **Azure Synapse Pipelines**: Azure Synapse Pipelines applies the capabilities of Azure Data Factory. Pipelines are the cloud-based ETL and data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale. You can include activities that transform the data as it's transferred, or you can combine data from multiple sources together.
-- **Azure Synapse Link**: This component allows you to connect to Azure Cosmos DB. You can use it to perform near real-time analytics over the operational data stored in an Azure Cosmos DB database.
-- **Azure Synapse Studio**: This element is a web-based IDE that can be used centrally to work with all capabilities of Azure Synapse Analytics. You can use Azure Synapse Studio to create SQL and Spark pools, define and run pipelines, and configure links to external data sources.
+- **Azure Synapse SQL pool**: Choose between dedicated SQL pools for consistent performance and cost, or serverless SQL endpoints for flexible, on-demand workloads.
+  
+- **Azure Synapse Spark pool**: Run Apache Spark clusters to process data using Python, Scala, SQL, or C#.
+  
+- **Azure Synapse Pipelines**: Use cloud-based ETL workflows to move and transform data at scale, combining multiple sources if needed.
+  
+- **Azure Synapse Link**: Connect to Azure Cosmos DB for near real-time analytics on operational data.
+  
+- **Azure Synapse Studio**: Work in a central web-based IDE to manage SQL and Spark pools, pipelines, and data links.
 
 #### Analytical options
 
@@ -50,14 +54,22 @@ The following table compares storage solution criteria for using Azure Data Fact
 | Compare                                    | Azure Data Factory                                           | Azure Synapse Analytics                                      |
 | ------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
 | **Data sharing**                           | Data can be shared across different data factories           | Not supported                                                |
-| **Solution templates**                     | Solution templates are provided with the Azure Data Factory template gallery | Solution templates are provided in the Synapse Workspace Knowledge center |
+| **Solution templates**                     | Solution templates are provided with the Azure Data Factory template gallery | Solution templates are provided in the Synapse Workspace Knowledge Center |
 | **Integration runtime cross region flows** | Cross region data flows are supported                        | Not supported                                                |
 | **Monitor data**                           | Data monitoring is integrated with Azure Monitor             | Diagnostic logs are available in Azure Monitor               |
 | **Monitor Spark Jobs for data flow**       | Not supported                                                | Spark Jobs can be monitored for data flow by using Synapse Spark pools |
 
 Azure Synapse Analytics is an ideal solution for many other scenarios. Consider the following options:
 
 - **Consider variety of data sources**. When you have various data sources that use Azure Synapse Analytics for code-free ETL and data flow activities.
+  
 - **Consider Machine Learning**. When you need to implement Machine Learning solutions by using Apache Spark, you can use Azure Synapse Analytics for built-in support for Azure Machine Learning.
+  
 - **Consider data lake integration**. When you have existing data stored on a data lake and need integration with Azure Data Lake and other input sources, Azure Synapse Analytics provides seamless integration between the two components.
+  
 - **Consider real-time analytics**. When you require real-time analytics, you can use features like Azure Synapse Link to analyze data in real-time and offer insights.
+  
+- **Consider Microsoft Fabric**. Microsoft recommends Microsoft Fabric over new Synapse deployments.
+
+> [!Tip]
+> Learn more with self-paced training, [Introduction to end-to-end analytics using Microsoft Fabric](/training/modules/introduction-end-analytics-use-microsoft-fabric/).