Skip to content

Commit 5256ba6

Browse files
authored
Update 2-solution-azure-data-factory.md
1 parent fcca784 commit 5256ba6

1 file changed

Lines changed: 16 additions & 4 deletions

File tree

learn-pr/wwl-azure/design-data-integration/includes/2-solution-azure-data-factory.md

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,12 @@
77
There are four major steps to create and implement a data-driven workflow in the Azure Data Factory architecture:
88

99
1. **Connect and collect**. First, ingest the data to collect all the data from different sources into a centralized location.
10-
2. **Transform and enrich**. Next, transform the data by using a compute service like Azure Databricks and Azure HDInsight Hadoop.
11-
3. **Provide continuous integration and delivery (CI/CD) and publish**. Support CI/CD by using GitHub and Azure Pipelines to deliver the ETL process incrementally before publishing the data to the analytics engine.
12-
4. **Monitor**. Finally, use the Azure portal to monitor the pipeline for scheduled activities and for any failures.
10+
11+
1. **Transform and enrich**. Next, transform the data by using a compute service like Azure Databricks and Azure HDInsight Hadoop.
12+
13+
1. **Provide continuous integration and delivery (CI/CD) and publish**. Support CI/CD by using GitHub and Azure Pipelines to deliver the ETL process incrementally before publishing the data to the analytics engine.
14+
15+
1. **Monitor**. Finally, use the Azure portal to monitor the pipeline for scheduled activities and for any failures.
1316

1417
The following diagram shows how Azure Data Factory orchestrates the ingestion of data from different data sources. Data is ingested into a Storage blob and stored in Azure Synapse Analytics. Analysis and visualization components are also connected to Azure Data Factory. Azure Data Factory provides a common management interface for all of your data integration needs.
1518

@@ -34,16 +37,25 @@ A significant challenge for a fast-growing home improvement retailer like Tailwi
3437
Let's review how the Azure Data Factory components are involved in a data preparation and movement scenario for Tailwind Traders. They have many different data sources to connect to and that data needs to be ingested and transformed through stored procedures that are run on the data. Finally, the data should be pushed to an analytics platform for analysis.
3538

3639
- In this scenario, the linked service enables Tailwind Traders to ingest data from different sources and it stores connection strings to fire up compute services on demand.
40+
3741
- You can execute stored procedures for data transformation that happens through the linked service in Azure-SSIS, which is the integration runtime environment for Tailwind Traders.
42+
3843
- The datasets components are used by the activity object and the activity object contains the transformation logic.
44+
3945
- You can trigger the pipeline, which is all the activities grouped together.
46+
4047
- You can use Azure Data Factory to publish the final dataset consumed by technologies, such as Power BI or Machine Learning.
4148

4249
### Things to consider when using Azure Data Factory
4350

4451
Evaluate Azure Data Factory against the following decision criteria and consider how the service can benefit your data integration solution for Tailwind Traders.
4552

4653
- **Consider requirements for data integration**. Azure Data Factory serves two communities: the big data community and the relational data warehousing community that uses SQL Server Integration Services (SSIS). Depending on your organization's data needs, you can set up pipelines in the cloud by using Azure Data Factory. You can access data from both cloud and on-premises data services.
54+
4755
- **Consider coding resources**. If you prefer a graphical interface to set up pipelines, then Azure Data Factory authoring and monitoring tool is the right fit for your needs. Azure Data Factory provides a low code/no code process for working with data sources.
48-
- **Consider support for multiple data sources**. Azure Data Factory supports 90+ connectors to integrate with disparate data sources.
56+
57+
- **Consider support for multiple data sources**. Azure Data Factory supports 100+ connectors, including Microsoft Fabric Warehouse and Fabric Lakehouse alongside Azure, AWS, Google Cloud, SaaS, and database sources.
58+
4959
- **Consider serverless infrastructure**. There are advantages to using a fully managed, serverless solution for data integration. There's no need to maintain, configure, or deploy servers, and you gain the ability to scale with fluctuating workloads.
60+
61+

0 commit comments

Comments
 (0)