Using Azure Data Factory Mapping Data Flows to populate Data Vault

(2019-May-24)Data Flow as a data transformation engine has been introduced to the Microsoft Azure Data Factory (ADF) last year as a private feature preview. This privacy restriction has been lifted during the last Microsoft Build conference and Data Flow feature has become a public preview component of the ADF.

There are many different use-case scenarios that can be covered by Data Flows, considering that Data Flows in SQL Integration Service (SSIS) projects are still playing a big role to fulfill Extracting-Loading-Transforming (ETL) patterns for your data.

In this blog post, I will share my experience of populating a Data Vault repository by using Data Flows in Azure Data Factory. 

First, I need to create my Data Vault model. 

Data Model

For this exercise I've taken a date warehouse sample of AdventureWorksDW2017 SQL Server database, twhere I limited a set of entities to a small set of dimension tables (DimProduct, DimCustomer, DimGeography) and one fact table (FactInternateSales).


Azure Data Factory: Extracting array first element

Validation activity in Azure Data Factory - Traffic light of your operational workflow

Delete activity in Azure Data Factory - Cleaning up your data files

Communication in Azure: using Data Factory to send messages to Azure Service Bus

Data story behind a Global Fishing Power BI report