Metadata-driven pipelines in Azure Data Factory | Part 4 - Analytical Processing

(2023-Feb-20)  The previous posts covered the following areas of Metadata-driven pipelines in Azure Data Factory: Part 1  - Data Copy Part 2  - Feed Configuration Part 3  - Column Metadata These 3 areas suggested that it is feasible to build a data solution in Azure Data Factory (ADF) where a small set of generic ADF pipelines can support a data load stream of multiple sourcing data feeds with different sets of data columns. This requires fewer efforts to create data load pipelines and more energy to configure the required metadata (i.e feed detailed specification) to support the expected flow of your data. What's next? Now you have all configured data feeds running independently and in parallel, thanks to the support Azure Data Factory functionality . Let's look at the case of using data from individual feeds to populate analytical data tables (they could also be called data warehouse tables) where one analytical table can depend on one or many sourcing feeds and one sourcing

Thinking about data points and more …

Azure DevOps: Deploying Power BI reports with a parameterized gateway-based Data Source

Azure DevOps: Merging code to Main branch from a specific branch only

Including reference data into a database deployment process or passing lookup tables to Production

Metadata-driven pipelines in Azure Data Factory | Part 3 - Column Metadata

Populating PostgreSQL JSONB column using Azure Data Factory Data Flow

Metadata-driven pipelines in Azure Data Factory | Part 2 - Feed Configuration

Metadata-driven pipelines in Azure Data Factory | Part 1 - Data Copy

Fixing a giant or running a SQL Server project deployment with Temporal tables

Trusting the Fine Print or connecting Azure Data Factory with Salesforce

Can I create a CI/CD pipeline to deploy Python Function to Azure Function App using Windows self-hosted Azure DevOps agent?

“Could not find the modules: 'Az.Accounts' with Version: ''” error message and a story to remember

Fail activity in Azure Data Factory and Why would I want to Fail

Error [IM002] [Microsoft][ODBC Driver Manager] "Data source name not found and no default driver specified" and who do you trust?

Lambda Architecture in data systems and possible meaning of this name

Data Modeling at your fingertips

New Azure Data Factory home page, what's in it for me?

Executing Azure Data Factory pipelines by Power App / Automate Flow

Azure Function Drain mode

Blog post about Copying Azure files to SharePoint, comparing Logic App and Power Automate Flow for this, and Trust No One

Creating a generic (template) pipeline in Azure Data Factory to send email messages with attached files from Azure Storage account

Recent updates on my work with Data Vault and Azure Data Factory

Using Logic App to send multiple attachments from Azure Storage Account

How to add a Constant value to Dynamic JSON array in Azure Data Factory

How I tested Parse data transformation in Azure Data Factory

Upgrading Durable Function Extension of my Azure Function App

Passing JSON arrays between pipelines in Azure Data Factory

About Friendship and Using Optional Stored Procedure parameters in Azure Data Factory

Power of Unlearning

Logical crossroads in Azure Data Factory (IF and Filter operations)

These little things about Azure Data Factory

Logging the history of my past SQL Saturday presentations