Data Adventures

(2023-Feb-20)  The previous posts covered the following areas of Metadata-driven pipelines in Azure Data Factory: Part 1  - Data Copy Part 2  - Feed Configuration Part 3  - Column Metadata These 3 areas suggested that it is feasible to build a data solution in Azure Data Factory (ADF) where a small set of generic ADF pipelines can support a data load stream of multiple sourcing data feeds with different sets of data columns. This requires fewer efforts to create data load pipelines and more energy to configure the required metadata (i.e feed detailed specification) to support the expected flow of your data. What's next? Now you have all configured data feeds running independently and in parallel, thanks to the support Azure Data Factory functionality . Let's look at the case of using data from individual feeds to populate analytical data tables (they could also be called data warehouse tables) where one analytical table can depend on one or many sourcing feeds and one sourcing