Posts

Transforming JSON data with the help of Azure Data Factory - Part 5 - Comparing it with Databricks

(2020-July-29) There is a well known and broadly advertised message from Microsoft that Azure Data Factory (ADF) is a code-free environment to help you to create your data integration solutions - https://azure.microsoft.com/en-us/resources/videos/microsoft-azure-data-factory-code-free-cloud-data-integration-at-scale/. I agree and support this approach of using drag and drop visual UI to build and automate data pipelines without writing code. However, I'm also interested to try if I can recreate certain ADF operations by writing code, just out of my curiosity.
Previously I have written a blog post about using ADF Data Flow Flatten operation to transform a JSON file - Part 1: Transforming JSON to CSV with the help of Azure Data Factory - Mapping Data Flows

Collecting custom parameters/variables for Azure Data Factory deployment

Send a Database Query Result as Email via Logic App in HTML format using Azure Data Factory

Transforming JSON data with the help of Azure Data Factory - Part 4 - Passing Arrays around

Replicating SQL IN operator in Azure Data Factory

Stacking Up Datasets in Azure Data Factory

Using Azure Functions in Azure Data Factory

Transforming JSON data with the help of Azure Data Factory - Part 3

Extracting data from Common Data Services (Microsoft Dynamics 365) using Azure Data Factory

Transforming JSON to CSV with the help of Azure Data Factory - Part 2 (Wrangling data flows)

Transforming JSON to CSV with the help of Flatten task in Azure Data Factory

Using Azure Data Factory Mapping Data Flows to populate Data Vault - PASS Video

Azure Databricks Performance Notes

Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub - Part 2

Notifications in Azure Data Factory: Email, Internal Alerts, Log Analytics Alerts

Wrangling data flows in Azure Data Factory: Calculating distance between two Geo-points

Event-driven architecture (EDA) with Azure Data Factory - Triggers made easy

Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub - PASS Video

Switch activity in Azure Data Factory: Container with many IFs

Working with Arrays in Azure Data Factory

Using Azure Data Factory Mapping Data Flows to populate Data Vault

Azure Data Factory: Extracting array first element

Validation activity in Azure Data Factory - Traffic light of your operational workflow