Skip to main content

Data Adventures

My personal journey in this intricate world of data and continuous efforts to make it more structured and well understood - Rayis Imayev.

Search This Blog

Subscribe to this blog

Follow by Email

Posts

Transforming JSON data with the help of Azure Data Factory - Part 5 - Comparing it with Databricks

Posted by Rayis Imayev on July 29, 2020

(2020-July-29) There is a well known and broadly advertised message from Microsoft that Azure Data Factory (ADF) is a code-free environment to help you to create your data integration solutions - https://azure.microsoft.com/en-us/resources/videos/microsoft-azure-data-factory-code-free-cloud-data-integration-at-scale/. I agree and support this approach of using drag and drop visual UI to build and automate data pipelines without writing code. However, I'm also interested to try if I can recreate certain ADF operations by writing code, just out of my curiosity.
Previously I have written a blog post about using ADF Data Flow Flatten operation to transform a JSON file - Part 1: Transforming JSON to CSV with the help of Azure Data Factory - Mapping Data Flows

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Collecting custom parameters/variables for Azure Data Factory deployment

Posted by Rayis Imayev on June 22, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Send a Database Query Result as Email via Logic App in HTML format using Azure Data Factory

Posted by Rayis Imayev on June 07, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON data with the help of Azure Data Factory - Part 4 - Passing Arrays around

Posted by Rayis Imayev on May 24, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Replicating SQL IN operator in Azure Data Factory

Posted by Rayis Imayev on May 01, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Stacking Up Datasets in Azure Data Factory

Posted by Rayis Imayev on April 24, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Using Azure Functions in Azure Data Factory

Posted by Rayis Imayev on April 19, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON data with the help of Azure Data Factory - Part 3

Posted by Rayis Imayev on April 06, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Extracting data from Common Data Services (Microsoft Dynamics 365) using Azure Data Factory

Posted by Rayis Imayev on March 30, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON to CSV with the help of Azure Data Factory - Part 2 (Wrangling data flows)

Posted by Rayis Imayev on March 26, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON to CSV with the help of Flatten task in Azure Data Factory

Posted by Rayis Imayev on March 19, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Using Azure Data Factory Mapping Data Flows to populate Data Vault - PASS Video

Posted by Rayis Imayev on March 15, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Azure Databricks Performance Notes

Posted by Rayis Imayev on February 04, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub - Part 2

Posted by Rayis Imayev on January 28, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Notifications in Azure Data Factory: Email, Internal Alerts, Log Analytics Alerts

Posted by Rayis Imayev on December 15, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Wrangling data flows in Azure Data Factory: Calculating distance between two Geo-points

Posted by Rayis Imayev on November 10, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Event-driven architecture (EDA) with Azure Data Factory - Triggers made easy

Posted by Rayis Imayev on October 27, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub - PASS Video

Posted by Rayis Imayev on October 23, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Switch activity in Azure Data Factory: Container with many IFs

Posted by Rayis Imayev on October 16, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Working with Arrays in Azure Data Factory

Posted by Rayis Imayev on June 06, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Using Azure Data Factory Mapping Data Flows to populate Data Vault

Posted by Rayis Imayev on May 24, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Azure Data Factory: Extracting array first element

Posted by Rayis Imayev on April 28, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Validation activity in Azure Data Factory - Traffic light of your operational workflow

Posted by Rayis Imayev on April 02, 2019

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Rayis Imayev

Follow @RayisImayev

Archive

2020 14

2019 19
- December 1
- November 1
- October 3
- June 1
- May 1
- April 2
- March 3
- February 4
- January 3
2018 16
- December 1
- November 2
- October 2
- August 1
- July 3
- May 2
- March 1
- February 2
- January 2
2017 12
- December 1
- November 1
- October 1
- September 1
- August 1
- July 1
- June 1
- May 1
- April 1
- March 1
- February 1
- January 1
2016 9
- November 1
- October 1
- September 1
- August 1
- July 1
- June 1
- May 1
- April 1
- February 1
2015 10
- December 1
- November 1
- October 1
- August 1
- July 2
- June 1
- May 1
- April 1
- March 1
2014 11
- December 1
- November 1
- October 2
- August 1
- July 3
- April 1
- March 1
- February 1

Show more Show less

Labels

3D Map ADF ADLS Alerts Analytics Animation Apache Spark API Append Variable ArcGIS

Architect Architecture ArgGIS Map Array Azure Azure Data Factory Azure Data Lake Store Azure Databricks Azure Function Azure Logic App Azure Monitor Azure Service Bus Big Data Bing Map Blob Blob Storage Container Build CDS CI/CD Common Data Services Condition continuous delivery continuous integration CRM CSV Custom Maps Custom Shapefile Custom Solution Data Data Architecture Data Factory Data Lake Data Modeling Data Vault Database Design Databricks DataForACause Delete Activity DevOps Distance Dual axes Email Notification EPSG 4326 ER Studio ESRI Excel File Management Filled Map Flatten Flow map Function Gallery Template Geometry GIS Git GitHub Global Fishing Globe Map HDFS Heatmap HTML Icon Map IF activity Ingestion Integration JSON Latitude Load Log Analytics Longitude M language Macro Mapbox Mapping Data Flows Maps Metrics Microsoft Dynamics 365 Monitor Narrative Science Network Open Data Parameter Parent Child Nodes PASS Performance Pipeline Polygon Power BI Power Map Power Query Predictive Analytics Primary Secondary axis Pulse Chart QGIS Queue Reference Layer Release REST API Send Message Shape Map Solution Template Source Control Spark Spatial Data Spatial Functions SQL SQL Azure SQL Azure Database SQL Server SSAS SSMS SSRS Statistics Switch Activity System Variables Tabular Template Template Gallery Time Time Slider TopoJSON Trigger Twitter Validation Variables Version Visualization Web Webhook WGS 84 Wrangling data flows XML

Show more Show less

Popular Posts

Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub

(2019-Feb-18) With Azure Data Factory (ADF) continuous integration, you help your team to collaborate and develop data transformation solutions within the same data factory workspace and maintain your combined development efforts in a central code repository. Continuousdelivery helps to build and deploy your ADF solution for testing and release purposes. Basically, the CI/CD process helps to establish a good software development practice and aims to build a healthy relationship between development, quality assurance, and other supporting teams.

Update 2020-Mar-15: Part 2 of this blog post published in 2020-Jan-28:
http://datanrg.blogspot.com/2020/01/continuous-integration-and-delivery.html

Update 2019-Jun-24:
Video recording of my webinar session on Continuous Integration and Delivery (CI/CD) in Azure Data Factory at a recent PASS Cloud Virtual Group meeting.

Working with Arrays in Azure Data Factory

(2019-June-06) I remember that I had a professor at my university who would often encourage his students to learn and get more experienced with simple things first. Learn the basics in and out and then move forward to more complicated concepts and practices; that was his motto, and he really tried to share this idea with us.

Previously published blog posts about using Variables in Azure Data Factory:
- Setting Variables in Azure Data Factory Pipelines
- Append Variable activity in Azure Data Factory: Story of combining things together
- System Variables in Azure Data Factory: Your Everyday Toolbox
- Azure Data Factory: Extracting array first element

Simple things sometimes can be overlooked as well. With the addition of Variables in Azure Data Factory Control Flow (there were not available there at the beginning), Arrays have become one of those simple things to me.

Image by Magnascan from Pixabay

Setting Variables in Azure Data Factory Pipelines

(2018-Oct-15) Working with Azure Data Factory you always tend to compare its functionality with well established ETL packages in SSIS. Data flow task have been recreated as Data Copy activities; logical components have found they cloud-based siblings; as well as new kids on the block, such as Databricks and Machine Learning activities could boost adoption rate of Azure Data Factory (ADF) pipelines.

Support for local variables hasn't always beenavailable in ADF and was only recently introduced to already available pipeline parameters. This addition makes more flexible to create interim properties (variables) that you can adjust multiple times within a workflow of your pipeline.

Append Variable activity in Azure Data Factory: Story of combining things together

(2018-Oct-29) There are only a few sentences in the official Microsoft web page that describe newly introduced activity task (Append Variable) to add a value to an existing array variable defined in Azure Data Factory - Append Variable Activity in Azure Data Factory But it significantly improves your ability to control a workflow of the data transformation activities of your Data Factory pipeline.

Using Azure Functions in Azure Data Factory

(2020-Apr-19) Creating a data solution with Azure Data Factory (ADF) may look like a straightforward process: you have incoming datasets, business rules of how to connect and change them and a final destination environment to save this transformed data. Very often your data transformation may require more complex business logic that can only be developed externally (scripts, functions, web-services, databricks notebooks, etc.).

In this blog post, I will try to share my experience of using Azure Functions in my Data Factory workflows: my highs and lows of using them, my victories and struggles to make them work. If you share the same pain points, if you find any mistakes or feel a total misrepresentation of facts, please leave your comments, there is no better opportunity to learn from positive critiques :-)

Azure Functions gives you the freedom to create and execute a small or moderate size of code in C#, Java, JavaScript, Python, or PowerShell. This freedom releases you from a need to …

Transforming JSON to CSV with the help of Flatten task in Azure Data Factory

(2020-Mar-19) Recently, Microsoft introduced a new Flatten task to the existing set of powerful transformations available in the Azure Data Factory (ADF) Mapping Data Flows - https://docs.microsoft.com/en-us/azure/data-factory/data-flow-flatten.

What this new task does it helps to transform/transpose/flatten your JSON structure into a denormalized flatten datasets that you can upload into a new or existing flat database table.

2020-Mar-26 Update:
Part 2: Transforming JSON to CSV with the help of Flatten task in Azure Data Factory - Part 2 (Wrangling data flows)

I like the analogy of the Transpose function in Excel that helps to rotate your vertical set of data pairs (name : value) into a table with the column names and values for corresponding objects. And when this vertical JSON structural set contains several similar sets (array) then ADF Mapping Data Flows Flatten does a really good job by transforming it into a table with several rows (records).

Azure Data Factory: Extracting array first element

(2019-Apr-28)Full credit goes toMicrosoft for the latest efforts updating Azure products with new features and documenting corresponding changes for the end users. Azure Data Factory (ADF) is a great example of this.

A user recently asked me a question on my previous blog post (Setting Variables in Azure Data Factory Pipelines) about possibility extracting the first element of a variable if this variable is set of elements (array).

So as a spoiler alert, before writing a blog post and adding a bit more clarity to the existing Microsoft ADF documentation, here is a quick answer to this question. The first element of an ArrayVariable is extracted by this expression: @variables('ArrayVariable')[0].

I created a simple ADF pipeline to test this out:

System Variables in Azure Data Factory: Your Everyday Toolbox

(2018-Nov-20) After working and testing the functionality of variables within Azure Data Factory pipelines, I realized that it's worth to explore existing system variables. That's basically could be my toolbox to collect and store control flow metrics of my pipelines.

Communication in Azure: using Data Factory to send messages to Azure Service Bus

(2019-Mar-10)A conversation between two or more people involves continuous efforts to listen and reflect on what other people have to say, i.e. you need to stay connected all the time, otherwise it's no longer a mutual communication. Emails provided us with the flexibility to provide timely and reasonably delayed responses, and I have already blogged about using Azure Data Factory with Emails. Azure Service Bus is one of the earliest or oldest components that was introduced in Azure. Can we use Data Factory to send messages to Azure Service Bus?

Image by Kranich17 on Pixabay

Using Azure Data Factory Mapping Data Flows to populate Data Vault

(2019-May-24)Data Flow as a data transformation engine has been introduced to the Microsoft Azure Data Factory (ADF) last year as a private feature preview. This privacy restriction has been lifted during the last Microsoft Build conference and Data Flow feature has become a public preview component of the ADF.

2020-Mar-15 update: a video recording of my webinar session on Using Azure Data Factory Mapping Data Flows to populate Data Vault at the recent PASS Data Architecture Virtual Group meeting - http://datanrg.blogspot.com/2020/03/using-azure-data-factory-mapping-data.html

There are many different use-case scenarios that can be covered by Data Flows, considering that Data Flows in SQL Integration Service (SSIS) projects are still playing a big role to fulfill Extracting-Loading-Transforming (ETL) patterns for your data.

In this blog post, I will share my experience of populating a Data Vault repository by using Data Flows in Azure Data Factory.

Twitter Widget

Tweets by RayisImayev

Search This Blog

Posts

Posts

All Comments

All Comments

Powered by Blogger