Skip to main content

Data Adventures

My personal journey in this intricate world of data and continuous efforts to make it more structured and well understood - Rayis Imayev.

Search This Blog

Subscribe to this blog

Follow by Email

Posts

These little things about Azure Data Factory

Posted by Rayis Imayev on January 12, 2021

(2021-Jan-12 ) Small data that is hard to be noticed but in a long run makes the greatest impact, little things that are not well apricated but eventually become those final pieces of a puzzle that fit perfectly and complete a problem solution. Photo by Pixabay from Pexels

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Logging the history of my past SQL Saturday presentations

Posted by Rayis Imayev on December 31, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Using Azure Durable Functions with Azure Data Factory - HTTP Long Polling

Posted by Rayis Imayev on December 21, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Using Durable Functions in Azure Data Factory - Support for long running processes in Azure Functions

Posted by Rayis Imayev on October 14, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Row Numbers in Azure Data Factory Data Flows

Posted by Rayis Imayev on October 05, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Custom deployment parameters in Azure Data Factory - Forget Me Not

Posted by Rayis Imayev on September 26, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Passing array of arrays as a parameter in Azure Data Factory and the role of an @item element in this

Posted by Rayis Imayev on September 13, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Setting default values for Array parameters/variables in Azure Data Factory

Posted by Rayis Imayev on August 14, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON data with the help of Azure Data Factory - Part 5 - Comparing it with Databricks

Posted by Rayis Imayev on July 29, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Collecting custom parameters/variables for Azure Data Factory deployment

Posted by Rayis Imayev on June 22, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Send a Database Query Result as Email via Logic App in HTML format using Azure Data Factory

Posted by Rayis Imayev on June 07, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON data with the help of Azure Data Factory - Part 4 - Passing Arrays around

Posted by Rayis Imayev on May 24, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Replicating SQL IN operator in Azure Data Factory

Posted by Rayis Imayev on May 01, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Stacking Up Datasets in Azure Data Factory

Posted by Rayis Imayev on April 24, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Using Azure Functions in Azure Data Factory

Posted by Rayis Imayev on April 19, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON data with the help of Azure Data Factory - Part 3

Posted by Rayis Imayev on April 06, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Extracting data from Common Data Services (Microsoft Dynamics 365) using Azure Data Factory

Posted by Rayis Imayev on March 30, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Transforming JSON to CSV with the help of Azure Data Factory - Part 2 (Wrangling data flows)

Posted by Rayis Imayev on March 26, 2020

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Labels

Azure Data Factory Power BI Azure Data ADF Maps Variables Data Modeling Open Data Power Query

Visualization Array Data Factory Excel JSON Mapping Data Flows SQL Server Analytics Parameter Pipeline continuous integration ArcGIS ArgGIS Map Azure Function Azure Logic App DevOps Email Notification GIS GitHub Latitude Longitude QGIS Shape Map continuous delivery API Azure Data Lake Store Big Data Bing Map CI/CD CSV Custom Shapefile Data Vault Databricks Durable Function ESRI Flatten Mapbox PASS SQL SQL Azure SQL Azure Database Trigger WGS 84 Webhook Wrangling data flows XML 200 202 3D Map ADLS ARM Template Alerts Animation Apache Spark Append Variable Architect Architecture Azure Databricks Azure Monitor Azure Service Bus Blob Blob Storage Container Build CDS CRM Common Data Services Condition Custom Maps Custom Solution Data Architecture Data Flow Data Lake DataForACause Database Design Default Value Delete Activity Distance Dual axes EPSG 4326 ER Studio Error File Management Filled Map Flow map Function Gallery Template Geometry Git Global Fishing Globe Map HDFS HTML HTTP Heatmap IF activity Icon Map Ingestion Integration Load Log Analytics M language Macro Metrics Microsoft Dynamics 365 Monitor Narrative Science Network Parent Child Nodes Performance Polling Polygon Power Map Predictive Analytics Primary Secondary axis Pulse Chart Queue REST API Reference Layer Release Resource Manager SQL Saturday SSAS SSMS SSRS Send Message Solution Template Source Control Spark Spatial Data Spatial Functions Statistics Status Code Switch Activity System Variables Tabular Template Template Gallery Time Time Slider TopoJSON Twitter Usability Validation Version Web arm-template-parameters-definition.json custom parameter deployment long running process

Show more Show less

Rayis Imayev

Follow @RayisImayev

Archive

2021 1
- January 1
  - These little things about Azure Data Factory

2020 21
- December 2
- October 2
- September 2
- August 1
- July 1
- June 2
- May 2
- April 3
- March 4
- February 1
- January 1
2019 19
- December 1
- November 1
- October 3
- June 1
- May 1
- April 2
- March 3
- February 4
- January 3
2018 16
- December 1
- November 2
- October 2
- August 1
- July 3
- May 2
- March 1
- February 2
- January 2
2017 12
- December 1
- November 1
- October 1
- September 1
- August 1
- July 1
- June 1
- May 1
- April 1
- March 1
- February 1
- January 1
2016 9
- November 1
- October 1
- September 1
- August 1
- July 1
- June 1
- May 1
- April 1
- February 1
2015 10
- December 1
- November 1
- October 1
- August 1
- July 2
- June 1
- May 1
- April 1
- March 1
2014 11
- December 1
- November 1
- October 2
- August 1
- July 3
- April 1
- March 1
- February 1

Show more Show less

Labels

2001 2021 3D Map1 ADF11 ADLS1 Alerts1 Analytics4 Animation1 Apache Spark1 API2

Append Variable1 ArcGIS3 Architect1 Architecture1 ArgGIS Map3 ARM Template1 arm-template-parameters-definition.json1 Array5 Azure14 Azure Data Factory35 Azure Data Lake Store2 Azure Databricks1 Azure Function3 Azure Logic App3 Azure Monitor1 Azure Service Bus1 Big Data2 Bing Map2 Blob1 Blob Storage Container1 Build1 CDS1 CI/CD2 Common Data Services1 Condition1 continuous delivery3 continuous integration4 CRM1 CSV2 Custom Maps1 custom parameter1 Custom Shapefile2 Custom Solution1 Data13 Data Architecture1 Data Factory5 Data Flow1 Data Lake1 Data Modeling6 Data Vault2 Database Design1 Databricks2 DataForACause1 Default Value1 Delete Activity1 deployment1 DevOps3 Distance1 Dual axes1 Durable Function2 Email Notification3 EPSG 43261 ER Studio1 Error1 ESRI2 Excel5 File Management1 Filled Map1 Flatten2 Flow map1 Function1 Gallery Template1 Geometry1 GIS3 Git1 GitHub3 Global Fishing1 Globe Map1 HDFS1 Heatmap1 HTML1 HTTP1 Icon Map1 IF activity1 Ingestion1 Integration1 JSON5 Latitude3 Load1 Log Analytics1 long running process1 Longitude3 M language1 Macro1 Mapbox2 Mapping Data Flows5 Maps8 Metrics1 Microsoft Dynamics 3651 Monitor1 Narrative Science1 Network1 Open Data6 Parameter4 Parent Child Nodes1 PASS2 Performance1 Pipeline4 Polling1 Polygon1 Power BI21 Power Map1 Power Query6 Predictive Analytics1 Primary Secondary axis1 Pulse Chart1 QGIS3 Queue1 Reference Layer1 Release1 Resource Manager1 REST API1 Send Message1 Shape Map3 Solution Template1 Source Control1 Spark1 Spatial Data1 Spatial Functions1 SQL2 SQL Azure2 SQL Azure Database2 SQL Saturday1 SQL Server5 SSAS1 SSMS1 SSRS1 Statistics1 Status Code1 Switch Activity1 System Variables1 Tabular1 Template1 Template Gallery1 Time1 Time Slider1 TopoJSON1 Trigger2 Twitter1 Usability1 Validation1 Variables8 Version1 Visualization6 Web1 Webhook2 WGS 842 Wrangling data flows2 XML2

Show more Show less

Popular Posts

Working with Arrays in Azure Data Factory

(2019-June- 06) I remember that I had a professor at my university who would often encourage his students to learn and get more experienced with simple things first. Learn the basics in and out and then move forward to more complicated concepts and practices; that was his motto, and he really tried to share this idea with us. Previously published blog posts about using Variables in Azure Data Factory: - Setting Variables in Azure Data Factory Pipelines - Append Variable activity in Azure Data Factory: Story of combining things together - System Variables in Azure Data Factory: Your Everyday Toolbox - Azure Data Factory: Extracting array first element Simple things sometimes can be overlooked as well. With the addition of Variables in Azure Data Factory Control Flow (there were not available there at the beginning), Arrays have become one of those simple things to me. Image by Magnascan from Pixabay

Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub

(2019-Feb- 18) With Azure Data Factory (ADF) continuous integration, you help your team to collaborate and develop data transformation solutions within the same data factory workspace and maintain your combined development efforts in a central code repository. Continuous delivery helps to build and deploy your ADF solution for testing and release purposes. Basically, the CI/CD process helps to establish a good software development practice and aims to build a healthy relationship between development, quality assurance, and other supporting teams. Update 2020-Mar-15: Part 2 of this blog post published in 2020-Jan-28: http://datanrg.blogspot.com/2020/01/continuous-integration-and-delivery.html Update 2019-Jun- 24: Video recording of my webinar session on Continuous Integration and Delivery (CI/CD) in Azure Data Factory at a recent PASS Cloud Virtual Group meeting.

Using Azure Functions in Azure Data Factory

(2020-Apr- 19) Creating a data solution with Azure Data Factory (ADF) may look like a straightforward process: you have incoming datasets, business rules of how to connect and change them and a final destination environment to save this transformed data. Very often your data transformation may require more complex business logic that can only be developed externally (scripts, functions, web-services, databricks notebooks, etc.). In this blog post, I will try to share my experience of using Azure Functions in my Data Factory workflows: my highs and lows of using them, my victories and struggles to make them work. If you share the same pain points, if you find any mistakes or feel a total misrepresentation of facts, please leave your comments, there is no better opportunity to learn from positive critiques :-) Azure Functions gives you the freedom to create and execute a small or moderate size of code in C#, Java, JavaScript, Python, or PowerShell. This freedom releases you from a

Transforming JSON to CSV with the help of Flatten task in Azure Data Factory

(2020-Mar- 19) Recently, Microsoft introduced a new Flatten task to the existing set of powerful transformations available in the Azure Data Factory (ADF) Mapping Data Flows - https://docs.microsoft.com/en-us/azure/data-factory/data-flow-flatten . What this new task does it helps to transform/transpose/flatten your JSON structure into a denormalized flatten datasets that you can upload into a new or existing flat database table. 2020-Mar-26 Update: Part 2 : Transforming JSON to CSV with the help of Flatten task in Azure Data Factory - Part 2 (Wrangling data flows) I like the analogy of the Transpose function in Excel that helps to rotate your vertical set of data pairs ( name : value ) into a table with the column name s and value s for corresponding objects. And when this vertical JSON structural set contains several similar sets (array) then ADF Mapping Data Flows Flatten does a really good job by transforming it into a table with several rows (records).

Setting Variables in Azure Data Factory Pipelines

(2018-Oct-15) Working with Azure Data Factory you always tend to compare its functionality with well established ETL packages in SSIS. Data flow task have been recreated as Data Copy activities; logical components have found they cloud-based siblings; as well as new kids on the block, such as Databricks and Machine Learning activities could boost adoption rate of Azure Data Factory (ADF) pipelines. Support for local variables hasn't always been available in ADF a nd was only recently introduced to already available pipeline parameters. This addition makes more flexible to create interim properties (variables) that you can adjust multiple times within a workflow of your pipeline.

Append Variable activity in Azure Data Factory: Story of combining things together

(2018-Oct-29) There are only a few sentences in the official Microsoft web page that describe newly introduced activity task ( Append Variable) to add a value to an existing array variable defined in Azure Data Factory - Append Variable Activity in Azure Data Factory But it significantly improves your ability to control a workflow of the data transformation activities of your Data Factory pipeline.

Transforming JSON to CSV with the help of Azure Data Factory - Part 2 (Wrangling data flows)

(2020-Mar- 26) There are two ways to create data flows in Azure Data Factory (ADF): regular data flows also known as " Mapping Data Flows " and Power Query based data flows also known as " Wrangling Data Flows ", the latter data flow is still in preview, so do expect more adjustments and corrections to its current behavior. Last week I blogged about using Mapping Data Flows to flatten sourcing JSON file into a flat CSV dataset: Part 1 : Transforming JSON to CSV with the help of Flatten task in Azure Data Factory Today I would like to explore the capabilities of the Wrangling Data Flows in ADF to flatten the very same sourcing JSON dataset. At the time of writing this blog post, ADF Wrangling Data Flows only supports two types of file formats: CSV and Parquet. JSON file type is not supported. I could have just stopped here and wait for Microsoft to enable the work with " J ava S cript O bject N otation" data structure. Origin: https://openso

Azure Data Factory: Extracting array first element

(2019-Apr- 28) Full credit goes to Microsoft for the latest efforts updating Azure products with new features and documenting corresponding changes for the end users. Azure Data Factory (ADF) is a great example of this. A user recently asked me a question on my previous blog post ( Setting Variables in Azure Data Factory Pipelines ) about possibility extracting the first element of a variable if this variable is set of elements (array). So as a spoiler alert, before writing a blog post and adding a bit more clarity to the existing Microsoft ADF documentation, here is a quick answer to this question. The first element of an ArrayVariable is extracted by this expression: @variables(' ArrayVariable ')[0] . I created a simple ADF pipeline to test this out:

Using Azure Data Factory Mapping Data Flows to populate Data Vault

(2019-May- 24) Data Flow as a data transformation engine has been introduced to the Microsoft Azure Data Factory (ADF) last year as a private feature preview . This privacy restriction has been lifted during the last Microsoft Build conference and Data Flow feature has become a public preview component of the ADF. 2020-Mar-15 update : a v ideo recording of my webinar session on Using Azure Data Factory Mapping Data Flows to populate Data Vault at the recent PASS Data Architecture Virtual Group meeting - http://datanrg.blogspot.com/2020/03/using-azure-data-factory-mapping-data.html There are many different use-case scenarios that can be covered by Data Flows, considering that Data Flows in SQL Integration Service (SSIS) projects are still playing a big role to fulfill Extracting-Loading-Transforming (ETL) patterns for your data. In this blog post, I will share my experience of populating a Data Vault repository by using Data Flows in Azure Data Factory.

Power BI with different Network Visualizations

(2018-Feb-10) A few days ago Microsoft provided another update for its desktop version of the Power BI data analytical tool - https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-february-2018-feature-summary/ . Among all the major updates in this release, there was one new custom network visualization added by ZoomCharts. It wasn't the first network visualization available in the Power BI and I played a little bit with some of them in the past. Currently, there are 5 network visual controls that you can add to your Power BI reports: - Network Navigator Chart by Microsoft - Force-Directed Graph by Microsoft - Journey Chart by MAQ Software - Social Network Graph by Arthur Graus - Network Visualization by ZoomCharts

Twitter Widget

Tweets by RayisImayev

Search This Blog

Posts

Posts

All Comments

All Comments

Powered by Blogger