Data Adventures

(2019-June- 06) I remember that I had a professor at my university who would often encourage his students to learn and get more experienced with simple things first. Learn the basics in and out and then move forward to more complicated concepts and practices; that was his motto, and he really tried to share this idea with us. Previously published blog posts about using Variables in Azure Data Factory: - Setting Variables in Azure Data Factory Pipelines - Append Variable activity in Azure Data Factory: Story of combining things together - System Variables in Azure Data Factory: Your Everyday Toolbox - Azure Data Factory: Extracting array first element Simple things sometimes can be overlooked as well. With the addition of Variables in Azure Data Factory Control Flow (there were not available there at the beginning), Arrays have become one of those simple things to me. Image by Magnascan from Pixabay

Executing Azure Data Factory pipelines by Power App / Automate Flow

(2021-Jun-30) One of my university professors liked to tell us a quote, “ The Sleep of Reason Produces Monsters ”, in a way to help us, his students, to stay active in our thinking process. I’m not sure if Francisco Goya , had a similar aspiration when he was creating his artwork with the same name. So, let me explain my reasons to create a solution to trigger Azure Data Factory (ADF) pipelines from a Power App and why it shouldn't be considered as a monster :-)

Azure Data Factory: Extracting array first element

(2019-Apr- 28) Full credit goes to Microsoft for the latest efforts updating Azure products with new features and documenting corresponding changes for the end users. Azure Data Factory (ADF) is a great example of this. A user recently asked me a question on my previous blog post ( Setting Variables in Azure Data Factory Pipelines ) about possibility extracting the first element of a variable if this variable is set of elements (array). So as a spoiler alert, before writing a blog post and adding a bit more clarity to the existing Microsoft ADF documentation, here is a quick answer to this question. The first element of an ArrayVariable is extracted by this expression: @variables(' ArrayVariable ')[0] . I created a simple ADF pipeline to test this out:

Transforming JSON data with the help of Azure Data Factory - Part 4 - Passing Arrays around

(2020-May- 24) It has never been my plan to write a series of articles about how I can work with JSON files in Azure Data Factory (ADF). While working with one particular ADF component I then had discovered other possible options to use richness and less constrained JSON file format, which in a nutshell is just a text file with one or more ( " key " : " value " ) pair elements. Sometimes you could have an array of those pair elements or an array of other arrays; now you see where I'm going with the name of my current blog post :-)

Azure Function Drain mode

(2021-Jun-16) Recently, Google started to notify me that my blog website ( http://datanrg.blogspot.com/ ) had been experiencing some ‘Server (5xx)’ errors. Nothing has been changed on my side, and I’m not sure how long this Google-powered Blogger platform will be active. Another “sucker punch” came from my youngest child :-). She is 9, however, she is aware of Azure Data Factory and other data platform components to a certain extent, that I attempt to write. A few days ago she suggested that I should stop writing about all this “boring” stuff and add some actions & fun stuff that she reads about in her chapter books or watch in her TV shows. So, if you stop seeing any new posts on my blog site: I’m either fighting with Blogger server errors or watching a new episode of the “Captain Underpants” TV show with my daughter :-) Image by PublicDomainPictures from Pixabay Here are some actions and fun stuff about this blog post idea. It came from my recent communication and assistan

Blog post about Copying Azure files to SharePoint, comparing Logic App and Power Automate Flow for this, and Trust No One

(2021-May-31) Recently, I created a prototype solution for one of the projects, that would copy data files from Azure to SharePoint location. The task is trivial, doesn't require a lot of effort to build such a workflow with the help of the Azure Logic App. However, SharePoint itself is like another ecosystem and it may throw a curveball to me and slow down my solution development timeline dramatically. This post is a reminder to myself on how I had spent 5% of my time to develop this solution and then the other 95% to understand why it wasn't working initially to later resolve it.

Send a Database Query Result as Email via Logic App in HTML format using Azure Data Factory

(2020-June- 07) It is a very simple concept, you have a database query result, either a direct database table output or a stored procedure returned result that needs to be sent as email notification. This email could serve as a status report of your daily data processing job or alert notification with some metrics that require immediate attention, and you want to be as user-friendly as possible with this message aka HTML format for your dataset is required.

Using Durable Functions in Azure Data Factory - Support for long running processes in Azure Functions

(2020-Oct-14 ) Ok, here is my problem: I have an Azure Data Factory (ADF) workflow that includes an Azure Function call to perform external operations and returns output result, which in return is used further down my ADF pipeline. My ADF workflow (1) depends on the output result of the Azure Function call; (2) plus a time efficiency of the Azure Function call is another factor to consider, if its time execution hits 230 seconds or more, ADF Azure Function will fail with a time-out error message and my workflow is screwed. Image by Ichigo121212 from Pixabay

Transforming JSON to CSV with the help of Flatten task in Azure Data Factory

(2020-Mar- 19) Recently, Microsoft introduced a new Flatten task to the existing set of powerful transformations available in the Azure Data Factory (ADF) Mapping Data Flows - https://docs.microsoft.com/en-us/azure/data-factory/data-flow-flatten . What this new task does it helps to transform/transpose/flatten your JSON structure into a denormalized flatten datasets that you can upload into a new or existing flat database table. 2020-Mar-26 Update: Part 2 : Transforming JSON to CSV with the help of Flatten task in Azure Data Factory - Part 2 (Wrangling data flows) I like the analogy of the Transpose function in Excel that helps to rotate your vertical set of data pairs ( name : value ) into a table with the column name s and value s for corresponding objects. And when this vertical JSON structural set contains several similar sets (array) then ADF Mapping Data Flows Flatten does a really good job by transforming it into a table with several rows (records).

Using Azure Functions in Azure Data Factory

(2020-Apr- 19) Creating a data solution with Azure Data Factory (ADF) may look like a straightforward process: you have incoming datasets, business rules of how to connect and change them and a final destination environment to save this transformed data. Very often your data transformation may require more complex business logic that can only be developed externally (scripts, functions, web-services, databricks notebooks, etc.). In this blog post, I will try to share my experience of using Azure Functions in my Data Factory workflows: my highs and lows of using them, my victories and struggles to make them work. If you share the same pain points, if you find any mistakes or feel a total misrepresentation of facts, please leave your comments, there is no better opportunity to learn from positive critiques :-) Azure Functions gives you the freedom to create and execute a small or moderate size of code in C#, Java, JavaScript, Python, or PowerShell. This freedom releases you from a

Twitter Widget

Tweets by RayisImayev

Search This Blog

Posts

Atom

Posts

All Comments

Atom

All Comments

Powered by Blogger