Data Adventures

(2019-Feb- 18) With Azure Data Factory (ADF) continuous integration, you help your team to collaborate and develop data transformation solutions within the same data factory workspace and maintain your combined development efforts in a central code repository. Continuous delivery helps to build and deploy your ADF solution for testing and release purposes. Basically, the CI/CD process helps to establish a good software development practice and aims to build a healthy relationship between development, quality assurance, and other supporting teams. Update 2020-Mar-15: Part 2 of this blog post published in 2020-Jan-28: http://datanrg.blogspot.com/2020/01/continuous-integration-and-delivery.html Update 2019-Jun- 24: Video recording of my webinar session on Continuous Integration and Delivery (CI/CD) in Azure Data Factory at a recent PASS Cloud Virtual Group meeting.

Executing Azure Data Factory pipelines by Power App / Automate Flow

(2021-Jun-30) One of my university professors liked to tell us a quote, “ The Sleep of Reason Produces Monsters ”, in a way to help us, his students, to stay active in our thinking process. I’m not sure if Francisco Goya , had a similar aspiration when he was creating his artwork with the same name. So, let me explain my reasons to create a solution to trigger Azure Data Factory (ADF) pipelines from a Power App and why it shouldn't be considered as a monster :-)

Append Variable activity in Azure Data Factory: Story of combining things together

(2018-Oct-29) There are only a few sentences in the official Microsoft web page that describe newly introduced activity task ( Append Variable) to add a value to an existing array variable defined in Azure Data Factory - Append Variable Activity in Azure Data Factory But it significantly improves your ability to control a workflow of the data transformation activities of your Data Factory pipeline.

System Variables in Azure Data Factory: Your Everyday Toolbox

(2018-Nov-20) After working and testing the functionality of variables within Azure Data Factory pipelines, I realized that it's worth to explore existing system variables. That's basically could be my toolbox to collect and store control flow metrics of my pipelines.

Transforming JSON to CSV with the help of Flatten task in Azure Data Factory

(2020-Mar- 19) Recently, Microsoft introduced a new Flatten task to the existing set of powerful transformations available in the Azure Data Factory (ADF) Mapping Data Flows - https://docs.microsoft.com/en-us/azure/data-factory/data-flow-flatten . What this new task does it helps to transform/transpose/flatten your JSON structure into a denormalized flatten datasets that you can upload into a new or existing flat database table. 2020-Mar-26 Update: Part 2 : Transforming JSON to CSV with the help of Flatten task in Azure Data Factory - Part 2 (Wrangling data flows) I like the analogy of the Transpose function in Excel that helps to rotate your vertical set of data pairs ( name : value ) into a table with the column name s and value s for corresponding objects. And when this vertical JSON structural set contains several similar sets (array) then ADF Mapping Data Flows Flatten does a really good job by transforming it into a table with several rows (records)...

Setting Variables in Azure Data Factory Pipelines

(2018-Oct-15) Working with Azure Data Factory you always tend to compare its functionality with well established ETL packages in SSIS. Data flow task have been recreated as Data Copy activities; logical components have found they cloud-based siblings; as well as new kids on the block, such as Databricks and Machine Learning activities could boost adoption rate of Azure Data Factory (ADF) pipelines. Support for local variables hasn't always been available in ADF a nd was only recently introduced to already available pipeline parameters. This addition makes more flexible to create interim properties (variables) that you can adjust multiple times within a workflow of your pipeline.

Send a Database Query Result as Email via Logic App in HTML format using Azure Data Factory

(2020-June- 07) It is a very simple concept, you have a database query result, either a direct database table output or a stored procedure returned result that needs to be sent as email notification. This email could serve as a status report of your daily data processing job or alert notification with some metrics that require immediate attention, and you want to be as user-friendly as possible with this message aka HTML format for your dataset is required.

Azure Data Factory: Extracting array first element

(2019-Apr- 28) Full credit goes to Microsoft for the latest efforts updating Azure products with new features and documenting corresponding changes for the end users. Azure Data Factory (ADF) is a great example of this. A user recently asked me a question on my previous blog post ( Setting Variables in Azure Data Factory Pipelines ) about possibility extracting the first element of a variable if this variable is set of elements (array). So as a spoiler alert, before writing a blog post and adding a bit more clarity to the existing Microsoft ADF documentation, here is a quick answer to this question. The first element of an ArrayVariable is extracted by this expression: @variables(' ArrayVariable ')[0] . I created a simple ADF pipeline to test this out:

Excel Power Map vs. Power BI Globe Map visualization

[ 2016-02-02 ] PowerMap in Excel is a great tool with good design and comprehensive user experience. I’ve started using it about a year ago when our team participated at first Canadian Microsoft Big Data hackathon ( http://blogs.technet.com/b/cansql/archive/2015/04/13/big-data-hackathon-a-story-from-a-winning-team.aspx ).

Power BI with different Network Visualizations

(2018-Feb-10) A few days ago Microsoft provided another update for its desktop version of the Power BI data analytical tool - https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-february-2018-feature-summary/ . Among all the major updates in this release, there was one new custom network visualization added by ZoomCharts. It wasn't the first network visualization available in the Power BI and I played a little bit with some of them in the past. Currently, there are 5 network visual controls that you can add to your Power BI reports: - Network Navigator Chart by Microsoft - Force-Directed Graph by Microsoft - Journey Chart by MAQ Software - Social Network Graph by Arthur Graus - Network Visualization by ZoomCharts

Twitter Widget

Tweets by RayisImayev

Search This Blog

Posts

Atom

Posts

All Comments

Atom

All Comments

Powered by Blogger