(2021-July-07) Back in the late 90s when I was a college student intern at a local power supply company, I was invited to a social event of our IT department. There, I have been introduced to a guest visitor, who had developed a database solution for another organization using “old” developments tools, even for that time: FoxPro & Clipper.
In my student pursuit to learn new things about computers, by reading all available PC Magazines and ComputerWorld newspapers, that guy stood out like Clint Eastwood's character from his Dirty Harry movie: confident, positive, with no regrets of his tools & techniques not being very modern. The most important thing was that his work was highly valued by the organization and he also was very well paid.
Microsoft has recently updated the look & feel of the Azure Data Factory by replacing the “Author & Monitor” button with the “Open Azure Data Factory Studio” and refreshing its home page: https://techcommunity.microsoft.com/t5/azure-data-factory/the-new-and-refreshing-data-factory-home-page/ba-p/2515076
The updated home page looks fresh. Rectangular buttons to start “Ingest”, “Orchestrate”, “Transform data”, “Configure SSIS” wizards are very handy for new ADF users to easily create new data transformation components; which is a less preferred way for existing data factories developers.
A list of recent resources is also helpful and can add more visibility to the changed items, like an audit log. Would I start interacting with this “recent resources” list (e.g. to resume editing previously updated resources), probably not? In my ADF projects, I tend to structure all my pipelines in folders, and folder names are not shown in the recent resource list. Also, when I plan to resume editing a particular data transformation pipeline, I go directly to the ADF and locate it by expanding a necessary folder.
Starting links to ADF training materials: Tutorials, Videos, Community Content, etc. is very good! I highly admire and respect Mark Kromer for his work to share Data Factory training materials, thank you!
What I didn’t like in this look & feel update, is the starting “Open Azure Data Factory Studio” button. No, I agree with the new title, it’s good and less confusing compared to “Author & Monitor”. However, the change to the interactive area of that button could be described as one step backward in usability.
Then I thought, who am I to complain about all those things. There might be other higher priorities for the ADF product team at Microsoft, people may move across the organization and focus may change. At the end of the day, it’s just a tool that can be used to orchestrate data transformation processes. Don’t complain, just use it!
During the last year and a half, I had an opportunity to participate in three different sessions to review and provide feedback about new UX Data Factory features with the Microsoft ADF team. Both control and data flow components were discussed, updated visuals were demonstrated and honest feedback was collected. I have also shared some additional comments and suggestions on how to improve ADF to make it easier for developers to adopt, and due to prioritization or a focus shift, those comments haven’t been taken into account.
These are some of the ideas that I had shared during those UX feature review meetings with Microsoft, and I believe there are still worthy to be considered:
- Local and Global variables, currently there is no scope for variables in ADF, everything is global and variable updates in the loop container affect each iteration.
- Local editor to check variables and expressions, it would handy to have a Debug editor and evaluate not just syntax but variable/expression values.
- Hide empty flows in the IF (false) or Switch (False) activities, if some of the conditional flows don’t have tasks, I don’t need to see the empty placeholder, it just takes very valuable space in my ADF pipeline canvas.
- Data Flow zooming problem, small and medium-size data flows are ok to be seen, however, if the logic becomes very complex, the number of sourcing and sinking datasets grows, zoomed-out view of a data flow is annoying, many seconds spent to zoom in whenever open such data flow.
- Expand of the Switch Activity, irritating collapse and expand behaviour.
- Newly deployed ADF pipelines are not visible and not selected in the Monitor after your trigger them, even if the "Pipeline" selection is set to 'All'. It's like a 'select * from table' in view, when newly added columns are not visible.
- Space characters are not allowed in the Data flow transformation tasks
- Data Preview results scrolling in data flows are not easy when you have many columns
- Passing ADF pipeline output to another pipeline is currently not possible
- Grouping tasks (a.k.a. SSIS Sequence container), will simplify the UI experience dramatically if your ADF pipeline has many tasks.
- Disabling tasks, sometimes I just need to keep the tasks in my pipelines but keep them from execution.
- Allowing Validation tasks with IF container, currently, it’s not possible too.
- Copy connections between Activities, when you copy several connected activities tasks, their connections are copied, but this would be very helpful as well.
No complaints to the ADF product team, they’ve done a very good job of moving their product from V1 to V2, and to where it is now. I wish them all the best, sincerely and with no sarcasm!
My kids are still young and we watch a lot of animation movies together. There is a nice line in the Robots movie that teaches to, “Look around for a need, and start coming up with ideas to fill that need. One idea will lead to another, and before you know it, you’ve done it! See a need, fill the need!”. So, it’s less about the technology, but more emphasis on solutions that can be developed with available technologies.
So, if I ever complain about that small “Open” button in the Azure Data Factory, then I can always move on and chose different tools, both old and new; because there are so many out there!
Comments
Post a Comment