(2025-June-15) Long gone are the days when a data engineer could simply focus on building a new data integration solution. You’d establish connections to one or more source datasets, review the expected target destination, and design the required transformation logic to meet secure standards and match formatting controls on the destination side. Then you'd hit execute on the new pipeline, no errors, it looked like data moved forward (though you weren’t quite sure how much), and you'd think, "Well, it worked... probably."
Today, logging and monitoring are no longer optional. They’re integral parts of any data flow, sometimes out of necessity, to verify whether an ETL/ELT job actually completed successfully, and sometimes to meet internal or external audit requirements by tracing every transformation step across all stages of the pipeline and collecting all required logs.
Azure Data Factory is no exception when it comes to logging options. All your debug or triggered pipeline executions—their parameters passed during execution, statuses, timings, durations, and more, can be monitored natively in Azure Data Studio. Once you immerse yourself in the realm of previously executed pipelines and start seeing all activities, passed input values, processed output results, and variables being transformed into something else that can only be understood by examining internal expressions and many other details, you begin to feel like an investigator meticulously analyzing everything.
When you realize that all your efforts to review a single pipeline execution have been exhausted and you still don’t understand why it either failed or ran successfully, Data Factory offers an additional level of logging detail: activity-level logs. Not all activities provide this option. In fact, based on my empirical observation, there is only one activity in ADF that exposes this level of logging: the Data Copy activity.
You can read more about this option in the official documentation source while it's still available and not replaced by not yet matching content from the Microsoft Fabric platform: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-log?tabs=data-factory
What’s real gold about this option is the level of support it provides to ensure that a very important Data Copy activity, involving thousands or even millions of records to be treated, doesn’t fail because of a few “black sheep” records. If all of your pipeline’s Data Copy steps work perfectly fine, and you don’t have this logging setting configured. It’s not the end of your data world; it just means your data world hasn’t been tested enough. With some potential data issues in the future, your Data Copy will just fail with a mere indication of good and bad records, and most likely, you wouldn’t know what was actually wrong with those few problematic ones.
However, when you enable this logging setting and configure the output, let’s say to your file storage in Azure, you gain another valuable tool for investigation. Every failed record will flow into your designated destination, along with the complete set of incoming data columns and explanations of why the data copy wasn’t successful. It could be a data issue on the source side, which may lead to resolving the problem with your data provider. Or, something might have happened either intermittently or persistently on the destination side, and that could be another story to resolve. Either way, this could be your golden ticket to understanding how a data flow may behave under different circumstances.
Going further, with the built-in monitoring and logging capabilities in Data Factory, you’re still stuck with a 45-day log retention policy that won’t let you see “very old” pipeline executions, even if you really want to: https://learn.microsoft.com/en-us/azure/data-factory/monitor-data-factory#store-data-factory-pipeline-run-data
The best way to divert your ADF pipelines, triggers, and activities logs to a log repository with a longer retention period that you can control and adjust is by using a Log Analytics workspace in Azure. This tool can serve as your centralized log repository for many other Azure resources as well, you just need to find the Diagnostic settings of your ADF resource and configure the types of events to be logged to your Log Analytics workspace.
This is one example of the level of detail you can see for just one particular activity among many others within a single pipeline execution, on a specific day, many days ago. Just incredible!
Now what? You feel that you may have covered all the bases, no gaps left in the logging ecosystem of your data integration solution. But once you start working seriously with data flows in Azure Data Factory, you begin to feel that something is missing.
You have a good grasp of your data flow execution: it's either a reassuring green or a concerning red. The Data Factory product team has also done a good job of providing real visibility into the actual row counts from the destination back to the source, essentially, a retrospective view of the data flow.
However, if your destination sink in a data flow is configured with many UPDATE or UPSERT statements, you’re still missing visibility into the actual incoming data just before it gets “sunk” into the data flow sink. In this case, especially when the volume of your UPSERT operations is not low, you can simply clone your incoming data stream, attach a job ID and timestamp attribute, and log it to a dedicated container in Azure Storage.
If everything works well, you might not be bothered or feel the need to look at those cloned datasets. However, if a request to investigate certain data anomalies arises later, you’ll have another valuable information resource at your disposal. Thus, you may find yourself creating your own custom logs for your data integration solution.So, in Azure Data Factory, you have: (1) native built-in logging and monitoring capabilities, (2) logs that can be configured at the Data Copy activity level, (3) additional logging details with a longer retention period via a Log Analytics workspace in Azure, and (4) custom logs that you can create using any approach that fits well with your architectural knowledge, tools, and available platforms. The example with Data Flow was just one of many possibilities.
PS. Yesterday, while I watched the World War Z movie with my youngest daughter, where Brad Pitt plays a former UN investigator seeking a solution for a sudden zombie apocalypse, I caught myself thinking that my appreciation of this movie has evolved over time. When I watched it for the first time, it gave the impression that the movie was all about zombies. Later viewings showed Pitt's character as a caring family man and a skilled resource who was in high demand during the most difficult time. Watching it again recently allowed me to see his character as a curious, well-informed investigator, someone still ready to search for more. And while the movie didn’t show that he was able to find patient zero, he continued his investigation, and it turned out to be good.
Comments
Post a Comment