(2020-Sep-26) Last week one of my Azure Data Factory (ADF) deployment pipelines failed with an error that it couldn’t find some of the deployment parameters that I try to override, and I wondered what might have caused the issue. Usually, after configuring and testing all the deployment steps, there are very few things that can topple a normal and successful deployment process of your data factory, and it’s got to be big.
My first step was to remove parameters overriding instances from a deployment pipeline. There is no need to reference and override a deployment pipeline parameter if it hadn’t come out of the process of creating my ADF Azure Resource Manager (ARM) template. That quick fix worked and my ADF deployment process successfully continued and new code was deployed to a testing environment.
However, when I checked one of the web-activity tasks in my testing Data Factory after the deployment was finished, that activity still had a URL reference to a resource from the Development environment and not the Testing. That URL reference correction (overriding) used to properly happen during the deployment process. You can change the default parameterization template and prescribe your Data Factory to include some specific properties as additional custom parameters to deploy - https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template. I have also blogged about it two months ago - https://datanrg.blogspot.com/2020/06/raking-custom-parametersvariables-for.html
My initial custom parameterization to collect all the web activities URLs from an ADF pipeline was configured the following way with the Pipelines/Activities path to my URL properties.
If those settings are still correct, then why I’m not getting any of the web activities URL properties extracted into my ADF ARM template. And then I realized after moving my web-activity task from the main pipeline level into a Switch Case activity, the original custom parameterization process couldn’t longer find them. I needed to make further adjustments to my custom parameters template.
By going to one level deeper into my ADF activities properties by including an additional “Cases” case Pipelines/Activities/Cases, I was able to capture my web-activities URL attributes.
This aha moment had shown me that modifying my ADF custom parameters template is a very good way to capture very specific attributes for deployment, however, it’s also very important to remember to review this parameter template file after making some major modification to your ADF code and see if some of the settings are no longer relevant.
That’s why I have tagged this post as “Forget Me Not” because I’m a regular person and tend to forget things :-)
Comments
Post a Comment