(2019-Feb-11) Azure Data Factory (ADF) provides you with a framework for creating data transformation solutions in the Microsoft cloud environment. Recently introduced Template Gallery for ADF pipelines can speed up this development process and provide you with helpful information to create additional activity tasks in your pipelines.
We naturally long to seek if something standard can be further adjusted. That custom design is like ordering a regular pizza and then hitting the "customize" button in order to add a few toppings of our choice. It would be very impressive then to save this customized "creation" for future ordering. And Azure Data Factory has a similar option to save your custom data transformation solutions (pipelines) as templates and reuse them later.
When you open your Data Factory UI, the option to see and create your own templates is available for you. Live factory mode doesn't allow you to add custom solution templates into the gallery. This "Templates" feature is only available for you in Azure Data Factory GIT integrated mode.
Here is how I was able to test this:
Step 1: Saving pipeline as a template
I've modified a pipeline that was originally created from the "Copy multiple files containers between File Stores" ADF template and added my custom Web Email Notification activity task at the end of the workflow. I have also made some additional changes to the Name and Description of this new "Copy Files with Email Notification" pipeline.
Then I can hit the "Save as template" button on the Pipeline tab to add this existing solution to the ADF Gallery. A new window will show up with an overview of all the activity tasks and Git location for the new template. In my case, this template was saved to my personal GitHub.
Step 2: Verifying your custom solution in the Template Gallery
After successfully saving your ADF pipeline to the Template Gallery its name will appear under the Template on the left-hand side of the Azure Data Factory UI. If you need to change this template you can either add those code changes in the GitHub template JSON files directly or drop and recreate it from the existing pipeline again.
I also was able to see my custom ADF pipeline solution "Copy File with Email Notification" in the Template Gallery. Which grabs your attention to see your own name along with the Microsoft among the list of template contributors. Even if your creation only exists within your own Data Factory workspace.
Step 3: Using your custom solution template
You can either hit the ellipsis button beside the Pipelines and select the "Pipeline from template" in order to open the Gallery Template or go right to the "Templates" and choose your new customer pipeline solution. It will open up a window and offer you to set linked services, or in a case of your own custom solution template, those options may vary.
Difference between a customer solution ADF template and a cloned pipeline:
You can always clone an existing pipeline in ADF and treat it as a template for your future development, however, there are some differences between this approach and pipeline from the Template gallery.
1) You custom solution template will be saved in a separate "/template" Git location and it will also be coupled with the manifest.json file.
2) A cloned pipeline will preserve all the original settings from its parent pipeline vs. interactive UI experience to select/create additional settings (e.g. linked services) when you add a new pipeline from a template.
Personally, I really liked this option of saving my custom ADF pipeline solution as a template. It helps me to think that you can now create a whole new data factory as a template :-)
Official Microsoft documentation for solution templates in Azure Data Factory:
https://docs.microsoft.com/en-us/azure/data-factory/solution-templates-introduction
My custom "Copy File with Email Notification" solution template in my personal GitHub:
https://github.com/NrgFly/Azure-DataFactory/tree/master/Samples/templates
We naturally long to seek if something standard can be further adjusted. That custom design is like ordering a regular pizza and then hitting the "customize" button in order to add a few toppings of our choice. It would be very impressive then to save this customized "creation" for future ordering. And Azure Data Factory has a similar option to save your custom data transformation solutions (pipelines) as templates and reuse them later.
When you open your Data Factory UI, the option to see and create your own templates is available for you. Live factory mode doesn't allow you to add custom solution templates into the gallery. This "Templates" feature is only available for you in Azure Data Factory GIT integrated mode.
Here is how I was able to test this:
Step 1: Saving pipeline as a template
I've modified a pipeline that was originally created from the "Copy multiple files containers between File Stores" ADF template and added my custom Web Email Notification activity task at the end of the workflow. I have also made some additional changes to the Name and Description of this new "Copy Files with Email Notification" pipeline.
Then I can hit the "Save as template" button on the Pipeline tab to add this existing solution to the ADF Gallery. A new window will show up with an overview of all the activity tasks and Git location for the new template. In my case, this template was saved to my personal GitHub.
Step 2: Verifying your custom solution in the Template Gallery
After successfully saving your ADF pipeline to the Template Gallery its name will appear under the Template on the left-hand side of the Azure Data Factory UI. If you need to change this template you can either add those code changes in the GitHub template JSON files directly or drop and recreate it from the existing pipeline again.
I also was able to see my custom ADF pipeline solution "Copy File with Email Notification" in the Template Gallery. Which grabs your attention to see your own name along with the Microsoft among the list of template contributors. Even if your creation only exists within your own Data Factory workspace.
Step 3: Using your custom solution template
You can either hit the ellipsis button beside the Pipelines and select the "Pipeline from template" in order to open the Gallery Template or go right to the "Templates" and choose your new customer pipeline solution. It will open up a window and offer you to set linked services, or in a case of your own custom solution template, those options may vary.
Difference between a customer solution ADF template and a cloned pipeline:
You can always clone an existing pipeline in ADF and treat it as a template for your future development, however, there are some differences between this approach and pipeline from the Template gallery.
1) You custom solution template will be saved in a separate "/template" Git location and it will also be coupled with the manifest.json file.
2) A cloned pipeline will preserve all the original settings from its parent pipeline vs. interactive UI experience to select/create additional settings (e.g. linked services) when you add a new pipeline from a template.
Personally, I really liked this option of saving my custom ADF pipeline solution as a template. It helps me to think that you can now create a whole new data factory as a template :-)
Official Microsoft documentation for solution templates in Azure Data Factory:
https://docs.microsoft.com/en-us/azure/data-factory/solution-templates-introduction
My custom "Copy File with Email Notification" solution template in my personal GitHub:
https://github.com/NrgFly/Azure-DataFactory/tree/master/Samples/templates
Comments
Post a Comment