(2019-October-16) Developing conditional logic of your Azure Data Factory control flow has been simplified with introducing of the Switch activity - https://docs.microsoft.com/en-us/azure/data-factory/control-flow-switch-activity. Official documentation resource states, this new data factory activity "provides the same functionality that a switch statement provides in programming languages". I would also add a more simplified definition of the Switch activity in Azure Data Factory: it is a container (or wrapper) for multiple IF conditions.
Image by Bruno Glätsch from Pixabay
If I didn't have Switch activity in the ADF, this would be my way to create logic with multiple conditions to process files based on a File_Action parameter-flag.
Where each of the IF conditions would have the following expressions to evaluate:
a) @equals(pipeline().parameters.FileAction,'move')
b) @equals(pipeline().parameters.FileAction,'copy')
c) @equals(pipeline().parameters.FileAction,'delete')
And each of IF activity containers would have the following list of sub-tasks:
a) Copy Data; Delete
b) Copy Data
c) Delete
However the new logical control flow for my ADF pipeline may look a bit simpler with only "Get Metadata" and "Switch" activities:
My Switch activity expression would be just set to my pipeline parameter value: @pipeline().parameters.FileAction
and on the next tab, I can define different cases that need to be evaluated:
You can also notice sub-tasks associated with each different case, "move" case has two tasks indeed.
And after executing this ADF pipeline with 'copy' value for FileAction parameter, you can even check how many files got copied.
The code of this ADF pipeline can be found here:
https://github.com/NrgFly/Azure-DataFactory/blob/master/Samples/pipeline/Workflow_Switch_activity_pl.json
Let me know what your thoughts are about this new ADF activity.
Image by Bruno Glätsch from Pixabay
If I didn't have Switch activity in the ADF, this would be my way to create logic with multiple conditions to process files based on a File_Action parameter-flag.
Where each of the IF conditions would have the following expressions to evaluate:
a) @equals(pipeline().parameters.FileAction,'move')
b) @equals(pipeline().parameters.FileAction,'copy')
c) @equals(pipeline().parameters.FileAction,'delete')
And each of IF activity containers would have the following list of sub-tasks:
a) Copy Data; Delete
b) Copy Data
c) Delete
However the new logical control flow for my ADF pipeline may look a bit simpler with only "Get Metadata" and "Switch" activities:
My Switch activity expression would be just set to my pipeline parameter value: @pipeline().parameters.FileAction
and on the next tab, I can define different cases that need to be evaluated:
You can also notice sub-tasks associated with each different case, "move" case has two tasks indeed.
And after executing this ADF pipeline with 'copy' value for FileAction parameter, you can even check how many files got copied.
The code of this ADF pipeline can be found here:
https://github.com/NrgFly/Azure-DataFactory/blob/master/Samples/pipeline/Workflow_Switch_activity_pl.json
Let me know what your thoughts are about this new ADF activity.
I am developing a new ADFv2 based Product for our Company. GREAT Blog! I didn't know the Switch Activity exists. I have multiple IFs in my pipeline. They seem to take alot of time to process; is the Switch faster? Thanks for your Blog! Mike Kiser
ReplyDeleteThanks Mike. I would say there shouldn't be a performance difference between multiple IFs and a single Switch, however I only tested on a small set of conditions. Usually those IF conditions take a few seconds to validate, so Switch should work the same.
Deletebut you can't nest IFs and SWITCHes, uugggghhh, and this type of hierarchical logic is so typical.
DeleteHi Rayis!! I love all your Blogs! They have greatly helped me in my ADF work. I have a question regarding the Switch Activity. I am trying to put a Switch inside a ForEach and within the Switch there is nothing but default showing. Can we not use a Switch inside a ForEach for some reason? Thanks! Mike Kiser
ReplyDeleteHi Mike, thank for your feedback about the blog, it describes in someway my own journey of understanding how certain data platform tools operate. Regarding the Switch activity, you can use it in the ForEach loop container. After adding it, just go to Activities tab of the Switch and add new Cases besides Default definition, it should work.
DeleteCan we put a OR condition in the case, example i have a single activity for two id's, so i want to put single case for both id's with a OR condition instead two cases, is is possible?
ReplyDeleteThis is possible as well, in this case you won't need a Switch activity, a single IF activity would suffice: just define your OR condition expression for the True case and add the activity that you want to be executed with the True expression evaluation.
Delete