Passing array of arrays as a parameter in Azure Data Factory and the role of an @item element in this

(2020-Sep-13) Array of arrays, in a JSON world, it’s a very common concept to have a set of sets of other elements in a dataset. Despite how strange it may sound, our real life is filled with similar analogies: books arranged by categories in a library, plates stacked up in a cupboard, or even someone’s closet has many sections that are filled with different clothing items. Arrays are part of our daily life too :-)

Photo by Lovefood Art from Pexels

In my previous blog post - Setting default values for Array parameters/variables in Azure Data Factory, I had helped myself to remember that arrays could be passed as parameters to my Azure Data Factory (ADF) pipelines. This time I’m helping myself to remember that an array of other arrays can also exist as ADF pipeline parameters’ values.

(1) Simple array of values:

This array will be passed into my par_meal_array parameter

Then Azure Data Factory ForEach will loop through this set of elements and each individual value will be referenced with the use of the @item() expression.

(2) Simple array with sub-elements:

Sub-elements’ values during each iteration of this array can be referenced with: 
- @item().meal_type
- @item().meal_time

(3) Array of arrays:

My meal array of items could be easily referenced as @item().meal_items (e.g. my first element of the meal_items would have this set of values: ["Egg","Greek Yogurt","Coffee"]). 

Since I’m already in the ForEach loop container then I can pass this array into another ForEach loop container. However, this is currently not possible in Azure Data Factory unless this lower-level ForEach loop container exists in another pipeline and I can just simply execute it from the top ForEach loop container and pass it as another pipeline array parameter.

Closing thoughts:

I just wish that Azure Data Factory would allow having nested ForEach loop containers in a single pipeline.