tag:blogger.com,1999:blog-5403386884901073007.post7442358342347124855..comments2024-03-29T02:22:33.688-04:00Comments on Data Adventures: Continuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHubRayis Imayevhttp://www.blogger.com/profile/03340391869523894676noreply@blogger.comBlogger83125tag:blogger.com,1999:blog-5403386884901073007.post-22186441130846293582023-01-14T10:23:59.558-05:002023-01-14T10:23:59.558-05:00This is simple, you will just need to override req...This is simple, you will just need to override required parameters from the ArmTemplate Parameter file and replace their initial DEV values for the QA and PROD stages during the deployment. I explain that in the video and my blog post (2.f Add new Stage to your Release pipeline).Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-37573629291372850662023-01-12T07:09:53.667-05:002023-01-12T07:09:53.667-05:00How do you stop the deploy flow from Dev to QA to ...How do you stop the deploy flow from Dev to QA to PRD, when the linkedsevices type sftp uses different hosts? In my scenario when I deploy DEV to QA to PROD the host that exists in QA and PROD are changed by the host that is in DEV, is there a way to block this?Admrehathttps://www.blogger.com/profile/10037272702530043576noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-87565539928098208772023-01-12T07:07:17.718-05:002023-01-12T07:07:17.718-05:00How do you stop the deploy flow from Dev to QA to ...How do you stop the deploy flow from Dev to QA to PRD, when the linkedsevices type sftp uses different hosts? In my scenario when I deploy DEV to QA to PROD the host that exists in QA and PROD are changed by the host that is in DEV, is there a way to block this?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-73275783732811850422022-04-29T12:49:59.201-04:002022-04-29T12:49:59.201-04:00You can read about more pre and post deployment sc...You can read about more pre and post deployment script to avoid any potential conflicts during the deployment: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-sample-scriptRayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-88012122613687596222022-04-29T12:49:12.465-04:002022-04-29T12:49:12.465-04:00These are still unsupported CICD features of ADF: ...These are still unsupported CICD features of ADF: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery#unsupported-featuresRayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-86333617458883599382022-04-29T10:44:23.659-04:002022-04-29T10:44:23.659-04:00once the code repo is synced , is there any way to...once the code repo is synced , is there any way to deploy the updated codes to the respective datafactory resources without any conflicts and deletion of artifacts that is not present in code repo but present in adf.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-29141393456534757322021-01-09T07:22:48.149-05:002021-01-09T07:22:48.149-05:00It is a great blog engaging DevOps practices. I wo...It is a great blog engaging DevOps practices. I would like to add one more things that DevOps is a set of software development practices that combines software development (Dev) and information technology operations (Ops) to optimise the delivery of a product, solution or platform.Deployflowhttps://www.deployflow.co/noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-69155811672042042062020-11-30T10:01:22.196-05:002020-11-30T10:01:22.196-05:00Hi Swapna, I understand your concern. Don't lo...Hi Swapna, I understand your concern. Don't look at a single Data Factory as a big resource that should combine all ETL processes of all the projects (small and big) that you can think of. I believe that ADF is similar to an SSIS projects, and all components in one ADF should share the same deployment cycle. Plus from the cost perspective it will be better to segregate workload for each project separately, i.e. having multiple projects in a single ADF is not a best practice. Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-57308566301001808282020-11-28T05:44:23.134-05:002020-11-28T05:44:23.134-05:00I have designed the DevOpS release pipeline with D...I have designed the DevOpS release pipeline with DEV ADF(Git configured), QA , UAT and PROD ADF. Each ADF environment will be deployed from adf_publish branch of DEV. <br />This is working fine if we work on one project at a time. But our scenario is multiple developers works in multiple projects by creating feature branch. one projects needs to go live before other. This scenario we are ending up having code that not ready for prod release in adf_publish. How to handle this? Our projects are not so big to create separate ADF. I also tried Revert PR option, but after Reverting not able to merge the same feature branch back to master in DEV.Swapnahttps://www.blogger.com/profile/14422243171992332304noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-64271505146197615582020-11-27T11:14:25.143-05:002020-11-27T11:14:25.143-05:00ADF deployment ARM template and parameter contain ...ADF deployment ARM template and parameter contain full ADF definition. It's a best practice to define one data factory per project and not to mix different project workflows into a single ADF since deployment cycles can be different.Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-36864562278745405832020-11-27T10:45:36.129-05:002020-11-27T10:45:36.129-05:00Hi ,
Im facing one issue in ADF deployment with D...Hi ,<br /><br />Im facing one issue in ADF deployment with DevOps. Please share your views.<br />Suppose, i have two Projects pipelines in master. These 2 projects merged into master from respective feature branches. Now i want to promote one project into QA and one into UAT.<br />How to handle this ? I shud remove first project code then deploy second project into UAT. ?Swapnahttps://www.blogger.com/profile/14422243171992332304noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-15563320606170204872020-10-28T17:53:05.738-04:002020-10-28T17:53:05.738-04:00If you connect all ADFs to the same git repository...If you connect all ADFs to the same git repository, it will create conflicts when you try to merge code from multiple factories. When you assign each ADF (let's say you have 3-4 data factories for a project) individual git repos, which doesn't make any good sense to me, that you will manage different version of the code in different ADFs separately and you won't need a deployment at all, it will be a mess.<br />Best practice: 1 git repo for a project which then you link to your Dev ADF, all the rest of ADF are updated via a automated deployment process using code in your Git Repository.Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-39414393901006244702020-10-28T16:42:23.994-04:002020-10-28T16:42:23.994-04:00Hi,
What if all my ADF’s are git enabled in all en...Hi,<br />What if all my ADF’s are git enabled in all environments, hiw this resource manager deployment would behave?Sunnyhttps://www.blogger.com/profile/11705224471745879198noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-15426442498442901592020-10-15T11:24:52.910-04:002020-10-15T11:24:52.910-04:00Hi Ashish, I've worked with Linked Templates t...Hi Ashish, I've worked with Linked Templates that had Logic Apps in them, but not with a Data Factory.Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-26371312630427522472020-10-13T12:53:31.349-04:002020-10-13T12:53:31.349-04:00Thanks for the details. Have you used the process ...Thanks for the details. Have you used the process for incremental deployment of ADF with more than 1000 components or using linked templates Ashishnoreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-20898087678075568302020-07-13T09:23:30.290-04:002020-07-13T09:23:30.290-04:00Hi Ravi, recently Microsoft introduced custom para...Hi Ravi, recently Microsoft introduced custom parameters in your ARM deployment templates. And this will help you naturally select all possible JSON elements (including web activities URL) into your ARM template parameter file that you can further override. I've also blogged about this: http://datanrg.blogspot.com/2020/06/raking-custom-parametersvariables-for.htmlRayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-9121388000039493432020-07-10T07:52:19.288-04:002020-07-10T07:52:19.288-04:00Hi Rayis, Thank you so much for providing such gre...Hi Rayis, Thank you so much for providing such great articles.<br />I have a question regarding CI CD deployment from Development Azure Data Factory to Production Data factory.<br />Example: Lets say i have a web activity used in pipelines the web activity URL is connected to logic apps.<br />In this case how we can customize override parameters for web activities while moving code from Dev to Prod.<br /><br />Thanks in advance for your response. <br />Regards,<br />Ravi.Ravihttps://www.blogger.com/profile/15281272145790989415noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-15549055057937643782020-06-23T10:11:43.877-04:002020-06-23T10:11:43.877-04:00I see two options: (1) decrease actual number of y...I see two options: (1) decrease actual number of your triggers, it may require some logic change in your solution as well, (2) with the use of new custom file (arm-template-parameters-definition.json) you can decrease the number of triggers being uploaded into your final ARM template parameter file: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-templateRayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-32497335328556072762020-06-10T22:53:45.426-04:002020-06-10T22:53:45.426-04:00Hi Rayis,
I have setup the devops pipeline to mig...Hi Rayis,<br /><br />I have setup the devops pipeline to migrate from dev to prod and it fails with exceed in parameters limit. Most of these parameters are scheduled triggers.<br />Question :<br />1. Is there a way I can avoid these scheduled parameters in my <br /> ArmTemplateParameters_master.json ?<br />2. Or is there any option to proceed with these Triggers.Giginoreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-76893513979817263082020-01-23T11:27:13.897-05:002020-01-23T11:27:13.897-05:00Yes, that's correct. Key Vaults with the secre...Yes, that's correct. Key Vaults with the secrets have to be provisioned first.Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-10587857368201977452020-01-23T07:35:06.805-05:002020-01-23T07:35:06.805-05:00If my understanding is correct, before creating re...If my understanding is correct, before creating release pipeline, we need to have Key vaults for each environment and create two secrets (one for holding storage access key and other for linked service connection string) and we need to place connection string in all three env key vaults before creating release pipeline right?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-47744805527071120402019-12-09T11:53:48.770-05:002019-12-09T11:53:48.770-05:00I have watched your video nothing linked. And even...I have watched your video nothing linked. And even if I wait, no resource groups arrived. By the way you don't create any Service Connection, I think it must be my issue this Service Connection. <br /><br />Did you do any video or test on it ? <br /><br />Because I can see on your videos that you can have access from 3 Resource Groups for the 3 environment, but I can't understand with Service Connection if you can do the same or how . <br /><br /><br /><br />Remyhttps://www.blogger.com/profile/12026230481618366862noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-84139801887282971562019-12-09T11:39:52.643-05:002019-12-09T11:39:52.643-05:00Sometimes, it takes a few seconds to refresh a lis...Sometimes, it takes a few seconds to refresh a list of available resource groups for your to select. Or you can type the exact name of the resource group, it allows this as well. Please watch my YouTube video on this, I think I've had this case as well.Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-41336214265738579302019-12-09T11:11:49.216-05:002019-12-09T11:11:49.216-05:00Hi Rayis,
I have an issue while I am doing your s...Hi Rayis,<br /><br />I have an issue while I am doing your step. <br />First I have a resource group where I have my Data Factory resource.<br />I created my "Service Connection" which is working. But first question, the service manager linked to this service connection should be in the resource group or the data factory resource ?<br /><br />So I have created my service connection (hopefully the right way), and I chose my Service Connection in the Azure Subscription when I create the Azure Deployment.<br />Look like I have the right and everything looks fine. But I can't select any "Resource Group", no results found. <br />Do you know what's wrong ? right issue, wrong service connection ? <br /><br />Well still I'm trying to put the data factory resource name and when I try to launch the pipeline I get a "Failed to check the resource group status. Error: {"statusCode":403}."<br /><br />I think the issue come from the fact I can't retrieve any Resource Group when I configure the pipeline.<br />Do you know where the issue can come from ?<br /><br />ThaksRemyhttps://www.blogger.com/profile/12026230481618366862noreply@blogger.comtag:blogger.com,1999:blog-5403386884901073007.post-11741297961885679142019-12-05T11:55:24.999-05:002019-12-05T11:55:24.999-05:00Please check if you've enabled Azure DevOps to...Please check if you've enabled Azure DevOps to access your KeyVault.Rayis Imayevhttps://www.blogger.com/profile/03340391869523894676noreply@blogger.com