ADF team recently announced that they support automated publish for ADF and it was something I wanted for a while in my day today ADF works. Before that when someone wants to create a release in DevOps, that person must go to the master branch and click the publish button manfully to generate ARM template files which are used within ARM template deployment in DevOps. With this announcement, whenever you make a pull request to your collaboration branch, in most cases the master branch, a new build is created and hence manual intervention is not required anymore.  You can read more about this in below Microsoft documentation.https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment-improvements

The reason why I wanted to write this post is that I felt that this documentation is not comprehensive enough so that someone new to DevOps cannot read this document and configure it. Therefore, I’m going to create a step by step guide for this.

In my case I have my master branch and a feature branch in my dev ADF instance.  The idea is whenever I make a pull request from my working branch (“AsankaP”) to master, it should trigger a new release and push the release to QA.

image
Branch set up in Dev ADF

STEP 1: Copy package.json file to your master branch repo.

Copy and paste below JSON code and save it as package.json inside your dev repository. You don’t need to change anything in this code.

{
     "scripts":{
         "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
     },
     "dependencies":{
         "@microsoft/azure-data-factory-utilities":"^0.1.3"
     }
}
image
Save package.json file

STEP2: Creating a Build Pipeline

Go go the DevOps and click on Pipelines and the create a new pipeline. Select Azure Repo as your code location and select where you want to save the YAML file.

image
Select Azure Repo Git for YAML file location

Then you need to select in which repo you want to save the YAML file, I select the same repo I have my ADF code.

Select a Repo to save file.

Then it will ask how you want to configure your release and select “Starter Pipeline”.

Select Starter Pipeline

Replace the default template YAML code with the code below and change the area I have mentioned within square brackets.

You will need to change:

  • Azure Subscription Id
  • Dev Resource Group Name
  • Dev ADF Instance Name

Apart from that, if your main branch is not master, change “master” to that branch name in trigger section.



trigger:
- master #change master to something else if master is not your main branch

pool:
  vmImage: 'ubuntu-latest'

steps:


- task: NodeTool@0
  inputs:
    versionSpec: '10.x'
  displayName: 'Install Node.js'

- task: Npm@1
  inputs:
    command: 'install'
    verbose: true
  displayName: 'Install npm package'



- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions/[xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-your subscription id]/resourceGroups/[testResourceGroup-your resource group name]/providers/Microsoft.DataFactory/factories/[yourFactoryName- your dev data factory name]'
  displayName: 'Validate'

# Validate and then generate the ARM template into the destination folder. Same as clicking "Publish" from UX
# The ARM template generated is not published to the ‘Live’ version of the factory. Deployment should be done using a CI/CD pipeline. 

- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/[xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-your subscription id]/resourceGroups/[testResourceGroup-your resource group name]/providers/Microsoft.DataFactory/factories/[yourFactoryName- your dev data factory name]"ArmTemplate"'
  displayName: 'Validate and Generate ARM template'

# Publish the Artifact to be used as a source for a release pipeline

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.Repository.LocalPath)/ArmTemplate'
    artifact: 'ArmTemplates'
    publishLocation: 'pipeline'

Once all the modifications are done, click “Save and Run” to see whether build pipeline is working correctly. If there is no error, it should generate artifacts in build folder. In my case ARM Template folder.

Build pipeline execution

If you click on the highlighted section.”1 artifact produced”, you should be able to see generated ARM template files by the build.

ARM template files generated from the build pipeline

STEP 3: Create a Release Pipeline

Next step is to create a release pipeline. Go to Releases in DevOps and create new release pipeline. Select Empty Job to start with.

Select empty job in release pipeline.

Click on Add an Artifact and select build as source type. Select the name of the build pipeline you created in previous step. You can keep Latest to the Default Version or can change based on your release requirement.

Select Build as source type

Then click on “Continuous Deployment Trigger” and enable continuous deployment as below. This will make sure when a new build is available, a release is trigger from the release pipeline. Make sure you select correct build branch. In my case master branch.

Enable continuous deployment trigger

Then go to stage and add an agent job. In the Agent Job, click + icon and search for ARM template. Select ARM template and click Add.

Add ARM template deployment task to agent

In ARM template deployment provide Subscription Information, Azure Resource Manager connection and Resource group. By this time, you should have create a ADF instance for QA and Prod. In my case QA ADF instance is asankap-QA

Click on Template and select “ARMTemplateForFactory.json” as the template file as below.

Select template file

Click on “Template parameters” and select “ARMTemplateParametersForFactory.JSON” as parameter file.

Select Template parameter file

Then click on Override template parameters and set the parameters manually. Unfortunately, since build is only accessible in run time, parameters are not generated and you will have to add them manually one by one. Check below image to see how to add parameters you want to override. You can get these parameter names from the JSON you specified previously. Make sure you change the factoryName to your QA or Production ADF instance name.

Manually adding template parameter values.

Save the release pipeline and do a modification to in DEV ADF instance and make a pull request to update the master branch.

Create a pull request to trigger a release

If you go and check DevOps, you will see that the build pipeline is triggered and once the build is ready, the release pipeline is triggered to move changes QA ADF instance.

Build pipeline is triggered by the pull request.

Release pipeline is triggered and a release is created.

I hope this post helps you to create fully automated release pipelines for your ADF environment. Thanks for reading and stay safe! Cheers!

3 thoughts on “Automated publish (CI/CD) for Azure Data Factory using DevOps

  1. Thanks for the thorough tutorial! One question – do you publish changes to Dev through the release pipeline too? I tried to do that, but it caused git-mode to become disabled 😕

    1. Not sure how you have set up the environment here. You don’t have to publish changes to Dev as you used Dev instance to create all the artifacts. Having ARM templates in Dev does not help as well.

  2. Thank you for taking the time to put a more comprehensive set of steps to implement CI/CD. Both pipeline build and release run successfully, however after I check my QA ADF environment, I don’t see any of my pipelines (I started with a blank repo and have been creating a simple pipeline with a wait activity for testing purposes). Not sure if it’s an access issue, or ARM Templates aren’t updating?

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s