Implementing CI/CD process can be a cumbersome task for a data engineer as they don’t like/ they are not familiar with concepts such as DevOps. DevOps is an essential part in software engineering, but for Data engineering it’s fairly new concept. Nevertheless, it is a part of DataOps and data engineers are expected to use DevOps in their project.

Since it’s bit of hassle to understand DevOps concepts and implement release pipeline, I thought of creating a YAML file which creates a multi-stage build pipeline. That way anyone who wants to create a CI/CD pipeline can copy this YAML file and change the sections as commented and create a pipeline, which does ARM template build as well as deployment to different environments.

SETP 1: Create environments for each stage

In typical project we will have at least 3 environment, Development, Test and Production. For each of these environment, you need to create environments within DevOps. Go to your project in Azure DevOps and create one environment for each.

Then within each environment click on Approvals and check and add the users/user groups who can approve the release for each environment. For example , if you add a user inside Production environment, only that user can approve the production ADF release.

STEP2: Modify the below YAML file

I have created a YAML file which does following task.

  1. Build ADF artifacts in to ARM template
  2. Deploy built ARM template into Test ADF instance
  3. Deploy built ARM template into Production instance

Please read the comments inside the YAML file and replace values marked in XXX characters with appropriate values for your environment. In my case, I use Azure Key vault to store all the credential information.

# Sample YAML file to validate and export an ARM template into a Build Artifact
# Requires a package.json file located in the target repository

trigger:
- master # name of the branch which trigger the build
stages:
- stage: 'Build' 
  displayName: 'Build ARM template files to deploy'
  jobs: 
  - job: 'Build'
    displayName: 'Build ARM Template'
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      wwwrootDir: 'Tailspin.SpaceGame.Web/wwwroot'
      dotnetSdkVersion: '3.1.300'
    steps:
    # Installs Node and the npm packages saved in your package.json file in the build

    - task: NodeTool@0
      inputs:
        versionSpec: '10.x'
      displayName: 'Install Node.js'

    - task: Npm@1
      inputs:
        command: 'install'
        verbose: true
      displayName: 'Install npm package'

    # Validates all of the ADF resources in the repository. You will get the same validation errors as when "Validate All" is clicked
    # Enter the appropriate subscription and name for the source factory 

    - task: Npm@1
      inputs:
        command: 'custom'
        customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions/XX[Subscription Id]XX/resourceGroups/XX[resource group name] XX/providers/Microsoft.DataFactory/factories/XX[name of the dev ADF instance]XX'
      displayName: 'Validate'

    # Validate and then generate the ARM template into the destination folder. Same as clicking "Publish" from UX
    # The ARM template generated is not published to the ‘Live’ version of the factory. Deployment should be done using a CI/CD pipeline. 

    - task: Npm@1
      inputs:
        command: 'custom'
        customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/XX[subscription id]XX/resourceGroups/XX[name of the resource group]XX/providers/Microsoft.DataFactory/factories/XX[dev adf instance name]XX "ArmTemplate"'
      displayName: 'Validate and Generate ARM template'

    # Publish the Artifact to be used as a source for a release pipeline

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(Build.Repository.LocalPath)/ArmTemplate'
        artifact: 'ArmTemplates'
        publishLocation: 'pipeline'
- stage: 'Test'
  displayName: 'Deploy Changes to Test Environment'
  dependsOn: Build
  condition: succeeded('Build')
  jobs: 
  - deployment: DeployToTest
    pool:
      vmImage: 'vs2017-win2016'
    environment: Test 
    strategy:
      runOnce:
        deploy:
          steps: 
          - task: DownloadPipelineArtifact@2 #downloading artifacts created in build stage
            inputs:
              source: 'current'
              path: '$(Pipeline.Workspace)'
          - task: AzureResourceManagerTemplateDeployment@3
            displayName: 'ARM Template deployment: Resource Group scope'
            inputs:
              azureResourceManagerConnection: 'XX[Azure service connection name]XX'
              subscriptionId: 'XX[subscription id]XX' #your subscription id
              resourceGroupName: 'XX[resource group of Test ADF instance]XX' #resource group of the target ADF instance 
              location: 'XX[Location of Test ADF instance]XX' #location of the target ADF instance 
              csmFile: '$(Pipeline.Workspace)/ArmTemplates/ARMTemplateForFactory.json' # Location of the ARM template in downloaded build
              csmParametersFile: '$(Pipeline.Workspace)/ArmTemplates/ARMTemplateParametersForFactory.json' #Location of the parameter ARM template of build
              overrideParameters: '-factoryName "XX[Test adf instance name]XX" -ls_azuresql_asankap_properties_typeProperties_connectionString_secretName "ConynectionString-ContosoSQL" -ls_blob_asankastorage_properties_typeProperties_serviceEndpoint "https://XXXX.blob.core.windows.net" -ls_keyvault_properties_typeProperties_baseUrl "https://kv-XXXX-dev.vault.azure.net/"' 
        #XXXX values needs to be replaced for target ADF instance, this include name of the target adf instance, key vault url and etc.  
- stage: 'Production'
  displayName: 'Deploy Changes to Production'
  dependsOn: Test
  condition: succeeded('Test')
  jobs: 
  - deployment: DeployToProduction
    pool:
      vmImage: 'vs2017-win2016'
    environment: Production 
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadPipelineArtifact@2 #downloading artifacts created in build stage
            inputs:
              source: 'current'
              path: '$(Pipeline.Workspace)'
          - task: AzureResourceManagerTemplateDeployment@3
            displayName: 'ARM Template deployment: Resource Group scope'
            inputs:
              azureResourceManagerConnection: 'XX[Service connection Name]XX'
              subscriptionId: 'XX[subscription id]XX' #your subscription id
              resourceGroupName: 'XX[Production RG name]XX' #resource group of the target ADF instance 
              location: 'XX[Prod ADF Instance location]XX' #location of the target ADF instance 
              csmFile: '$(Pipeline.Workspace)/ArmTemplates/ARMTemplateForFactory.json' # Location of the ARM template in downloaded build
              csmParametersFile: '$(Pipeline.Workspace)/ArmTemplates/ARMTemplateParametersForFactory.json' #Location of the parameter ARM template of build
              overrideParameters: '-factoryName "XX[Prod ADF instance name]XX" -ls_azuresql_asankap_properties_typeProperties_connectionString_secretName "ConnectionString-ContosoSQL" -ls_blob_asankastorage_properties_typeProperties_serviceEndpoint "https://XX.blob.core.windows.net" -ls_keyvault_properties_typeProperties_baseUrl "https://kv-XX-dev.vault.azure.net/"'
              #values needs to be replaced for target ADF instance, this include name of the target adf instance, key vault url and etc.  
        

STEP 3: Create build pipeline

Next step is to create a YAML pipeline. Go to pipeline in Azure DevOps and select create new pipeline. Then select Azure Repos Git as code location and select where you want to store the YAML file you modified in previous step.

Next in the configure section select “Starter Pipeline”. This will open you an editor window. Paste the YAML file you modified in STEP2.

Click on Save and run and if you do modify the YAML file correctly , you will see DevOps build the ARM template and deploy to the different environment based on the approval process you set.

Thanks for reading. Cheers!

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s