Azure Data Factory–Templates

Templates; Latest addition to the ADF. Few days ago, when I open ADF development environment, there was a new option available in left side panel and I did not have time to looking into it at that time. However,here am I after a week or so, checking what this new feature is all about. Normally I don’t write post about new features because there are plenty of documents/ blogpost available when someone want to learn about those. But in this case I couldn’t find any and hence thought of writing something about it.

First of all,where is can you see this new option?  3 places I would say. First in home page of the ADF, you get a option to create a pipeline using template called “Create pipeline from template”( Image: 01)

image

Image: 01

Secondly, when you go to “Author” area , where you create pipelines , dataset, etc.., you have a menu option called “Templates” just bellow “Datasets”.  ( Image: 02)

image

                Image: 02

Then when you click + icon to create a new pipe line , ADF gives you an option to create a pipeline from existing template ( Image: 03)

image

             Image: 03

Now what is a template? It is nothing but a pipeline which is configured to perform a certain task using certain components. Which means if you know that you are going to perform same set of actions using same set of pipeline activities with minimum changes , then you can re use that pipeline to create one or more pipelines .

Lets take an example.When it comes to ADF pipe line, the most common used action is to copy data from one source to another. If you are to create a 10 copy pipelines , you will have to create 10 source datasets, 10 destination datasets and 10 pipelines for this purpose. However, using Template, you can do it must easier manner. Staring from Microsoft created templates which covers most commonly used pipeline activities such as coping data from on premise SQL server to Azure SQL database.

If you click “Pipeline from Template”, It will show you template created by other vendors as well as template created by your self (Image: 04).

image

Image:04

Once you select a template, based on your need, in this case copy data from On-Prem SQL Server to Azure SQL database, it only asks for the source connection and the destination connection. Check image bellow.

image

Image: 05

If you already have a connection you can pick that connection or it gives a option to create a connection from that window it self. Once connections are selected, it will create two datasets,  one for source, one for destination and along with a new pipeline. (Image: 06)

imageimage

Image : 06

Now only thing you have to do is to go datasets and select source table as well as destination table.

imageimage

Image: 07

But for me, the real value of Template is not that. I believe that it will save lot of time if you create a your own template and reuse it whenever possible. For example, I have created a pipeline with a Copy activity as well as Notebook activity.In this pipeline, data is copied to Azure Data Lake Store from on premise SQL Server database and using Azure Databricks that data is processed and then saved inside an Azure SQL database. Check the image bellow.

image

Image: 08

Since I know that, this pattern will be used over and over again inside my project, I can save this as a template by clicking “Save as template” button( Image:09) .However, in order to do that, my Data Factory should be integrated with Git.

image

            Image: 09

Once that is done, it will be available under Templates section .

image

           Image: 10

Now only thing you have to do when you want to create a new pipeline with same pattern , is to click on your custom template and it will open up a window to select user inputs, in this case connections to a on-prem SQL server,a data lake as well as a Databricks.

image

Image: 11

As I mentioned before, it will automatically create all the required datasets for this pipeline and you only have to configure those.

Thanks for reading, Cheers !!.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s