Templates; Latest addition to the ADF. Few days ago, when I open ADF development environment, there was a new option available in left side panel and I did not have time to looking into it at that time. However,here am I after a week or so, checking what this new feature is all about. Normally I don’t write post about new features because there are plenty of documents/ blogpost available when someone want to learn about those. But in this case I couldn’t find any and hence thought of writing something about it.
First of all,where is can you see this new option? 3 places I would say. First in home page of the ADF, you get a option to create a pipeline using template called “Create pipeline from template”( Image: 01)
Secondly, when you go to “Author” area , where you create pipelines , dataset, etc.., you have a menu option called “Templates” just bellow “Datasets”. ( Image: 02)
Then when you click + icon to create a new pipe line , ADF gives you an option to create a pipeline from existing template ( Image: 03)
Now what is a template? It is nothing but a pipeline which is configured to perform a certain task using certain components. Which means if you know that you are going to perform same set of actions using same set of pipeline activities with minimum changes , then you can re use that pipeline to create one or more pipelines .
Lets take an example.When it comes to ADF pipe line, the most common used action is to copy data from one source to another. If you are to create a 10 copy pipelines , you will have to create 10 source datasets, 10 destination datasets and 10 pipelines for this purpose. However, using Template, you can do it must easier manner. Staring from Microsoft created templates which covers most commonly used pipeline activities such as coping data from on premise SQL server to Azure SQL database.
If you click “Pipeline from Template”, It will show you template created by other vendors as well as template created by your self (Image: 04).
Once you select a template, based on your need, in this case copy data from On-Prem SQL Server to Azure SQL database, it only asks for the source connection and the destination connection. Check image bellow.
If you already have a connection you can pick that connection or it gives a option to create a connection from that window it self. Once connections are selected, it will create two datasets, one for source, one for destination and along with a new pipeline. (Image: 06)
Image : 06
Now only thing you have to do is to go datasets and select source table as well as destination table.
But for me, the real value of Template is not that. I believe that it will save lot of time if you create a your own template and reuse it whenever possible. For example, I have created a pipeline with a Copy activity as well as Notebook activity.In this pipeline, data is copied to Azure Data Lake Store from on premise SQL Server database and using Azure Databricks that data is processed and then saved inside an Azure SQL database. Check the image bellow.
Since I know that, this pattern will be used over and over again inside my project, I can save this as a template by clicking “Save as template” button( Image:09) .However, in order to do that, my Data Factory should be integrated with Git.
Once that is done, it will be available under Templates section .
Now only thing you have to do when you want to create a new pipeline with same pattern , is to click on your custom template and it will open up a window to select user inputs, in this case connections to a on-prem SQL server,a data lake as well as a Databricks.
As I mentioned before, it will automatically create all the required datasets for this pipeline and you only have to configure those.
Thanks for reading, Cheers !!.