Email: Password:       Forgot Password 
    .netCodeSG
A Saarsha Group Online Community for dot net codes group like C#, Asp.NET, VB.NET, Sharepoint, JavaScript, JQuery, Ajax, SQL, WCF, WPF.
 
TECHNOLOGIES:
 

AZURE DATA FACTORY

Article:
Viewed:  722 
Posted On:  01/09/2019 09:16:17 
What is Azure Data Factory? How we can create a Azure Data Factory? 

 

In this article, I am going to explain what Azure Data Factory is by explaining an example.

What is Azure Data Factory?

The Azure Data Factory service is a fully managed service for composing data storage, processing, and movement services into streamlined, scalable, and reliable data production pipelines.
It provide access to on-premises data in SQL Server and cloud data in Azure Storage (Blob and Tables) and Azure SQL Database.
 

In this article, I am going to show how we can transfer data from One Blob Folder to another Blob Folder using Data Factory.

Below are the steps, need to follow for this:

Step 1: Create a Storage Account

Step 2: Create a Blob (Container) => Source

Step 3: Create another Blob => Destination

Step 4: Create a Data Factory

Step 5: Create a Pipeline

Step 6: Create 2 Data Set

Step 7: Create a Linked Service

Step 8: Validate Pipeline

Step 9: Debug Pipeline

Step 10: Trigger Pipeline Manually

Step 11: Add Schedule Trigger

Assuming that you have a Storage Account in your Azure account. In this Storage account.

1.png

Image 1.

Here Select Blobs -> Create a Container (adfdemo)

2.png

Image 2.

Now upload a .txt file inside this like below:

3.png

Image 3.

4.png

Image 4.

5.png

Image 5.

Now time to create a Data Factory:

Go to Azure Portal: https://portal.azure.com/

Click on  -> +Crate a Resource -> Select Analytics -> Select Data Factory

6.png

Image 6.

Here enter all required details:

7.png

 

Image 7.

After creating New Data Factory "AzureDataFactoryDemoOne" Click on "Author & Monitor".

8.png

Image 8.

New window will open. On Left Side click on Author Option.

9.png

Image 9.

Now Time to create a Linked service so that we can link our Azure Storage Account to Data Factory.

Click on Connection in Left Side menu.

10.png

Image 10.

11.png

Image 11.

Below is our newly created Linked Service.

12.png

Image 12.

Now time to create Data Sets as we are going to move data from source (One Blob) to destination (to Another blob) so we will create 2 data Sets, type of Azure Blob, here names are:

1. InputDataSet (Azure Blob Type) => Source

2. OutputDataSet (Azure Blob Type) => Destination

To Add DataSet click on + in Factory Resource:

13.png

Image 13.

14.png

Image 14.

Click on Continue -> Select Format -> Binary

15.png

Image 15.

16.png

Image 16.

Click on Continue:

17.png

Image 17.

Now by following above steps we have to crate OutputDataSet also.

18.png

Image 18.

Select DataSet -> Azure Blob Storage -> Binary=>

19.png

Image 19.

20.png

Image 20.

Now time to create a Pipeline with Copy Activity

21.png

Image 21.

Click on Pipeline -> In General Tab Name it "CopyDataPipeLine"

From Activities section => In Move & Transform, Drag CopyData to Pipeline Design Surface.

22.png

Image 22.

Now click on Source Tab:

23.png

Image 23.

Sink Tab:

24.png

Image 24.

Now validate your Pipeline:

25.png

Image 25.

 

Now we need to Debug our Pipeline:

26.png

Image 26.

27.png

Image 27.

Now check your Storage account. Data has been copied.

28.png

Image 28.

29.png

Image 29.

 

Now time to Trigger Pipeline:

Here it will deploy all entities ie.: Linked Service, data sets, pipeline to data factory.

30.png

Image 30.

We can manually trigger our pipeline like below:

Click on Add Trigger => Trigger Now

31.png

Image 31.

32.png

Image 32.

Now click on Monitor from Left side Menu: 

33.png

Image 33.

You can view your Activity Runs

34.png

Image 34.

35.png

Image 35.

Click on Detail View:

36.png

Image 36.

Now add  New Trigger to trigger pipeline on a schedule basis:

37.png

Image 37.

38.png

Image 38.

Now publish all your activity J

  Comment:
         HOME   |   Submit Article   |   Contact Us   |   About Us   |   Terms & Condition   |   Advertise With us