Create Data Pipeline in Data Factory of Microsoft Fabric

Hello everyone! In this video, I'm going to show you how to create a data pipeline in Data Factory of Microsoft Fabric

So let's get started! Basically, a data pipeline allows us to copy, move or transform data from its source to its destination

Let's see how to do this in the Data Factory So this is Microsoft Fabric welcome page

I'm going to click on this Power BI and create a workspace

So click on my workspaces and choose new workspace I'm just going to call it new data pipeline

And then click apply That's going to create the workspace Alright, so I'm going to switch from the Power BI experience to the data factory

Click on that and then we can see we have the data flow gen 2 which is power query online and the data pipeline

so I'm gonna click on data pipeline and Then I need to give a name for the pipeline. Let's just call it a

forced data Pipeline and then click create and save, start building your data pipeline

now we can add pipeline activity or copy data or choose a task to start

so we just want to copy data from a source to a destination click on copy data

and then in the copy data dialog box we can choose the source

now there are so many sources here from sample data in package csv json file and so on and so forth and of course you can even connect to different sources like the amazon rds for sql server we can connect to amazon s3 azure blob storage and so on anyway we want

to connect to this data covid19 data lake so click on that and choose next covid19 data file

we can see different kind of tables such as BINC COVID-19 and we can see COVID-19 tracker

European Center for Disease Prevention and Control and of course Oxford COVID-19 Government

Response Tracker we're connected to this European Center for Disease Prevention and Control

and then we can choose the format either from CSV which is 1.5 megabytes or we can choose from the

json json line or packet file we want to go with the csv so click on that and choose next and then

we need to choose the data destination we can see the first is the data source connect to the actual

data source and then choose the data destination now i want to actually load into a lake house so

i'm going to choose lake house and then click next now we can choose to dump the data in an existing

lake house if we have any we can choose to create a new lake house i'm going to just create a new

lake house and just call it covid19 data set and then click next in the next page we have the root

folder now i can choose tables or file so i want to go to the table and of course i can optionally

load to a new table or an existing table so i want to dump it in a new table and then i can rename

this default name but this is fine and very importantly we can see the column mapping so we can perform some things here we can see these are the source columns and then the type Now we can change some of them from string to date and some other data types

I'm just going to click on next. And then we can see the step

I want to copy from this file, from this location. And then we have this going to be the destination, the lake house

And then we can see the source and the sample data set, the destination, connection name as well as the table name

So we can choose to start data transfer immediately or maybe later

But this is fine, just click on save and run And then we can see gathering your new changes and start

We can see successfully running force data pipeline And then we can see under the output at the bottom here

we can see the pipeline run ID, we can see the pipeline status and then we can

see the activity name which is to copy data we can see the status the run

start the duration the input and output so just gonna wait for some few seconds

to see the activity status There we go, so we can see succeeded, successfully ran the first pipeline, that's quite amazing

So we've been able to transfer the data from the source into the lake house and then we

can go back to the workspace, yeah, you can come back here and double click

And there we go, we can see the pipeline that we established and the data is now in the

lake house. click on it and then wait for the data to load Okay so there we go so you can see covid19 underscore data set and then you can see all the tables which is loading anyway

okay so there we go i'm going to come to the workspace we can begin to even write sql in the

SQL ytics endpoints so when I click on that that's going to open the SQL

endpoints to write queries against the data so you can see loading your SQL

ytics endpoint metadata and there we go so I can click on new SQL query so

I'm going to write select star from ecdc underscore cases I can see where

country and territories equal to inside single code Australia right and then we

can terminate the statement and click on wrong okay so there we go so this is the

output of the query and then we can see there are 350 records where the

countries and territories is equal to Australia so this is basically how we

can connect from a data source and load the data into a destination using a data

pipeline in data factory so I hope you enjoyed this video if you do like share

to friends and comments thank you and bye for now cheers