Azure Data Factory - Chapter 1

 

Azure data factory is the cloud based ETL and 

data integration service that allows to create data driven workflows for 

orchestrating data movement, and transforming data at scale. 

Using Azure data factory, one can create and schedule data driven workflows 

called pipelines, that can ingest data from disparate data stores. B

uild complex ETL processes, that transform data visually, 

with data flows or by using compute services such as Azure HD insight, 

Hadoop, Azure data bricks and Azure synapse analytics. 

Much of the functionality of Azure data factory appears in Azure synapse 

analytics, as a feature referred to as pipelines w

hich enables to integrate data pipelines between SQL pools, Spark 

pools and SQL serverless, providing a one stop shop for all your analytical needs. 

Azure data factory also provides rich visualizations to display the lineage and 

dependencies between your data pipelines, and monitor all your data pipelines from 

a single unified view, to easily pinpoint issues and set up monitoring alerts.

The steps for the ELT process are the same as for the ELT process, 

they just follow a different order. 

In ELT process data is extracted and loaded in its native format. 

This change reduces the time required to load the data into a destination system. 

The change also limits resource contention on the data sources. 

Azure Data Factory Process

The pipelines are data-driven workflows, 

in Azure Data factory typically perform the following four steps, 

connect and collect, transform and enrich, publish, and monitor. 

Connect and collect - 

The first step in building an orchestration system is to define and 

connect all the required sources of data together such as databases, 

file shares, and FTP Web services. 

The next step is to ingest the data as needed to a centralized location for 

subsequent processing. 

Transform and enrich - compute services such as data bricks and machine learning can be used to prepare or produce transformed data on a maintainable and 

control schedule to feed production environments with cleansed and 

transform data. 

Publish - after the raw data has been refined into a business ready consumable 

form from the transform and enrich phase,

 load the data into Azure data warehouse as your SQL database, 

Azure Cosmos DB  

Or whichever analytics engine your business users can point to from their 

business intelligent tools. 

Monitor -  Azure data factory has built in support for 

pipeline monitoring via Azure Monitor, API Power Shell, 

Azure Monitor Logs and health panels on the Azure portal. 

To monitor the scheduled activities and pipelines for success and failure rates. 

Azure data factory is composed of components that work together to 

provide the platform on which you can compose data driven 

workflows with steps to move and transform data. 

Azure Data Factory is composed of below key components.

• Pipelines
• Activities
• Datasets
• Linked services
• Data Flows
• Integration Runtimes

Pipeline

A data factory might have one or more pipelines. A pipeline is a logical grouping of activities that performs a unit of work. Together, the activities in a pipeline perform a task. For example, a pipeline can contain a group of activities that ingests data from an Azure blob, and then runs a Hive query on an HDInsight cluster to partition the data.

Activity

Activities represent a processing step in a pipeline. For example, you might use a copy activity to copy data from one data store to another data store. Similarly, you might use a Hive activity, which runs a Hive query on an Azure HDInsight cluster, to transform or analyze your data. Data Factory supports three types of activities: data movement activities, data transformation activities, and control activities.

Datasets

Datasets represent data structures within the data stores, which simply point to or reference the data you want to use in your activities as inputs or outputs.

Linked services

Linked services are much like connection strings, which define the connection information that's needed for Data Factory to connect to external resources. For example, an Azure Storage-linked service specifies a connection string to connect to the Azure Storage account. 

Integration Runtime

An integration runtime provides the bridge between the activity and linked Services. It's referenced by the linked service or activity, and provides the compute environment where the activity either runs on or gets dispatched from. This way, the activity can be performed in the region closest possible to the target data store or compute service in the most performant way while meeting security and compliance needs.

Default setting

There are three types of integration runtime, 

including Azure, self hosted, and Azure-SSIS. 

Azure Integration runtime

When copying between two Cloud data sources where 

both source and sink linked services are using Azure IR, 

ADF will use the regional Azure IR if specified, 

or also determine a location of Azure IR if you choose 

the auto-resolve IR or default 

as described in integration runtime location section. 

Azure automatically provisions an integration runtime 

which can connect to Azure resources such as Azure SQL, 

Azure Synapse Analytics 

storage accounts without any issues.

                                        

Self-hosted IR

Perform data integration securely in 

a private network environment 

shielded from the public Cloud environment by

 installing 

a self-hosted IR inside your virtual private network. 

The self-hosted integration runtime only makes 

outbound HTTP-based connections to open Internet. 

When you need to copy between 

a Cloud data source and 

a data source in a private network,

if

 either source or sink 

linked service points to a self-hosted IR, 

the copy activity is executed 

on that self-hosted integration runtime

                                     

Azure - SSIS IR

If you are copying between two data sources in 

private network and both the source 

and sink linked service 

must point to the same instance of integration runtime 

and that integration runtime is 

used to execute the copy activity. 

You can natively execute SSIS packages by 

creating an Azure SSIS integration runtime, 

which creates an integration services catalog 

in Azure SQL database where the packages are stored. 

An ADF pipeline run sends commands to the Azure SSIS IR, 

which executes the SSIS packages. 

Data Factory Triggers

Pipeline runs are typically instantiated by passing arguments to parameters that you define in the pipeline. You can execute a pipeline either manually or by using a trigger.

1. Pipelines and triggers have many to many relationships(except for tumbling window trigger).                  Multiple triggers can kick off a single pipeline or a single trigger can kick off multiple pipelines.

2. Different types of triggers - 

• Schedule Trigger : A trigger that invokes a pipeline on a wall-clock schedule. It supports periodic and advanced calendar options. 
• Event based trigger - A trigger that responds to an event. 
• Tumbling window trigger - A trigger that operates on a periodic interval, while also  retaining state. Creates windows, Works well for historical dates Self Dependency Trigger - Using of offsets becomes compulsory

Read more about Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics

Data flows 

These are one of the features inside the Azure Data Factory which allows data engineers to develop data transformation logic in a graphical approach without writing code. The resulting data flows can then be executed as activities within Azure Data Factory pipelines that use scaled-out Spark clusters. Data flows will run on your own execution cluster for scaled-out data processing. ADF internally handles all the code translation, spark optimization and execution of transformation. Data flow activities can be operationalized via the existing Data Factory scheduling, control, flow, and monitoring capabilities.

There are two types of Data flows:

• Mapping Data Flow
• Wrangling Data Flow

Other terminologies -

Control flow is an orchestration of pipeline activities that include chaining 

activities in a sequence, branching, defining parameters at the pipeline 

level and passing arguments while invoking the pipeline on demand are from a trigger. 

It also includes custom state passing and looping containers and for each iterators. 

Pipeline runs

A pipeline run is an instance of the pipeline execution. Pipeline runs are typically instantiated by passing the arguments to the parameters that are defined in pipelines. The arguments can be passed manually or within the trigger definition.

Parameters are key value pairs of read only configuration. 

Parameters are defined in the pipeline. 

The arguments for the defined parameters are passed during execution from the run 

context that was created by a trigger or a pipeline that was executed manually. 

Activities within the pipeline consumed the parameter values. 

 

Variables

Variables can be used inside of pipelines to store temporary values and can also be used in conjunction with parameters to enable passing values between pipelines, data flows, and other activities.

Once all the work is complete, use data factory to publish 

the final data set to another linked service that can be then 

consumed by technologies such as Power BI or machine learning. 

ADF security 

To create Data Factory instances, 

the user account that you use to 

sign into Azure must be a member of 

the contributor or owner role 

or administrator of the Azure subscription. 

To create and manage 

Data Factory objects including datasets, linked services, 

pipelines, triggers, and integration runtimes, 

the following requirements must be met. 

To create, and manage 

child resources in the Azure portal, 

you must belong to 

the Data Factory contributor role 

at the resource group level or above. 

To create and manage resources 

with PowerShell or the SDK, 

the contributor role at 

the resource level or above is sufficient. 

As contributor, you can create, edit, 

and delete data factories and 

child resources, including datasets, 

linked services, pipelines, 

triggers, and integration runtimes and other activities listed in the below image.

Data can be copied from more than 80 connectors or in between 80 different applications in Azure. These applications may be Software as a Service kind of applications like Dynamic 365 or Salesforce. Or these applications may be on-premises data stores like SQL Server or Oracle. Or these applications may be cloud data stores like Azure SQL Database or maybe Amazon S3. 

Thus it enables to copy data from application in any environment to different application in any other environment. While copying, it can convert file formats, like zip or unzip file , map the columns between the source and the destination. 

In addition to copying data, transformation of the data is also possible.  Previously the only way to do this was using the external services like HDInsight, or Databricks, or maybe SQL Server stored procedures. But very recently in 2019, Azure Data Factory added the Data Flow task. Now this Data Flow task is very much like a Data Flow in the SSIS and it has all possible kind of transformations.

Now both copy as well as transforming the data is possible right from the same interface in Data Factory. This makes Data Factory a complete ETL tool or Data ingestion tool. 

 

Different ways of working with Azure Data Factory -

1. Azure Portal UI

2. Azure PowerShell

3.  .NET

4. Python

5. REST

6. Resource Manager Template

 Summary :

1. ADF allows to create workflows for transforming and orchestrating the data movement.

2. Data movement triggers can be timed or event driven.

3. The management of these strategic events is called as orchestration. The better to say is the management of these events, triggers, workflows, data pipelines dependencies is called as orchestration.

4. ADF is a cloud based software as a service (ETL) tool. Also called as code free ETL service.

5. Operations associated with ADF are : Ingest, Control, Data flow, Schedule and Monitor.

6. Parameterize Datasets in ADF : You can parameterize a dataset and pass dynamic values at run time.

7. System Variables in ADF : Available at three scopes (Pipeline scope, tumbling window trigger scope and schedule trigger scope)

8.Connectors in ADF :

Data can be copied from more than 80 connectors or in between 80 different applications in Azure.

9.Supported formats for datasets :

Avro, Binary (.txt) , Delimited text format (csv), JSON format, ORC format, Parquet format

ORC, Parquet and Avro format are all part of Apache Hadoop eco-system. They work on compression algorithm , data is stored in compression and hence query results are faster