Data storage in Azure Storage

Microsoft Azure storage is a managed service that provides durable, secure and scalable storage in the cloud. 

Durable, this means that the redundancy ensures that your data is safe in the event of transient hardware failures and also replicate data across data centers or geographical regions for extra protection from local catastrophe or natural disaster. Data replicated in this way remains highly available in the event of an unexpected outage. 

Security, all data written to Azure storage is encrypted by the service as your storage provides you with fine grained control over who has access to your data. 

Scalability, Azure storage is designed to be massively scalable to meet the data storage and performance needs of today's applications. 

Management, Microsoft Azure handles maintenance and any critical problems for you. 

A single Azure subscription can host up to 200 storage accounts, each of which can hold 500 terabytes of data. 

Azure storage offers four configuration options.

• Azure blob, a scalable object store for text and binary data.
• Azure files, managed file shares for cloud or on premises deployments(for traditional SMB file shares)
• Azure queue, a messaging store for reliable messaging between application components(persistent messaging)
• Azure table and NoSQL store for no schema storage of structured data (for structured storage of key/values)

Azure blob storage is an object storage solution optimized for storing massive amounts of unstructured data, such as text or binary data. 

Ideal for serving images or documents directly to a browser, including full static websites, storing files for distributed access, streaming video and audio, storing data for backup and restoration. Disaster recovery and archiving and storing data for analysis by on premises or Azure hosted service. 

If you need to provision a data store that will store but 

not query data your cheapest option is to set up a storage account as a blob store. 

Blob storage works well with images and unstructured data, and 

it's the cheapest way to store data in Azure. 

It also provides rest API and SDK for Azure storage in various languages. 

And supported code languages include .net, java, node, python PHP , Ruby and Go.

Microsoft Azure Blobs are files for the Cloud. 

The term BLOB is an acronym for Binary Large Object. 

For example, a Blob can hold a PDF document, 

a JPEG image, a JSON file, 

video content, and so on. 

Blobs aren't limited to common file formats. 

A Blob could contain gigabytes of 

binary data streamed from a scientific instrument, 

an encrypted message for another application, 

or data in a custom format for an app you're developing. 

Blobs are usually not appropriate for 

structured data that needs to be queried frequently. 

They have higher latency than 

memory and local disk and don't have 

the indexing features that make 

databases efficient at running queries. 

Blob Storage is ideal for storing up 

to 8 terabytes of data for virtual machines, 

storing data for analysis by an 

On-premises or Azure hosting service, 

storing data for backup and 

restore disaster recovery and archiving. 

Streaming video and audio, 

storing files for distributed access, 

and serving images or documents directly to a browser. 

Many Azure components use Blobs behind the scenes. 

For example, Azure Cloud Shell 

stores your files and configuration in 

Blobs and Azure Virtual Machines 

uses Blobs for hard disk storage. 

Also, Blobs are frequently used in combination with 

databases to store non-query-able data. 

In Blob Storage, 

every Blob lives inside a Blob container. 

You can store an unlimited number of Blobs in a container 

and an unlimited number of 

containers in a storage account. 

Remember that containers are flat. 

They can only store Blobs, not other containers. 

Blobs and containers support metadata 

in the form of name-value string pairs. 

Your apps can use metadata for anything you like. 

A human-readable description of 

a Blob's contents to be displayed by 

the app or a string that your app uses to 

determine how to process the Blob's data and so on. 

Finally, Blob Storage does not provide 

any mechanism for searching or sorting Blobs by metadata. 

 

The Blob Storage API is REST-based and 

supported by client libraries in many popular languages. 

It lets you write apps that 

create and delete Blobs and containers, 

upload and download Blob data, 

and lists the Blobs in a container

Azure storage supports three kinds of blobs. 

Block blobs are used to hold text or binary files up to 5 terabytes, 50,000 blocks of 100 megabytes in size. The primary use case for block blobs is the storage of files that are read from beginning to end, such as media files or image files for websites. And they are named block blobs because files larger than 100 megabytes must be uploaded a small blocks. These blocks are then consolidated or committed into the final blob. 

Page blobs are used to hold random access files up to 8 terabytes in size. Page Blobs are used primarily as the backing storage for the virtual hard disks or VHDs used to provide durable discs for Azure virtual machines or Azure VMs. They are named page blobs because they provide random read write access to 512-byte pages. These blobs are frequently used for logging information from one or more sources into the same blob, and to give an example, you might write all your trace logging to the same append blob for an application running on multiple VMs. 

Append blobs are made up of blocks like block blobs, but they are optimized for append operations. A single append blob can be up to 195 gigabyte

Azure files enables you to set up highly available network file shares that can be accessed using the standard server message block or SMB protocol. This means that multiple VMs can share the same files with both read and write access. You can also read the files using the REST interface and the storage client libraries. Finally, you can also associate a unique URL to any file to allow fine grained access to a private file for a set period. File shares can be used for many common scenarios, storing shared configuration files for VMs, tools or utilities so that everyone is using the same version. Log files such as diagnostics, metrics and crash dumps and shared data between on premises applications and Azure VMs to allow migration of apps to the cloud over a period. 

Azure Queue service is used to store and retrieve messages. Queue messages can be up to 64 kilobytes in size and a queue can contain millions of messages. Finally, queue are used to store lists of messages to be processed asynchronously. You can use queue to loosely connect different parts of your application together. For example, we could perform image processing on the photos uploaded by our users. Perhaps we want to provide some sort of face detection or tagging capability so people can search through all the images they have stored in our service. And we could use queues to pass messages to our image processing service to let it know that new images have been uploaded and are ready for processing. This sort of architecture would allow you to develop and update each part of the service independently.

Azure storage offers several types of storage accounts. Each type supports different features and has its own pricing model.

Standard General Purpose v2 is the standard storage account type for blobs, file shares, queues and tables, it is recommended from most scenarios using Azure storage. And the premium storage account type is for block blobs and a pen blobs, this is recommended for scenarios with high transaction rates, scenarios that use smaller objects or those that require consistently low storage latency. 

Premium storage account type is for file shares only. It is recommended for enterprise or high performance scale applications. This is useful if you require both SMB and NFS, file share support. Finally, the premium storage account type is for page blobs only. 

Azure storage also supports scripting in Azure Power Shell and 

in the Azure command line interface. 

Data ingestion, to ingest data into your system use Azure Data factory 

storage Explorer, the AzCopy tool, Power Shell or visual studio. 

If you use the file upload feature to import file sizes above two gigabytes, 

use Power Shell or Visual Studio. 

AzCopy supports a maximum file size of one terabyte and 

automatically splits Data files that exceed 200 GB. 

If you create a storage account as a blob store, you can't query the data directly. 

To directly query the data, either move the data to a store that supports 

queries or set up the Azure storage account for a data lake storage. 

Azure storage encrypts all data that's written to it. 

Azure storage also provides you with fine grain control over who has access to 

your data. 

You'll secure the data by using keys or shared access signatures. 

Azure resource manager provides a permissions 

model that uses role-based access control, RBAC. 

Use this functionality to set permissions and assign roles to users groups or 

applications.

Microsoft Azure Storage APIs

Azure Storage provides a REST API to work with the containers and data stored in each account. Each type of data you can store has its own independent API. Recall that we have four specific data types: Blobs, Queues, Tables, Files.

The storage Rest APIs are accessible 

from anywhere on the Internet by any app 

that can send a HTTP or HTTPS 

request and receive a HTTP or HTTPS response. 

For example, if you wanted to 

list all the blobs in a container, 

you would send a simple GET command. 

This would return an XML block 

with data specific to the account. 

However, this approach requires 

a lot of manual parsing and 

the creation of HTTP packets to work with each API. 

For this reason, Azure 

provides pre-built client libraries 

that make working with the service 

easier for common languages and frameworks. 

 

Client libraries can save a significant amount of work 

for app developers because the API is tested, 

and it often provides nicer wrappers around 

the data model sent and received by the Rest API. 

Microsoft has Azure client libraries 

that support several languages and frameworks, 

including.Net, Java, Python, Node.js, and Go. 

For example, to retrieve 

the same list of blobs in C Sharp, 

we could use a code snippet or 

perform a similar action in JavaScript. 

The client libraries are 

just thin wrappers over the Rest API. 

They are doing exactly what you would do 

if you use the web services directly. 

These libraries are also open source, 

making them very transparent.

To work with data in a storage account, 

your app will need two pieces of data; 

access key and REST API endpoint. 

Each storage account has 

two unique access keys that 

are used to secure the storage account. 

If your app needs to connect 

to multiple storage accounts, 

your app will require 

an access key for each storage account. 

In addition to access keys for 

authentication to storage accounts, 

your app will need to know 

the storage service endpoints to issue the REST requests. 

The REST endpoint is a combination 

of your storage account name, the data type, 

and a known domain, for example, 

blobs, queues, table, and files. 

If you have a custom domain tied to Azure, 

then you can also create 

a custom domain URL for the endpoint. 

The simplest way to handle access keys and endpoint URLs 

within applications is to use 

storage account connection strings. 

A connection string provides all needed 

connectivity information in a single text string. 

Azure storage connection strings 

looks similar to the example provided, 

but with the access key and account name 

of your specific storage account. 

There are several best practices to 

follow when working with secure access keys. 

Access keys are critical to 

providing access to your storage account, 

and as a result, 

should not be given to any system or person that 

you do not want to have access to your storage account. 

Each storage account has two access keys. 

The reason for this is to allow 

keys to be rotated or regenerated 

periodically as part of security 

best practice in keeping your storage account secure. 

This process can be done from the Azure Portal, 

or the Azure CLI, 

or PowerShell command-line tool. 

Rotating a key will 

invalidate the original key value immediately, 

and will revoke access to 

anyone who obtained the key inappropriately. 

With support for two keys, 

you can rotate keys without causing 

downtime in your applications that use them. 

Your app can switch to using 

the alternate access key 

while the other key is regenerated. 

If you have multiple apps using this storage account, 

they should all use the same key 

to support this technique. 

Let's run through the basic idea. 

First, update the connection strings in 

your application code to reference 

the secondary access key of the storage account. 

Next, regenerate the primary access key for 

your storage account using 

the Azure Portal or command-line tool. 

Then, update the connection strings in 

your code to reference the new primary access key. 

Finally, regenerate 

the secondary access key in the same manner. 

It's highly recommended that you periodically 

rotate your access keys to ensure they remain private, 

just like changing your passwords. 

If you are using the key in a server application, 

you can use an Azure Key Vault 

to store the access key for you. 

Key Vaults include support to synchronize directly to 

the storage account and 

automatically rotate the keys periodically. 

Finally, using a Key Vault 

provides an additional layer of security, 

so your app never has to 

work directly with an access key. 

You can also maintain 

a secure account with shared access signatures. 

Access keys are the easiest approach 

to authenticating access to a storage account. 

They provide full access 

to anything in the storage account, 

like a root password on a computer. 

Storage accounts offer 

a separate authentication mechanism 

called shared access signatures 

that support exploration and limited permissions for 

scenarios where you need to grant limited access.

SAS

Never share storage account keys with external third party applications. 

And if these apps need access to your data, then secure their 

connections without using storage account keys. 

For untrusted client, use a Shared Access Signature or SAS. 

The SAS is a string that contains a security 

token that can be attached to a URL. 

And use an SAS to delegate access to storage objects. 

And specify constraints such as the permissions and the time range of access. 

You can give a customer an SAS token, for example, so 

they can upload pictures to a file system in blob storage. 

And separately you can give a web app permission to read those pictures. 

In both cases, you allow only the access that the application needs to do the task. 

Two different types of SAS, service level SAS and account level SAS. 

 Use a service-level SAS to allow access to specific resources 

in a storage account. 

And you would also use this type of SAS for example to 

allow an app to retrieve a list of files in a file system or to download a file. 

Use an account level SAS to allow access to anything that a service level SAS 

can allow plus additional resources and abilities. 

And you can use an account level SAS to allow the ability to create file systems. 

Further Reading - Azure-data-lake-storage-gen2

Control network access

By default, storage accounts 

accept connections from clients on any network. 

To limit access to selected networks, 

you must first change the default action. 

You can restrict access to specific IP addresses, 

ranges and virtual networks. 

Be careful as changing network rules can affect 

your application's ability to connect to Azure Storage. 

If you set the default network rule to deny, 

you'll block all access to the data unless 

specific network rules grant access. 

Before you change the default rule to deny access, 

be sure to use network rules to 

grant access to any allowed networks.

Advanced threat protection

Microsoft Azure defender for storage provides an extra layer of security 

intelligence that detects unusual and potentially harmful attempts to access or 

exploit storage accounts. 

This layer of protection allows you to address threats without being a security 

expert or managing security monitoring systems. 

This does incur an additional cost with its own pricing structure.

Security alerts are triggered when anomalies in activity occur. 

And these security alerts are integrated with Azure security center and 

are also sent via email to subscription administrators with details of suspicious 

activity and recommendations on how to investigate and remediate threats. 

Azure defender for storage is currently available for blob storage. 

Azure Files and Azure data Lake Storage Gen2. 

Account types that support Azure defender include general purpose V2, 

block blob, and blob storage accounts. 

How does the Azure environment handle security anomalies?

When storage activity anomalies occur, you receive an email notification with 

information about the suspicious security event. 

A typical email will include the nature of the anomaly, the storage account, 

name, the event time and storage type, potential causes, and 

guidance around investigation and remediation steps. 

Finally, the email will also include details about possible causes and 

recommended actions to investigate and mitigate the potential threat. 

You can review and manage your current security alerts from 

Azure security centers security alerts tile. 

Selecting a specific alert provides details and actions for 

investigating the current threat and addressing future threats.

Reference and Credits : Microsoft.com