Introduction

In this post, we talk about Azure Batch Service. This service was created to execute parallels batch workloads without having the stress of the resources that they need to run these jobs. And that happens because Azure Batch is responsible to create and manage pools of compute nodes. Imagine a lot of parallel batch workloads separated each from another. In a quick thought we can say that is similar to another Azure service VMSS (Virtual Machine Scale Sets).

We can start processing parallels workloads via Azure Batch service ONLY by using
APIs and tools. These can be used to Create and manage a pool of compute nodes, and then we can schedule to run different jobs and tasks.

For Azure Batch service we need to pay for the used resources (compute & storage).

Azure Batch Use Cases

In the table below we can see some of the Azure Batch use cases with an example to understand where Batch service used in every case.

Use CaseExample
Financial Data AnalysisImagine the financial data analysis for a bank
Software TestingA developing software team can run multiple parallel tests for an application
ΑI SolutionA good example is a Face recognition service

Azure Batch Architecture

At the image below we can see a sample of Azure Batch architecture.

  Note
A Pool execute a job and the Nodes(VMs) of the Pool executes one or more tasks/jobs.

Azure Batch Concepts

Service Quotas And Limits

In this point of the post, we are going to read about the Quotas and the Limits of Azure Batch Service. We should read carefully the following tables because if we don’t understand the meaning of the values for the quotas and the limitations might have future
problems with the workloads.

Resource Quotas

Service Quota is quite important for the Azure Batch workloads because it is very likely in a rough design might be reached this limit.

ResourceDefault LimitMaximum Limit
Batch accounts per region per subscription1-350
Dedicated cores per Batch account10-100N/A
Low-priority cores per Batch account10-100N/A
Active jobs and job schedules3 per Batch account100-3001000
Pools per Batch account20-100500

Pool Size Limits

The Pool Size is the number of the nodes, as a single node consider a virtual machine.
The next table shows the limits for the pool size.

ResourceMaximum Limit
Compute nodes in
inter-node communication enabled pool
Batch service pool allocation mode100
Batch subscription pool allocation mode80
Compute nodes in
Dedicated nodes2000
Low-priority nodes1000
  Note
We can choose high-priority nodes, which are dedicated VMs and low-priority nodes, of course there some limitations which can found in this section of the post.

Other Limits

All the other limits are relevant with the Azure Batch Workloads details.

ResourceMaximum Limit
Concurrent tasks per compute node4 x number of node cores
Applications per Batch account20
Application packages per application40
Maximum task lifetime180 days

If the workloads need to increase the quota on an Azure Batch Account, then we can follow the directions at this link.

Supported VM Sizes

When we create an Azure Batch Pool, it is very important to select the correct VM size for the nodes of the Pool.
At the tables below we can see what are the sizes that the Azure Batch Pool DOES NOT support.

FamilyUnsupported sizes
Basic A-SeriesBasic_A0(A0)
A-SeriesStandard_A0
B-SeriesAll

VM size which are supported for Low-Priority nodes

FamilySupported Sizes
M-SeriesStandard_M64ms
M-SeriesStandard_M128ms

Virtual Machine Image Type

There are two types of images that we can choose between Pre-configured and
Custom. Of course, there are some differences between those two types :

Pre-configured ImageCustom Image
The image already existsNeed to create a new one
No need for updating & patchingNeed patching & updating
All custom software need to be installed via pool configNo need for large changes in the pool config

How It Works

In the following steps, we will see a quick demo of the Azure Batch Service.

Prerequisites

To proceed further with the demo we must be sure that we have all the following:

Create The Azure Batch Account

Search for the Azure Batch Service

From the Azure Portal left main blade, select + Create a resource, type [Batch Service], and select to Create the Batch Service.

Basics Tab

In the Basics Tab we have to  fill in few fields and move to the tab “Advanced”

SettingValue
SubscriptionSelect a valid subscription
Resource groupSelect an existing or create a new Resource group
Account nameType a name for the Azure Batch Account, MUST be unique
LocationSelect a Location for the Batch Instance
Select a Storage accountSelect an existing storage account or after deployment complete, create a

storage account
and link it with the Azure Batch account.

Advanced Tab

In the Advanced tab, we must choose a Pool allocation mode. The choices are two Batch service and User subscription, for the demo purposes we select Batch service.

Batch serviceThe pool VMs are created using behind-the-scenes Batch service subscriptions.
User subscriptionThe pool VMs are created directly in the same subscription as the Batch account.

Check this
blog post
about Azure Batch capabilities for more details.

Review + Create Tab

In the Review + create tab, we just need to check if the validation passed and click
Create to start the Azure Batch Account deployment.

After the Azure Batch account is created we are ready to see how this works. And that is the juicy part of this post.

 Important
For the part of the demo we will use an existing project from GitHub, which is coded from the user dlepow.

Azure Batch Sample

First, we must connect to GitHub and move on the “Azure-Samples/batch-dotnet-ffmpeg-tutorial” section, by clicking
here.

Run the file BatchDotnetTutorialFfmpeg.sln

As we can see there are some Dependencies missing, for that reason we select Built – Rebuild Project

We must download and install the
.Net Core SDK
, and then Build the Project (BuildBuild Solution).

After the Build is complete successfully, the Dependencies looks fine.

  Note
If after the build we get the following error, then we should close and open the VS.

Download & Install Application Packages

One of the prerequisites for this Azure Batch app is to download and upload the ffmpeg3.4 to the Azure Batch Service. This can be done by following the next steps:
  1. Download the 64bit ffmpeg 3.4 file from this
    here
    .
  2. Upload the zip file “ffmpeg-3.4-win64-static.zip” to Azure Batch Service, from the left menu blade
    FeaturesApplications+ Add

The Code Part

After we successfully complete the Build of the solution then we must make some changes to the code.

public
class
Program
    {
        // Update the Batch and Storage account credential strings below with the values unique to your accounts.
        // These are used when constructing connection strings for the Batch and Storage client objects.
 
        // Batch account credentials
        private
const
string
BatchAccountName = "xxxxxxxxx";
        private
const
string
BatchAccountKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==";
        private
const
string
BatchAccountUrl = "https://xxxxxxxxxxxx.westeurope.batch.azure.com";
 
        // Storage account credentials
        private
const
string
StorageAccountName = "xxxxxxxxxxxxxxxxxxxxx";
        private
const
string
StorageAccountKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==";

The Batch and Storage Account Credentials are in the Batch Account dashboard in Azure Portal (see the image below).

Running The App

We complete all the necessary steps with download, installations, configurations etc. And the next thing to do is just Running the Application.

  Note
This App is processing media files in parallel using the ffmpeg tool

The first thing when the app runs is the following cmd prompt console

Create Containers [Input] – [Output]

The Console App creates two Storage Containers (Input, Output).

Upload The Media Files

The second step is to begin media files uploading

Create The Batch Pool

The App creates 5 low-priority nodes inside the Batch Pool and the 5 Tasks that will run parallel in every node. At the two next images we can see exactly the Pool with the nodes in the Azure Portal.

The Running Tasks

At the image below we can clearly see, what about Azure Batch parallel workload works

The Final Results

For the final step we don’t have something to do, all the 5 files are processed and created in the Output folder.

Conclusion

In this post, we made a quick intro to Azure Batch a service that is basically addressed to developers but also can be useful and a very important tool for other groups like IT or in our days much better DevOps.

See Also


Share This