Understanding Azure container offerings

The Microsoft Azure platform offers several different solutions for container management as well as container orchestration:

What should I pick??

We'll go through each of the different services, and provide some detail about what they're used for, and also note what you probably shouldn't use due to planned sunsetting.

What are the different options?

  • Azure Container Instances - Also referred to as "ACI" or "Container instances"
  • Azure Container Registry - Also referred to as "ACR" or "Container registry"
  • Azure Container Services - Also referred to as "ACS" or "Container services"
  • acs-engine - An open source project used to generate ARM templates to deploy Azure resources with orchestrator binaries already configured
  • Azure Kubernetes Services - Also referred to as "AKS" or "Kubernetes services"
  • aks-engine - An open source project used to generate ARM templates to deploy Azure resources with Kubernetes binaries already configured
  • Virtual kubelet - An experimental open source project used in conjunction with AKS and ACI

Azure Container Instances:

Azure Container Instances can be thought of as "container-infrastructure-as-a-service".  With ACI, you don't really have to worry about what container runtime you're dealing with (docker, rkt, etc.), or what container orchestrator (Kubernetes, DC/OS, etc.) is running them.   All you do is supply a container image and the specs (CPU/RAM/etc.) you need, and Azure takes care of the rest for you.

One upside to this is it's extremely easy to get started hosting a container in the cloud.  However, digging in further you can see this really is built more for one-off types of workloads, not for something more complex where you need the control.  For example, in a Kubernetes environment, you can easily specify that you need X instances of a container image to run.  However, with ACI, you only get one container in a container group, and to "scale up", you either need to use vertical scaling by increasing the size (and cost) of your container group, or by adding more container groups with the same image.

Azure container instances would probably not be the go-to if you have a full distributed application that you wanted to host.  However, if you have something like an infrequent batch processing job, or something that is triggered by an event in say an Azure queue or Azure function, ACI might be more useful to you there.

NOTE: ACI gets more interesting when used in conjunction with AKS and the Virtual Kubelet, which we'll talk about later in the article.

Azure Container Registry:

Azure Container Registry is pretty much what it sounds like: a container registry to store your Docker images in, similar to the Docker hub.  If you just need something simple to store your images, and you don't care if they're private or not, it's probably easier to just use the Docker hub.  However, you only get one "private" image on the Docker registry, unless you start paying more for their upgraded version.

One benefit of using ACR is you get your images very close to the data center where you're going to deploy them.  If you're using ACI/ACS/AKS to deploy your containers/clusters, you can have a ge0-replicated container registry that's in the same datacenter as your VM's.  This would result in faster image download times when you spin up new containers or add new nodes to your cluster.

Azure Container Services

Azure Container Service (ACS) is an offering that will deploy a container orchestrator system for you on Azure resources.  You can spin up a Kubernetes, DC/OS, or Docker Swarm cluster with this option.

This spins up a fully functional and deployed cluster.  For example, if we look at the Kubernetes option, it will deploy the master nodes, as well as the worker nodes.  The worker nodes can be split into multiple "pools", and can run either Windows or Linux OS's.  However, Kubernetes itself still doesn't officially support Windows containers, so you may run into issues with that deployment.  You should also note that YOU are responsible for maintaining both the master and worker nodes in this environment.

This is a great option to quickly get a fully-functional Kubernetes cluster up and running in Azure.  It's also your only option if you want to get a DC/OS or Docker Swarm cluster running in Azure.

However - you should NOT use this moving forward.  In December 2018, Microsoft announced that they would be ending support for ACS on January 31, 2020.

acs-engine:

acs-engine is an open source tool that is used to generate Azure ARM templates to deploy clusters on Azure VM's, using either Kubernetes, DC/OS, OpenShift, Docker Swarm, or Swarm orchestrators.  It gives you a lot of variety and options in how you want to configure your cluster and resources.  In fact, the ACS offering actually utilizes some of the acs-engine tool behind the scenes.

Getting started is pretty easy - first we generate a resource group and create a service principal with access to that resource group:

	
# Fill in your variables:
$tenantId = 'Your-Azure-Tenant-ID'
$subscriptionName = 'Your-Subscription-Name'
$resourceGroupName = 'Name-of-resource-group-to-create'
$resourceGroupLocation = 'Azure-location'
$password = 'Your-password'
$applicationName = 'kubernetes-acs-engine-ABC'
$homePage = "http://{0}/{1}" -f $tenantId,$applicationName
$identifierUri = $homePage

# Login
Login-AzureRmAccount

# Select your desired subscription
Select-AzureRmSubscription -SubscriptionName $subscriptionName

# Create the resource group
New-AzureRmResourceGroup -Name $resourceGroupName -Location $resourceGroupLocation

# create secure password
$securePassword = $password | ConvertTo-SecureString -AsPlainText -Force

# Create the Azure AAD application
$aadApplication = New-AzureRmADApplication -DisplayName $applicationName -HomePage $homePage -IdentifierUris identifierUri -Password $securePassword -Verbose

# Create the SPN and sleep for a while to ensure SPN propagates through AAD
$servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $aadApplication.ApplicationId
Start-Sleep 30

# Assign contributor access for the AAD application to your resource group
New-AzureRmRoleAssignment -ResourceGroupName $resourceGroupName -ObjectId $servicePrincipal.Id -RoleDefinitionName Contributor

# Output these values and note them down - you will need them later
Write-Host "clientId: $aadApplication.ApplicationId"
Write-Host "secret: $password"
	

Once we have this, we can create our cluster definition JSON file.  We can start with something simple - this will deploy a Kubernetes Linux cluster on version 1.12 with 1 master node and 3 worker nodes, all using the "Standard_D2_V2" VM image (make sure you fill in your dnsPrefix, ssh public key, clientId, and secret):

	
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.12"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "your-custom-dns-prefix",
      "vmSize": "Standard_D2_v2"
    },
    "agentPoolProfiles": [
      {
        "name": "linuxagentpool1",
        "count": 3,
        "vmSize": "Standard_D2_v2",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "your-ssh-public-key-data"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "clientId from powershell output",
      "secret": "secret from powershell output"
    }
  }
}
	

At this point it's just a few simple commands to generate the ARM template from the JSON file and deploy it to your resource group:

	
    $dnsPrefix = 'your-custom-dns-prefix'
    $resourceGroup = 'your-resource-group-name'
    acs-engine.exe generate path-to-cluster-definition.json
    
    $templateFile = '_output\{0}\azuredeploy.json' -f $dnsPrefix
    $parametersFile = '_output\{0}\azuredeploy.parameters.json' -f $dnsPrefix
    
    New-AzureRmResourceGroupDeployment -Name  -ResourceGroupName $resourceGroup -TemplateFile $templateFile -TemplateParameterFile $parametersFile
	

Wait a while (usually takes 20-30 minutes to fully deploy), and you'll have a complete Kubernetes cluster up and running!

The only real differences between ACS and acs-engine are:

  • acs-engine gives you much more flexibility
  • ACS actually creates an "ACS" resource in the Azure portal, but all you can do with it is scale your node count up or down.

NOTE: While ACS is being de-supported in 2020, a cluster created by acs-engine can't really be de-supported since it's just creating some Azure resources for you.  However, Microsoft has effectively killed the acs-engine project at this point.  All future development, features, and bug fixes will instead be done in the aks-engine project, which we will talk about later.

Azure Kubernetes Services:

Azure Kubernetes Service (AKS) is the next evolution from ACS.  There are two major differences.  First of all, unlike ACS that supported Kubernetes, DC/OS, and Swarm, AKS ONLY supports Kubernetes as an orchestrator.  Additionally, AKS abstracts away the Kubernetes control plane ("master nodes") from you.  All setup, configuration, security, maintenance, etc. of the master nodes is handled from you - you don't even have access to SSH into them yourself.  You are, however, still responsible for maintaining your worker nodes.

Also, at the time of this writing, ONLY Linux worker nodes are supported.  You are not able to deploy Windows worker nodes into an AKS cluster.  Hopefully Microsoft will add this support once Kubernetes officially supports Windows nodes, which is currently planned for Kubernetes version 1.14 - see GitHub issue here.

As previously noted, Microsoft is ending support for ACS in January 2020, so you should plan to move to this (or use the acs-engine/aks-engine options) before then.

aks-engine

aks-engine is the evolution of acs-engine.  From my evaluation, it looks like they just cloned the acs-engine repo and made some very minor changes.  The most important change is they've removed support for all the other orchestrators besides Kubernetes.

If you want to get started using it, you can literally use the exact same sample I provided up above in the acs-engine section, just replacing the call to "acs-engine.exe" with "aks-engine.exe".  This is the open source tool Microsoft will be actively developing in, so if you're currently using acs-engine to generate your templates, you should move to this ASAP.

I'm not exactly sure how the AKS managed service uses this tool.  It would be neat if you could use aks-engine to deploy a managed system with the control plane managed for you, but I'm not sure of a way to do that with aks-engine.  Perhaps Microsoft will add that in the future.

Virtual Kubelet

The virtual kubelet is a pretty cool open source project.  It essentially is a virtual implementation of the kubelet that is the main binary in any kubernetes node.  In theory, you can use this to mesh in VM's from many different providers and have them "act" like an available node in kubernetes.   The example Microsoft provides is using them for "burst" workloads in conjunction with an AKS cluster.

When you install the virtual kubelet into your cluster, it shows up as a node in 'kubectl get nodes', and you can schedule pods onto it by providing a specific nodeSelector and tolerations in your YAML files.  This uses Azure Container Instances (ACI) behind the scenes to spin up your containers.  More specific details on how to hook it up with AKS can be found here.

One thing to note, directly from the GitHub page though:
"Please note this software is experimental and should not be used for anything resembling a production workload."

So while it might be neat to play around with and see how you could use it, you should probably hold off on using it in production.

Summary/Recommendations:

  • If you just want to run a few containers and don't need a full scale orchestrator - use Azure Container Instances (ACI)
  • If you want to run strictly Linux containers and don't need to stray too far from the defaults provided - use Azure Kubernetes Service (AKS)
  • If you need to run Windows containers or need a higher level of configuration over what your cluster configuration looks like - use aks-engine

Hopefully this helps you understand some of the Azure offerings a bit better - feel free to reach out if you have any questions!

Justin Carlson

Read more posts by this author.

Wisconsin