This technical blog explains how NetApp® Astra™ Control Service (ACS) can be used to protect applications on private Azure Kubernetes Service (AKS) clusters. Private AKS clusters restrict Kubernetes network traffic to an internal network. Familiarity with AKS clusters and general Azure principles is recommended.
Co-authors: Sayan Saha and Balasubramanian Ramesh Babu, NetApp
Security is a growing concern for customers. For organizations that would like to steer clear of public IPs, Azure offers a convenient way of spinning up AKS clusters that are private. A standard AKS cluster provides a public IP address for the API server. A private AKS cluster, on the other hand, will use a private IP address for its API server endpoint. This protects the API server from being exposed to the outside world. In both cases, nodes present in the cluster will use a private network for assigning IPs to the worker nodes.
This blog covers the following:
Reference: https://docs.microsoft.com/azure/aks/private-clusters
Deploying a private AKS cluster requires using the Azure CLI. There are several options available to customize a private AKS cluster. For the sake of simplicity, this blog creates a private AKS cluster with basic networking, using the Kubenet plugin. If desired, clusters can be created using the Azure network plugin. This will allow clusters to leverage an existing Azure Vnet and subnets.
# Create a resource group and a private AKS cluster
az group create -l eastus -n rg-brameshb
az aks create -n private-cluster-eastus -g rg-brameshb --load-balancer-sku standard --enable-private-clusterOnce the cluster is deployed and is in the “Succeeded” status, additional details about the cluster can be obtained.
As can be seen from the output, the AKS cluster is a resource present in `rg-brameshb` RG. Nodes present in the AKS cluster, however, are placed in a separate RG. `nodeResourceGroup` indicates the name of this RG. It can be obtained from the JSON output of the AKS cluster.
`MC_rg-brameshb_private-cluster-eastus_eastus` contains the Azure resources that make up `private-cluster-eastus`.
Since the API server is not publicly routable, additional steps must be taken. Microsoft recommends choosing an approach from the following:
This blog uses (1). A bastion host is created in the `MC_rg-brameshb_private-cluster-eastus_eastus` RG.
To manage private AKS clusters with Astra Control Service, the Astra Connector Operator is used. The Astra Connector Operator needs to be deployed on the AKS cluster to enable communications with NetApp Astra Control Service. This step requires using the manifests provided here. Proceed to login to the bastion host and configure access to the AKS cluster.
#Access bastionhost
ssh -i brameshb-key.pem brameshb@20.232.168.67
#Login to Azure and access private AKS cluster
brameshb@bastionhost:~$ az login
brameshb@bastionhost:~$ az account set --subscription <subscription-id>
brameshb@bastionhost:~$ az aks get-credentials --resource-group rg-brameshb --name private-cluster-eastus
#Download kubectl
brameshb@bastionhost:~$ curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
brameshb@bastionhost:~$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
brameshb@bastionhost:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-25392440-vmss000000 Ready agent 22h v1.21.9
aks-nodepool1-25392440-vmss000001 Ready agent 22h v1.21.9
aks-nodepool1-25392440-vmss000002 Ready agent 22h v1.21.9Deploy the Astra Connector operator.
brameshb@bastionhost:~$ kubectl create ns astra-connector-operator
brameshb@bastionhost:~$ git clone https://github.com/NetApp/astra-connector-operator.git
brameshb@bastionhost:~$ cd astra-connector-operator
#Apply operator manifest
brameshb@bastionhost:~$ kubectl apply -f astraconnector_operator.yaml -n astra-connector-operator
#Check if astra-connector-operator is up and running
brameshb@bastionhost:~$ kubectl get pods -n astra-connector-operator
NAME READY STATUS RESTARTS AGE
operator-controller-manager-bbf65d7cd-cpcfb 2/2 Running 0 4h18mThe next step is to create an AstraConnector object. It provides connectivity information required by Astra to discover the AKS cluster it is created on. To do this, an API token must be generated first:
Once the API token is generated, it is plugged into the AstraConnector spec. The name of the private cluster [clusterName] and Astra account ID [accountID] are also required. A private image registry [imageRegistry] can be used if needed. A sample config is available in the config/samples directory. For a complete description of the parameters in use, refer to the table provided here.
#Create AstraConnector to discover private AKS cluster
$ cat config/samples/astraconnector_v1.yaml
apiVersion: netapp.astraconnector.com/v1
kind: AstraConnector
metadata:
name: astra-connector
spec:
natssync-client:
image: natssync-client:0.9.202202170408
cloud-bridge-url: https://astra.netapp.io
nats:
image: nats:2.6.1-alpine3.14
httpproxy-client:
image: httpproxylet:0.9.202202170408
echo-client:
image: echo-proxylet:0.9.202202170408
imageRegistry:
name: theotw
astra:
token: <token>
clusterName: <aks-cluster-name>
accountId: <astra-account-id>
acceptEULA: yesCreate the AstraConnector. `kubectl get astraconnector` informs if the private cluster is registered with Astra. If running into issues, examine logs from the `operator-controller-manager` pod in the astra-connector-operator namespace.
# Create a namespace and apply AstraConnector config
$ kubectl create ns astra-connector
$ kubectl apply -f config/samples/astraconnector_v1.yaml -n astra-connector
# Observe its status
$ kubectl get astraconnector -n astra-connector
NAME REGISTERED ASTRACONNECTORID
astra-connector true a740d092-d272-4b88-a605-a396d11373a4
# Examine logs from operator-controller-manager
$ kubectl logs operator-controller-manager-bbf65d7cd-cpcfb -n astra-connector-operator -c manager
.
.
.
1.6496804658713624e+09 INFO controller.astraconnector Registering cluster with Astra {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}
1.649680465871367e+09 INFO controller.astraconnector CloudBridgeURL for Astra Host {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "CloudBridgeURL": "https://preview.astra.netapp.io"}
1.6496804658713708e+09 INFO controller.astraconnector Checking for a valid SA credential for cloud {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "cloudType": "Azure"}
1.6496804659914987e+09 INFO controller.astraconnector Found a valid SA credential for cloud {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "cloudType": "Azure", "credName": "brameshb-azure-sp"}
1.649680465991535e+09 INFO controller.astraconnector Fetching cloud ID {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}
1.649680466105789e+09 INFO controller.astraconnector Found cloud ID {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "cloudID": "883191fe-0cd1-499d-b2da-ce9104d49b74"}
1.6496804661058552e+09 INFO controller.astraconnector Finding cluster ID {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}
1.649680466224342e+09 INFO controller.astraconnector Found cluster ID {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "clusterId": "dae554c0-0c2e-49f9-b26c-841f3d89936d"}
1.6496804663686578e+09 INFO controller.astraconnector successfully registered astraConnectorID {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector", "astraConnectorID": "v1:a740d092-d272-4b88-a605-a396d11373a4"}
1.649680466368695e+09 INFO controller.astraconnector Registered cluster with Astra {"reconciler group": "netapp.astraconnector.com", "reconciler kind": "AstraConnector", "name": "astra-connector", "namespace": "astra-connector"}To provide persistent storage for applications, Azure NetApp Files is recommended. An NetApp account is created in the resource group (rg-brameshb) that contains the private AKS cluster. In addition, virtual networks used by Azure NetApp Files and the AKS cluster must be peered.
With all required dependencies in place, the private cluster can be discovered by Astra. Log in to the Astra dashboard and add the cluster. This requires an Azure credential. Refer to the Astra docs to understand how an Azure Service Principal can be created.
Applications can be deployed with persistent storage provided by Azure NetApp Files, Azure Disk, and/or Azure File. Astra Control protects application workloads. Protection policies can be created that specify backup schedules. To learn more about how this works, take a look at Disaster Recovery of AKS workloads with Astra Control Service and Azure NetApp Files.
This article explained the management of private AKS clusters with Astra Control, using the Astra Connector operator.