Storage
Mobility
Protection
Analysis and control
TOPICShare
As public cloud adoption increases year after year and enterprises continue to migrate production data to the cloud, it’s important for stakeholders to devise a backup strategy. Data stored in the public cloud needs to be able to be restored and Azure and AWS snapshots are key to that process.
In this article, we look at the mechanisms behind the snapshot technologies at the two leading public cloud providers, Azure and AWS. In the other parts of this series, we saw how Azure snapshot and AWS snapshot costs compare, and took a deep dive into NetApp’s Cloud Volumes ONTAP snapshots.
Azure Snapshots: Azure Backups and More
Azure Storage has two types of virtual machine disk storage: unmanaged and managed. In unmanaged disks, the tenant maintains and manages the disks which are stored in page blobs which in turn are stored in Storage Accounts. With managed disks on the other hand, Microsoft manages the service, freeing the tenant from the responsibilities of managing the storage account and its imposed limits such as disk size, IOPS, and storage bandwidth limits. Below we’ll take a look at how snapshots work in Azure managed vs. unmanaged disks.
Unmanaged Disks
With unmanaged disks, we want to create a snapshot of the blob object. When a snapshot of the Azure blob storage is created, the snapshot has the same URI (Uniform Resource Identifier) as the base blob with a value attached to it formatted as DateTime that indicates when the snapshot was taken.
For example, if we look at a base page blob URI https://netappaz.blob.core.windows.net/netapppb/netappsnap.vhdx, the snapshot URI will be: https://netappaz.blob.core.windows.net/netapppb/netappsnap.vhdx?snapshot=2018-06-02T04:04:51.9827557Z.
When a snapshot is created, the blob’s system properties and metadata are copied to the snapshot, excluding the blob’s lease values, as there cannot be any leases on the snapshot. You also have the option to modify the metadata of the snapshot during creation in order to uniquely identify the snapshot. To ensure an application-consistent snapshot, it is recommended to detach the disk from the VM or shut down the VM while taking the snapshot. Alternatively, for Azure backup and recovery, you can use tools like Azure Backup that will coordinate with the VSS on the VMs to ensure that the snapshots taken are application-aware while the VM is running. Although the VSS can only be used for Windows VMs, in Azure Backup a file-consistent backup is offered for Linux VMs and you can also create your own script to perform flushing or similar operations so that the snapshot is application-consistent.
As a backup strategy, snapshots are taken periodically and copied to another storage account using CopyBlob or AzCopy tools. This copied snapshot is stored in the form of page blobs. The incremental changes are copied by comparing the snapshot of the base blob and the backup blob using GetPageRanges and PutPage.
During the restore process, the point-in-time snapshot to be restored is promoted as the backup base page blob. After creating a snapshot of these newly promoted page blob, the page blob is restored as a disk in the source storage account. This restored disk is attached to the Azure VM and the old disk is detached and decommissioned.
Azure Managed Disks
With managed disks, disks are treated as first class objects, which means as far as the user is concerned, there is no storage account to maintain. It is very easy to create snapshots of managed disk VHDs by simply clicking on the “Create Snapshot” option in the Managed Disk blade. Alternatively, you can use PowerShell cmdlets like New-AzureRmSnapshotConfig and New-AzureRmSnapshot to create the snapshots. At the time of writing this article, only full snapshots are allowed, and these snapshots will be stored as VHDs based on the choice of resource group and storage type (e.g. LRS).
Creation of disks from these snapshots is also straightforward. You can use the Azure portal to create managed disks and point to the snapshot taken to create a new managed disk. Alternatively, here is an example of how you can achieve this through PowerShell.
AWS Snapshots
One of the most common use cases for AWS snapshots is backing up Amazon EBS volumes to Amazon S3 by taking point-in-time snapshots. AWS snapshots are incremental and consist of changes in blocks from the most recent snapshot. On deletion, only the data unique to the snapshots are removed. These snapshots are created asynchronously, and the data is loaded lazily to the replicated volume in the background; while snapshot creation is immediate, its status will be pending until completion of the transfer. This data or block transfer process means it can take hours to create the initial replica or to update if there have been numerous block changes since the last snapshot.
As applications and processes can cache data in RAM and flush to the disk on their own schedule, it is pertinent to either stop the Amazon EC2 instance or detach the Amazon EBS volume to take an application-consistent snapshot.
By default, Amazon EBS snapshots are stored in Amazon S3 accounts maintained by AWS. Once the lazy copy has finished and the status of the snapshot is marked completed, you can copy the snapshots to another region or within the same region. Data in transit and at rest during this transfer are encrypted by AES 256-bit encryption. These copies can also be incremental where the first copy is a full replica and subsequent copies are incremental in terms of changed blocks. The copy operation is useful in use cases such as migrating an application to a new region, disaster recovery, and data retention. Amazon CloudWatch Events can provide notifications on events like completion of a snapshot creation. This in turn can trigger an AWS Lambda function to copy it to another region for DR. It is also possible to set up automated EBS snapshots creation.
Amazon EBS volumes can be restored from a snapshot by the owner or by sharing the snapshots with another account. If the snapshot is a shared snapshot, then the account must create a copy of the snapshot so that they own the resultant snapshot in order to restore the volume.
Amazon EBS volumes can be restored easily using the Amazon EC2 console, AWS CLI, or AWS Tools for Windows PowerShell. A restored volume can be attached as soon it is created. The data from the Amazon EBS snapshots are loaded lazily but if data is accessed from the restored volume that has not been loaded yet, the requested data is immediately downloaded, and the lazy copy operation continues. Unlike in Azure, the snapshots cannot be downloaded or exported outside the AWS portal.
Final Note and Cloud Volumes ONTAP Snapshots
It is important to understand the lifecycle of a snapshot and how that affects your RTO and RPO objectives. Although periodic snapshots are possible, and it can be automated, the process becomes a bit complex and cumbersome. The costs of Azure snapshots and AWS snapshots are also a consideration, as we saw in the second part of our series on snapshots, and in the first part, which covered NetApp’s Cloud Volumes snapshots.
Cloud Volumes snapshots offer ways to overcome the various limitations of cloud-native snapshots, such as the full snapshots used by Azure managed disks and the full initial replicas of Amazon EBS snapshots. In addition to this, Cloud Volumes ONTAP also provides other features such as cost-cutting storage efficiencies, data tiering to object storage on Amazon S3 or Azure Blob, instant clones with minimal storage ideal for DevOps workflows, a centralized management console in Cloud Manager, and integration with third-party applications to provide application-aware snapshots.
