There are a few different Azure storage types, and each can benefit specific workloads. While the block storage options available in Azure can be used for storage capacity in more traditional constructs, such as hard disks for virtual machines, object storage via Azure Blob storage enables more economical data storage for specific data sets. However, Azure doesn’t offer the more advanced and cost saving capability for tiering between block and object storage—now NetApp has a solution for that which will bring your Azure cost management to the next level.
In this blog, we’ll explore the native data tiering features in Azure and how NetApp’s Cloud Volumes ONTAP can help manage the overflow of data from block storage to object storage with its own data tiering capabilities.
Before we talk about tiering data between block storage and object storage, let’s take a look at what the two offerings mean.
Block storage keeps data in large groups, called blocks, that it stores in different places. The block storage construct in the cloud is similar to the version available on-premises, where a fixed size storage disk is made available to an operating system. This disk could be mounted from an independent storage system such as NAS, SAN, iSCSI SAN, or from the host server itself.
In Azure, page blobs in Azure Storage are used to store the VHDs used by virtual machines. They can be used to store files of up to 8 TB in size. Azure Premium storage offers the best performance in this category and is ideal for I/O intensive workloads with a maximum of 7500 IOPS per disk. Block level storage is useful for applications that need service-side processing as well as for databases that need good disk read/write performance.
Object storage considers all data equally, with each object independent of the others, relying on metadata to organize the information. Object storage targets modern application architectures where stored data can be directly accessed by applications through API calls or simple http/https requests. It offers the flexibility required by decoupled architectures where location of data storage should not be a concern.
What kind of data is object storage ideal for? Large video and audio files, input data for big data analytics applications, backup and archival data, log files are just some of the possible use cases for cloud-based object storage.
As stated above, there is no native Azure feature to tier data between block and object storage. However, Azure Blob storage is available in three storage tiers—hot access tier, cool access tier and archive access tier—each of which can be integrated to your applications based on the use case.
Hot access tier can be used to store data that is frequently accessed. Production applications could use this tier to store live data for immediate processing. The storage cost is slightly higher, but access costs are lower. In the East US region for example, storing 5 TB of data in hot access tier would cost around $1248 annually, with data retrieval free of charge.
Cool access tier is intended to store data that is not accessed frequently and will be stored for a minimum of 30 days. It is ideal for short-term backups, data collection for future processing, log file storage for analytics, etc. Storage charges are lower when compared to hot tier but with higher data access rates. If the same 5 TB of data in the above example is stored in the cool access tier, the annual storage cost would come down to $912, but there will be an additional charge involved for data retrieval at rate of $0.010 per GB.
Archive access tier, as the name indicates, is Azure cold storage tier for archive data. It is used for storage of archival data that is rarely accessed and can be stored in offline mode for a minimum storage period of 180 days. Data retrieval from the archive access tier could take hours as it must go through a rehydration process. This Azure archive storage tier is ideal for storage of long-term data retention for DR purposes, rarely-accessed archival data that should be made available for compliance requirements, and other similar cases. For example, storing 5 TB of data will cost only $120 annually, but the data retrieval will cost $0.020 per GB, which is on the high side. It means that retrieval of 5 TB data from Archive tier could cost you an additional $100.
Although all three tiers of storage can reside in the same storage account, it is up to the customer to evaluate and choose the right tier for specific data. Note, the way Azure tiers is not automatic: changing the storage tier is a manual process and should be done from the Azure portal. There is no native feature to automatically tier between the three different tiers based on the access pattern. Attempting to move data from block storage to object storage to leverage the Azure Blob’s tiering functionality is a manual process as well, and would require the use of custom-built tools and complicated configuration.
There are many enterprise use cases that would benefit from an auto-tiering feature to automatically move data between tiers once it becomes cold, especially the data that resides on costlier block storage devices. Cloud Volumes ONTAP for Azure storage brings this functionality with automated tiered storage.
Cloud Volumes ONTAP data tiering allows infrequently-accessed data to automatically move from the performance tier on Premium SSD or Standard HDD managed disks to the capacity tier on Azure Blob storage. Cloud Volumes ONTAP uses the hot access tier for storing tiered data by default, but users can decide to automatically tier the data further to the cool access tier once it remains un-accessed by a predefined date.
This feature provides a significant cost advantage, as Azure Blob storage rates are much lower than HDD/SSD disk price rates. Rarely-accessed data sets, like those in backup archives and DR use cases, can be automatically moved to even cheaper storage on cool access tier, allowing companies to store massive amounts of data for as low as $0.01 per GB. For example: the cost for storing 4 TB of data in cool access tier is $5000 less than storing the data in block format on Premium SSD disks.
The data tiering capability is built into Cloud Volumes ONTAP, so there’s no need for any additional complex configurations or tools. Customers can choose from any of the four tiering policies: Snapshot only, Auto-tiering, DR, or no tiering. Snapshot only tiering helps in reducing the performance tier capacity used by snapshots by moving snapshots to Azure Blob storage. The Auto-tiering policy moves data that is not accessed for a specific period of time to the capacity tier. The DR tiering policy uses the capacity tier by default for DR copies. Thus, cost optimization can be implemented for a majority of enterprise use cases with minimal or no impact to the performance as data can be tiered back to performance tier as soon as usage patterns indicate the data is in use.
Leveraging Azure object storage can be an easy way for companies to cut storage costs for infrequently-used data. However, Azure only allows data tiering within its object storage offering, not between Azure block storage and object storage.
Managing overflow of data from block storage to object storage using Cloud Volumes ONTAP helps users allocate the right storage tier while meeting the performance demands of target workloads. One NetApp customer, EngageYa, used Cloud Volumes ONTAP’s storage efficiencies to cut its storage costs and achieve a 70% reduction in Premium storage disk usage.