Storage
Mobility
Protection
Analysis and control
TOPICShare
To successfully meet the challenge of storing data in the cloud using AWS, IT professionals need to familiarize themselves with the management of Elastic Block Store (EBS) volumes and snapshots.
This article will briefly outline the important facts of EBS storage in AWS, and walk through two methods for replicating EBS volumes across AWS regions.
What Are EBS Volumes?
Amazon Elastic Block Store (EBS) is a raw block-level storage service designed to be used with Amazon EC2 instances. When mounted to EC2 instances, EBS volumes can be used like any other raw block device: they can be formatted with a specific file system, host operating systems and applications, and have snapshots or clones made from them.
Every EBS volume that is provisioned will be automatically replicated to other storage devices in the same Availability Zone inside the AWS region to offer redundancy and high availability (guaranteed 99.999% by Amazon).
AWS also offers seamless encryption of data at rest (both boot and data volumes) using Amazon-managed keys or keys customers create through Amazon Key Management Service (KMS).
Two Types of EBS volumes
There are two Amazon EBS volume type categories: SSD-backed volumes and HDD-backed volumes.
SSD-backed volumes are optimized for transactional workloads, where the volume performs a lot of small read/write operations. The performance of such volumes is measured in IOPS (input/output operations per second).
HDD-backed volumes are designed for large sequential workloads where throughput is much more important (and the performance is measured with MiB/s). Each category has two subsets.
| Solid State Drives (SSD) | Hard Disk Drives (HDD) |
|
General Purpose SSD Balanced for economy and performance |
Throughput Optimized HDD: Inexpensive, for high use, intensive workloads |
|
Provisioned IOPS SSD High performance, for important applications |
Cold HDD Cheap, used for infrequent access |
General Purpose SSD (gp2) is balanced for price and performance, and is recommended for most use cases.
General purpose SSD is a good choice for boot volumes, applications in development and testing environments, and even for low latency production apps that don’t require high IOPS. Baseline performance is directly correlated with volume size, since customers receive three IOPS per GB of volume size.
Volumes also have I/O “credits” which represent the available bandwidth that the volume can use to burst up to a higher IOPS value over a certain period of time.
Credits are accumulated over time (the larger the initial volume, the more credits it will earn over time) up to a maximum of 3,000 IOPS, that can be saved for whenever the EBS volumes may need it. When a credit is depleted, the EBS volume goes back to its initial baseline performance rate.
Sizes of gp2 volumes can vary from 1 GiB to 16 TiB, while maximum throughput is capped at 160 MiB/s.
Provisioned IOPS SSD (io1) are the SSD volume type to use for critical production applications and databases that need the high performance (up to 20,000 IOPS) EBS storage.
Instead of credits, io1volumes can provision a desired IOPS value, with a maximum ratio of 50:1. That means a performance rate of 5,000 IOPS can be set when provisioning a 100 GiB volume. All volumes bigger than 400 GiB can provision a maximum of 20,000 IOPS. Io1 volume sizes vary from 4 GiB to 16 TiB, while throughput is maxed at 320 MiB/s.
AWS guarantees customers that their Provisioned IOPS SSD volumes will operate at maximum 10% of loss of performance at 99.9% SLA on a yearly basis.
The recommended practice is to maintain at least a 2:1 ratio between IOPS and volume size.
Throughput Optimized HDD (st1) is designed for applications that require larger storage and bigger throughput, such as big data or data warehousing, where IOPS is not that relevant.
St1 volumes, much like SSD gp2, use a burst model, where the initial baseline throughput is tied to the volume size, and credits are accumulated over time.
Bigger volumes have higher baseline throughput and will acquire credits faster over time.
Maximum throughput is capped at 500 MiB/s, sizes vary from starting 500 GiB to 16 TiB, and volumes can hold a maximum of 1 TiB in credits (which is more than 30 minutes of workload under highest throughput).
Cold HDD (sc1) is a magnetic storage format suitable for scenarios where storing data at low cost is the main criteria.
Sizes vary from 500 GiB to 16 TiB, throughput can reach 250 MiB/s, and a similar burst model is used as with st1 volumes, but the credits fill at a slower value (12 MiB/s opposing to 40 MiB/s with st1 volumes).
Both st1 and sc1 storage types cannot be used as a boot volume, so it is recommended to use gp2 volumes for that purpose.
Copying EBS Volumes Between Regions
Since February 2017, a new EBS feature called Elastic Volumes allows AWS customers to easily switch between EBS volume types on the fly and dynamically grow volume sizes.
In order to manage EBS volumes, customers can utilize AWS Management console in their web browser, or type in commands using AWS Command Line Interface (CLI).
The following two sections will demonstrate both techniques for transferring EBS volumes between different AWS regions.
Using AWS Management Console to Copy EBS Between Regions
Before creating the EBS volume, login to an AWS account and select the desired region in the top right corner of the browser.
Once the region is selected, click on “Services” at the top left corner of the screen, find the “Compute” area, and select “EC2.”
In the left navigation pane, under “EC2 Dashboard,” locate the “Elastic Block Store” subsection and select “Volumes.” Once the new window displays, click the blue “Create Volume” button.
In the pop-up window, customize the volume (type, size, IOPS or throughput), select the desired Availability Zone, and decide whether or not to use encryption.
Next, proceed to create the volume.
Once the volume is ready, attach it to the EC2 instance by right-clicking it and selecting the “Attach” option. Please note that the volume will need to be formatted inside the operating system being used.
To move the EBS volume to another region, first create a snapshot. Right-click on the volume and select “Create Snapshot.” The following screen will appear:
Enter a name for the snapshot and, optionally, add a description. If the volume wasn’t originally encrypted, the snapshot won’t be either.
By the same token, encrypted volumes will yield encrypted snapshots.
Once the snapshot is created, move it to the other region. In the Elastic Block Storage subsection in the “EC2 Dashboard” navigation menu on the left side of the screen, select “Snapshots.” Find the desired snapshot, and select “Copy.”
The following pop-up window will appear:
In the pop-up window, select “Destination Region,” as well as the option to encrypt the new snapshot.
In order to utilize the new snapshot in the destination region, change the region in the top right corner of AWS Management Console.
Using AWS CLI to Copy EBS Between Regions
Before using AWS CLI, it will be necessary to configure the environment.
To create an EBS volume, use the “aws ec2 create-volume” command. Important arguments are --size, --region, --availability-zone, --volume-type, --iops (when creating SSD volumes) and --encrypted (if securing the data is important).
When the volume is created, attach it with the aws ec2 attach-volume command, and supply --volume-id and --instance-id arguments).
To move the volume to another region, first create a snapshot with “aws ec2 create-snapshot” with one mandatory argument of --volume-id. To move the snapshot, use aws ec2 copy-snapshot command. This command requires the following parameters:
- --source-region
- --source-snapshot-id
- --destination-region (optional, but necessary for cross-region copying)
- --encrypted
All the above-mentioned commands with more examples can be found in the official documentation from Amazon.
Impacts of Cross-Region EBS Replication
Copying EBS snapshots from one region to another is a handy feature, both as an easy way to geographically migrate data and as a simple solution for data recovery.
However, there are two things to consider when deciding to create such copies:
- Pricing impact: When storing volumes and snapshots in both the source and destination region, users are charged for both copies, based on the size in GiBs. It doesn’t matter that both store the same data.
There are no data deduplication mechanisms, and there is also an added price for data transfers, per every GiB moved between regions.
- Seamless synchronization impact: Data replicated across regions isn’t synced at all times. Snapshots will capture the data currently on the volume, and when moved to new region, there may be new data on the original source volume.
Even scripting the entire process utilizing CLI or APIs cannot guarantee that the EBS volumes in both regions will always be in sync.
NetApp’s Cloud Volumes ONTAP (formerly ONTAP Cloud) Optimizes Cross-region EBS Migration
NetApp’s Cloud Volumes ONTAP enables companies to leverage enterprise storage capabilities in cloud environments. By deploying Cloud Volumes ONTAP instances in desired regions and utilizing NetApp’s SnapMirror technology, customers can take snapshots of their data without requiring additional storage or impacting an application's performance, and then replicate those snapshots to another region while keeping them synchronized automatically per a user-defined schedule.
If needed, Cloud Volumes ONTAP can also leverage physical on-premises ONTAP systems as a source or destination, allowing users to backup or restore data to and from the AWS cloud.
Other benefits for using Cloud Volumes ONTAP in AWS is because of NetApp’s storage efficiencies like thin provisioning, data deduplication and compression. These features allow ONTAP consume much less underlying EBS capacity that you would have to natively in AWS for facilitating your data requirements.
These efficiencies also persist for the SnapMirror relationships, meaning that the data is replicated in a space efficient manner, which saves you on data egress charges.
Another key thing to note about SnapMirror, is that it only needs to baseline once. After the baseline only the space efficient changes need to be replicated.
The end result is that Cloud Volumes ONTAP can reduce your costs of storing and replicating data in the Amazon cloud, as well as provide you enhanced data management capabilities.
Related Articles:
- Migrating MySQL Database: EC2 Hosted vs. Amazon RDS
- The S3 Outage: Fundamental Building Blocks to Prepare for Unavoidable Cloud Failure
- 5 Functions of EBS Volumes You're Not Using
