Storage
Mobility
Protection
Analysis and control
TOPICShare
An AWS instance store is a temporary storage type located on disks that are physically attached to a host machine. Instance stores are made up of single or multiple instance store volumes exposed as block devices. Block storage on AWS is available with AWS EBS.
Longtime users of AWS are well-aware of AWS instance store, and are probably using this type of storage quite frequently. However, new AWS users may not have the same familiarity with this storage type.
One challenge with instance store-backed storage is that the data is not persistently stored. Once an instance is terminated, all of its data is lost.
In the world of the cloud, where you want persistent data, instance store backed instances are a challenge for certain applications, such as databases. If you want data to be persistent then it must be backed up.
This article will introduce you to the AWS instance store storage type, compare it to AWS Elastic Block Storage (AWS EBS), and show you how to backup data stored on instance stores to AWS EBS.
AWS Instance Store at a Glance
Depending on the instance type, the size of an instance store and the number of devices available will vary by instance type. While an instance store is dedicated to a particular instance, the disk subsystem is shared among instances on a host computer.
The diagram below shows how an instance store volume is run on a host machine:
Instance store volumes are ideal for temporary backup, and for storing an application’s cache, logs, or other random data. Instance store volumes are also useful for applications that focus on processing data rather than storing that data.
They can reformat original instance storage and use part of that for SWAP. Instance stores are also good for RAID 10 with 6 disks (4 EBS and 2 Ephemeral disks) to improve overall instance performance and provide high availability.
For temporary storage of information that changes frequently, such as data that is replicated across a fleet of instances, or run a load-balanced pool of web servers, instance stores are also a good choice.
Instance Store Volume Types
There are multiple types of instance store volumes and each of them are known for their own unique high performance capability. There are three types of instance store volumes: ephemeral, non-volatile memory (NVMe) SSD, and TRIM.
- Ephemeral: Instance store volumes also known as virtual devices have underlying hardware physically attached to the host computer for the instance. Instance store volumes support ephemeral [0-23]. Whenever you launch an EC2 instance, the instance store volume type is ephemeral by default. Learn more about ephemeral volumes here.
- Non-Volatile Memory (NVMe): NVMe SSD instance store volumes are known for fast IO operation. To access the NVMe volumes, you must use an operating system that supports NVMe. Only I3 instances offer NVMe SSD instance store volumes. You can learn more about NVMe volumes here.
- TRIM: Instance store volumes that support TRIM are known to reduce write amplification and increase performance. TRIM volumes are not formatted with a file system when they launch, so you have to format them before they can be mounted and used. Get to know more about TRIM here.
Instance Store-Backed vs AWS EBS-Backed
Being a block-level storage, instance store-backed volumes are a good option when you need storage with very low latency. Instance store devices are temporary in state: in other words that means the moment you stop or terminate an instance, every block of storage in the instance store is reset.
Because of that, your data cannot be accessed through the instance store of another instance.
Instance store-backed volumes are not fault tolerant, so if you’re looking for a more durable and persistent storage solution, AWS EBS-backed volumes are a better choice. EBS-backed volumes preserve your data through instance stops and terminations and can be easily backed up with EBS snapshots.
They also support full-volume encryption, which is not an option for instance store-backed volumes. This makes EBS an ideal option to store valuable and long-term data. Of course, there are other alternatives, such as AWS S3 and AWS EFS. It's worth getting to know the differences between these options so you can make the optimal storage choice for you.
AWS recommends that if you aren't certain which storage type best suits for your needs, you should start with Amazon EBS instances. Although the service comes at a modest cost, there is less risk of unintended data loss.
Limitations of Instance Store Volumes
Instance store volumes are complicated and it is important for you to get to know their limitations before you decide to use it as a storage option. Below are some of the limitations of instance store volumes:
- In order to deliver very high random I/O performance, instance store volumes only run on solid state drives (SSD).
- They only support C3, G2, HI1, I2, I3, M3, R3, and X1 instance families.
- Data in the instance store can be lost if the underlying disk drive fails or if the instance stops, or if the instance terminates.
- Instance store devices have only root volume, which can’t be extended. You will have storage space of 350GB on instance store volumes. Depending on OS you can manage 350GB space by partitioning root volume if OS supports.
- It's not possible to resize instance-backed root volumes on the fly. You'll need to create an AMI and then launch a new instance from that AMI and giving it a bigger instance store.
- Taking a snapshot or AMI of an instance store volume is not as straightforward as taking snapshot of EBS volume.
- Instance store volumes attached to certain instances will suffer a first-write penalty which might affect I/O of the storage when a new volume is attached. The only way this won’t happen is if they are already initialized before they are in-use.
- Some special instance types use NVMe or SATA based solid state drives (SSD) to deliver very high random I/O performance.
- A few instance types such as c1.medium and m1.small include a 900 MB instance store swap volume, which may not be automatically enabled at boot time.
- Instance store volumes are included as part of the instance's hourly cost.
Another major drawback for using instance store is that it does not provide persistent storage. Even though instance store is good option for storing data which is required on temporary basis, there may be a case where you want to store data on a long-term basis.
In such cases, you would have to transfer the data from your instance store volume to a permanent storage solution like AWS EBS.
Backing Up Ephemeral Storage to AWS EBS
This section will provide a step-by-step guide to backing up instance store-backed ephemeral storage to an EBS volume on AWS.
For engineers who want to know the ideal way to launch instance store instances, opt for a community AMI when launching an instance store instance, as shown below (Note: root device type should be “instance-store”):
Step 1: Make sure that your instance store instance is in the “running” state.
Step 2: Create an EBS volume. Make sure to choose a size large enough to hold the data you’re migrating.
Step 3: Attach the EBS volume to your instance.
Step 4: Make the volume available to your operating system, and create a compatible file system on the volume. To do so:
List all volumes. Here the EBS volume we attached is denoted as xvdf.
View file system on EBS volume
sudo file -s /dev/xvdf
Ideally it should not have any file system configured, as shown in image above. If you get a response as a Linux file system then you can skip next step.
Setup file system on EBS volume:
sudo mkfs -t ext4 /dev/xvdf
The above command will help format the EBS volume to a ext4 file system ideal for Linux instances. For Windows, refer to this link.
Create mount point/directory for EBS volume.
You can choose any location for mount directory, here its /opt.
Next, mount EBS volume with help of fstab. Keep in mind that before you make changes it recommended to back up your existing fstab file.
In fstab file, enter device name followed by mount point.
The last three fields on this line are the file system mount options, the dump frequency of the file system, and the order of file system checks done at boot time.
Once you’ve made the changes in the fstab file, now you can execute the following command to mount all the entries in the fstab file:
sudo mount -a
After successfully mounting the volume you should see following output for the df command:
Now transfer all your data to mount directory on EBS volume. You can use the rsync utility to transfer all the data on Linux instances:
sudo rsync -zvh /path-to-your-data.tar.gz /path-to-mount-directory
That’s it! This will transfer all your data. Once you're done with the transfer you can unmount and detach the EBS volume. To unmount, use the following command:
sudo umount /dev/xvdf
Things to keep in mind: It’s a best practice to keep both volumes available until you’re sure that the copy operation has succeeded.
If you’re using Windows OS, refer to the following document to achieve same on an Windows instance store volume.
If you would like to continually back up data from ephemeral storage to EBS you can use tools such as rsync, lsyncd for Linux, and robocopy for Windows. You can also make automated backups of an EBS volume by taking snapshots.
Best Practices for Data Backups
Instance store volumes are good for specific use cases but does not fit the use case of persistent storage.
Even though instance store volumes provide better performance, you can’t create snapshots of them the way you can take snapshots of EBS volumes. The only way to retain your data is by attaching and migrating the data to a new EBS volume.
If you are using Amazon EC2 instance store-backed instances, It is highly recommend that you distribute the instance store instances across multiple availability zones and back up critical data on your instance store volumes to persistent storage on a regular basis.
It’s a best practice to keep backups of all data stored in your AWS infrastructure, allowing you to roll back changes and helping to protect your data against failures of underlying infrastructure. To gain even more control over how you store your backup data on AWS, check out Cloud Volumes ONTAP for AWS.
Cloud Volumes ONTAP adds additional features to your AWS deployment, including automatic storage tiering that sends data seamlessly between Amazon S3 and Amazon EBS to optimize usage, data cloning capabilities that aren’t available natively, storage efficiency features like deduplication and compression that cut down cloud data storage costs.
