As the high-performance and high throughput block storage on AWS for Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Block Storage (AWS EBS) is a core part of how companies run their applications on AWS. However, the number of volumes in use, especially in enterprise IT deployments can quickly spiral out of control, leading to increased cloud storage costs.
Tracking down all these unused resources can be a time-consuming task, so an automatic solution for cleaning them up works best. Luckily, there is just a way to do that using AWS Lambda functions.
In this post we’ll give you a step-by-step walkthrough on how to clean up your unused Amazon EBS volumes using Lambda functions.
Before we get into how to remove an unused volume, let’s look at some of the reasons why you would want to do that in the first place. Not a lot of people realize it, but unused AWS EBS volumes do cost you money. These charges are based on the EBS volume size.
Usually, what you see on production systems is that each application or service running on an EC2 instance will have its own volume. This is configured in such a way to facilitate sandboxing, and to avoid system failure. Consider that there’s only one volume and two services are using that volume. If for some reason the volume fails, both the services using that volume will be affected, and that will mean trouble, for any business. To avoid this, each service will usually have its own separate volume, which limits the scale of failure, but increases the amount of Amazon EBS volumes that are in use.
So, while it’s good to have a volume for each application on an EC2 instance which could be used whenever the application runs, when that application or service isn’t running, the volume is still provisioned, and still racking up charges on AWS. Even though it’s good to have the volume ready for when it is needed, it could be increasing your monthly bill.
These “parked” EBS volumes still cost money. So, for AWS cost optimization, it becomes important to keep track of how many volumes are being used and how many are parked. But this could get very tedious, very quickly. There’s a way to automatically take care of this: using an AWS Lambda function to remove the unused volumes.
You can easily take care of unused AWS EBS volumes by writing a simple Lambda function that runs periodically and removes all the unused volumes in your deployment. For this demonstration, we created a Lambda function in Java.
1. Before starting with the code, make sure you include the AWS EC2 Java dependency in the pom.xml file.
Fig. 1: The dependencies in the Lambda function.
2. Once you have the dependency added and resolved, the AWS SDK for Java will be added to the project. This will allow you to create an EC2 client instance which will be used to get the list of unused volumes and also to remove these volumes.
3. Next, configure the credentials for the EC2 client instance. There are two options for how to do this. Ideally, during development, the credential will be read from the system’s environment, as environment variables. The code for this is as follows:
Fig. 2: Code snippet to get AWS access credentials from the environment.
There are two environment variables here. One of them is AWS_ACCESS_KEY and the other is for AWS_SECRET_KEY. This is good enough during development, but when you are deploying this to production as a Lambda function, you need to create an IAM role which will have the necessary permissions to run the Lambda function and not be using these credentials.
4. Next, create an instance of the AWSCredentialsProvider class to use as the credential provider:
Fig. 3: Code snippet to create an instance of the AWSCredentialsProvider class.
This is also required only in the development phase, not for the production setup.
5. Using this credential provider, create an instance of the Amazon EC2 class:
Fig 4: Code snippet to create an Amazon EC2 client.
Make sure you change the region in the code snippet above to match your setup.
6. Using this amazonEC2Client, get the list of all the volumes that are unused. To do this, first create a filter so that the query returns only the volumes that are unused. For this filter, use the status field, the value of which will be available for unused volumes.
7. Next, call the describeVolumes() method on the amazonEC2Client object to get the list of volumes:
Fig. 5: Code snippet to get the list of unused EBS volumes.
8. Next, loop through the result and just remove the volumes one by one:
Fig. 6: Code snippet to delete a volume from the list.
9. The code snippet above will also print out a list of IDs of all the volumes that were removed.
Fig. 7: Output of the Lambda function after deleting two unused EBS volumes.
As you see, the Lambda function was able to fetch the list of unused EBS volumes, and the list contained two volumes. For debugging purposes, the Lambda function will print the ID of each volume before deleting that volume. You can modify the code to log any error that the function could encounter while trying to delete the volume. That would further help in debugging in case of failure.
You now have a Lambda function which can delete unused volumes and prevent you from wasting money on unused EBS volumes. But how and when are you going to run this Lambda function?
It is impractical to think that you’ll go to the AWS console every morning and run it manually. Running this Lambda function at any frequency you want, automatically, is easy to do with the help of CloudWatch Events. Let’s see how to do this.
1. From the Configuration tab in the Lambda function page, click the “Add Trigger” button.
2. Select the “CloudWatch Events/EventBridge” option, as shown in the screenshot below.
Fig. 8: CloudWatch Event Trigger configuration example.
3. From here, you can create a new rule with a schedule of your choice. As you can see from the screenshot, there’s an option in the Rule type section where you can provide a cron expression, based on which the Lambda function will be triggered.
4. For example, suppose that you want to trigger the Lambda function every morning at 6. To do this, the cron expression will be cron(0 6 * * *). Similarly, you can specify any cron expression which suits your business needs. And if a cron is not something you’re comfortable with, you can use the rate() function to trigger the Lambda function every day. For example, rate(1 day) will trigger the Lambda function every day.
5. Make sure the “Enable trigger” checkbox is checked before you save this trigger.
Once you do that, you will be all set. The Lambda will trigger automatically thanks to CloudWatch based on whichever parameters you defined.
If you keep unused volumes around, your monthly AWS bill is going to increase. To avoid this, you can easily go to the AWS console and delete all unused volumes. But this can’t be a manual process that you add to your schedule every day. An automatic solution is required.
You can create a Lambda function that will get the list of all unused EBS volumes in the given region, and then delete all of them, one by one. Then, you can configure a CloudWatch Event to trigger this Lambda function automatically at given intervals to completely automate the process, while keeping the cost low. This is a level of cloud automation that will not only save money but a lot of headaches.
But another way to alleviate the costs of active or unused EBS volumes in your deployment is by using NetApp Cloud Volumes ONTAP. By leveraging storage cost-cutting efficiency features such as thin provisioning, deduplication, compaction, and compression, the cost of storage volumes can be reduced.