Terraform is a solution that provides infrastructure as code (IaC) capabilities, and is commonly used for Infrastructure as Code automation on AWS. Terraform integrates with a wide range of data sources, including AWS storage services.
In this post, we’ll show how to use Terraform to automate EBS and EFS, and add EBS volumes or EFS file shares to your automated deployments. We will also show how NetApp Cloud Volumes ONTAP can help simplify the management of multi-vendor and hybrid cloud environments.
In this article, you will learn:
Terraform is an IaC solution that you can use to build, modify, and version your infrastructure. It is created by Hashicorp and is an Advanced Technology Partner in the AWS Partner Network (APN). Terraform is also part of the AWS DevOps Competency. You can use it in place of AWS CloudFormation to manage your AWS infrastructure.
Several features of Terraform can make it an appealing solution, including:
Learn more in our article about Terraform AWS, which explains how to deploy a Terraform enterprise cluster on AWS.
Amazon Elastic Block Store (EBS) is a service that provides block storage. Block storage enables you to store large amounts of data in blocks that serve as virtualized hard drives. This type of storage can provide high performance and is ideal for volatile or transactional data.
When using EBS you can store a wide variety of data types, including file systems, databases, big data engines, containers, and applications. The service is scalable, flexible, and includes built-in data replication across Availability Zones (AZ). It also includes a snapshot feature for easy backup and duplication of volumes.
AWS EBS volumes are the individual machines you create in EBS. The most common use cases for EBS volumes include:
EBS was originally designed to complement EC2 and is the default storage choice for EC2 instances. However, you can use EBS freely regardless of whether you are using the EC2 service. For example, it is possible to use EBS as your primary storage or with other AWS services.
AWS EFS is a scalable storage service that enables you to create file systems in AWS. It is based on the NFSv4 file system protocol, which mirrors standard, on-premises systems. This makes transfer and access of files easier and faster.
EFS supports Linux-based workloads and applications, and can be combined with on-premises resources and other AWS services. It is available in two storage classes — Standard and Infrequent Access. Standard access is more expensive but provides lower latency for active files. Infrequent access is cheaper in exchange for higher latency and is meant for archived or rarely used files.
Terraform data sources let you access data stored outside of Terraform or defined in independent Terraform configurations. When a data source, such as EBS, is connected, you can fetch and compute data without relying on Terraform’s resources. This helps stabilize performance on the Terraform server, because it does not need to transfer or compute data directly, and enable you to share configuration data across your system or teams more easily.
To access data sources, you need to declare your data resource in a data block. Data resources are different from managed resources in that the resource is read-only and you cannot create, update, or delete objects. You can see an example block below:
data "aws_ami" "output" {
most_recent = true
owners = ["self"]
tags = {
Name = "app-server"
Tested = "true"
}
}
In this block, Terraform is requesting to read from the "aws_ami" data source and to export the results to the local name "output". You can then use this local name throughout your Terraform module to access the data. The data source and local name serve as an ID for the data and are unique to the module.
After the source and export location are defined, and within the block body, you can define any query constraints associated with the source. In the above case, the constraints are most_recent, owners, and tags.
Specific to using EBS, you can use the following data sources to make configuration and management of your volumes and data easier:
Like EBS, EFS also has dedicated data sources you can use. Below are the most commonly used sources:
Below is a brief tutorial showing a common deployment task, attaching an EBS volume to an EC2 instance. For this tutorial you should already have an AWS account set up with both EBS and EC2 services. It also assumes that you have Terraform installed and configured to work with AWS.
provider "aws" {
region = "us-east-1" // Replace with your desired region
}
resource "aws_security_group" "morning-ssh-http" {
name = "morning-ssh-http"
description = "allow ssh and http traffic"
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_instance" "good-morning" {
ami = "ami-5b673c34" #Red Hat RHEL V7
instance_type = “t3.micro”
availability_zone = "us-east-1"
security_groups = ["${aws_security_group.morning-ssh-http.name}"]
key_name = "zoomkey"
user_data = <<-EOF
#! /bin/bash
sudo yum install httpd -y
sudo systemctl start httpd
sudo systemctl enable httpd
echo "<h1>Sample Webserver" | sudo tee /var/www/html/index.html
EOF
tags = {
Name = "webserver"
}
}
resource "aws_ebs_volume" "data-vol" {
availability_zone = "us-east-1"
size = 1
tags = {
Name = "data-volume"
}
}
#
resource "aws_volume_attachment" "good-morning-vol" {
device_name = "/dev/sdc"
volume_id = "${aws_ebs_volume.data-vol.id}"
instance_id = "${aws_instance.good-morning.id}"
}
Below is a brief tutorial showing how to create an EFS file share. Like the EBS tutorial, this tutorial assumes that you already have an AWS account set up with EFS services. It also assumes that you have Terraform installed and configured to work with AWS.
This is adapted from a more in-depth tutorial by Earl Ruby which you can find here.
To begin, you need to define your resource in your EFS Terraform file. Your code should resemble the following:
resource "aws_efs_file_system" "example-efs" {
creation_token = "example-efs"
performance_mode = "generalPurpose"
throughput_mode = "bursting"
encrypted = "true"
tags = {
Name = "TestEFS"
}
}
After your resource is defined, you also need to set up a mount target so you can attach volumes to the file share. The following block gives an example of how your target is defined.
resource "aws_efs_mount_target" "example-efs-mt" {
file_system_id = "${aws_efs_file_system.example-efs.id}"
subnet_id = "${aws_subnet.subnet-efs.id}"
security_groups = ["${aws_security_group.ingress-efs.id}"]
}
In the above block, your file_system_id ties your mount target to your file share. Subnet_id is a separately defined connection that you can create. The security-groups designation is also separately defined. You can see how to create these below.
EFS subnet
The following definition should be added to your network Terraform file. You can choose any available subnets when defining this resource. The below example uses /16 for a virtual private cloud and /24 for EFS shares and machine clusters. The subnet is tied to the US East 1a availability zone.
resource "aws_vpc" "test-env" {
cidr_block = "10.0.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags {
Name = "test-env"
}
}
resource "aws_subnet" "subnet-efs" {
cidr_block = "${cidrsubnet(aws_vpc.test-env.cidr_block, 8, 8)}"
vpc_id = "${aws_vpc.test-env.id}"
availability_zone = "us-east-1a"
}
EFS security group
Creating a security group enables you to control traffic between your EFS file share and your test environment. A block like the following example should be added to your security Terraform file.
This example assumes an existing security group (ingress-test-env) and creates a new group to enable restricted (to port 2049) inbound traffic and unrestricted outbound traffic. This restriction improves security and eliminates the need for the cidr_blocks attribute.
resource "aws_security_group" "ingress-efs-test" {
name = "ingress-efs-test-sg"
vpc_id = "${aws_vpc.test-env.id}"
ingress {
security_groups = ["${aws_security_group.ingress-test-env.id}"]
from_port = 2049
to_port = 2049
protocol = "tcp"
}
egress {
security_groups = ["${aws_security_group.ingress-test-env.id}"]
from_port = 0
to_port = 0
protocol = "-1"
}
}
Once all your files are defined, you can apply the definitions by running Terraform. Provided the definitions are applied successfully, you should be returned an output that lists your mounted folders and the details of each.
NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, Kubernetes integration, and more.
In particular, Cloud Volumes ONTAP provides Cloud Manager, a UI and APIs for management, automation and orchestration, supporting hybrid & multi-cloud architectures, and letting you treat pools of storage as one more element in your Infrastructure as Code setup.
Cloud Manager is completely API driven and is highly geared towards automating cloud operations. Cloud Volumes ONTAP and Cloud Manager deployment through infrastructure- as- code automation helps to address the DevOps challenges faced by organizations when it comes to configuring enterprise cloud storage solutions. When implementing infrastructure as code, Cloud Volumes ONTAP and Cloud Manager go hand in hand with Terraform to achieve the level of efficiency expected in large scale cloud storage deployment.