Elasticsearch is a popular open-source NoSQL database. Amazon’s managed Elasticsearch Service (Amazon ES) simplifies its deployment, scaling, and operation of clusters in the Amazon Web Service (AWS) cloud environment. The service runs the same code and APIs as on-premises Elasticsearch, ensuring that existing applications are fully compatible.
Amazon ES launches Elasticsearch clusters, provisions resources as needed, and continues to monitor for failed Elasticsearch nodes—automatically replacing them. It thereby enables real-time application monitoring, clickstream analysis, and log analytics.
You can create an Amazon Elasticsearch domain (the Amazon term for an Elasticsearch cluster) through the Amazon console, CLI, or Amazon SDKs.
In this article, you will learn:
Amazon ES features include:
Related content: read our guide to Elasticsearch architecture
Let’s see how to start a new Amazon Elasticsearch Service (Amazon ES) domain, which represents a hosted Elasticsearch cluster with its settings and resources.
To create an Amazon ES domain using the Amazon console:
5. Under Availability Zones, select the number of AWS Availability Zones you want the cluster to run in, between 1-3. Deploying on two or three AZs increases availability, but will also increase your costs because you are running redundant compute and storage resources.
6. Under Instance Type, select the Amazon EC2 instance type for data nodes.
7. Under Number of nodes, select the number of data nodes in your cluster. Each will run on its own EC2 instance.
a. Under Data nodes storage, select Instance, meaning data will be stored in local, non-durable storage on the instance, or EBS, meaning data will be stored in an Amazon EBS volume.
8. Under Network configuration section, select whether you want to enable only VPC access to your instances, or Public access over the Internet.
9. Under Fine-grained access control, to control authorization for your Elasticsearch cluster using Amazon Identity and Access Management (IAM), select Set IAM role as master user. Otherwise, to use the Elasticsearch user database, select Create a master user.
10. Verify the domain configuration, then select Confirm to deploy.
Related content: read our guide to Elasticsearch in Production on AWS
In Elasticsearch, you feed data into the index in JSON format. In Amazon ES, you don’t need to manually create an index, because the service creates an index when you add your first document.
To create a new document and index it, use an HTTP console such as the Kibana dev console, to make an HTTP call like the following. Remember that if you use another HTTP console, you will need to provide a full URL and credentials.
PUT /customers/_doc/1
{
"firstname":"John",
"lastname":"Smith"
}
This command prompts the service endpoint to automatically create an index named customers, and add a single document into it, with an ID of 1. The string _doc is the Elasticsearch mapping type—if you are using Elasticsearch 7 and up, it is not required.
To add documents in bulk, use a call like the following. A bulk call bundles multiple calls, each of which can perform a different operation on a different index. For example:
You can perform a basic search using a query like the following:GET /customers/_search?q=name:l*
This returns the first customer document we created in the previous section.
To perform more advanced searches, use a more complex query using the Elasticsearch domain specific language (DSL). Departing from the simple customer data examples above, here is a more complex query provided in the Amazon ES documentation:
The Amazon Elasticsearch service manages setup of an Elasticsearch domain (cluster), including infrastructure provisioning within the network environment and installing Elasticsearch software.
After setup, the service performs automated backups, instance monitoring, software patching, and other administrative tasks for as long as the domain runs. You can specify storage settings and select EC2 instances to run your cluster on, in accordance with application requirements.
Yes. The code, applications, and tools you currently use with Elasticsearch will work seamlessly with Amazon’s Elasticsearch Service. See the official list of Elasticsearch supported operations.
Migrating data from one Elasticsearch cluster to other entails creating a snapshot of the existing cluster and storing it in an Amazon S3 bucket. After creating a new domain, you can use the Elasticsearch restore API to load data into it from the snapshot.
NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, Kubernetes integration, and more.
Cloud Volumes ONTAP supports advanced features for managing SAN storage in the cloud, catering for NoSQL database systems, as well as NFS shares that can be accessed directly from cloud big data analytics clusters.
In addition, Cloud Volumes ONTAP provides storage efficiency features, including thin provisioning, data compression, and deduplication, reducing the storage footprint and costs by up to 70%.
For more on optimizing Elasticsearch deployment with NetApp, download our free eBook Optimize Elasticsearch Performance and Costs with Cloud Volumes ONTAP today.