Storage
Mobility
Protection
Analysis and control
TOPICShare
If you’re using the AWS cloud, there are two ways that you can leverage the free-form search engine, Elasticsearch: either signing up for the managed service for Elasticsearch on AWS, AWS Elasticsearch Service, or deploying Elasticsearch on AWS compute instances.
In this blog post we’ll take a look at both of these deployment models, giving you a point-by-point breakdown of the benefits and limitations of both approaches.
This article will cover:
- Elasticsearch Up Close
- What Is AWS Elasticsearch Service?: The Managed Service Option
- Running Elasticsearch on EC2: The Self-Managed Option
- Elasticsearch on Amazon EC2 with Cloud Volumes ONTAP: The Best of Both Worlds
Elasticsearch Up Close
Elasticsearch, an open-source project originally developed by Elastic, is one of the most popular solutions when it comes to providing free-form search capabilities. Based on Apache Lucene, its distributed analytics engine allows you to store, search and analyze massive volumes of all types of data, such as textual, numerical, geospatial, structured and unstructured. It is often used together with Logstash for data ingestion and processing, and Kibana for data visualization and management, commonly referred to as the ELK Stack or Elastic Stack.
Thanks to Elasticsearch's ability to quickly search and analyze all kinds of data, returning results in near real-time, it can address a wide range of use cases, anywhere from a simple search bar to an enterprise-scale logging solution.
Some of the most common use cases are:
- Full Text Search
- Logging and Log Analysis
- Application and Infrastructure monitoring
- Security information and event management (SIEM)
What Is AWS Elasticsearch Service?: The Managed Service Option
AWS Elasticsearch Service is a fully managed service from Amazon that allows you to easily deploy and manage Elasticsearch clusters, and provides features that can significantly decrease the operational overhead in software installation and patching, monitoring, failure recovery, and backups.
Managed Service Benefits for Elasticsearch
- Easy to set up, less operational overhead: As a managed service, AWS Elasticsearch takes care of many of the setup, configuration, and maintenance tasks
- Support for multiple versions: AWS Elasticsearch Service supports multiple versions of Elasticsearch up to version 7.9 and the most common open-source Elasticsearch APIs allowing your existing code and applications to use the service without any code changes.
- Cross-AZ availability: Deployments can span across as many as three Availability Zones (AZ) for high availability with failed nodes replaced automatically for failure recovery.
- Native integration with AWS services
- Support for external services: Built-in managed Kibana for data visualization, and Logstash support for data ingestion and processing into your clusters.
- Automated snapshots: AWS Elasticsearch clusters automatically perform snapshots hourly without additional costs for up to 14 days.
Managed Service Limitations
As useful as the managed service benefits are, there are a number of caveats that come with AWS Elasticsearch as a service. These include:
- Additional premium cost: You need to pay by the hour according to rates that vary per instance type and AWS region.
- Version and Instance types: You will only be able to choose Elasticsearch versions (up to 7.9) and certain instance types that define the amount of memory and CPU each Elasticsearch cluster node has.
- Limited plugins: You are unable to install additional plugins such as the popular X-Pack. AWS Elasticsearch comes prepackaged with a limited set of plugins that you are unable to modify or expand.
- Configuration limitations: While some configuration settings are available (e.g., version, custom domain name, data storage nodes, and number of nodes in the cluster), without full control over your Elasticsearch clusters settings and underlying storage, there can be some limitations in terms of scalability—hybrid and multicloud clustering or leveraging certain AWS regions—for your particular use cases.
- Oversight still required: Even with the advantages the AWS Elasticsearch provides in being easy to set up and lowering the operational aspects of managing your Elasticsearch clusters, you will still need expert knowledge to manage day-to-day operations.
Elastic Cloud on AWS
Alternatively, you can use the managed service from Elastic Cloud on AWS, a third-party SaaS solution available on the AWS Marketplace, which gives you access to the latest version of the Elastic Stack, additional exclusive features and direct support from Elastic. You can deploy Elastic Cloud on AWS using pre-defined templates on instances optimized for specific use cases, but keep in mind that being a managed service, it comes with additional cost on top of the used instances.
It is worth noting that because of the recent Elasticsearch open-source license change, which restricts some of its usage, AWS forked the project and named it “OpenSearch”. This Elasticsearch fork maintains the same original and more permissive Apache 2.0 license and will become the new baseline for Open Distro and the AWS managed service.
Running Elasticsearch on EC2:The Self-Managed Option
When opting to self-manage your Elasticsearch clusters on AWS, you will have to deploy your clusters on Amazon Elastic Compute Cloud (Amazon EC2) instances.
Self-Managed Elasticsearch Benefits
- More instance options: All the instance types available with different CPU and memory configurations to choose from bare metal instance type or instances up to 128 virtual CPUs and 1952GB of memory
- Greater control over clusters: Access to all Elasticsearch APIs and settings, and underlying resources, giving you full control and flexibility over the number of data nodes per cluster or the number of dedicated master nodes
- More flexibility: Enabling more complex cluster topologies that spread over multiple AWS Regions and leverage non-AWS environment resources
- Full use of AWS platform: AWS service integrations are often also possible, with some added initial setup effort
- Fully compatible with all plugin options: You will also be able to install every plugin available such as the popular X-Pack and use different monitoring and storage solutions
- Lower Cost: Considerably less expensive compared to managed service fees
- Elasticsearch version options: Choose from the upstream open-source version of Elasticsearch from Elastic.co or the Open Distro for Elasticsearch
By opting to use Open Distro for your self-managed Elasticsearch deployments, you will benefit from some of the features included in AWS Elasticsearch—since it’s the same engine AWS uses in its managed service—such as enhanced security, alerting, index management and more, lowering the operational overhead, while having full control and flexibility over your Elasticsearch clusters.
Self-Managed Elasticsearch Cons
- Domain expertise: Your admins will need the expertise to make sure security, backups, monitoring and failure recovery are in place
- Added management work: Managing your own instances will require additional setup work, which may increase overheads
Elasticsearch on Amazon EC2 with Cloud Volumes ONTAP: The Best of Both Worlds
AWS Elasticsearch Service allows you to easily deploy and manage Elasticsearch clusters with built-in capabilities that ease off the burden of day-to-day operations, however that comes with some limitations and at an additional premium cost. On the other hand, deploying Elasticsearch on EC2 instances offers complete control and flexibility over your Elasticsearch clusters, installation of plugins, access to features that aren’t supported by AWS Elasticsearch at the expense of increased operational overhead.
When you want to get the most out of your self-managed Elasticsearch clusters, one deployment option to consider is Cloud Volumes ONTAP, the cloud-based data management storage solution from NetApp.
With Cloud Volumes ONTAP, Elasticsearch gains a number of benefits:
- Cluster management, high availability, and backup management are easy to implement and operate using Cloud Manager.
- Achieve higher performance on Elasticsearch read-intensive workloads through intelligent caching, dramatically reducing latency.
- Storage efficiency features, you will be able to reduce your data footprint while also reducing storage costs up to 70%.
- Ability to migrate, replicate and synchronize data on the fly across hybrid and multicloud environments.
- High availability for RPO=0 and RTO<60 seconds business continuity
- Seamless disaster recovery backed by Snapshot™ and SnapMirror® technology help reduce operational overhead.
When self-managing your Elasticsearch clusters, you should consider Cloud Volumes ONTAP, which provides you with out-of-the box features that reduce the operational overhead, storage costs, and increase performance of your Elasticsearch clusters.
Learn more in our solution guide, Optimize Elasticsearch Performance and Costs with Cloud Volumes ONTAP.
