Storage
Mobility
Protection
Analysis and control
TOPICShare
The ability to input free-form text into a search bar and quickly obtain relevant results is now a constant part of our day-to-day usage of web services. Elasticsearch is a key technology that makes that possible. For AWS big data use cases, it can be a powerful resource.
Here are the lessons I learned while using Elasticsearch on AWS:
- Lesson 1: Unexpected Elasticsearch Management Overhead
- Lesson 2: Leverage AWS-specific Elasticsearch Features
- Lesson 3: Explore the AWS Elasticsearch Pricing Dimensions
- Lesson 4: Data Retention and Elasticsearch Storage Tiering
- Lesson 5: Elasticsearch License and Inherent Limitations
What Is Elasticsearch and AWS ES?
As a distributed search and analytics engine, Elasticsearch enables large volumes of data to be searched and indexed in various formats—textual, numerical, structured, and unstructured. Originally created by Elastic and built on Apache Lucene, the open-source project was quickly adopted due to its speed, scalability, and API-driven approach.
On top of it, there are complementary tools that can be used such as Kibana for data visualization dashboards and Logstash for data collection and processing that together provide a powerful end-to-end solution pipeline known as ELK (Elasticsearch, Logstash, and Kibana).
AWS Elasticsearch Service (ES) is a managed service from Amazon that gives the ability to deploy and run Elasticsearch clusters at scale. Integrated within the AWS ecosystem, Amazon Elasticsearch eases off the burden of performing Elasticsearch management operations such as scaling, monitoring, and performing backups by yourself.
What Are Typical AWS Elasticsearch Use Cases?
The unique Elasticsearch features make it suitable for a wide range of use cases, from operational and business analytics to security intelligence applications.
Key Elasticsearch Use Cases:
- Centralized Logging
- Analytics
- Security information and event management (SIEM) solutions
The most popular use case for Elasticsearch is centralized logging and analytics. Software systems generate application logs at a fast pace and thus, it’s important to ingest and store them in a central place for audit, troubleshooting and quality assurance purposes. What made Elasticsearch a perfect tool for this use case was the full-text search and visualization capabilities that enabled engineering teams to actually make the most out of their operational data that was often left stored and unused.
Another typical use case is to leverage Elasticsearch features to build a security information and event management (SIEM) solution. The Elasticsearch stack makes it easy to collect and ingest events at scale from the multiple organization devices, while also providing the ability to correlate data points and integrate with security tools that can enrich and provide valuable threat insights.
So here are the lessons I learned while using Elasticsearch:
Lesson 1: Unexpected Elasticsearch Management Overhead
Customers opt to use Amazon Elasticsearch because they want to focus on using Elasticsearch to create business value without the inherent operational overhead.
Users might find that the expectations of a fully managed SaaS are not really aligned with the reality of using one. While AWS Elasticsearch provides capabilities that make the operational efforts of managing an Elasticsearch cluster significantly lower, there is still a big need for the engineering teams to be deeply involved in its day-to-day management.
A few examples of common operational tasks that will require your efforts while using AWS Elasticsearch are:
- Manual upgrades. There is a fixed list of Elasticsearch versions you can choose from and AWS provides you a fairly easy upgrade experience. However, you can expect some months of delay between the open-source version release and its availability in the AWS Elasticsearch Service. Moreover, upgrading is a manual process, which of course gives you more control on the timing, but can bring real headaches if postponed for too long between major versions.
- Scaling and Resizing Clusters. AWS Elasticsearch simplifies the cluster deployment and management, but it is entirely up to the engineering teams to define the cluster size and type required. This implies that you need a good amount of Elasticsearch operational knowledge and expertise to be able to define the optimal setup, including actively planning for the need to scale the cluster size or resize data storage capacity in the day to day.
- Craft custom solutions to fill-in the managed service gaps. The amount of control and access you have to the AWS Elasticsearch cluster is more limited than in a self-managed option. Also, you are limited to the features provided. This is expected in a managed service but quite often also means that you need to come up with additional solutions and tools to overcome those limitations. One prime example was controlling the access to private clusters by deploying a custom proxy.
Lesson 2: Leverage AWS-specific Elasticsearch Features
The second lesson I learned is something that’s often overlooked: AWS Elasticsearch has its own set of additional features that can really help you make the most out of it.
There are a few differences between the Elasticsearch upstream version open-sourced by Elastic.co versus the one offered by AWS. Amazon ES clusters are based on Open Distro for Elasticsearch (an enhanced open-source version of ES derived from the upstream version) and provides additional capabilities such as Anomaly Detection, security enhancements, and more.
Also, as an Amazon managed service, there are out-of-the-box integrations with other AWS services that are worthwhile to explore. As an example, you can use the native AWS Cognito integration for end-user authentication. The credentials can be stored in Cognito user pools or simply delegated to an external identity such as a Google or Microsoft Active Directory using SAML 2.0. Another great example is data ingestion. Using the built-in integrations, data can be pushed to the AWS Elasticsearch cluster directly from sources such as AWS Kinesis Firehose and CloudWatch Logs without having the need to develop custom business logic.
Lesson 3: Explore the AWS Elasticsearch Pricing Dimensions
My third Amazon ES lesson is something that everyone is probably worried about: how much it costs.
When it comes to pricing, AWS Elasticsearch follows a typical cloud pay-as-you-go model without the need to commit to a minimum spending or upfront fees. There are different dimensions that affect the cost of each cluster and are worth spending some time learning:
- AWS Elasticsearch instance cost per hour, which varies according to the instance type (e.g. medium, large, etc.) used in the cluster.
- AWS EBS storage capacity per GB/month varies depending on the amount allocated to each instance in the cluster.
- Provisioned Storage IOPS per month defined based on the needed performance/throughput of the EBS storage volumes.
- Data transfer follows the regular AWS costs associated with inbound and outbound data transfers.
Understanding these different levers is critical when using AWS Elasticsearch Service at scale and plays a huge role when engineering teams are planning the cluster capacity and scaling options. This often raises good discussions and requires careful consideration, such as choosing to scale the cluster by adding additional instance nodes or increasing the data density per node.
Lesson 4: Data Retention and Elasticsearch Storage Tiering
The fourth lesson is another issue related to costs, and something that is common for many services.
The native distributed characteristics of the Elasticsearch engine make it straightforward to increase the cluster size—with more nodes or bigger data volumes—as the dataset keeps growing. However, the increased compute and storage costs are far more costly when you are using a managed service like AWS Elasticsearch, which comes at a premium cost, than when self-managing deployments.
Simply continuously expanding the cluster is not a cost-effective strategy to handle data growth. It is a good idea to leverage Elasticsearch’s built-in capabilities such as “time-based indices” to delete data after a certain period of time. However, a different possibility worth exploring is UltraWarm, an AWS Elasticsearch storage tier that complements the typical storage node “hot tier”, by using AWS S3 to store data in a more cost-effective manner. This enables older and less frequently accessed data to continue being used for Elasticsearch queries (read-only) at a very low cost per GB. Note that this tiering feature is only available for use with Amazon ES, and not by other AWS services.
Lesson 5: Elasticsearch License and Inherent Limitations
Different vendors have managed Elasticsearch offerings that leverage the upstream open-source version and add their own features on top. As a result, it is worth keeping in mind that you won’t be able to purchase and use those plugins / extra functionalities (e.g. X-Pack by Elastic.co) for use within AWS Elasticsearch Service clusters.
There have been many ups and downs in the history of the Elasticsearch open-source project when it comes to its license. Both Elastic.co, the company behind the Elasticsearch project, and AWS have been involved in a controversial dispute for quite some time over the usage of the project in their competing managed-service offerings.
Recent events will further change the future direction of Elasticsearch. Earlier this year, Elastic.co made a decision to change the open-source project license, making it less permissive and impractical to be used by managed service providers like AWS. While AWS reiterated its commitment to support and contribute to keeping “Open Distro for Elasticsearch” with a permissive open-source license, this effectively means that the two open-source projects that were closely aligned—Elasticsearch by Elastic.co and Open Distro for Elasticsearch—will start to diverge in the near future.
Conclusion
All things considered, Amazon Elasticsearch Service is a great option for teams looking to make easier the deployment and management of Elasticsearch clusters in AWS. It does come with its own caveats and, although it’s a managed service, be prepared to spend a fair amount of time learning and managing Elasticsearch.
If your team already has a good knowledge of Elasticsearch, it might be worth considering self-manage cluster deployments on top of virtual instances. This approach brings additional benefits in terms of flexibility when it comes to configuration, installation of plugins, faster access to latest features, and reducing costs. To lower the operational overhead, you can opt to use Open Distro for Elasticsearch (that already includes some additional features) and Cloud Volumes ONTAP, the data management platform from NetApp.
With Cloud Volumes ONTAP, you can make Elasticsearch management easier (such as backing up and cloning cluster data), while benefiting from additional features such as data protection, cost-cutting storage efficiencies, and data sharing across multiple environments (AWS, Azure, Google Cloud, and on-premises).
For more on optimizing Elasticsearch deployment with NetApp, download our free eBook Optimize Elasticsearch Performance and Costs with Cloud Volumes ONTAP today.
