Self-Managed Elasticsearch vs. Elastic Cloud Managed Service

From a high-level, it can be a little confusing what Elasticsearch does. When asked, people might give you a number of different answers, including “it’s an index”, an “analytics database”, “a search engine”, or that “it's a fast and scalable big data store.”

All of these answers are actually somewhat correct. That’s one of the many reasons Elasticsearch is so popular. Elasticsearch and what has been expanded to the “Elastic Stack” has been used for a growing number of use cases. There are basic use cases such as implementing a simple search on a website or document and more analytical use cases such as analyzing log data and real-time reporting for data analysis. In some cases, developers are building visualizations and APIs on top of Elasticsearch, using it as a fast data retrieval method that does not compromise the traditional system database and storage availability.

Regardless of your use case, there are only two methods to deploy and use Elasticsearch in the cloud: using a managed service or a self-managed approach. In this article we will look at the self-managed Elasticsearch vs. Elastic Cloud managed service deployment models.

Read on below as we take a look at:

• Elasticsearch 101
• The Managed Service Option: Using Elastic Cloud for Elasticsearch Services
• Using a Self-Managed Elasticsearch Instance / Cluster
• How Cloud Volumes ONTAP Add Value to a Self-Managed Elasticsearch

Elasticsearch 101

At its core, Elasticsearch is a server-side engine that uses JSON style querying to pull JSON data that is indexed. Elasticsearch is also an open-source search engine that allows developers to utilize its distributed architecture to run said queries quickly and efficiently.

Elasticsearch is based on Lucene, an open-source search framework. With Elasticsearch, developers added the ability to horizontally scale Lucene indices. This makes it possible to easily store, analyze, and search large volumes of data efficiently and quickly, all nearly in real time.

There are pros and cons to using Elasticsearch. As a real-time search engine, documents can be searched almost as soon as they are added into the data store. However, since all interactions are driven by APIs, there is a learning curve to get started with Elasticsearch—unlike your standard SQL-enabled relational database.

The Managed Service Option: Using Elastic Cloud for Elasticsearch Services

One way that organizations are gaining the benefits of Elasticsearch is through the managed service, Elastic Cloud. While some cloud providers have their own managed Elasticsearch service, Elastic Cloud remains a popular choice for Elasticsearch services.

What Is Elastic Cloud?

Elastic Cloud is a managed service provided by Elastic.co, the company founded by the core Elasticsearch authors to provide services and products around the ES ecosystem.

Elastic Cloud enables engineering teams to take advantage of the features and benefits of Elasticsearch in the cloud without requiring the expertise and maintenance involved.

It follows a subscription, pay-as-you-go model that enables teams to deploy and manage an Elasticsearch cluster in AWS, Azure or Google Cloud.

Elastic Cloud Pros

Here are some of the benefits of using Elastic Cloud:

• Less management overhead: As a managed service, the main benefit is that it frees you up from having to handle management of your Elasticsearch deployment.
• Easier to deploy: Optimized memory, storage and compute ratios. The memory and storage settings are adjusted by Elastic Cloud based on the deployment size you select.
• Managed security: The underlying infrastructure is managed by Elastic Cloud. As such, data encryption, OS hardening and other security requirements are handled by Elastic.co.
• Automatic updates: Having a managed version of a service like Elasticsearch allows your company’s development team to focus on other tasks without having to handle periodic cluster upgrades.

What Are Elastic Cloud’s Cons?

• Added Costs: Managed service comes with an additional premium service cost compared to a self-managed approach.
• Configuration Limitations: Because of the lack of access to the underlying infrastructure, there are inherent configuration challenges. Software engineers are limited to the features, options, and parameters available in Elastic Cloud. Depending on your needs, it might be simply impossible to customize the way you require—such as installing ES and Kibana plugins, the latter of which is not supported at all—or integrate with third-party services for things like logging and monitoring.

Using a Self-Managed Elasticsearch Instance / Cluster

The alternative to a managed service such as Elastic Cloud, is to deploy and operate Elasticsearch on cloud storage and compute using virtual instances or containers.

Self-Managed Elasticsearch Pros

There are numerous benefits to a self-managed Elasticsearch deployment in the cloud:

• Reduced costs: Running your own Elasticsearch clusters will avoid the premium costs of a managed Elastic Cloud service.
• Avoiding lock-in: Managing your own clusters allows an engineering team to migrate and switch various components without the lock-in of a given service provider.
• New License Possibilities: With the changes Elastic.co introduced to the Elasticsearch open-source license, there are significant differences in the project going forward and forked Elasticsearch projects are already appearing. A self-managed approach enables a better adaptation to the inherent changes happening in the project and also opens the possibility to a present or future transition to other ES forks.
• Flexibility: A self-managed Elasticsearch cluster gives you full flexibility over all the configuration and management aspects of it. One common example is the ability to install additional plugins or set up integrations with third party services that are usually not possible in a managed Elasticsearch service.

Cons with Self-Managed Elasticsearch

Despite the benefits you gain from using your own Elasticsearch, there are drawbacks:

• More overhead: Your teams will need to manage the day-to-day operations.
• Data Protection: Contrary to a managed Elasticsearch service, where management features such as daily snapshots are usually built-in, the self-managed approach requires additional setup work in making sure the data is backed up and protected, and monitoring and logging are in place.

Overall, it's important to consider these differences when comparing self-managed Elasticsearch vs. Elastic Cloud.

How Cloud Volumes ONTAP Add Value With Self-Managed Elasticsearch

One way that running Elasticsearch on native instances can be improved is with Cloud Volumes ONTAP.

Cloud Volumes ONTAP is the data management platform from NetApp that is built on top of native cloud provider storage from AWS, Azure, or Google Cloud. Its refined capabilities can give a self-managed Elasticsearch deployment the feature-set that you expect to get with a fully managed service like Elastic Cloud, and much more.

Cloud Volumes ONTAP Benefits for Self-Managed Elasticsearch

• Cross-region high availability with zero RPO and RTO under 60 seconds.
• Cost-effective data protection and DR with NetApp Snapshots™ and SnapMirror.
• Easy cluster and backup management through the Cloud Manager GUI or RESTful API calls.
• Cost-reducing storage efficiencies can cut down your cloud storage footprint and spending as much as 70%.
• Intelligent caching reduces latency by providing Elasticsearch with higher performance for read-heavy jobs.
• Full multicloud and hybrid data mobility and synchronizing with SnapMirror®.

While it’s true that Elasticsearch is highly optimized for fast search performance, that performance depends on the underlying storage used by Elasticsearch. When using Cloud Volumes ONTAP together with Elasticsearch, developers can find a cost reduction for storage of 40% and an increase in query speed by 66% thanks to the performance improvements that Cloud Volume ONTAP provides.

Between the operational and performance improvements, Cloud Volume ONTAP is proving to be a strong choice when it comes to deploying your Elasticsearch instance.

Learn more about optimizing Elasticsearch deployment with NetApp in our free eBook Optimize Elasticsearch Performance and Costs with Cloud Volumes ONTAP.