Elasticsearch on Google Cloud: Your First Managed Cluster

What is Elasticsearch on Google Cloud?

Elasticsearch is a popular NoSQL database based on the open-source Lucene search engine, which facilitates fast search across large datasets. Elasticsearch provides the Elastic on Google Cloud solution, which lets you deploy Elasticsearch  clusters on the Google Cloud Platform. It supports four deployment options, which provide different capabilities:

• Elastic Enterprise Search—lets you create customized search experience, ingesting any data format, from storage services based on Google Cloud or elsewhere.
• Elastic Observability—lets you visualize log and metrics data, analyze it, identify anomalies in real time and respond to them.
• Elastic Security—lets you monitor IT environments, integrating with SIEM integration and endpoint protection solutions. This Elasticsearch deployment model provides a machine learning engine that helps identify misconfigurations and attacks against your infrastructure.
• Elastic Stack—a “plain vanilla” deployment with the traditional Elasticsearch, Logstash and Kibana (ELK) stack.

Read more about these options in the Deployment Options  section below.

In this article, you will learn:

• Getting Started with Elastic on Google Cloud
  • Deployment Options
  • Customize your Deployment
  • Launch Kibana
  • View and Analyze Data
• Elasticsearch on Google Cloud with Cloud Volumes ONTAP

Getting Started with Elastic on Google Cloud

To get started, visit the solution page on the Google Cloud Platform Marketplace. The Elastic on Google Cloud solution is a commercial offering, so first confirm that costs are suitable for your budget, then click Purchase, then Subscribe.

On the opening page, click Manage via Elastic and create a new account.

Image Source: Elastic

After you create an account, you can select Manage to view Google Cloud charges for your Elasticsearch resources, or click Manage on Provider to manage your clusters using the Elastic Cloud console.

Click Create deployment to set up your first cluster on Google Cloud.

Deployment Options

Now you can choose one of four deployment options. Each deploys a different variant of the Elasticsearch infrastructure, providing different functionality.

Source: Elastic

• Elastic Enterprise Search—enables powerful search over massive datasets. This deployment option lets you design and deploy a search experience for websites, mobile applications or enterprise datasets. Included within the Enterprise Search solution is Elastic Workplace Search, which provides a unified search experience across your organization’s collaboration, productivity, and storage tools.
• Elastic Observability—integrates logs, metrics, and traces from your entire ecosystem into a unified, high-performance, scalable stack. Provides a rich UI to monitor, search, and analyze all operational data to quickly troubleshoot issues and improve user experience.
• Elastic Security—effective information security requires effective analysis of big data. This deployment option makes it easy to manage and scale security event data. It integrates with SIEM, endpoint security, threat hunting, cloud monitoring tools and many more data sources. It provides an intuitive user interface for searching, visualizing and analyzing security data.
• Elastic Stack—a generic Elasticsearch, Logstash and Kibana (ELK) stack that lets you build your custom solutions. This deployment option provides a variety of useful hardware profile templates, including hot-warm topologies.

Related content: read our guide to Elasticsearch architecture

After selecting the deployment type, assign it a unique name, then click Create Deployment.

Customize your Deployment

You can customize your deployment before or after creating it. Here are several customizations you can achieve:

• Increase or decrease capacity—you can modify capacity by changing the total amount of storage and memory reserved or each component of your deployment.
• Autoscaling—this feature automatically scales resources according to current demands.
• Improve high availability—you can adjust the number of AZs your deployment is running on.
• Resize instance configurations—to enable features like application performance monitoring (APM) or machine learning (ML).
• Add warm or cold data tier nodes—you can do this by adding capacity for each data tier.
• Enable Elasticsearch plugins—if not enabled by default.
• Set configuration parameters—for Kibana instances or Elasticsearch nodes.

Launch Kibana

In the Elasticsearch Console, click Open Kibana.

When you first deploy Elasticsearch on Google Cloud, Elasticsearch loads sample data, which can help you get started quickly and familiarize yourself with the tooling.

To load sample data and get started with Kibana:

1. On the Kibana main page, click Try our sample data. There are several cards using sample datasets. Choose the one closest to your real datasets, then click View data > Add data.

2. From the main menu, select Discover. This shows a histogram with the distribution of the data over time, and a table listing the fields of each document that match the index. By default, all fields in each matching document are displayed.

3. Change the time filter to Last 7 Days.

4. Select documents to view using Kibana Query Language (KQL). For example, if you selected the Sample eCommerce Orders dataset, you can try the following KQL query, which shows orders for women’s clothing with a total of over $60.

products.taxless_price >= 60 AND category : Women's Clothing5. Use the list of Available fields and add fields to view them in the table below.

View and Analyze Data in Kibana

Let’s briefly review how to work with Kibana dashboards in Elasticsearch for Google Cloud. This assumes you have loaded sample data as shown in the previous section.

To view a dashboard based on sample data:

1. From the main menu, select Dashboard.

2. Select the dashboard automatically created for the sample data you selected. If you selected the Ecommerce dataset, click E-Commerce Revenue Dashboard.

Image Source: Elastic

3. Use the filters at the top-right of the dashboard to focus on the data you want to display.

4. Try to manually add a filter by clicking Add filter.

5. Experiment with adding filters and applying filters to see how data is updated in your sample dashboard.

Elasticsearch on Google Cloud with Cloud Volumes ONTAP

NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, Kubernetes integration, and more.

Cloud Volumes ONTAP supports advanced features for managing SAN storage in the cloud, catering for NoSQL database systems, as well as NFS shares that can be accessed directly from cloud big data analytics clusters.

In addition, Cloud Volumes ONTAP provides storage efficiency features, including thin provisioning, data compression, and deduplication, reducing the storage footprint and costs by up to 70%.

For more on optimizing Elasticsearch deployment with NetApp, download our free eBook Optimize Elasticsearch Performance and Costs with Cloud Volumes ONTAP today.