Elastic Stack on Azure is a deployment template that lets you deploy Elasticsearch clusters, including the full ELK Stack (Elasticsearch, Logstash and Kibana) in a fully automated manner.
It provides a user interface that guides you through the required steps and feeds your inputs to an Azure Resource Manager (ARM) template, which then deploys the required resources to a resource group.
You can also access the ARM template independently, either from the Azure portal, or using Azure’s command-line interface (CLI) or PowerShell command-line tools.
In this article, you will learn:
Deploying the required Azure resources for running an Elasticsearch cluster requires Azure’s Resource Manager API. This is an infrastructure-as-code solution that deploys infrastructure as a service (IaaS) resources within the Azure environment using a consistent API.
The ARM template provided by Elasticsearch can create the resources needed for an Elasticsearch cluster, either an incremental deployment targeted at an Azure resource group with existing Elasticsearch workloads, or as a full deployment, which replaces Elasticsearch resources that are currently deployed.
Related content: read our guide to Azure Resource Manager (ARM) Benefits and Best Practices
This reference architecture was provided by contributors to the Azure documentation.
Here is how this topology is generated automatically using an ARM template:
Related content: read our guide to Elasticsearch architecture
5. Select whether you are deploying a new virtual network (VNet) or using an existing one. If networking resources exist in a resource group, and you want the cluster to access resources on the same network, deploy them to an existing VNet.
6. Click on the Nodes Configuration tab to determine the number of nodes of each type you require for the cluster. You may specify the disk size and type, as well as the number of disks for data nodes. If there are two or more disks, the system will create a RAID 0 array across them for improved performance.
Image Source: Elastic
7. Click on the Kibana & Logstash tab. Here you can decide whether you wish to deploy Kibana and/or Logstash alongside Elasticsearch. You can also determine the cluster access mode from outside the virtual network.
Note: Kibana deploys to a separate VM, which receives a public IP address and network security group. It is accessible over TCP using ports 5601 and 22. Logstash also installs to a separate VM, deployed in an Availability Set. You can use a custom Logstash configuration file and install additional Logstash plugins.
8. Define users and role configuration by clicking on the Security tab. This feature is only available during the 30-day trial period offered by the Azure Marketplace. Here you can set up six built-in user accounts, providing credentials for each:
● Elastic user account—the built-in account for the superuser
● Kibana user account—the Kibana account for connecting and communicating with Elasticsearch (from version 7.8.0 andkibana_system)
● Logstash system user account—the Logstash account for saving monitoring data in Elasticsearch
● Beats system user account—used by Beats, an open source data shipper that sends data to Logstash or Elasticsearch
● APM system user account—the application performance monitoring (APM) server’s account for saving APM data to Elasticsearch
● Remote monitoring user system user account—used by Metricbeat to collect monitoring data
9. Click on the Certificates and Review + Create tabs to confirm your input values and click OK to launch your new deployment.
Congratulations: you are now running an Elasticsearch cluster on Azure!
Deployment prices through Azure’s marketplace depend on your desired deployment, as well as the VM types (SKU) used to deploy the cluster. Prices also depend on the selected Azure region.
You should take into account the resources deployed by the template—Elastic Stack Gold, Enterprise and Platinum features, and their required licenses. Azure calculates the final cost on a case-by-case basis, which can be estimated using Azure’s pricing calculator.
Within Azure, connect to the virtual network that contains the desired cluster, through an internal load balancer. This will round robin requests directly to data nodes, or to coordinating nodes, if these are deployed. From outside Azure, you can access Elasticsearch through an external load balancer, configured using the External access tab in the Marketplace solution configuration.
After configuring your Elasticsearch deployment using the tab steps above, you will be able to access all virtual network deployed VMs through a Kibana or Jumpbox VM. Both of these operate in a network security group, which can be accessed via TCP through port 22. This allows you to use SSH to securely connect to VMs.
You can use either a password or SSH key to connect, as defined in Basics step of configuration. You will need to know the private IP address, or the hostname, if DNS is defined, to connect to the VMs.
Related content: read our guide to Elasticsearch in production
NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, Kubernetes integration, and more.
Cloud Volumes ONTAP supports advanced features for managing SAN storage in the cloud, catering for NoSQL database systems, as well as NFS shares that can be accessed directly from cloud big data analytics clusters.
In addition, Cloud Volumes ONTAP provides storage efficiency features, including thin provisioning, data compression, and deduplication, reducing the storage footprint and costs by up to 70%.
For more on optimizing Elasticsearch deployment with NetApp, download our free eBook Optimize Elasticsearch Performance and Costs with Cloud Volumes ONTAP today.