The original vision of cloud computing was automated, on-demand services that scale dynamically to meet demand. While this vision is now a reality, it doesn’t happen on its own. Cloud automation is complex and requires specialized tools, expertise, and hard work. Learn why cloud automation is important, where it is used, and how it works.
In this post, we’ll review the potential of cloud automation, use cases in modern organizations such as infrastructure as code and DevOps, and common tools used to implement cloud automation strategies. We’ll also show how NetApp Cloud Manager and Cloud Volumes ONTAP can help with the automation of cloud data flows.
This is part of an extensive series of guides about FinOps.
In this article, you will learn:
Cloud automation enables IT teams and developers to create, modify, and tear down resources on the cloud automatically. One of the major promises of cloud computing was that services could be used on demand, if and when needed. But in reality, someone needs spin up those resources, test them, identify when they are no longer needed, and take them down, and this can represent a huge manual effort.
Cloud automation is not built into the cloud; it requires expertise and the use of specialized tools. You can leverage cloud automation tools and capabilities offered by your public cloud vendor (such as AWS or Azure), automation features in your private cloud platform (such as OpenStack or Cloud Foundry), or third party cloud tools that can perform automation in cloud environments (such as Puppet, Chef, Kubernetes, or Cloudify).
Cloud automation requires hard work, but it pays off when you get through the initial pain and gain the ability to perform complex tasks at the click of a button. Beyond the obvious benefit of reducing manual work, cloud automation has additional benefits:
Orchestration is one more step beyond automation. Orchestration involves scheduling and integrating automated tasks between different systems.
For example, consider a system with three nodes running the application, a central database, and a load balancer. Deploying this complex system requires a carefully coordinated series of actions—starting up the database, starting up the nodes, ensuring they connect to the database successfully, then setting up the load balancer.
While automation would involve performing each of these isolated tasks automatically, an orchestration system would be able to perform them in the correct order and timing, and ensure that the entire system is up and running. Another important use of orchestration is to manage connectivity. An orchestration system can ensure components are connected to the local network or the Internet, and are communicating with the remote services they need to function.
For example, orchestrating an app means not only deploying an application, but also connecting it to the network so it can communicate with users and other apps. In the cloud, orchestration is often key to ensuring that automated spin-up activities like auto-scaling take place in the right order, with the right security rules and permissions in place.
Cloud automation has a variety of uses in modern enterprises and development workflows. Here are some of the most common scenarios.
Infrastructure as Code (IaC)
In the IaC paradigm, IT infrastructure should be defined in configuration files and launched automatically according to that configuration. Configuration files should be kept in version control, managed, tested and evolved, just like source code. Cloud automation makes IaC much more powerful, because public and private clouds have control over almost all aspects of the infrastructure, allowing orchestration of complex systems. Read our articles about IaC on AWS and IaC on Azure.
Workload management
The most basic form of workload management is autoscaling, a cloud automation feature that is a staple of any public cloud provider. In addition to control costs on public clouds, it is essential to track what is running, clear unused resources, and adapt the infrastructure to actual operating requirements. Cloud automation helps administrators monitor their environment and automatically adjust workloads as needed.
Application development and testing
Continuous delivery and continuous deployment, which are very widely practiced in agile development organizations, require realistic dev/test environments that are automatically provisioned as part of automated development pipelines. Cloud automation on the public cloud has been, and remains, the primary way to serve the continuous delivery pipeline and generate the environments developers need. Read our article on DevOps benefits and strategies for automating DevOps in a hybrid cloud environment.
Hybrid cloud
Hybrid cloud environments are complex and require intricate integrations between on-premise and public cloud systems. Cloud automation makes it possible to synchronize assets between local data centers and cloud resources, to perform “cloud bursting”—shifting loads to the cloud when local resources are exceeded, enable disaster recovery scenarios with a remote DR site mirroring the on-premise environment, and more. Learn more about how NetApp can help automate hybrid cloud storage.
Cloud automation can be done using throwaway scripts, but the industry has realized that due to the complexity of cloud environments and the need for intricate orchestration of many day-to-day tasks, it is better to rely on a mature automation platform. Here are a few systems commonly used to automate and orchestrate tasks in the cloud.
Puppet
Puppet is a veteran configuration management tool, and one of the pioneers of Infrastructure as Code (IaC). Its commercial version, Puppet Enterprise, is especially suited to cloud workloads and supports huge sale, up to 20,000 Puppet nodes. It can be used for public, private, and hybrid clouds, and offers a GUI interface for defining configuration and automation tasks, as well as Domain Specific Language (DSL) scripts.
Ansible
Ansible is another configuration management product that is synonymous with IaC, developed by Red Hat. Its commercial version, Ansible Tower, supports full orchestration capabilities, with job scheduling, management of a resources inventory, and the concept of “playbooks” that let you automate complex series of actions. You can define configurations using its GUI, DSL, or a REST API. It provides strong support for all public cloud providers and is a robust option for private cloud as well.
Kubernetes
Kubernetes is not by definition a cloud automation solution—it doesn’t allow you to automate all your cloud resources. However, Kubernetes manages and orchestrates Docker containers, which you can use to run pretty much any workload on the cloud. It manages deployment of services within containers, resource consumption, networking within the Kubernetes cluster and with the outside world, and more. Kubernetes is widely used on the public cloud; all public cloud providers offer robust Kubernetes as a service offering, such as Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS).
Learn more in our series of articles about managing and automating Kubernetes storage.
AWS Elastic Beanstalk
Elastic Beanstalk is an automation platform that lets you deploy applications developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker, across EC2 instances. You upload your code, and Elastic Beanstalk handles provisioning of the application, autoscaling, load balancing, monitoring, and more.
AWS CloudFormation
AWS CloudFormation is Amazon’s Infrastructure as Code platform, which allows you to define declarative templates that specify what you want to run in the cloud, and have Amazon automatically spin up the required resources. You can use CloudFormation templates to automate anything from routine deployment of an S3 bucket or EC2 instance group, to highly complex, large-scale applications combining multiple Amazon services.
Terraform
Terraform is an open source tool by Hashicorp, which lets you build, modify and version Infrastructure as Code. It does not provide the configuration management aspect, so it works together with CM tools like Puppet or Chef. Terraform takes configuration files and generates execution plan that describes how it will reach the desired state. When configuration changes, it can create new execution plans to adapt the resources to the new desired state. It can manage anything from computer, storage and networking, to DNS and higher-level application features.
NetApp Cloud Volumes ONTAP, the leading enterprise-grade storage management solution, delivers secure, proven storage management services on AWS, Azure and Google Cloud. Cloud Volumes ONTAP supports up to a capacity of 368TB, and supports various use cases such as file services, databases, DevOps or any other enterprise workload, with a strong set of features including high availability, data protection, storage efficiencies, cloud automation, Kubernetes integration, and more.
In particular, Cloud Volumes ONTAP provides Cloud Manager, a UI and APIs for management, automation and orchestration of cloud storage, supporting hybrid & multi-cloud architectures.
There’s a lot more to learn about cloud automation. To continue your research, take a look at the rest of our blogs on this topic.
There’s a lot more to learn about cloud automation. To continue your research, take a look at the rest of our blogs on this topic.
DevOps vs SRE: Can SRE Make DevOps Better?
Site reliability engineering (SRE) applies automation to IT operations tasks like support and maintenance, while DevOps teams are responsible for collaboratively performing both development and operations tasks. Understand the impact SRE programs can have on DevOps organizations, and how you can leverage SRE to improve DevOps.
Read more: DevOps vs SRE: Can SRE Make DevOps Better?
Automating Cloud Operations with Cloud Volumes ONTAP and REST
NetApp OnCommand® Cloud Manager gives users a modern and easy-to-use graphical UI that makes deploying and getting started with Cloud Volumes ONTAP very straightforward. Take a look at the functionality provided by NetApp OnCommand Cloud Manager and how the same cloud operations can be automated using command line scripts.
Read: Automating Cloud Operations with Cloud Volumes ONTAP and REST
How to Automate Storage Volume Provisioning with Ansible and Cloud Volumes ONTAP
Enterprises scale makes cloud automation a necessity for storage volumes provisioning. In this post we give code examples for how to combine NetApp Cloud Manager APIs and Ansible to create volumes in NetApp Cloud Volumes ONTAP without challenging manual overhead.
This cloud automation process for provisioning storage volumes helps limit the costs of manual effort required to create volumes, speeds up compliance practices, and reduces errors.
Read: How to Automate Storage Volume Provisioning with Ansible and Cloud Volumes ONTAP
Cloud Automation with Cloud Volumes ONTAP and REST
NetApp Cloud Manager provides a convenient UI that lets you work with storage managed in Cloud Volumes ONTAP very straightforward. Learn how the same operations can be performed programmatically using the Cloud Manager REST API.
Read: Cloud Automation with Cloud Volumes ONTAP and REST
Infrastructure as Code: DevOps Done Right
Infrastructure as Code (IaC) is one of the key enablers of the DevOps revolution. Together with cloud automation technology, It provides the ability to turn complex systems and environments into a few lines of code, which can be deployed at the click of a button. Learn how IaC is one of the four steps to achieving an agile, DevOps-friendly environment, how IaC dramatically changes operations work, and which technologies you can use to implement it.
Read: Infrastructure as Code: DevOps Done Right
Cloud Cloning with FlexClone: Cloud Volumes ONTAP Customer Case Studies
Testing is a vital component in the developmental process for both startups and the largest enterprises. DevOps teams are charged with the task of setting up and maintaining testing environments. Read case studies about Cloud Volumes ONTAP’s FlexClone feature, which provides space and cost-efficient cloud cloning to decrease dev/test time and lower costs.
Read: Cloud Cloning with FlexClone: Cloud Volumes ONTAP Customer Case Studies
NetApp Cloud Storage Manager Case Study: How Willis Towers Watson Unlocked the Full Potential of Cloud Storage
Running a business in the cloud can be a challenging task. Enterprises often have limited visibility of their cloud resources and find themselves faced with management overhead, trying to deal with limited capacity, expanding volumes of data that are stored inefficiently, and tedious routine tasks such as backup and disaster recovery. As one of the largest insurers in the world, Willis Towers Watson’s cloud storage management challenges were significant. Find out how Cloud Manager helped.
Terraform & Cloud Manager: How to Use Cloud Manager Terraform Provider
The cloud demands agility through automation, whether you’re deploying code or infrastructure. That has made infrastructure-as-code cloud automation an integral part of DevOps processes, by letting infrastructure be maintained the same way as code. Terraform Cloud Manager Learn how provider allows you to deploy infrastructure as code in your Cloud Volumes ONTAP deployments.
Read: Terraform & Cloud Manager: How to Use Cloud Manager Terraform Provider
How to Deploy Cloud Volumes ONTAP Using NetApp Cloud Manager Terraform Provider
With its platform-agnostic approach, Terraform makes programmatically managing cloud resources easy as part of a DevOps or IaC pipeline. With the NetApp Cloud Manager Terraform provider, NetApp users have a native Terraform resource for IAC storage provisioning with Cloud Volumes ONTAP.
This post will show you how to deploy a Cloud Manager Connector and Cloud Volumes ONTAP step by step using the Cloud Manager Terraform provider.
Read: How to Deploy Cloud Volumes ONTAP Using NetApp Cloud Manager Terraform Provider
Top 12 Site Reliability Engineering (SRE) Tools
SRE is a methodology that applies software engineering principles to IT operations. Learn how SRE tools can help software reliability engineers monitor critical systems, collaborate on issues, automate incident response and manage infrastructure as code (IaC).
Read more: Top 12 Site Reliability Engineering (SRE) Tools
Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of FinOps.
Authored by Spot
Authored by Spot
Authored by Spot