Storage
Mobility
Protection
Analysis and control
TOPICShare
Large organizations tend to deploy cloud infrastructure across multiple AWS accounts. They do this for a number of reasons—from better segregation between company divisions or environments to compliance with regulatory restrictions such as PCI or SOC 2. In many cases, use of multiple accounts will also result from the organic growth of deployments to the cloud rather than a deliberate choice.
Regardless of how or why the multiple accounts came about, there will always be a need to provide unified monitoring across all of them, and this can be tricky. For example, to report on performance metrics and KPIs across accounts via a central repository would require working through a variety of networking and security headaches. After populating the repository, monitoring data would then need to be organized and presented in a way that supports more effective decision making and troubleshooting by centralized infrastructure and DevOps teams.
In a previous article, we showed how Cloud Insights can be used to rapidly deploy infrastructure monitoring for individual AWS accounts. This article will build upon that foundation to illustrate how the features can also extend to multi-account environments.
Multi-Account AWS Monitoring
The use of multiple AWS accounts provides a robust boundary between cloud infrastructure deployments. It creates complete separation between IAM users and roles, VPCs and provisioned AWS services - such as the kind that would exist with accounts belonging to unrelated organizations. Controlled interoperation among these AWS accounts can be used to support the sharing of data and resources.
Building a central monitoring repository, however, presents a number of challenges that require a significant degree of technical expertise to address. After creating a central AWS account for monitoring, the next step would be to set up a peering relationship between the VPC in the monitoring account and the VPCs in all of the other accounts containing resources that need to be monitored. Routing rules and security groups would then need to be correctly configured to ensure that only the minimal level of required access was granted. Maintaining these types of network configurations in large organizations quickly becomes a burden.
After dealing with network connectivity, IAM roles would be used to delegate access from the monitoring account to each of the other accounts using IAM policies to define the permissions granted in the destination account. A user from the monitoring account would then be set up with permissions to assume the role. In a similar way to the networking setup, this security configuration would need to be managed over time, with the potential for human error leaving the door wide open to security issues.
Then, when these peripheral issues of networking and security have been resolved, the real task of collecting the monitoring data and transporting it to the monitoring account can begin. This process may require stitching together a variety of AWS services, such as Amazon Kinesis, Data Firehose, AWS Lambda, and Amazon S3, in order to create an asynchronous transport process for the monitoring data generated within a given source AWS account to the monitoring AWS account.
AWS CloudWatch
Since the end of 2019 it’s been possible to centralize monitoring across multiple accounts and regions with AWS CloudWatch without needing to manually construct the type of monitoring data repository described above. This new feature of allows you to link various accounts, such as CloudWatch logs, metrics, and alarms, and share data among them. Metrics data from different accounts can also be aggregated to create an overall view of performance across an organization.
In order to make full use of this functionality, users must ensure that CloudWatch has been correctly and consistently configured within each account. It’s also important to invest time into creating the dashboards that allow you to draw value from the available monitoring data. This step requires a significant degree of technical knowledge and skill. If centralized monitoring has to be extended to multi-cloud environments, a more flexible and cloud agnostic solution, such as NetApp Cloud Insights, will likely be required.
NetApp Cloud Insights
Cloud Insights is a cloud-based solution for infrastructure monitoring that works for on-premises, private cloud, and public cloud environments, including AWS, Azure, and Google Cloud. Monitoring data from all of these environments can be centralized in the cloud, providing you with a truly unified view of your infrastructure deployments. This ability to integrate with a diverse set of operational platforms also extends to supporting multiple AWS accounts.
Cloud Insights provides a huge library of preconfigured data collectors, with a collector extracting the relevant metrics for the specific application or infrastructure component to which it applies. This feature allows you to easily collect the data that will produce the most useful insights into your AWS environments.
Infrastructure data collectors are associated with an acquisition unit. This acquisition unit (or AU) collects metrics from cloud providers’ native monitoring APIs, CloudWatch in the case of AWS, and forwards them to the Cloud Insights service. The AU could be deployed in an Amazon EC2 instance within the VPC containing the resources that you need to monitor, with no peering relationships or special IAM permissions that need to be granted. The only requirement is that it must be able to send monitoring data back to Cloud Insights over the internet.
Cloud Insights is all about flexibility and ease of deployment though. As data is collected using a cloud provider’s native APIs, a single AU could even be used to monitor all of your resources, across multiple clouds and on-prem devices, so long as it’s able to access those APIs.
This greatly reduces the potential for opening up access permissions or security vulnerabilities in your network as a result of your efforts to support monitoring. Removing the requirement to change your VPC or IAM configuration also makes maintaining environments monitored with Cloud Insights much easier in the long term.
The Cloud Insights data model makes understanding and reporting on the data collected from multiple AWS accounts, or environments across multiple cloud providers, a breeze. Instead of viewing and monitoring these accounts individually, a consistent data model allows you to create dashboards and reports, queries and policies using a consistent approach. A policy created to monitor virtual machine utilization will do so for virtual machines across multiple AWS accounts, just as it will for Azure VMs, or VMs sitting in your down datacenters on VMware, Hyper-V or OpenStack, to name a few. Similarly, dashboards showing storage utilization can include resources from anywhere – a gigabyte is a gigabyte regardless of whose datacenter it’s sitting in. This lets you use tags and metadata to carve up your views, instead of having restricted monitoring views based on how the environments are built. Siloed resources don’t have to mean siloed monitoring.
Collecting and Centralizing Monitoring Data
Multiple AWS accounts are commonly used in enterprise environments to silo cloud infrastructure deployments. While this structure provides many advantages, it can also complicate the task of setting up centralized monitoring. NetApp Cloud Insights is a compelling solution to this problem. It provides secure support for monitoring a wide variety of application and infrastructure services without unnecessarily compromising the security of your network and access controls by permitting access to external services.
Collecting and centralizing the required monitoring data is only the first stage of building a viable monitoring platform. Cloud Insights’ added value lies in its ability to analyze the collected data to produce real insights that aid in decision making. Having a global view of performance metrics, utilization, and costs across the AWS accounts you are using—including multi-cloud accounts in Azure and Google Cloud Platform—gives you much greater control over your cloud assets.
To find out more, try out Cloud Insights for yourself today with our 30-day free trial.
