Storage
Mobility
Protection
Analysis and control
TOPICShare
What is Cloud Logging?
Cloud logging enables you to centrally manage log data collected from multiple cloud resources. Log data is essential for measuring and optimizing cloud performance and security. However, it can be extremely difficult to effectively leverage log data from complex multi-cloud architectures. Often, simplicity and granular visibility are critical for ensuring efficient cloud logging.
In this post, we’ll examine key components any multi-cloud logging strategy should account for, including tips for choosing tooling, determining structure, and creating logging processes. We will also show how NetApp Cloud Insights can help you leverage the power of log-based insights to improve cloud performance and billing.
In this article, you will learn:
- What is cloud logging
- How to build a multi-cloud logging strategy
- Top cloud log management services
- Cloud logging best practices
- Cloud logging with NetApp Cloud Insights
What Is Cloud Logging?
Cloud logging is a practice that enables you to collect and correlate log data from cloud applications, services, and infrastructure. It is performed to help identify issues, measure performance, and optimize configurations.
Cloud logging relies on the creation of log files, collections of data that document events occurring in your systems. Log files can contain a wide variety of data, including requests, transactions, user information, and timestamps. The specific data that logs collect is dependent on how your components are configured.
When performing cloud logging, there are several types of logs you should collect. These include event logs, transaction logs, message logs, and audit logs. To make the collection and aggregation process easier, you can use log management tools to ingest, process, and correlate data.
Building a Multi-Cloud Logging Strategy: Tooling, Structure, and Processes
While all cloud logging can be complex, multi-cloud logging strategies can present unique issues due to the distribution of or incompatibilities between services. Consider the following aspects, based on the expertise of Securosis, when creating your logging strategy.
Tooling
The types of tools you use and the methods those tools provide for ingesting data are a significant consideration.
- Data sources—multi-cloud deployments create a wide range of data sources that you need to ingest centrally. This means having the ability to handle a wide range of log formats and aggregating data in real or near-real time. In particular, you need to ensure that tooling supports logs for API calls since most cloud services rely heavily on APIs to communicate.
- Identity and origin—your various cloud services may or may not have consistent identity controls and markers. If not, it can be challenging to track user or application events throughout your services. Additionally, if you are using ephemeral resources, such as serverless infrastructures, you need to differentiate between instances based on time and events. This is because IPs are often and sometimes rapidly reused.
- Scalability and flexibility—tools need to scale to meet incoming data volumes. This means increasing both storage and processing power to ensure that incident detection times are not negatively affected by increased logging. Additionally, tools need to include features for service discovery to ensure that dynamic resources are identified and logged from the time of creation.
Structure
The structure of your multi-cloud logging strategy is determined by the infrastructure you have in place and how your various components are related.
- Accounts and architectures—logging across your accounts and architectures adds complexity in terms of permissions, secure transmission, and log types. Depending on which services you’re using and how they are connected, you may be limited in the sorts of logs that are natively available. This can mean that additional tooling is necessary to collect comparable data from across services.
- Monitoring “up the stack”—many cloud services use managed infrastructure, meaning you are not responsible for and may not have access to infrastructure logs. This means that your logging focus needs to shift up your tech stack, centering on application and resource performance, data availability and security, and costs. While this may create less of a burden on your IT teams, it also limits your visibility and can make it harder to identify or resolve issues.
- Storage and ingestion—where and how you are storing log data is a critical decision. While you can store data on-premises, you are likely to run out of space quickly. In comparison, object storage in the cloud is cheap and highly scalable. However, keep in mind that while storage is cheap, ingestion and transfer may not be. You need to verify how the services or tools you are using access data and how transfer rates may differ for native or non-native services.
Processes
With tooling and infrastructure in place, you can begin accounting for your logging processes. These determine how your data is handled and how systems and teams use data during and after collection.
- Selecting data—the type of data you need to collect is dependent on what you are trying to measure or monitor and what regulatory or contractual agreements you fall under. You also need to account for which specific environments you need to collect data for. For example, it is generally less important to collect comprehensive logs for temporary dev environments than for production ones.
- Speed and timing—the closer you can get to real-time processing and analysis of data, the more meaningful many of your logging processes are. This means you need to prioritize how data is consumed and correlated. For example, logs from dev environments should not take priority over those for mission critical services. Additionally, you may need to adapt new pipelines for processes, such as ones that perform analysis before loading data in storage.
- Automated responses—the ability to automatically respond to alerts and log analyses is vital to preventing service outages and minimizing damage caused by cybersecurity issues. Automation is also key to managing the vast amount of data created by logging and monitoring systems. Without automation, IT teams would not be able to correlate or respond to data efficiently..
Top Cloud Log Management Services
Depending on the makeup of your cloud environment, there are several cloud log management services you can choose from. All major cloud providers offer native log monitoring services or there are third-party cloud logging services to choose from.
- Azure Monitor—a proprietary tool from Microsoft that you can use to monitor Azure applications and services as well as on-premises resources. You can use this service to collect logs and metrics data, analyze trends, and visualize your results. Monitor integrates with a variety of Azure services, including Security Center and Azure Automation.
- AWS Centralized Logging—a proprietary tool from Amazon that enables you to collect, analyze, and visualize log data across deployment regions and accounts. It is based on Elasticsearch and Kibana and can integrate with a variety of AWS services, including CloudFormation. You can also use Centralized Logging to ingest external VPC flow or host-level logs.
- Google Cloud Operations—formerly known as Stackdriver, this is a freemium solution from Google Cloud Platform (GCP) that you can use to monitor cloud resources and manage logging. It includes features for storage, queries, and analysis of both internal and external logs, including on-premises and multi-cloud resources. Operations integrates with other GCP services, including Trace, Debugger, and Error Reporting.
- InsightOps—is a proprietary service from Rapid7 that you can use to centralize, visualize, and monitor log data. You can use it to ingest data from a wide range of cloud services or from on-premises servers. It includes features for normalizing data, queries, visualization, and automation.
Cloud Logging Best Practices
When implementing cloud logging, consider the following best practices:
Simplify your toolset
Try to condense your tooling as much as possible to reduce points of failure and the number of interfaces that teams need to access. Ideally, you should be able to ingest all logs into a centralized solution that supports analysis, visualization, and alerting.
Abstract away log differences
Select tools that can normalize your log data or analyses through abstraction. This makes it easier to query and compare log data and can provide a more comprehensive view of your system conditions and status.
Aim for comprehensive and granular visibility
Logging practices should provide an overview of your cloud resources while allowing you to drill down for specifics. This requires having data sources clearly and consistently identified and collecting data consistently across resources.
Plan for compliance
Logs often contain highly sensitive information that you need to account for when meeting compliance. Additionally, audit logs are typically required to prove compliance through your systems. Having systems in place to secure your log data and ensure that it hasn’t been tampered with are essential.
Cloud Logging with NetApp Cloud Insights
NetApp Cloud Insights is an infrastructure monitoring tool that gives you visibility into your complete infrastructure. With Cloud Insights, you can monitor, troubleshoot and optimize all your resources including your public clouds and your private data centers.
Cloud Insights helps you find problems fast before they impact your business. Optimize usage so you can defer spend, do more with your limited budgets, detect ransomware attacks before it’s too late and easily report on data access for security compliance auditing.
In particular, NetApp Cloud Insights helps you discover your entire hybrid infrastructure, from the public cloud to the data center. With NetApp Cloud Insights you can centralize cloud logging, optimize cloud costs and save money across your environment, by identifying unused resources and right-sizing opportunities.
Recommended Reading
- Optimize Your IT Infrastructure with NetApp Cloud Insights
- What's New With Cloud Insights
- Effective troubleshooting in Hybrid Cloud
