Storage
Mobility
Protection
Analysis and control
TOPICShare
What Is Cloudwatch Log Insights?
CloudWatch Logs Insights is a feature of CloudWatch, a central part of the AWS monitoring ecosystem. You can use Log Insights to search and analyze your log data interactively. It enables you to query your logs and can assist you in responding to operational issues.
In this article, you will learn:
- How to Feed Log Data into CloudWatch Logs Insights?
- CloudWatch Logs Insights Query Syntax
- How to Run and Modify a Sample Query
- Cloudwatch Log Insights with NetApp Cloud Insights
How to Feed Log Data into CloudWatch Log Insights?
Depending on the services you have deployed and how you are running application code, there are different methods for transferring log data to Logs Insights. Below are two methods you can use.
Collect Logs from EC2 or Serverless Functions
If you are running EC2 instances you need to use the CloudWatch Agent to access logs. You must install and configure this agent for each instance you are using. After configuration, the agent monitors your local log files and forwards them to CloudWatch Logs.
This agent works for instances running application code, Linux syslogs, and web servers. You can also use it to forward logs from on-premises servers.
If you are using containers for your operations, you need to redirect your logs manually.. This is true for containers managed by both Elastic Kubernetes Service (EKS) and Elastic Container Service (ECS). If you are using Kubernetes on its own, you can redirect logs from the control plane.
If you are running application code in Lambda functions your logs are automatically sent to CloudWatch Logs. This is by default and you cannot change this feature. You can, however, stream your logs to an additional service for processing if you want.
Collect Logs from Other AWS Services
Many of your AWS services automatically send logs to CloudWatch or enable you to set up forwarding easily. Below are three commonly used services that you may need to forward log data from:
- CloudTrail—you can stream event logs to CloudWatch Logs to collect information about API calls and responses. This requires configuring the service to send events to CloudWatch Logs. You can then use these events to generate alarms.
- VPC—you can stream flow logs to CloudWatch Logs to collect information about traffic health. Keep in mind, this creates a significant amount of data depending on your traffic volume so this should only be used for analytics or auditing purposes.
- Relational Database Service (RDS)—you can configure your individual database engines to forward logs to CloudWatch Logs.
CloudWatch Logs Insights Query Syntax
As mentioned above, Logs Insights comes with a native query language for evaluating your logs. With this language, you can use multiple commands at a time along with supported functions and operations. To string query commands, you need to separate each with a pipe character (|).
Operations and functions supported include generic, string, datetime, or numeric functions, comparison or arithmetic operations, and regular expressions. The language also supports comments using the hash character (#).
Below you can see descriptions of the primary commands.
|
Command |
Description |
|
display |
Defines the fields to display in a query. You should only use this command once per query since only the last display command is applied. |
|
fields |
Lists the available fields for display from a log. You can also use this command, along with supported operations or functions to create new fields for the query or modify field values. |
|
filter |
Enables you to filter your query according to defined conditions. |
|
stats |
Aggregates statistics of your field values. With this command you can specify groups of values to aggregate by. |
|
sort |
Enables you to sort your returned values in either descending or ascending order. |
|
limit |
Enables you to restrict how many values are returned by your query. |
|
parse |
Enables you to extract data from queried fields for additional queries. This command works with regular and glob expressions. |
How to Run and Modify a Sample Query
When you’re first learning to use Logs Insights, it’s helpful to experiment with sample queries. These can help you gain familiarity with commands, refining your queries, and how results are returned.
Before you can perform a query, you need to have logs available in CloudWatch Logs. If you do not already have logs available you can import a sample log to practice with.
Run a Sample Query
- From the CloudWatch console select Insights and locate the query editor at the top of the page. By default your 20 most recent log events are returned.
- Choose the log groups you want to query. You can do this by searching for logs in the available search bar. Once selected, the service automatically detects your log fields.
- Define your time period through the time selector on your screen's upper right, the number of events you want to return, and select Run.
- After your results are returned, you can view your results along with a bar graph of your events over time. If you want to see all fields returned in your events, you can select the icon on the left.
Modify the Sample Query
Modifying queries involves altering values as needed from the query editor menu. From the query results that you want to modify, change your values as needed and re-run the query. Your modified results are then displayed.
AWS Monitoring with NetApp Cloud Insights
NetApp Cloud Insights is an infrastructure monitoring tool that gives you visibility into your complete infrastructure. With Cloud Insights, you can monitor, troubleshoot and optimize all your resources including your public clouds and your private data centers.
Cloud Insights helps you find problems fast before they impact your business. Optimize usage so you can defer spend, do more with your limited budgets, detect ransomware attacks before it’s too late and easily report on data access for security compliance auditing.
In particular, NetApp Cloud Insights provides Active IQ predictive analytics, which let you take advantage of prescriptive guidance. You can use this feature to ensure your resources are operating optimally at all times.
Learn more: Effective Troubleshooting in Hybrid Cloud
Start a 30-day free trial of NetApp Cloud Insights. No credit card required
Schedule time to speak with a specialist about how NetApp Cloud Insights can help your organization. Learn how you can better optimize your IT Infrastructure with NetApp Cloud Insights here.
