CloudWatch Logs Insights is a feature of CloudWatch, a central part of the AWS monitoring ecosystem. You can use Log Insights to search and analyze your log data interactively. It enables you to query your logs and can assist you in responding to operational issues.
In this article, you will learn:
Depending on the services you have deployed and how you are running application code, there are different methods for transferring log data to Logs Insights. Below are two methods you can use.
If you are running EC2 instances you need to use the CloudWatch Agent to access logs. You must install and configure this agent for each instance you are using. After configuration, the agent monitors your local log files and forwards them to CloudWatch Logs.
This agent works for instances running application code, Linux syslogs, and web servers. You can also use it to forward logs from on-premises servers.
If you are using containers for your operations, you need to redirect your logs manually.. This is true for containers managed by both Elastic Kubernetes Service (EKS) and Elastic Container Service (ECS). If you are using Kubernetes on its own, you can redirect logs from the control plane.
If you are running application code in Lambda functions your logs are automatically sent to CloudWatch Logs. This is by default and you cannot change this feature. You can, however, stream your logs to an additional service for processing if you want.
Many of your AWS services automatically send logs to CloudWatch or enable you to set up forwarding easily. Below are three commonly used services that you may need to forward log data from:
As mentioned above, Logs Insights comes with a native query language for evaluating your logs. With this language, you can use multiple commands at a time along with supported functions and operations. To string query commands, you need to separate each with a pipe character (|).
Operations and functions supported include generic, string, datetime, or numeric functions, comparison or arithmetic operations, and regular expressions. The language also supports comments using the hash character (#).
Below you can see descriptions of the primary commands.
|
Command |
Description |
|
display |
Defines the fields to display in a query. You should only use this command once per query since only the last display command is applied. |
|
fields |
Lists the available fields for display from a log. You can also use this command, along with supported operations or functions to create new fields for the query or modify field values. |
|
filter |
Enables you to filter your query according to defined conditions. |
|
stats |
Aggregates statistics of your field values. With this command you can specify groups of values to aggregate by. |
|
sort |
Enables you to sort your returned values in either descending or ascending order. |
|
limit |
Enables you to restrict how many values are returned by your query. |
|
parse |
Enables you to extract data from queried fields for additional queries. This command works with regular and glob expressions. |
When you’re first learning to use Logs Insights, it’s helpful to experiment with sample queries. These can help you gain familiarity with commands, refining your queries, and how results are returned.
Before you can perform a query, you need to have logs available in CloudWatch Logs. If you do not already have logs available you can import a sample log to practice with.
Modifying queries involves altering values as needed from the query editor menu. From the query results that you want to modify, change your values as needed and re-run the query. Your modified results are then displayed.
NetApp Cloud Insights is an infrastructure monitoring tool that gives you visibility into your complete infrastructure. With Cloud Insights, you can monitor, troubleshoot and optimize all your resources including your public clouds and your private data centers.
Cloud Insights helps you find problems fast before they impact your business. Optimize usage so you can defer spend, do more with your limited budgets, detect ransomware attacks before it’s too late and easily report on data access for security compliance auditing.
In particular, NetApp Cloud Insights provides Active IQ predictive analytics, which let you take advantage of prescriptive guidance. You can use this feature to ensure your resources are operating optimally at all times.
Learn more: Effective Troubleshooting in Hybrid Cloud
Start a 30-day free trial of NetApp Cloud Insights. No credit card required
Schedule time to speak with a specialist about how NetApp Cloud Insights can help your organization. Learn how you can better optimize your IT Infrastructure with NetApp Cloud Insights here.