Using ContainIQ
Access metrics, logs, traces, and profiling instantly and with little overhead. With eBPF, our agent-less approach provides deeper insights and comes with meaningful performance benefits.
Engineering and DevOps teams use ContainIQ to monitor cluster health at a high level and drill down to more detailed views to identify problems and find root causes.
In this section, we provide a quick overview of:
- Continuous Profiling
- Request Tracing
- Latency
- Metrics
- Events
- Logs
Continuous profiling applications help engineering teams spot performance bottlenecks and troubleshoot issues faster. And for companies running applications in Kubernetes, profiling is a modern and essential practice for catching and improving upon resource-consuming application code.
ContainIQ offers an eBPF-based profiler for continuous profiling with minimal overheard. With ContainIQ’s profiler, engineers can improve the mean time to resolution, quickly spot performance issues, and identify ways to reduce unnecessary cloud spending. And it works out of the box without additional instrumentation.
As seen above, with ContainIQ, users can view all stack traces from individual applications. This helps identify the functions which are being called most frequently and identify performance bottlenecks. On the horizontal axis, you have the sample population, and on the vertical axis, you have the stack trace. By clicking on an individual stack frame, users can see all of the children of the stack. Or in other words, users can identify slow functions and functions that are consuming the majority of their CPU.
ContainIQ offers support for C/C++, Go, Rust, Python, Ruby, and Node.
You can hover over an individual stack and see the process name as well as the name of the specific function. And you can regular expression search on both the pod and the process.
Use the Node and Namepace filters to filter data by node and namespaces, either individually or as a grouping.
Filtering using the date-range filter to view data over time and in specific time periods.
ContainIQ’s tracing feature set and dashboard give users real-time tracing data for all URL paths, without having to instrument each microservice at the application level. Graph P99, P95, average latency, and error rate over time for each URL path or use the table to analyze trace details such as HTTP method, HTTP status code for each request. To correlate logs to a specific path, click into trace from the table to reveal the corresponding logs for a given point in time.
To correlate logs for a specific trace, users will need to select a trace from the table below the graph or search for a specific trace using the search feature located at the top of the dashboard. Once the trace is found, click and a drop-down will appear with all associated logs for that trace at the given point in time.
Users can view request and response payloads for each trace. To view payload details, users can click on an individual request and see both the request and response payloads. Additionally, payloads with larger volumes of data can be opened into a separate modal to he
Users can search traces by timestamp, or URL path. Paths can be searched for individually or grouped with regular expression. Users are also able to filter traces by cluster, namespace, or time range.
Advanced filtering allows users to filter by a number of different variables: path, pod name, protocol, method, status code, and latency timestamps. Users can build queries using single variables or by combining multiple variables.
ContainIQ’s latency dashboard allows users to graph and set alerts P95 latency, P99 latency, avg. latency and requests per second across all of their different Kubernetes services. No application package installations necessary.
To view latency by microservice, users can navigate to our Latency dashboard where they are shown a list of all active microservices and their corresponding P95 latency, P99 latency, avg. latency and requests per second.
Our graph view can help users quickly identify spikes in latency across each microservice. To graph these metrics over time, users can first select the desired microservice from the drop-down inside the graph. Then users can specify the time range to look into by using the time range selector, located in the top right corner of the navigation. Here users can specify the exact time range they’d like to look into, as well as, view data by minute, hour, or day.
By clicking on a microservice from our Latency dashboard, users will be presented with a modal where they can see requests per second and various latency metrics for each URL path or API endpoint within the given microservice. Users are able to search through endpoints by utilizing the search bar, or by manually paginating through, in addition to search, users can filter endpoint metrics by the past hour, day.
ContainIQ’s metrics dashboards give users a comprehensive view of cluster health. Helping users understand both the real-time and historical state of their clusters. These dashboards offer three charts: our hive chart (real-time metrics, conditions, events), historical pod and node metrics (view CPU and Memory usage over time, view CPU and Memory utilization), and Avg. CPU and Memory (view spikes in CPU or Memory over the last hour).
By using our hive chart, users can view real-time pod and node CPU and memory, color-coded based on relative usage. From here, users are able to view pod/node conditions, view events related to a specific pod or node, and dive into what pods are assigned to what node.
The two graphs under the hive chart show available vs. used CPU and Memory limits over time. These metrics can be used to rightsize clusters or monitor resource limits that have already been set. Users can view this data by hour, day, or week.
The bottom two graphs can be used to identify any spikes in CPU and Memory usage across all pods or nodes over the last hour.
ContainIQ’s events dashboard enables users to view events over time, search for specific events, filter based on event type, filter based on namespace, and set alerts for specific events like crash loops, pod evictions, and failed Kubernetes jobs.
For debugging purposes, users are able to easily correlate pod events to logs directly from the events dashboard. Simply click on the description of the pod event and ContainIQ will take users to the logs for that specific point in time.
By default, the events dashboard shows all events (normal and warning) users can choose to view both or look into all warning events or normal events. To do this simply click the event type filter in the secondary navigation bar and choose either event type: all, normal, or warning.
Users can view events by namespace. Instead of sifting through events, users can drill down into a specific namespace and view and search events for a particular namespace.
Along with our filters users can search events by name or description. Users can search events across their entire cluster or filter by namespace, event type, and time range.
ContainIQ automatically collects anything logged inside a user’s cluster encompassing both internal applications and Kubernetes system components. Users can search through logs by cluster, pod, container, message, timestamp, regex, or JSON path.
ContainIQ offers both basic and advanced log search functionality. Basic log search allows users to search across their clusters by pod, message, and time range. To do this users can use the search located in the navigation bar or click into a pod to get all corresponding logs for that specific pod.
Users can also take their search queries one step further with our advanced search functionality. Here users are able to build log queries by pod name, message, container name, stream, or JSON message.
Searching by JSON message allows users to parse by key-value pairs, making it easier to aggregate and filter JSON formatted log messages.
Users can view surrounding logs to help identify the root cause of a error or anomaly, while filtering out all non-matching lines. View between 10-30 log rows leading up to and after a specific error message for faster remediation. To view surorunding logs directly related to an issue:
- Use basic or advance search to target the the log line
- Click into the line to view the full message and all corresponding details
- To view surrounding logs click the ellipses button (furthest to the right)
- You will then be taken to a secondary dashboard where you can see all surrounding logs, from here you can view between 10- 30 surrounding logs
Users can leverage the Health dashboard to get a clear view of the health and status of each deployment, StatefulSet, and DaemonSet (workloads). The Health dashboard allow users to map deployment health directly to changes in their codebase, view corresponding deployment events as well as, compare CPU and Mem limits of each deployment over specific periods of time.
To view corresponding deployment events users should select a deployment and click the pop-out icon in the upper-right corner of the deployment card. Once clicked the user will be navigated to a secondary dashboard that shows all events for the specific deployment. To return back to the health dashboard, simply click the back arrow in the secondary nav (upper-left corner).
When a user encounters unhealthy deployments or workloads, they are able to map these back to changes in a Github repository and commit. To map these changes users will need to add the annotations below to each workload:
To view these changes in Github, after the annotations are added to the workload, click the label tagged with "app" and you will be directed directly to the commit.
Users can set up monitors to send alerts across all data points (ex. P95 latency, a Kubernetes job failing, a pod eviction, etc).
To deliver alerts, users must integrate ContainIQ with Slack and set up a channel for alerts to feed into. To do this navigate to the user settings page, on the left navigation click into the “Integration” tab and then click “Activate” under actions. From here, the user will be prompted to choose the appropriate Slack channel they would like to integrate with, as well as, selecting or creating the channel the alerts should feed into. Once a channel is selected or a new channel is created, click allow to enable the ContainIQ and Slack integration.
Monitors are easy to set up. Click the “New Monitor” button located in the uppermost navigation bar. From here, the user will be prompted to create a new monitor. Simply choose the alert type (event, metric, or logs) and then set appropriate variables.
Creating an Event Monitor
Event monitors are the most straightforward to set up. To create an event monitor users will need to do the following:
- Create a “New Monitor”
- Choose alert type - Event
- Select a condition (contains or any)
- Input the event reason to be alerted on i.e. Backoff
- To target a specific object (container, pod name, node, etc) enter the name of the object
- *To target all events across all clusters, select “Any” from the "Event Object Name" dropdown
- Select a condition (contains or any)
- Name the monitor
- Press "Create Monitor"
Any new monitor that is created will be turned on by default.
Creating a Log Monitor
To create a log monitor users will need to do the following:
- Create a “New Monitor”
- Choose alert type - Log
- Input the message query to be alerted on
- Set the alert trigger
- input number of occurances over a specific timeframe (minutes)
- Name the monitor
- Press "Create Monitor"
Any new monitor that is created will be turned on by default.
Creating a Metric Threshold Monitor
To create a metric threshold monitor users will need to do the following:
- Create a “New Monitor”
- Choose between metric alert types
- Metric threshold
- Metric % change
- Select what metric to be alerted on (pod CPU, node CPU, pod memory, node memory, avg. latency, P95 latency, or P99 latency)
- Input the name of the object you are targeting (pod name or node name)
- Enter the threshold conditions to trigger the alert
- CPU or Mem threshold over a specific timeframe (minutes)
- Name the monitor
- Press "Create Monitor"
Any new monitor that is created will be turned on by default.
Creating a Metric % Change Monitor
To create a metric % change monitor users will need to do the following:
- Create a “New Monitor”
- Choose between metric alert types
- Metric threshold
- Metric % change
- Select what metric to be alerted on (pod CPU, node CPU, pod memory, node memory)
- Input the name of the object you are targeting (pod name or node name)
- Enter the threshold conditions to trigger the alert
- % change over a specific timeframe (minutes)
- Name the monitor
- Press "Create Monitor"
Any new monitor that is created will be turned on by default.
Monitors can be turned off by way of the Monitors dashboard. This dashboard provides a list of all monitors, from here users can see monitor name, type, toggle notifications on or off, and edit existing monitors.
To turn off a monitor, simply click the toggle located on the right side of the table. ContainIQ also allows users to perform bulk actions on this dashboard, users can select multiple monitors and turn them on or off, or delete them.
