Logs
Mission Control provides centralized logging in its Control Plane. A centralized location allows querying for errors, monitoring messages, and finding root causes to issues.
Logging is configured during the installation of Mission Control.
Architecture
Mission Control aggregates, indexes, stores, and accesses logs with queries through Grafana’s User Interface (UI) or Grafana’s Loki API.
A Vector DaemonSet on each Kubernetes node actively scrapes the logs from the local pods.
Apache Cassandra® and DataStax Enterprise (DSE) pods monitor the logs on their pods through a Vector Sidecar.
Nodes added to racks, datacenters, and clusters are automatically discovered and scraped for logging and metric information.
Examine logs
Mission Control utilizes the Grafana and Loki tools. In Grafana, two dashboards deploy automatically with Mission Control. Access these from the browser window to which you ported the services.
Grafana queries the system collectd
logs and exposes the results in static dashboard visualizations. For a cluster of up to 500 nodes and 90 days of stored logs, query responses return within 100 ms for a query spanning the previous hour with a pod name label or value.
Choose steps in your User Interface (UI) preference or Command Line Interface (CLI).
-
Mission Control UI
-
KOTS Admin UI
-
CLI to Grafana service
-
In the Home Clusters dialog, click the target cluster namespace.
-
Click the Observability tab.
-
Click the Logs tab.
-
Review log activity for your cluster.
-
(Optional): To filter the results:
-
In the Search field, enter a specific node name.
-
In the Node field, select a specific node name.
-
In the Start or End fields, enter the dates for a specific time period.
-
-
(Optional): Click the clipboard icon on the row of your target message to copy and locally view log details.
Access the Grafana UI from the browser window to which you ported the services. Grafana queries the collected logs and exposes the results in static dashboard visualizations.
Grafana Loki displays each stream that is sourcing a log. A stream is a unique label that is used to index a log.
-
From the Kubernetes cluster, verify from the
mc-loki
namespace that the Loki components are installed and in aRunning
status. Look for a Loki gateway ReplicaSet and both read and write StatefulSets. -
In the Admin UI, under Network, open Services and port forward the Loki gateway service and Mission Control Grafana service to Open in Browser.
-
From the same browser window to which you ported Mission Control microservices, click General.
-
Choose the appropriate step to view either DSE Logs or Other Logs:
-
Click DSE Logs.
-
In the tabs, choose from the following filtering indexes and their values in the pulldown selections:
-
Cluster.
-
Datacenter.
-
Rack.
-
File.
-
Search text. Mission Control labels the Cassandra or DSE logs with
dse
, to aid with queries.
-
-
-
Click Other Logs.
-
In the tabs, choose from the following filtering indexes and their values in the pulldown selections:
-
Namespace.
-
Pod.
-
Container.
-
Search text.
-
-
-
-
Port forward to the Grafana service running in Mission Control:
kubectl port-forward -n mc-mimir svc/mission-control-grafana
-
From the same browser window to which you ported Mission Control microservices, click General.
-
Choose the appropriate step to view either DSE Logs or Other Logs:
-
Click DSE Logs.
-
The tabs represent the filtering boundary used to index the data. Choose one and select from its drop-down list.
-
Cluster.
-
Datacenter.
-
Rack.
-
File.
-
Search text. Mission Control labels the Cassandra or DSE logs with
dse
, to aid with queries.
-
-
-
Click Other Logs.
-
The tabs represent the filtering boundary used to index the data. Choose one and select from its drop-down list.
-
Namespace.
-
Pod.
-
Container.
-
Search text.
-
-
-
Interprete Grafana log level indicators
Grafana displays a color to indicate the log level. It bases the color on a specified level label, or a best-guess match after parsing the log’s content.
Log level |
Color |
---|---|
Critical/Crit/Alert/Fatal/Emerg |
Purple |
Error/Err/Eror |
Red |
Warning/Warn |
Yellow |
Information/Info/Notice |
Green |
Debug/dbug |
Blue |
Trace |
Light blue |
Unknown |
Grey |
Types of logs
- gc
-
also known as garbage collection. Aids in memory management by listing deallocated programming objects no longer used.
- systemd
-
centralized logs of all Linux kernel and user processes system messages.
- debug
-
aids in diagnosing and fixing bugs and errors.
Mission Control automatically deploys two Grafana dashboards:
-
DSE logs - all the logs collected from DSE.
-
Other logs - the logs collected from Kubernetes (K8s) pods.