Logs

Mission Control provides centralized logging in its Control Plane. A centralized location allows querying for errors, monitoring messages, and finding root causes to issues.

Logging is configured during the installation of Mission Control.

Architecture

Mission Control aggregates, indexes, stores, and accesses logs with queries through Grafana’s User Interface (UI) or Grafana’s Loki API.

Mission Control logging architecture

A Vector DaemonSet on each Kubernetes node actively scrapes the logs from the local pods.

Apache Cassandra® and DataStax Enterprise (DSE) pods monitor the logs on their pods through a Vector Sidecar.

Nodes added to racks, datacenters, and clusters are automatically discovered and scraped for logging and metric information.

Examine logs

Mission Control utilizes the Grafana and Loki tools. In Grafana, two dashboards deploy automatically with Mission Control. Access these from the browser window to which you ported the services.

Grafana queries the system collectd logs and exposes the results in static dashboard visualizations. For a cluster of up to 500 nodes and 90 days of stored logs, query responses return within 100 ms for a query spanning the previous hour with a pod name label or value.

Choose steps in your User Interface (UI) preference or Command Line Interface (CLI).

  • Mission Control UI

  • KOTS Admin UI

  • CLI to Grafana service

  1. Access Mission Control’s UI.

  2. In the Home Clusters dialog, click the target cluster namespace.

  3. Click the Observability tab.

  4. Click the Logs tab.

  5. Review log activity for your cluster.

  6. (Optional): To filter the results:

    • In the Search field, enter a specific node name.

    • In the Node field, select a specific node name.

    • In the Start or End fields, enter the dates for a specific time period.

  7. (Optional): Click the clipboard icon on the row of your target message to copy and locally view log details.

Access the Grafana UI from the browser window to which you ported the services. Grafana queries the collected logs and exposes the results in static dashboard visualizations.

Grafana Loki displays each stream that is sourcing a log. A stream is a unique label that is used to index a log.

  1. Access KOTs' Admin UI.

  2. From the Kubernetes cluster, verify from the mc-loki namespace that the Loki components are installed and in a Running status. Look for a Loki gateway ReplicaSet and both read and write StatefulSets.

  3. In the Admin UI, under Network, open Services and port forward the Loki gateway service and Mission Control Grafana service to Open in Browser.

  4. From the same browser window to which you ported Mission Control microservices, click General.

  5. Choose the appropriate step to view either DSE Logs or Other Logs:

    • Click DSE Logs.

      • In the tabs, choose from the following filtering indexes and their values in the pulldown selections:

        • Cluster.

        • Datacenter.

        • Rack.

        • File.

        • Search text. Mission Control labels the Cassandra or DSE logs with dse, to aid with queries.

    • Click Other Logs.

      • In the tabs, choose from the following filtering indexes and their values in the pulldown selections:

        • Namespace.

        • Pod.

        • Container.

        • Search text.

  1. Port forward to the Grafana service running in Mission Control:

    kubectl port-forward -n mc-mimir svc/mission-control-grafana
  2. From the same browser window to which you ported Mission Control microservices, click General.

  3. Choose the appropriate step to view either DSE Logs or Other Logs:

    • Click DSE Logs.

      • The tabs represent the filtering boundary used to index the data. Choose one and select from its drop-down list.

        • Cluster.

        • Datacenter.

        • Rack.

        • File.

        • Search text. Mission Control labels the Cassandra or DSE logs with dse, to aid with queries.

    • Click Other Logs.

      • The tabs represent the filtering boundary used to index the data. Choose one and select from its drop-down list.

        • Namespace.

        • Pod.

        • Container.

        • Search text.

Interprete Grafana log level indicators

Grafana displays a color to indicate the log level. It bases the color on a specified level label, or a best-guess match after parsing the log’s content.

Log level color indicators

Log level

Color

Critical/Crit/Alert/Fatal/Emerg

Purple

Error/Err/Eror

Red

Warning/Warn

Yellow

Information/Info/Notice

Green

Debug/dbug

Blue

Trace

Light blue

Unknown

Grey

Types of logs

gc

also known as garbage collection. Aids in memory management by listing deallocated programming objects no longer used.

systemd

centralized logs of all Linux kernel and user processes system messages.

debug

aids in diagnosing and fixing bugs and errors.

Mission Control automatically deploys two Grafana dashboards:

  • DSE logs - all the logs collected from DSE.

  • Other logs - the logs collected from Kubernetes (K8s) pods.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com