Install Mission Control with Helm
To prepare a Kubernetes cluster for Mission Control, you must create a namespace, configure an IBM Container Registry (ICR) image pull secret, and then install Mission Control with Helm from the OCI registry.
If you have an existing Replicated installation, see Migrate your Mission Control installation from Replicated to ICR.
Image pulls require a Kubernetes docker-registry secret that carries an IBM entitlement key.
Contact IBM Support if you don’t have an entitlement key or you receive container authorization errors.
The Helm chart is published to the OCI registry at:
oci://icr.io/mission-control-helm/mission-control
The chart references image pull secrets through the following value in values.yaml:
global:
imageConfig:
defaults:
pullSecrets:
- "{{ .Release.Name }}-registry"
By default, the chart derives the pull secret name from the Helm release name at installation time.
You do not need to follow this pattern.
You can use any secret name by overriding global.imageConfig.defaults.pullSecrets in a values file.
For more information, see Install Mission Control with Helm.
|
The value of |
Automatic pull secret replication
Mission Control automatically replicates every pull secret listed in global.imageConfig (both defaults.pullSecrets and any per-image pullSecrets) into project namespaces as clusters are provisioned.
You only need to create the secret once in the namespace where the operator is installed.
The operator propagates it wherever it is needed.
If you need to install Mission Control in environments where cluster-scoped resources must be managed separately from the main installation, see Install Mission Control with Helm using separate cluster resources.
For organizations that require separation of cluster administration and application management, see Install Kubernetes cluster-level resources separately for information about installing Kubernetes cluster-level resources in a separate chart.
Prerequisites
Before you begin, ensure that you have:
-
kubectlinstalled and configured to point at a cluster. -
A prepared installation environment on your existing Kubernetes cluster.
-
An IBMid: An IBMid account with MFA enabled is required. If you don’t have one, create an IBMid account. If your organization uses Enterprise Federation (EF) for authentication with corporate credentials, see the IBMid EF documentation.
-
Helm version 3.14.0 to 3.18.0 installed.
-
An IBM entitlement key.
-
Go to IBM Container Library.
-
Sign in with an IBM ID.
-
Click Get entitlement key (or Copy key if one already exists).
-
-
cert-managerinstalled in the cluster (see Configure cert-manager).
For information about security configurations, see Security overrides.
Set environment variables
Set these variables once in a shell. Every command in this guide uses them, so you can copy and paste commands directly without editing placeholders.
|
Environment variables set in a terminal session do not persist when you open a new terminal. If you close your terminal or start a new session, re-export all variables before running any commands. |
export MC_NAMESPACE=NAMESPACE
export MC_RELEASE_NAME=RELEASE_NAME
export MC_VERSION=VERSION
export PULL_SECRET_NAME=SECRET_NAME
export ENTITLEMENT_KEY=ENTITLEMENT_KEY
export IBM_EMAIL=EMAIL
Replace the following:
-
NAMESPACE: Kubernetes namespace for Mission Control installation. -
RELEASE_NAME: Helm release name. -
VERSION: Mission Control Helm chart version to install. -
SECRET_NAME: Name for the image pull secret.By default, the chart expects a secret named
${MC_RELEASE_NAME}-registry. If you setPULL_SECRET_NAMEto that value (for example,export PULL_SECRET_NAME=${MC_RELEASE_NAME}-registry), you do not need a values file override. If you choose a different name, you must set it inoverrides.yamlas described in Install Mission Control with Helm. -
ENTITLEMENT_KEY: IBM entitlement key. -
EMAIL: Email address associated with the IBM account.
Create the target namespace
All resources, including the pull secret and the Helm release, must reside in the same namespace.
-
Create the namespace:
kubectl create namespace $MC_NAMESPACE -
Verify that the namespace exists:
kubectl get namespace $MC_NAMESPACE
Configure pod-to-pod routing
Ensure that all database pods can route to each other. This is a critical requirement for proper operation and data consistency.
The requirement applies to:
-
All database pods within the same region or availability zone.
-
All database pods across different availability zones within the same region.
-
All database pods across different regions for multi-region deployments.
-
All database pods across different racks in the same datacenter for multi-region deployments.
The way you configure pod-to-pod routing depends on your cluster architecture:
- Single-cluster deployments
-
The cluster’s Container Network Interface (CNI) typically provides pod-to-pod network connectivity for database pods within a single Kubernetes cluster. You usually need no additional configuration beyond standard Kubernetes networking.
- Security considerations for shared clusters
-
If your database cluster shares a Kubernetes cluster with other applications, implement security controls to prevent unauthorized access to database internode ports (7000/7001):
-
NetworkPolicy isolation (required): Use Kubernetes NetworkPolicy to restrict access to internode ports to only authorized database pods. NetworkPolicy prevents other applications in the cluster from accessing these ports even if underlying firewall rules are broad.
-
Internode TLS encryption (required): Enable internode TLS to protect data in transit and prevent unauthorized nodes from joining the cluster.
-
Dedicated node pools (recommended): Consider dedicated node pools or subnets for database workloads to enable more granular firewall controls at the infrastructure level.
-
- Multi-cluster deployments
-
For database pods that span multiple Kubernetes clusters, NetworkPolicy alone doesn’t provide sufficient connectivity. You must establish Layer 3 network connectivity or overlay connectivity between the database pod networks (pod CIDRs or node subnets depending on your deployment). Kubernetes NetworkPolicy operates only within a single cluster boundary and can’t provide cross-cluster connectivity.
-
Choose one of the following approaches to establish pod network connectivity across clusters:
-
Routed pod CIDRs (recommended): Use cloud provider native routing solutions when your platform supports them. This approach provides the best performance and simplest operational model.
-
AWS: VPC Peering, Transit Gateway, or AWS Cloud WAN.
-
Azure: VNet Peering or Virtual WAN.
-
GCP: VPC Peering or Cloud VPN.
-
-
Submariner: Open-source multi-cluster connectivity solution, common in OpenShift multi-cluster deployments. Submariner provides encrypted tunnels between clusters. For more information, see the Submariner documentation.
-
Cilium Cluster Mesh: For clusters that use Cilium CNI. For more information, see the Cilium documentation.
-
Cilium Cluster Mesh provides native multi-cluster networking with Cilium.
-
Cilium Cluster Mesh enables pod-to-pod connectivity across clusters.
-
-
-
After you establish cross-cluster connectivity, implement the following security measures. Traditional firewall rules alone lack application awareness and can’t distinguish between different pods or services within a cluster. Use Kubernetes NetworkPolicy for pod-level access control within clusters, and combine it with network-level firewalls for defense in depth.
-
Enable internode TLS encryption to protect data in transit between clusters.
-
Configure firewall rules at the network level to restrict traffic between cluster pod CIDRs.
-
Use NetworkPolicy within each cluster to further restrict access to database ports.
-
Consider using dedicated subnets or VPCs for database clusters to enable network-level isolation.
-
-
To verify that pod-to-pod routing has been configured properly, do the following:
-
Test connectivity between database pods using
nodetool statusorcqlsh. -
Check that all nodes can see each other in the cluster topology.
-
Monitor for connection errors or timeouts in database logs.
-
Verify that gossip protocol communication functions correctly.
If pod-to-pod routing isn’t implemented correctly, you might experience the following:
-
Connectivity issues between database pods.
-
Cluster instability.
-
Data consistency issues.
-
Failed replication.
-
Incomplete or failed cluster operations.
Configure cert-manager
Mission Control uses Cert Manager to handle TLS certificate issuance and automation, making cert-manager a prerequisite for installation.
Before installing Mission Control, you must install and configure cert-manager to ensure proper cleanup of certificate secrets when Mission Control clusters are deleted.
|
For OpenShift environments, install |
Mission Control deletes the certificate objects when you delete a MissionControlCluster resource, which then deletes all the certificate secrets.
However, `cert-manager’s default settings leave the generated secrets behind when you delete the upstream certificates.
To ensure proper cleanup of certificate secrets when you delete Mission Control clusters, configure cert-manager to add ownership references on the secrets it generates.
- Helm installation
-
helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --version VERSION \ --set 'extraArgs[0]=--enable-certificate-owner-ref=true'Replace
VERSIONwith the version ofcert-manageryou want to install. Mission Control requires version 1.18.5 or later. - Kustomize installation
-
Add the following argument to the
cert-managerdeployment in thecert-manager-controllercontainer:- '--enable-certificate-owner-ref=true' - OpenShift installation
-
The OpenShift
cert-managerOperator doesn’t automatically delete certificate secrets when you removeCertificateresources. You must clean up secret resources manually withoccommands.-
Find the name of the
Certificateresources you want to delete:oc get certificate CERTIFICATE_NAME -n NAMESPACE -o yaml | grep "secretName"Replace the following:
-
CERTIFICATE_NAME: The name of theCertificateresource -
NAMESPACE: The namespace of theCertificateresource
-
-
Check if any resources are using the secret:
oc get all -n NAMESPACE -o custom-columns=KIND:.kind,NAME:.metadata.name --all-namespaces | xargs -L1 oc get -n NAMESPACE -o yaml | grep -B 2 -A 5 SECRET_NAMEReplace the following:
-
SECRET_NAME: The name of the secret resource -
NAMESPACE: The namespace of the secret resource
-
-
If no other resources are using the secret, delete it:
oc delete secret SECRET_NAME -n NAMESPACEReplace the following:
-
SECRET_NAME: The name of the secret resource -
NAMESPACE: The namespace of the secret resource
-
-
Wait for cert-manager to be ready
cert-manager deploys three components.
Wait for each rollout to complete before you proceed:
kubectl rollout status deployment cert-manager -n cert-manager
kubectl rollout status deployment cert-manager-cainjector -n cert-manager
kubectl rollout status deployment cert-manager-webhook -n cert-manager
All three commands should exit with successfully rolled out before you continue.
Use a custom Certificate Authority
You can use a custom Certificate Authority (CA) to generate certificates for TLS communication between Mission Control components (excluding webhooks and databases).
To use a custom CA, do the following:
-
Create a secret in the same namespace where you install Mission Control. The secret must contain the certificate data under the default
tls.crt,tls.key, and optionallyca.crtkeys.kubectl create secret tls CUSTOM_ROOT_CA_SECRET \ --cert=path/to/tls.crt \ --key=path/to/tls.key \ -n mission-controlReplace
CUSTOM_ROOT_CA_SECRETwith your desired secret name. -
In the Mission Control
values.yamlfile, disable root CA generation and reference your secret:tls: generateCa: false rootCaSecretName: "CUSTOM_ROOT_CA_SECRET"Replace
CUSTOM_ROOT_CA_SECRETwith the name of your secret.
During Helm installation, Mission Control creates an issuer that points to your secret and uses it to generate all required leaf certificates.
Configuration values
You can configure a Helm chart installation using command-line flags or a supplied value file.
Helm structures its values.yaml file with the top-level chart’s values placed at the root of the file and sub-charts placed under a top-level key matching the chart’s name or alias.
DataStax recommends placing all required configuration values within a values.yaml file.
This allows simple versioning and configuration iteration.
You can download a sample values.yaml file with default Mission Control settings.
Common configurations
Helm simplifies Kubernetes application management by using configuration files to define and deploy resources.
Use the values.yaml file to customize your deployment and configure each component to your specific needs.
The following sections describe common configuration keys and values for each chart and its dependencies. See the linked default templates for additional configuration options.
# -- Determines if the mission-control-operator should be installed as the control plane
# or if it's simply in a secondary cluster waiting to be promoted
controlPlane: true
disableCertManagerCheck: true
# -- Node labels for operator pod assignment.
# Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
# When set, Mission Control components (UI, API, observability stack) are scheduled on nodes with this label.
# Configure your sub-charts (Loki, Mimir, Grafana) with matching nodeSelector values to ensure all platform components use the same nodes.
nodeSelector:
mission-control.datastax.com/role: platform
# -- Controls whether operator components (like Reaper) can run on database-labeled nodes.
# When false (default), these components require the platform label.
# Set to true to allow operator components to schedule on database nodes.
allowOperatorsOnDatabaseNodes: false
client:
# -- Automatically handle CRD upgrades
manageCrds: true
# -- Platform services ingress configuration
# Enables unified access to Mission Control UI, Grafana, and Vector aggregator through domain-based routing
ingress:
# -- Enable ingress for all platform services
enabled: false
# -- top-level domain for Mission Control interface (e.g., mc.region.example.com)
regionDomain: ""
# -- Wildcard domain for service and database subdomains (for example, *.mc.region.example.com)
# Used for Grafana (grafana.mc.region.example.com) and database (my-cluster.my-project.mc.region.example.com)
wildcardDomain: ""
ui:
enabled: true
# -- Base URL for the UI, required when using Ingress or external authentication (OIDC)
# Example: https://mission-control.example.com
baseUrl: ""
ingress:
# -- Enable Ingress for UI access (recommended for production)
enabled: false
# hosts:
# - host: mission-control.example.com
# paths:
# - path: /
# pathType: Prefix
# tls:
# - secretName: mission-control-tls
# hosts:
# - mission-control.example.com
https:
# -- Enable HTTPS for the UI using self signed certificate
enabled: true
Configure custom alerting rules
In version 1.15.0 and later, Mission Control uses a two-tier alerting rules structure that separates automatically managed default rules from user-managed custom rules.
You can configure custom alerting rules through Helm values using the customRules field:
alerting:
customRules: |
- alert: CustomAlert
annotations:
context: ""
description: "Custom alert triggered"
summary: "Custom alert triggered"
expr: your_promql_expression
for: 5m
labels:
group: ""
severity: warning
For more information, see Create custom alerting rules.
Configure sub-charts
This section details the configuration options for various sub-charts included in Mission Control.
Refer to the upstream Helm chart repository for each subchart for a complete list of available configuration options.
Dex IdP
You can find Dex IdP upstream configuration keys in the Dex IdP Helm chart repo.
|
Dex requires at least one authentication connector to start.
Without it, the Dex pod fails with |
Place these entries under the dex key in your values.yaml file.
dex:
config:
enablePasswordDB: true
staticPasswords:
- email: admin@example.com
hash: "HASH"
username: admin
userID: "USER_ID"
Replace the following:
-
HASH: The bcrypt hash of the password.On Unix and Linux systems, you can generate this with the following command:
echo yourPassword | htpasswd -BinC 10 admin | cut -d: -f2 -
USER_ID: A unique identifier for the user.
Grafana
You can find Grafana upstream configuration keys in the Grafana Helm chart repo.
Place these entries under the grafana key in your values.yaml file.
grafana:
enabled: false
K8ssandra operator
Place these entries under the k8ssandra-operator key in your values.yaml file.
k8ssandra-operator:
disableCrdUpgraderJob: true
cass-operator:
disableCertManagerCheck: true
Loki
You can find Loki upstream configuration keys in the Loki Helm chart repo.
Place these entries under the loki key in your values.yaml file.
loki:
enabled: true
loki:
storage:
bucketNames:
chunks: LOKI_CHUNKS_BUCKET
limits_config:
retention_period: 7d
read:
persistence:
enabled: true
size: 10Gi
storageClassName: ""
replicas: 1
write:
persistence:
enabled: true
size: 10Gi
storageClassName: ""
replicas: 1
backend:
replicas: 1
This configuration uses a local storage backend. You can extend it with one of the following backends: S3, GCS, or Azure blob storage.
Configure Loki with S3 storage
To back Loki with an S3 bucket using Kubernetes secrets for credentials, do the following:
-
Create a Kubernetes secret with your S3 credentials:
kubectl create secret generic SECRET_NAME -n mission-control \ --from-literal=SECRET_KEY_ACCESS_KEY_ID=ACCESS_KEY_ID \ --from-literal=SECRET_KEY_SECRET_ACCESS_KEY=SECRET_ACCESS_KEYReplace the following:
-
SECRET_NAME: The name of your secret. For example,loki-s3-secrets. -
SECRET_KEY_ACCESS_KEY_ID: The key for your access key ID. For example,s3-access-key-id. -
ACCESS_KEY_ID: Your AWS access key ID. -
SECRET_KEY_SECRET_ACCESS_KEY: The key for your secret access key. For example,s3-secret-access-key. -
SECRET_ACCESS_KEY: Your AWS secret access key.
-
-
Configure the
backendsection to reference the secret values through environment variables:loki: enabled: true loki: storage: bucketNames: chunks: S3_BUCKET_NAME s3: accessKeyId: "${ENV_VAR_ACCESS_KEY}" secretAccessKey: "${ENV_VAR_SECRET_KEY}" endpoint: S3_ENDPOINT insecure: false region: AWS_REGION s3: s3.AWS_REGION.amazonaws.com s3ForcePathStyle: false type: s3 backend: replicas: 1 extraArgs: - '-config.expand-env=true' extraEnv: - name: ENV_VAR_ACCESS_KEY valueFrom: secretKeyRef: name: SECRET_NAME key: SECRET_KEY_ACCESS_KEY_ID - name: ENV_VAR_SECRET_KEY valueFrom: secretKeyRef: name: SECRET_NAME key: SECRET_KEY_SECRET_ACCESS_KEYReplace the following:
-
S3_BUCKET_NAME: The name of your S3 bucket. -
ENV_VAR_ACCESS_KEY: The environment variable name for the access key. For example,AWS_ACCESS_KEY_ID. -
ENV_VAR_SECRET_KEY: The environment variable name for the secret key. For example,AWS_SECRET_ACCESS_KEY. -
S3_ENDPOINT: The endpoint for your S3 bucket. This value is optional for Amazon S3 buckets. The default endpoint iss3.AWS_REGION.amazonaws.com. -
AWS_REGION: The AWS region for your S3 bucket. -
SECRET_NAME: The name of your Kubernetes secret. For example,loki-s3-secrets. -
SECRET_KEY_ACCESS_KEY_ID: The key in the secret for the access key ID. For example,s3-access-key-id. -
SECRET_KEY_SECRET_ACCESS_KEY: The key in the secret for the secret access key. For example,s3-secret-access-key.
The
extraArgs: ['-config.expand-env=true']configuration underbackendenables Loki to expand environment variables in the configuration. This is required when using Kubernetes secrets for S3 credentials. -
For more information on configuring Loki with S3, including service account settings, see the Grafana documentation.
Configure Loki with GCS storage
To back Loki with a GCS bucket, first create a secret named loki-secrets in the mission-control namespace with the GCP service account JSON stored as a gcp_service_account.json key:
apiVersion: v1
kind: Secret
metadata:
name: loki-secrets
namespace: mission-control
data:
gcp_service_account.json: >-
IHsgICAid..........vbSIgfQ==
type: Opaque
Adjust the Loki values using the following snippet.
If you created the secret with a different name, update the secretName references accordingly.
loki:
backend:
extraEnv:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/loki_secrets/gcp_service_account.json
extraVolumeMounts:
- mountPath: /etc/loki_secrets
name: loki-secrets
extraVolumes:
- name: loki-secrets
secret:
items:
- key: gcp_service_account.json
path: gcp_service_account.json
secretName: loki-secrets
persistence:
size: 30Gi
storageClass: ""
volumeClaimsEnabled: "1"
enabled: true
gateway:
extraEnv:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/loki_secrets/gcp_service_account.json
extraVolumeMounts:
- mountPath: /etc/loki_secrets
name: loki-secrets
extraVolumes:
- name: loki-secrets
secret:
items:
- key: gcp_service_account.json
path: gcp_service_account.json
secretName: loki-secrets
loki:
commonConfig:
replication_factor: 1
compactor:
retention_enabled: true
shared_store: gcs
working_directory: /var/loki/retention
limits_config:
ingestion_burst_size_mb: 2000
ingestion_rate_mb: 1000
retention_period: 7d
rulerConfig:
storage:
local:
directory: /var/loki/rules
type: local
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
storage:
bucketNames:
chunks: GCS_BUCKET_NAME
gcs:
insecure: true
type: gcs
read:
extraEnv:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/loki_secrets/gcp_service_account.json
extraVolumeMounts:
- mountPath: /etc/loki_secrets
name: loki-secrets
extraVolumes:
- name: loki-secrets
secret:
items:
- key: gcp_service_account.json
path: gcp_service_account.json
secretName: loki-secrets
persistence:
enabled: "1"
size: 30Gi
storageClassName: ""
replicas: 1
sidecar:
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
write:
extraEnv:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/loki_secrets/gcp_service_account.json
extraVolumeMounts:
- mountPath: /etc/loki_secrets
name: loki-secrets
extraVolumes:
- name: loki-secrets
secret:
items:
- key: gcp_service_account.json
path: gcp_service_account.json
secretName: loki-secrets
persistence:
size: 30Gi
storageClass: ""
volumeClaimsEnabled: "1"
replicas: 1
Replace GCS_BUCKET_NAME with the name of your GCS bucket.
Configure Loki with Azure storage
To back Loki with Azure blob storage, add the following to your Loki configuration:
loki:
loki:
storage:
type: azure
bucketNames:
chunks: LOKI_BUCKET_NAME
azure:
accountName: STORAGE_ACCOUNT_NAME
accountKey: STORAGE_ACCOUNT_KEY
endpoint_suffix: STORAGE_ACCOUNT_ENDPOINT_SUFFIX
structuredConfig:
storage_config:
azure:
container_name: LOKI_BUCKET_NAME
Replace the following:
-
LOKI_BUCKET_NAME: The name of your Azure blob storage container. -
STORAGE_ACCOUNT_NAME: The name of your Azure storage account. -
STORAGE_ACCOUNT_KEY: The access key for your Azure storage account. -
STORAGE_ACCOUNT_ENDPOINT_SUFFIX: The endpoint suffix for your Azure storage account.
Mimir
You can find Mimir upstream configuration keys in the Mimir Helm chart repo.
Place these entries under the mimir key in your values.yaml file.
mimir:
alertmanager:
enabled: true
extraArgs:
alertmanager-storage.backend: local
alertmanager-storage.local.path: /etc/alertmanager/config
alertmanager.configs.fallback: /etc/alertmanager/config/default.yml
alertmanager.sharding-ring.replication-factor: "2"
extraVolumeMounts:
- mountPath: /etc/alertmanager/config
name: alertmanager-config
- mountPath: /alertmanager
name: alertmanager-config-tmp
extraVolumes:
- name: alertmanager-config
secret:
secretName: alertmanager-config
- emptyDir: {}
name: alertmanager-config-tmp
persistentVolume:
accessModes:
- ReadWriteOnce
enabled: "1"
size: 10Gi
replicas: "2"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
ingester:
extraArgs:
ingester.max-global-series-per-user: "0"
ingester.ring.replication-factor: "1"
persistentVolume:
size: 64Gi
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
store_gateway:
persistentVolume:
size: 64Gi
compactor:
extraArgs:
compactor.blocks-retention-period: 30d
persistentVolume:
enabled: "1"
size: 64Gi
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
distributor:
extraArgs:
ingester.ring.replication-factor: "1"
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
mimir:
structuredConfig:
activity_tracker:
filepath: /data/activity.log
limits:
ingestion_burst_size: 100000
ingestion_rate: 50000
max_label_names_per_series: 120
out_of_order_time_window: 5m
These values omit configuration of a storage backend. Augment this configuration with one of the following backends: GCS, S3, or Azure blob storage.
Configure Mimir with GCS storage
To use a GCS bucket to store metrics, update the mimir.structuredConfig definition with the following and replace the placeholders:
mimir:
structuredConfig:
activity_tracker:
filepath: /data/activity.log
blocks_storage:
backend: gcs
bucket_store:
sync_dir: /data/tsdb-sync
gcs:
bucket_name: GCS_BUCKET_NAME
service_account: 'GCP_SERVICE_ACCOUNT_JSON_CONTENT'
tsdb:
dir: /data/tsdb
limits:
ingestion_burst_size: 100000
ingestion_rate: 50000
max_label_names_per_series: 120
out_of_order_time_window: 5m
Replace the following:
-
GCS_BUCKET_NAME: The name of your GCS bucket. -
GCP_SERVICE_ACCOUNT_JSON_CONTENT: The JSON content for your GCP service account.
Configure Mimir with S3 storage
To use an S3 bucket to store metrics with Kubernetes secrets for credentials, do the following:
-
Create a Kubernetes secret with your S3 credentials:
kubectl create secret generic SECRET_NAME -n mission-control \ --from-literal=SECRET_KEY_ACCESS_KEY_ID=ACCESS_KEY_ID \ --from-literal=SECRET_KEY_SECRET_ACCESS_KEY=SECRET_ACCESS_KEYReplace the following:
-
SECRET_NAME: The name of your secret. For example,mimir-s3-secrets. -
SECRET_KEY_ACCESS_KEY_ID: The key for your access key ID. For example,s3-access-key-id. -
ACCESS_KEY_ID: Your AWS access key ID. -
SECRET_KEY_SECRET_ACCESS_KEY: The key for your secret access key. For example,s3-secret-access-key. -
SECRET_ACCESS_KEY: Your AWS secret access key.
-
-
Configure Mimir to reference the secret values through environment variables:
mimir: structuredConfig: activity_tracker: filepath: /data/activity.log blocks_storage: backend: s3 bucket_store: sync_dir: /data/tsdb-sync s3: access_key_id: "${ENV_VAR_ACCESS_KEY}" bucket_name: S3_BUCKET_NAME endpoint: s3.AWS_REGION.amazonaws.com insecure: false secret_access_key: "${ENV_VAR_SECRET_KEY}" tsdb: dir: /data/tsdb limits: ingestion_burst_size: 100000 ingestion_rate: 50000 max_label_names_per_series: 120 out_of_order_time_window: 5m extraEnvFrom: - secretRef: name: SECRET_NAME extraEnv: - name: ENV_VAR_ACCESS_KEY valueFrom: secretKeyRef: name: SECRET_NAME key: SECRET_KEY_ACCESS_KEY_ID - name: ENV_VAR_SECRET_KEY valueFrom: secretKeyRef: name: SECRET_NAME key: SECRET_KEY_SECRET_ACCESS_KEYReplace the following:
-
S3_BUCKET_NAME: The name of your S3 bucket. -
AWS_REGION: The AWS region for your S3 endpoint. -
ENV_VAR_ACCESS_KEY: The environment variable name for the access key. For example,AWS_ACCESS_KEY_ID. -
ENV_VAR_SECRET_KEY: The environment variable name for the secret key. For example,AWS_SECRET_ACCESS_KEY. -
SECRET_NAME: The name of your Kubernetes secret. For example,mimir-s3-secrets. -
SECRET_KEY_ACCESS_KEY_ID: The key in the secret for the access key ID. For example,s3-access-key-id. -
SECRET_KEY_SECRET_ACCESS_KEY: The key in the secret for the secret access key. For example,s3-secret-access-key.
-
For more information on configuring Mimir with S3, see the Grafana Mimir documentation.
Configure Mimir with Azure storage
To use Azure blob storage to store metrics, update the mimir.structuredConfig definition with the following and replace the placeholders:
mimir:
mimir:
structuredConfig:
common:
storage:
backend: azure
azure:
account_name: STORAGE_ACCOUNT_NAME
account_key: STORAGE_ACCOUNT_KEY
endpoint_suffix: STORAGE_ACCOUNT_ENDPOINT_SUFFIX
blocks_storage:
backend: azure
azure:
container_name: MIMIR_BUCKET_NAME
Replace the following:
-
STORAGE_ACCOUNT_NAME: The name of your Azure storage account. -
STORAGE_ACCOUNT_KEY: The access key for your Azure storage account. -
STORAGE_ACCOUNT_ENDPOINT_SUFFIX: The endpoint suffix for your Azure storage account. -
MIMIR_BUCKET_NAME: The name of your Azure blob storage container.
Vector
The Vector chart is deployed multiple times in different contexts. Each instantiation has a different alias, allowing for multiple configurations.
Place these entries under the agent and aggregator keys in your values.yaml file.
Agent
Vector running in agent mode collects structured logs from each Kubernetes worker and the underlying container runtime, passing them along to the centralized aggregator.
agent:
enabled: true
Aggregator
Vector running in aggregator mode collects and processes all metrics and logs before sending them to downstream persistence systems like Loki, Mimir, or external sinks.
aggregator:
enabled: true
service:
type: ClusterIP
ports:
- name: vector
protocol: TCP
port: 6000
targetPort: 6000
# For external access to Vector aggregator, use Ingress instead of NodePort:
# ingress:
# enabled: true
# className: nginx
# hosts:
# - host: vector.example.com
# paths:
# - path: /
# pathType: Prefix
|
Starting with v1.18.0, the default Vector aggregator service type changed from NodePort to Ingress for accepting metrics and logs from external data planes. If you have external data planes with observability enabled, you must update your data plane configuration:
|
Configure airgap Helm installations
To install Mission Control using Helm in an airgapped environment, you must override coordinates of all the images in the values.yaml file and configure image pull secrets.
|
Update the placeholders to match your registry and namespace. Load all images into the private registry before installing. Update image tags for each new version. |
For registry override locations, see Override registry credentials for airgap installations.
For configuring image pull secrets, see Configure image pull secrets.
The following is a sample values file you can use as a starting point:
# -- Determines if the mission-control-operator should be installed as the control plane
# or if it's simply in a secondary cluster waiting to be promoted
controlPlane: true
disableCertManagerCheck: true
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
repository: datastax/mission-control
pullPolicy: IfNotPresent
tag: v1.4.0
imageConfigs:
registryOverride: REGISTRY_ADDRESS:REGISTRY_PORT
reaper:
repository: thelastpickle/cassandra-reaper
medusa:
repository: k8ssandra/medusa
# -- Node affinity for operator pod assignment.
allowOperatorsOnDatabaseNodes: false
client:
# -- Automatically handle CRD upgrades
manageCrds: true
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
repository: k8ssandra/k8ssandra-client
tag: latest
# -- Configuration of the job that runs at installation time to patch the conversion webhook in the CRD.
crdPatchJob:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
repository: bitnami/kubectl
tag: 1.30.1
ui:
enabled: true
# -- Base URL that client browsers will use to access the UI.
# If Dex only uses static passwords and/or the LDAP connector, this can be left empty, and the UI will work via any
# routable URL.
# If Dex uses an external provider (e.g. OIDC), this must be set, and the UI can only be accessed via this canonical
# URL.
baseUrl: ''
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
repository: datastax/mission-control-ui
tag: v1.4.0
ingress:
enabled: false
# Configure Ingress for UI access in air-gapped environments
# className: nginx
# hosts:
# - host: mission-control.example.com
# paths:
# - path: /
# pathType: Prefix
https:
enabled: true
# https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml
grafana:
enabled: true
imageRegistry: REGISTRY_ADDRESS:REGISTRY_PORT
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/grafana
sidecar:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/k8s-sidecar
downloadDashboardsImage:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/curl
initChownData:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE
plugins: []
# https://github.com/k8ssandra/k8ssandra-operator/blob/main/charts/k8ssandra-operator/values.yaml
k8ssandra-operator:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
cass-operator:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
imageConfig:
systemLogger: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/system-logger:v1.22.1
configBuilder: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/cass-config-builder:1.0-ubi8
k8ssandraClient: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/k8ssandra-client:v0.5.0
loki:
#enabled: false
kubectlImage:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
sidecar:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/k8s-sidecar
global:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
minio:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/minio
mcImage:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/mc
loki:
storage:
type: "s3"
s3:
region: eu-west-1
bucketNames:
chunks: chunks-bucket
limits_config:
retention_period: 7d
read:
persistence:
enabled: true
size: 10Gi
replicas: 1
write:
persistence:
enabled: true
size: 10Gi
replicas: 1
backend:
replicas: 1
mimir:
alertmanager:
enabled: true
extraArgs:
alertmanager-storage.backend: local
alertmanager-storage.local.path: /etc/alertmanager/config
alertmanager.configs.fallback: /etc/alertmanager/config/default.yml
alertmanager.sharding-ring.replication-factor: "2"
extraVolumeMounts:
- mountPath: /etc/alertmanager/config
name: alertmanager-config
- mountPath: /alertmanager
name: alertmanager-config-tmp
extraVolumes:
- name: alertmanager-config
secret:
secretName: alertmanager-config
- emptyDir: {}
name: alertmanager-config-tmp
persistentVolume:
accessModes:
- ReadWriteOnce
enabled: "1"
size: 10Gi
replicas: "2"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
ingester:
extraArgs:
ingester.max-global-series-per-user: "0"
ingester.ring.replication-factor: "1"
persistentVolume:
size: 64Gi
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
store_gateway:
persistentVolume:
size: 64Gi
enabled: "1"
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
compactor:
extraArgs:
compactor.blocks-retention-period: 30d
persistentVolume:
enabled: "1"
size: 64Gi
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
ruler:
enabled: true
extraArgs:
ingester.ring.replication-factor: "1"
ruler-storage.backend: local
ruler-storage.local.directory: /etc/rules
ruler.alertmanager-url: http://mission-control-mimir-alertmanager:8080/alertmanager
ruler.query-frontend.address: mission-control-mimir-query-frontend:9095
extraVolumeMounts:
- mountPath: /etc/rules/anonymous
name: ruler-config
extraVolumes:
- name: ruler-config
projected:
defaultMode: 420
sources:
- configMap:
name: mission-control-ruler-config
- configMap:
name: mission-control-ruler-custom-config
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
distributor:
extraArgs:
ingester.ring.replication-factor: "1"
replicas: "1"
resources:
limits:
memory: 2Gi
requests:
cpu: 100m
memory: 128Mi
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/mimir
memcached:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE
memcachedExporter:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/memcached-exporter
nginx:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
gateway:
nginx:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
enterprise:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/enterprise-metrics
mcImage:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/mc
mimir:
structuredConfig:
activity_tracker:
filepath: /data/activity.log
limits:
ingestion_burst_size: 100000
ingestion_rate: 50000
max_label_names_per_series: 120
out_of_order_time_window: 5m
agent:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/vector
aggregator:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/vector
kube-state-metrics:
image:
registry: REGISTRY_ADDRESS:REGISTRY_PORT
dex:
image:
repository: REGISTRY_ADDRESS:REGISTRY_PORT/REGISTRY_NAMESPACE/mission-control-dex
config:
enablePasswordDB: true
staticPasswords:
- email: EMAIL_ADDRESS
hash: "HASH"
username: admin
userID: "USER_ID"
Replace the following:
-
REGISTRY_ADDRESS: The address of your registry. -
REGISTRY_PORT: The port of your registry. -
EMAIL_ADDRESS: The email address for the admin user. -
HASH: The bcrypt hash of the password.On Unix and Linux systems, you can generate this with the following command:
echo yourPassword | htpasswd -BinC 10 admin | cut -d: -f2
GCP configuration
The following sections describe configuration values required for GCP deployments.
GCP control plane configuration
Download the GCP control plane YAML configuration to use as a starting point.
Replace the following:
-
OIDC_CLIENT_ID: The OIDC client ID for authentication. -
OIDC_CLIENT_SECRET: The OIDC client secret for authentication. -
OIDC_ISSUER: The OIDC issuer URL. -
MC_DOMAIN: The domain name for Mission Control. -
ADMIN_EMAIL: The email address for the admin user. -
HASHED_ADMIN_PASSWORD: The bcrypt hash of the admin password. -
SERVICE_ACCOUNT_JSON: The GCP service account JSON content. -
REGISTRY_ADDRESS: The address of your registry. -
REGISTRY_PORT: The port of your registry. -
REGISTRY_NAMESPACE: The namespace within your registry.
GCP data plane online values configuration
Download the GCP data plane online YAML configuration to use as a starting point.
Replace the following:
-
VECTOR_AGGREGATOR_URL: The URL for the vector aggregator. -
VECTOR_VOLUME_SIZE: The size of the vector volume. -
VECTOR_STORAGE_CLASS: The storage class for vector. -
REGISTRY_ADDRESS: The address of your registry. -
REGISTRY_PORT: The port of your registry. -
REGISTRY_NAMESPACE: The namespace within your registry.
GCP data plane local observability configuration
Download the GCP data plane local observability YAML configuration to use as a starting point.
Replace the following:
-
VECTOR_AGGREGATOR_URL: The URL for the vector aggregator. -
VECTOR_VOLUME_SIZE: The size of the vector volume. -
VECTOR_STORAGE_CLASS: The storage class for vector. -
REGISTRY_ADDRESS: The address of your registry. -
REGISTRY_PORT: The port of your registry. -
REGISTRY_NAMESPACE: The namespace within your registry.
Create and verify the image pull secret
There are two options to create the image pull secret, using kubectl or a YAML manifest.
kubectl(Recommended)-
-
Create the secret:
kubectl create secret docker-registry $PULL_SECRET_NAME \ --docker-server=cp.icr.io \ --docker-username=cp \ --docker-password=$ENTITLEMENT_KEY \ --namespace=$MC_NAMESPACEParameters Variable Value Description --docker-servercp.icr.ioIBM entitled registry host (fixed)
--docker-usernamecpFixed username for IBM entitled registry
-
Confirm the secret exists in the correct namespace:
kubectl get secret $PULL_SECRET_NAME -n $MC_NAMESPACEExpected output:
NAME TYPE DATA AGE SECRET_NAME kubernetes.io/dockerconfigjson 1 ... -
Inspect the decoded content to confirm credentials are correct:
kubectl get secret $PULL_SECRET_NAME \ -n $MC_NAMESPACE \ -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
-
- YAML manifest
-
-
Generate the base64-encoded Docker config:
kubectl create secret docker-registry $PULL_SECRET_NAME \ --docker-server=cp.icr.io \ --docker-username=cp \ --docker-password=$ENTITLEMENT_KEY \ --namespace=$MC_NAMESPACE \ --dry-run=client -o jsonpath='{.data.\.dockerconfigjson}'Copy the output to use as the value of
.dockerconfigjsonin the next step. -
Create a file named
pull-secret.yamlwith the following content:apiVersion: v1 kind: Secret metadata: name: SECRET_NAME namespace: NAMESPACE type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: BASE64_DOCKER_CONFIGReplace the following:
-
SECRET_NAME: The image pull secret name, typically in the format${MC_RELEASE_NAME}-registry. -
NAMESPACE: The Kubernetes namespace for your Mission Control installation. -
BASE64_DOCKER_CONFIG: The base64 string copied from the previous step.
-
-
Apply the manifest:
kubectl apply -f pull-secret.yaml -
Confirm the secret exists in the correct namespace:
kubectl get secret $PULL_SECRET_NAME -n $MC_NAMESPACEExpected output:
NAME TYPE DATA AGE secret-name kubernetes.io/dockerconfigjson 1 ... -
Inspect the decoded content to confirm credentials are correct:
kubectl get secret $PULL_SECRET_NAME \ -n $MC_NAMESPACE \ -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
-
Pull secrets for multi-cluster deployments
You only need to create the pull secret in the operator namespace (the namespace where Mission Control is installed).
The operator automatically replicates all pull secrets referenced in global.imageConfig into every project namespace as clusters are provisioned.
You don’t need to create the secret manually in each namespace.
If you are deploying Mission Control across multiple clusters, create the pull secret in the operator namespace of each cluster (update MC_NAMESPACE and the kubeconfig context for each):
kubectl create secret docker-registry $PULL_SECRET_NAME \
--docker-server=cp.icr.io \
--docker-username=cp \
--docker-password=$ENTITLEMENT_KEY \
--namespace=$MC_NAMESPACE
Install Mission Control with Helm
The Mission Control Helm chart is published to the OCI registry. You can install Mission Control directly without adding a Helm repo.
An overrides.yaml file is required because the chart enforces validation on several components at install time, and a bare install without values will fail.
At a minimum, your overrides.yaml must:
-
Configure Loki storage or disable Loki.
-
Configure Dex with at least one authentication connector (see Dex IdP).
To install Mission Control with Helm, do the following:
-
Optional: Pull the chart for offline inspection:
helm pull oci://icr.io/mission-control-helm/mission-control --version $MC_VERSIONThis downloads a
.tgzarchive locally so you can extract and inspectvalues.yamlbefore installing. -
Create an
overrides.yamlfile. Download the sample values file as a starting point, or use the minimum viable configuration below:# overrides.yaml loki: enabled: false dex: config: enablePasswordDB: true staticPasswords: - email: EMAIL hash: "BCRYPT_HASH" username: USERNAME userID: "USER_ID"Replace the following:
-
EMAIL: Login email address for the admin user. -
BCRYPT_HASH: bcrypt hash of the password. Generate one with:echo yourPassword | htpasswd -BinC 10 admin | cut -d: -f2 -
USERNAME: Display name for the admin user. -
USER_ID: A unique identifier for the user (any UUID).
If your pull secret name differs from the default (
${MC_RELEASE_NAME}-registry), also add:global: imageConfig: defaults: pullSecrets: - "SECRET_NAME"Replace
SECRET_NAMEwith the value of$PULL_SECRET_NAME. -
-
Install the chart:
helm install $MC_RELEASE_NAME \ oci://icr.io/mission-control-helm/mission-control \ --version $MC_VERSION \ --namespace $MC_NAMESPACE \ --create-namespace \ -f overrides.yaml -
Check that all pods are running:
kubectl get pods -n $MC_NAMESPACEIf any pods are in
ImagePullBackOfforErrImagePullstate, proceed to the Troubleshooting section. -
Check the Helm release status:
helm status $MC_RELEASE_NAME -n $MC_NAMESPACETo inspect all available chart values before installing, run:
helm show values oci://icr.io/mission-control-helm/mission-control --version $MC_VERSION
Relationship summary
| Item | Value |
|---|---|
Pull secret name |
|
Helm values key to set |
|
Default value (if not overridden) |
|
Secret namespace |
|
Registry server |
|
Registry username |
|
Registry password |
|
Troubleshooting
Follow these troubleshooting steps to diagnose and resolve common issues.
ImagePullBackOfforErrImagePull-
-
Confirm the secret exists in the correct namespace:
kubectl get secrets -n $MC_NAMESPACE -
Confirm the secret name in
global.imageConfig.defaults.pullSecretsmatches the secret you created:helm get values $MC_RELEASE_NAME -n $MC_NAMESPACE -
Decode and inspect the secret to confirm credentials:
kubectl get secret $PULL_SECRET_NAME \ -n $MC_NAMESPACE \ -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d -
Check pod events for the exact error:
kubectl describe pod POD_NAME -n $MC_NAMESPACEReplace
POD_NAMEwith the name of the pod.
-
- Authentication failures
-
-
The
--docker-usernamemust always be exactlycp(lowercase). -
There must be no leading or trailing whitespace in
$ENTITLEMENT_KEY. -
The registry server must be
cp.icr.io(noticr.iodirectly). -
Regenerate the key at IBM Container Library if the current key has expired.
-
- Helm OCI pull fails
-
If
helm pullorhelm installfrom the OCI URL fails:-
Ensure that Helm is installed with OCI support:
helm version. -
Verify OCI registry access:
helm show chart oci://icr.io/mission-control-helm/mission-control --version $MC_VERSION
-
Next steps
After installation, configure and access your deployment: