Install Mission Control in an airgap environment

Use this guide to install Mission Control in an air-gapped environment where the Kubernetes cluster has no direct internet access. You use Hauler to create a portable archive containing all required container images and Helm charts, transfer it to the air-gapped environment, and load it into a private registry.

Prerequisites

On the internet-connected system, ensure that you have:

Hauler installed.
Docker installed and running.
An IBM entitlement key from the IBM Container Library.
The Mission Control CLI tool, mcctl, is installed. If you need a tarball to install this tool in a Linux environment, contact IBM Support.
A Hauler manifest file. For example, hauler-manifest.yaml. The manifest is included with each release in the release notes.
Sufficient disk space for the archive (typically 5 to 10 GB).

On the air-gapped system, ensure that you have:

Hauler installed.
kubectl configured to access the target Kubernetes cluster.
helm installed.
Access to a private container registry.
Registry credentials (if authentication is required).
cert-manager installed and configured in the cluster. For installation instructions, see Install cert-manager.

Create the archive

Perform these steps on a system with internet access.

Fetch all images and charts listed in the manifest:

hauler store sync \
  --store ./hauler-store \
  --filename hauler-manifest.yaml

Hauler downloads all images and charts referenced in the manifest and writes them to ./hauler-store.

After syncing, Hauler attempts to render the Helm chart to discover additional images. If the chart has required values (such as Loki storage configuration), you might see a warning like: failed to render chart [mission-control]: execution error … Loki validation failed. This warning is expected and does not affect the store contents.

Optional: Verify that all artifacts were pulled successfully:
```
hauler store info --store ./hauler-store
```
Hauler lists all artifacts in the store.
Create a portable archive from the store:
```
hauler store save \
  --store ./hauler-store \
  --filename mission-control.tar.zst
```
Hauler writes the archive to mission-control.tar.zst.
Transfer mission-control.tar.zst to the air-gapped environment using your organization’s approved method. For example, with a USB drive or a secure file transfer to a jump host.

Load the archive into the target registry

Perform these steps on the air-gapped system after transferring mission-control.tar.zst.

Restore the store from the archive:

hauler store load \
  --store ./store \
  --filename mission-control.tar.zst

Hauler restores the store to ./store.

Confirm that all artifacts were loaded:
```
hauler store info --store ./store
```
Copy all images and charts from the store to your private registry:
```
hauler store copy \
  --store ./store \
  --plain-http \
  registry://REGISTRY
```
Replace REGISTRY with your registry’s address and port. For example, registry.example.com:5000. If your registry requires authentication, add credentials:
```
hauler store copy \
  --store ./store \
  --plain-http \
  --username USER \
  --password PASSWORD \
  registry://REGISTRY
```

Install Mission Control from the registry

After loading the chart and images into your private registry, install Mission Control using Helm with registry overrides.

Create the namespace:

kubectl create namespace mission-control

If your private registry requires authentication, create an image pull secret:

kubectl create secret docker-registry registry-credentials \
  --docker-server=REGISTRY \
  --docker-username=USER \
  --docker-password=PASSWORD \
  --namespace mission-control

For more information, see Configure image pull secrets.

Create a file named helm-overrides.yaml with the following content:

global:
  imageConfig:
    overrides:
      registry: "REGISTRY"
  imagePullSecrets:
    - name: registry-credentials

image:
  registry: "REGISTRY"

client:
  image:
    registry: "REGISTRY"

ui:
  image:
    registry: "REGISTRY"
    repository: "mission-control-ui"

aggregator:
  image:
    repository: "REGISTRY/timberio/vector"

k8ssandra-operator:
  image:
    registry: "REGISTRY"
  cass-operator:
    image:
      registry: "REGISTRY"

dex:
  image:
    repository: "REGISTRY/mission-control-dex"

grafana:
  enabled: true
  image:
    registry: "REGISTRY"
    repository: "grafana/grafana"
  sidecar:
    image:
      registry: "REGISTRY"
      repository: "kiwigrid/k8s-sidecar"

kube-state-metrics:
  image:
    registry: "REGISTRY"
    repository: "kube-state-metrics/kube-state-metrics"

loki:
  global:
    image:
      registry: "REGISTRY"
  image:
    registry: "REGISTRY"
    repository: "grafana/loki"
  gateway:
    image:
      registry: "REGISTRY"
      repository: "nginxinc/nginx-unprivileged"
  sidecar:
    image:
      registry: "REGISTRY"
      repository: "kiwigrid/k8s-sidecar"

mimir:
  image:
    repository: "REGISTRY/grafana/mimir"
  gateway:
    nginx:
      image:
        registry: "REGISTRY"
        repository: "nginxinc/nginx-unprivileged"

Replace REGISTRY with your registry host and port. For example, registry.example.com:5000.

If your private registry requires authentication, log in so that Helm can pull the chart:
```
helm registry login REGISTRY \
  --username USER \
  --password PASSWORD
```
Install Mission Control from your private registry.

Hauler publishes the chart under the hauler/ path in the registry.
```
helm install mission-control oci://REGISTRY/hauler/mission-control \
  --version CHART_VERSION \
  --namespace mission-control \
  -f helm-overrides.yaml
```
Replace the following:
- REGISTRY: Your private registry address
- CHART_VERSION: The chart version included in the Hauler bundle. For example, 1.20.1.
Verify that all pods are running:
```
kubectl get pods -n mission-control
```
Wait until all pods show Running status.

Confirm that no pods pull images from outside your private registry:

kubectl get pods -n mission-control \
  -o jsonpath='{range .items[]}{range .spec.containers[]}{.image}{"\n"}{end}{range .spec.initContainers[*]}{.image}{"\n"}{end}{end}' \
  | sort -u \
  | grep -v '^REGISTRY/'

Replace REGISTRY with your registry address. If this command produces output, an image is pulling from outside your private registry.

Install a data plane from the registry

If you are deploying Mission Control across multiple Kubernetes clusters, each data plane cluster must also pull images from a private registry. The Hauler archive you already transferred contains all required data plane images. No additional bundling is needed. The private registry accessible from the data plane cluster must contain those images before you run the data plane install.

If the data plane cluster is in a different network zone, push the Hauler archive contents to a registry that is reachable from that cluster, following the same steps in Load the archive into the target registry.

Create the namespace on the data plane cluster:
```
kubectl create namespace mission-control --context DP_CONTEXT
```
Replace DP_CONTEXT with the kubectl context name for the data plane cluster.

If the data plane’s private registry requires authentication, create an image pull secret:

kubectl create secret docker-registry registry-credentials \
  --docker-server=REGISTRY \
  --docker-username=USER \
  --docker-password=PASSWORD \
  --namespace mission-control \
  --context DP_CONTEXT

Create a file named dp-helm-overrides.yaml with the following content:

# This cluster is a data plane only
controlPlane: false

# Redirect all image pulls to your private registry
global:
  imageConfig:
    overrides:
      registry: "REGISTRY"
  imagePullSecrets:
    - name: registry-credentials

image:
  registry: "REGISTRY"

client:
  image:
    registry: "REGISTRY"

aggregator:
  image:
    repository: "REGISTRY/timberio/vector"
  customConfig:
    sources:
      mimir-self-monitoring: {}
    sinks:
      mimir: {}
      loki: {}
      control_plane_aggregator:
        type: vector
        inputs: ["vector", "internal_metrics", "kube_state_metrics", "cass_operator_metrics"]
        address: "CONTROL_PLANE_IP:30600"

k8ssandra-operator:
  image:
    registry: "REGISTRY"
  cass-operator:
    image:
      registry: "REGISTRY"

# Disable control-plane-only components
ui:
  enabled: false

dex:
  enabled: false

grafana:
  enabled: false

loki:
  enabled: false

mimir:
  enabled: false

agent:
  enabled: false

Replace the following:

REGISTRY: Your private registry address for the data plane cluster.
CONTROL_PLANE_IP: The IP address or hostname of a node in the control plane cluster. The control plane exposes the Vector aggregator on port 30600. Use a hostname that load-balances across all nodes for high availability.

If your private registry requires authentication, log in so that Helm can pull the chart:
```
helm registry login REGISTRY \
  --username USER \
  --password PASSWORD
```
If your registry uses plain HTTP, add --insecure.
Install Mission Control in data plane mode:
```
helm install mission-control oci://REGISTRY/hauler/mission-control \
  --version CHART_VERSION \
  --namespace mission-control \
  --kube-context DP_CONTEXT \
  -f dp-helm-overrides.yaml
```
Use the same release name (mission-control) as the control plane installation. Mismatched release names break communication between the planes.

Replace the following:
- REGISTRY: Your private registry address.
- CHART_VERSION: The chart version, matching the control plane version. For example, 1.20.1.
- DP_CONTEXT: The kubectl context for the data plane cluster.
Verify that all pods are running on the data plane cluster:
```
kubectl get pods -n mission-control --context DP_CONTEXT
```
Wait until all pods show Running status.

Confirm that no data plane pods pull images from outside your private registry:

kubectl get pods -n mission-control --context DP_CONTEXT \
  -o jsonpath='{range .items[]}{range .spec.containers[]}{.image}{"\n"}{end}{range .spec.initContainers[*]}{.image}{"\n"}{end}{end}' \
  | sort -u \
  | grep -v '^REGISTRY/'

Replace REGISTRY with your registry address. If this command produces output, an image is pulling from outside your private registry.

Register the data plane with the control plane using the mcctl CLI:
```
mcctl register --source-context DP_CONTEXT --dest-context CP_CONTEXT
```
Replace the following:
- DP_CONTEXT: The kubectl context for the data plane cluster
- CP_CONTEXT: The kubectl context for the control plane cluster

Troubleshoot installation issues

The following issues might occur during installation.

Hauler store sync fails

If hauler store sync fails to download images, do the following:

Verify internet connectivity.
Check that the manifest file path is correct.
Ensure sufficient disk space.

Registry push fails

If hauler store copy fails, do the following:

Verify registry connectivity.
Check registry credentials.
Ensure the registry supports OCI artifacts.

Helm install fails with image pull errors

If pods fail to start with ImagePullBackOff, do the following:

Verify that all images are present in your private registry.
Check that the registry overrides in helm-overrides.yaml are correct.
If your registry requires authentication, verify that the pull secret exists in the namespace where the failing pod is running:
```
kubectl get secret registry-credentials -n NAMESPACE
```
For more information, see Configure image pull secrets.

Pods pull from public registries

If the verification commands show images from public registries, do the following:

Review the helm-overrides.yaml file for missing or incorrect registry overrides.
Check that all image references in the chart are properly overridden.