Pin workloads to specific hosts in shared clusters

This guide explains how to pin workloads to specific hosts in shared Kubernetes clusters using affinity rules, node selectors, and tolerations in Mission Control. This is particularly useful when you need to ensure that database workloads run on specific nodes with particular hardware characteristics, storage, or network configurations.

Workload pinning methods

In shared Kubernetes clusters, multiple workloads may compete for the same resources. To ensure optimal performance and resource isolation, you can pin specific workloads to designated nodes using:

Node affinity
Node selectors
Tolerations
Pod anti-affinity

Prefer or require workloads to run on nodes with specific labels. Use this for complex scheduling rules with hard requirements or soft preferences.

Simple key-value pairs to match node labels. Use this for basic node selection when you don’t need complex affinity rules. Node selectors enforce hard requirements that must be satisfied. If no node matches the selector, the scheduler keeps the pods in the Pending state.

Allow workloads to run on nodes that have been tainted. Use this when you want to reserve certain nodes for specific workloads.

Prevent multiple instances of the same workload from running on the same node. Use this for high availability and resource isolation.

Node affinity

Node affinity allows you to specify rules that determine which nodes are preferred or required for your workloads. Mission Control supports both hard requirements with requiredDuringSchedulingIgnoredDuringExecution and soft preferences with preferredDuringSchedulingIgnoredDuringExecution.

Simple nodeAffinityLabels
Advanced affinity rules

You can pin workloads to specific nodes using the nodeAffinityLabels field in your rack configuration:

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: CLUSTER_NAME
spec:
  k8ssandra:
    cassandra:
      datacenters:
        - metadata:
            name: DATACENTER_NAME
          k8sContext: K8S_CONTEXT
          size: NUMBER_OF_NODES
          racks:
            - name: RACK_NAME_1
              nodeAffinityLabels:
                topology.kubernetes.io/zone: AVAILABILITY_ZONE_1
                mission-control.datastax.com/role: "database"
            - name: RACK_NAME_2
              nodeAffinityLabels:
                topology.kubernetes.io/zone: AVAILABILITY_ZONE_2
                mission-control.datastax.com/role: "database"
            - name: RACK_NAME_3
              nodeAffinityLabels:
                topology.kubernetes.io/zone: AVAILABILITY_ZONE_3
                mission-control.datastax.com/role: "database"

Replace the following:

CLUSTER_NAME: The name of your cluster
DATACENTER_NAME: The name of your datacenter
K8S_CONTEXT: The name of your Kubernetes context
NUMBER_OF_NODES: The number of nodes in your datacenter
RACK_NAME_1: The name of your first rack
AVAILABILITY_ZONE_1: The name of your first availability zone
RACK_NAME_2: The name of your second rack
AVAILABILITY_ZONE_2: The name of your second availability zone
RACK_NAME_3: The name of your third rack
AVAILABILITY_ZONE_3: The name of your third availability zone

This configuration ensures that:

Rack 1 nodes are scheduled in the AVAILABILITY_ZONE_1 availability zone
Rack 2 nodes are scheduled in the AVAILABILITY_ZONE_2 availability zone
Rack 3 nodes are scheduled in the AVAILABILITY_ZONE_3 availability zone
All nodes have the mission-control.datastax.com/role: "database" label

For more complex affinity rules, you can use the full affinity configuration:

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: CLUSTER_NAME
spec:
  k8ssandra:
    cassandra:
      datacenters:
        - metadata:
            name: DATACENTER_NAME
          k8sContext: K8S_CONTEXT
          size: NUMBER_OF_NODES
          racks:
            - name: RACK_NAME
              affinity:
                nodeAffinity:
                  requiredDuringSchedulingIgnoredDuringExecution:
                    nodeSelectorTerms:
                    - matchExpressions:
                      - key: topology.kubernetes.io/zone
                        operator: In
                        values:
                        - AVAILABILITY_ZONE
                      - key: mission-control.datastax.com/role
                        operator: In
                        values:
                        - database
                  preferredDuringSchedulingIgnoredDuringExecution:
                  - weight: WEIGHT_VALUE
                    preference:
                      matchExpressions:
                      - key: storage-type
                        operator: In
                        values:
                        - STORAGE_TYPE

Replace the following:

CLUSTER_NAME: The name of your cluster
DATACENTER_NAME: The name of your datacenter
K8S_CONTEXT: The name of your Kubernetes context
NUMBER_OF_NODES: The number of nodes in your datacenter
RACK_NAME: The name of your rack

This configuration requires nodes to be in the AVAILABILITY_ZONE zone and have the mission-control.datastax.com/role: database label.It also gives preference to nodes with STORAGE_TYPE storage, with the weight value determining how strongly the scheduler favors these nodes during scheduling.

Node selectors

Node selectors provide a simple way to match nodes based on labels. They are less flexible than node affinity but easier to configure:

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: CLUSTER_NAME
spec:
  k8ssandra:
    cassandra:
      datacenters:
        - metadata:
            name: DATACENTER_NAME
          k8sContext: K8S_CONTEXT
          size: NUMBER_OF_NODES
          nodeSelector:
            mission-control.datastax.com/role: "database"
            storage-type: STORAGE_TYPE

Replace the following:

CLUSTER_NAME: The name of your cluster
DATACENTER_NAME: The name of your datacenter
K8S_CONTEXT: The name of your Kubernetes context
NUMBER_OF_NODES: The number of nodes in your datacenter
STORAGE_TYPE: The type of storage you want to use

This configuration ensures that all database pods are scheduled only on nodes with the mission-control.datastax.com/role: "database" label and with the specified storage type.

Tolerations

Tolerations allow workloads to run on nodes that have been tainted. This is useful when you want to reserve certain nodes for specific workloads in Mission Control.

To use tolerations, you must first apply taints to the nodes you want to reserve for specific workloads and then add tolerations to your workload configuration.

Apply taints to nodes you want to reserve for specific workloads:

# Reserve nodes for database workloads
kubectl taint nodes NODE_NAME_1 mission-control.datastax.com/component=database:NoSchedule
kubectl taint nodes NODE_NAME_2 mission-control.datastax.com/component=database:NoSchedule
kubectl taint nodes NODE_NAME_3 mission-control.datastax.com/component=database:NoSchedule

# Reserve nodes for observability workloads
kubectl taint nodes NODE_NAME_4 mission-control.datastax.com/component=observability:NoSchedule
kubectl taint nodes NODE_NAME_5 mission-control.datastax.com/component=observability:NoSchedule

Replace the following:

NODE_NAME_1: The name of your first database node
NODE_NAME_2: The name of your second database node
NODE_NAME_3: The name of your third database node
NODE_NAME_4: The name of your first observability node
NODE_NAME_5: The name of your second observability node

Add tolerations to your workload configuration:

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: CLUSTER_NAME
spec:
  k8ssandra:
    cassandra:
      datacenters:
        - metadata:
            name: DATACENTER_NAME
          k8sContext: K8S_CONTEXT
          size: NUMBER_OF_NODES
          tolerations:
          - key: "mission-control.datastax.com/component"
            operator: "Equal"
            value: "database"
            effect: "NoSchedule"

Replace the following:

CLUSTER_NAME: The name of your cluster
DATACENTER_NAME: The name of your datacenter
K8S_CONTEXT: The name of your Kubernetes context
NUMBER_OF_NODES: The number of nodes in your datacenter

This configuration allows database pods to run on nodes that have been tainted with mission-control.datastax.com/component=database:NoSchedule.

Pod anti-affinity

Pod anti-affinity prevents multiple instances of the same workload from running on the same node, which is important for high availability.

Mission Control applies pod anti-affinity rules to database pods by default. This prevents the scheduler from placing two pods on the same worker node, ensuring high availability and resource isolation. You cannot change these default anti-affinity rules after the cluster is initially deployed.

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: CLUSTER_NAME
spec:
  k8ssandra:
    cassandra:
      datacenters:
        - metadata:
            name: DATACENTER_NAME
          k8sContext: K8S_CONTEXT
          size: NUMBER_OF_NODES
          racks:
            - name: RACK_NAME
              affinity:
                podAntiAffinity:
                  requiredDuringSchedulingIgnoredDuringExecution:
                  - labelSelector:
                      matchExpressions:
                      - key: app.kubernetes.io/name
                        operator: In
                        values:
                        - APP_NAME
                    topologyKey: TOPOLOGY_KEY

Replace the following:

CLUSTER_NAME: The name of your cluster
DATACENTER_NAME: The name of your datacenter
K8S_CONTEXT: The name of your Kubernetes context
NUMBER_OF_NODES: The number of nodes in your datacenter
RACK_NAME: The name of your rack
APP_NAME: The name of the application you want to pin
TOPOLOGY_KEY: The key of the topology you want to use

This configuration ensures that no two APP_NAME pods run on the same node.

Platform component pinning

For platform components like observability services, you can pin them to specific nodes using Mission Control’s standard labeling strategy:

Mimir components
Loki components
Vector aggregator

# Example for Mimir components in Mission Control
mimir:
  alertmanager:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: mission-control.datastax.com/role
              operator: In
              values:
              - platform
    tolerations:
    - key: "mission-control.datastax.com/component"
      operator: "Equal"
      value: "observability"
      effect: "NoSchedule"

This configuration ensures that Mimir alertmanager pods are scheduled only on nodes labeled with mission-control.datastax.com/role: platform and can tolerate nodes with the observability taint.

# Example for Loki components in Mission Control
loki:
  backend:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: mission-control.datastax.com/role
              operator: In
              values:
              - platform
    tolerations:
    - key: "mission-control.datastax.com/component"
      operator: "Equal"
      value: "observability"
      effect: "NoSchedule"

This configuration ensures that Loki backend pods are scheduled only on nodes labeled with mission-control.datastax.com/role: platform and can tolerate nodes with the observability taint.

# Example for Vector aggregator in Mission Control
aggregator:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: mission-control.datastax.com/role
            operator: In
            values:
            - platform
  tolerations:
  - key: "mission-control.datastax.com/component"
    operator: "Equal"
    value: "observability"
    effect: "NoSchedule"

This configuration ensures that Vector aggregator pods are scheduled only on nodes labeled with mission-control.datastax.com/role: platform and can tolerate nodes with the observability taint.

Best practices

When implementing workload pinning in shared Kubernetes clusters, follow these best practices to ensure optimal performance, resource utilization, and maintainability:

Establish a consistent node labeling strategy.
Specify resource requirements alongside affinity rules.
Monitor and validate workload placement.
Plan for scalability and maintenance.

Mission Control-specific recommendations

When working with Mission Control, consider these platform-specific recommendations:

Separate platform and database nodes: Use the mission-control.datastax.com/role label to separate platform components like Mimir, Loki, and Vector from database workloads.
Use component taints: Apply mission-control.datastax.com/component taints to reserve nodes for specific workload types.
Consider observability requirements: Platform components like Mimir and Loki have specific resource and storage requirements that you must consider when pinning.
Plan for multi-datacenter deployments: When deploying across multiple datacenters, ensure proper zone distribution using topology.kubernetes.io/zone labels.
Monitor platform component health: Use the built-in observability stack to monitor the health and performance of pinned workloads.

Node labeling strategy

Use Mission Control’s consistent labeling strategy for your nodes:

# Label nodes by {product} role
kubectl label nodes NODE_NAME_1 mission-control.datastax.com/role=platform
kubectl label nodes NODE_NAME_2 mission-control.datastax.com/role=database

# Label nodes by storage type
kubectl label nodes NODE_NAME_1 storage-type=STORAGE_TYPE_1
kubectl label nodes NODE_NAME_2 storage-type=STORAGE_TYPE_2

# Label nodes by availability zone
kubectl label nodes NODE_NAME_1 topology.kubernetes.io/zone=AVAILABILITY_ZONE_1
kubectl label nodes NODE_NAME_2 topology.kubernetes.io/zone=AVAILABILITY_ZONE_2

# Apply taints for workload isolation
kubectl taint nodes NODE_NAME_1 mission-control.datastax.com/component=observability:NoSchedule
kubectl taint nodes NODE_NAME_2 mission-control.datastax.com/component=database:NoSchedule

Replace the following:

NODE_NAME_1: The name of your first node
NODE_NAME_2: The name of your second node
STORAGE_TYPE_1: The type of storage for your first node
STORAGE_TYPE_2: The type of storage for your second node
AVAILABILITY_ZONE_1: The availability zone for your first node
AVAILABILITY_ZONE_2: The availability zone for your second node

This labeling strategy follows Mission Control’s conventions:

mission-control.datastax.com/role=platform for platform components like Mimir, Loki, and Vector
mission-control.datastax.com/role=database for database workloads
mission-control.datastax.com/component=observability taint for reserving nodes for observability components
mission-control.datastax.com/component=database taint for reserving nodes for database workloads

Resource requirements

Always specify resource requirements along with affinity rules:

apiVersion: missioncontrol.datastax.com/v1beta2
kind: MissionControlCluster
metadata:
  name: CLUSTER_NAME
spec:
  k8ssandra:
    cassandra:
      datacenters:
        - metadata:
            name: DATACENTER_NAME
          k8sContext: K8S_CONTEXT
          size: NUMBER_OF_NODES
          resources:
            requests:
              cpu: CPU_REQUEST
              memory: MEMORY_REQUEST
            limits:
              cpu: CPU_LIMIT
              memory: MEMORY_LIMIT
          racks:
            - name: RACK_NAME
              nodeAffinityLabels:
                topology.kubernetes.io/zone: AVAILABILITY_ZONE

Replace the following:

CLUSTER_NAME: The name of your cluster
DATACENTER_NAME: The name of your datacenter
K8S_CONTEXT: The name of your Kubernetes context
NUMBER_OF_NODES: The number of nodes in your datacenter
RACK_NAME: The name of your rack
AVAILABILITY_ZONE: The name of your availability zone
CPU_REQUEST: The CPU request for your workload
MEMORY_REQUEST: The memory request for your workload
CPU_LIMIT: The CPU limit for your workload
MEMORY_LIMIT: The memory limit for your workload

Monitor and validate

Monitor your workload placement using Mission Control’s labeling conventions:

# Check pod placement for database workloads
kubectl get pods -o wide -l app.kubernetes.io/name=cassandra

# Check pod placement for platform components
kubectl get pods -o wide -l app.kubernetes.io/name=mimir
kubectl get pods -o wide -l app.kubernetes.io/name=loki
kubectl get pods -o wide -l app.kubernetes.io/name=vector

# Check node labels
kubectl get nodes --show-labels | grep mission-control.datastax.com

# Check node taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints | grep mission-control.datastax.com

# Check specific role labels
kubectl get nodes -l mission-control.datastax.com/role=platform
kubectl get nodes -l mission-control.datastax.com/role=database

Validate the following:

Nodes are labeled with the expected mission-control.datastax.com/role labels.
Nodes have the expected mission-control.datastax.com/component taints.
Pods are scheduled to the expected nodes based on their workload type.
Platform components are running on nodes with the platform role.
Database workloads are running on nodes with the database role.

Node availability: Ensure nodes with required labels exist.
Resource availability: Check if nodes have sufficient resources.
```
kubectl describe node NODE_NAME
```
Replace NODE_NAME with the name of the node you want to check.
Taint/toleration mismatch: Verify tolerations match node taints.

# Check pod events
kubectl describe pod POD_NAME

# Check node resources
kubectl describe node NODE_NAME

Replace the following:

POD_NAME: The name of the pod you want to check
NODE_NAME: The name of the node you want to check

Affinity rule conflicts

If affinity rules conflict:

Review all affinity rules in your configuration.
Ensure node labels are correctly applied.
Check for conflicting requiredDuringSchedulingIgnoredDuringExecution rules.

Performance issues

If performance is poor after pinning:

Verify nodes have the expected hardware characteristics.
Check for resource contention on pinned nodes.
Monitor node resource utilization.

Pin workloads to specific hosts in shared clusters

Workload pinning methods

Node affinity

Node selectors

Tolerations

Pod anti-affinity

Platform component pinning

Best practices

Mission Control-specific recommendations

Node labeling strategy

Resource requirements

Monitor and validate

Plan for scalability and maintenance

Troubleshoot

Pods stuck in pending state

Affinity rule conflicts

Performance issues

See also

Was this helpful?

Give Feedback