Configure task retry behavior

In Mission Control versions 1.18.0 and later, you can configure the number of times Mission Control retries a failed task before it marks it as failed.

Prerequisites

Access to your Kubernetes cluster with kubectl
A database cluster deployed in Mission Control

Configure retry count

To add the retries field to your K8ssandraTask manifest to specify the maximum number of retry attempts, do the following:

Create or edit your K8ssandraTask manifest file.
In your K8ssandraTask manifest file, find the spec.template section.
Set restartPolicy to OnFailure.

This is required to enable the retries field.
Set the retries field.

This specifies the maximum number of retry attempts for the task. Set this value based on your operational requirements. Consider the following when you set the retry count:
- Higher values: Increase the likelihood of task success but extend the time before final failure and block other tasks from running if the concurrencyPolicy is set to Forbid.
- Lower values: Fail faster but might not account for transient infrastructure issues.
- Infrastructure reliability: Less reliable infrastructure might benefit from higher retry counts.
- Task criticality: Critical operations might warrant more retry attempts.

Define a job that runs the desired command:

apiVersion: control.k8ssandra.io/v1alpha1
kind: K8ssandraTask
metadata:
  name: TASK_NAME
  namespace: NAMESPACE
spec:
  cluster:
    name: CLUSTER_NAME
    namespace: CLUSTER_NAMESPACE
  template:
    concurrencyPolicy: Forbid
    restartPolicy: OnFailure
    retries: 5
    ttlSecondsAfterFinished: 0
    jobs:
      - command: restart
        name: JOB_NAME

Replace the following:

TASK_NAME: A unique name for this task operation
NAMESPACE: The Kubernetes namespace where you deploy your database cluster
CLUSTER_NAME: The name of your cluster
JOB_NAME: The specific job within the task

Apply the manifest:
```
kubectl apply -f task-manifest.yaml
```
Verify the task creation:
```
kubectl get k8ssandratask TASK_NAME -n NAMESPACE
```
Replace the following:
- TASK_NAME: The name of the task you created
- NAMESPACE: The namespace where the task is deployed

Configure task retry behavior

Prerequisites

Configure retry count

See also

Was this helpful?

Give Feedback