Troubleshoot OpsCenter

Visit IBM Support for OpsCenter troubleshooting articles.

Startup logs for OpsCenter

A preliminary startup.log file is created when you start the opscenterd process. If opscenterd fails before the opscenterd.log is created, you can use the debug information and stack traces in startup.log to troubleshoot the issue. The location of startup.log depends on your OpsCenter installation type:

  • Package installations: /var/log/opscenter/startup.log

  • Tarball installations: INSTALL_DIRECTORY/opscenterd/log/startup.log

If opscenterd fails or experiences an issue after startup, use opscenterd.log to troubleshoot the issue. This log file is stored in the same directory as startup.log.

Increase the log level for debugging

For more information when troubleshooting, change the log level from INFO to DEBUG in the <logger/> line of logback.xml. The location of this file depends on the type of installation:

  • Package installations: /etc/opscenter/logback.xml

  • Tarball installations: INSTALL_DIRECTORY/conf/logback.xml

<!-- Default log level -->
<logger name="org.apache.directory" level="INFO" additivity="false"/>

<!-- Log level for troubleshooting -->
<logger name="org.apache.directory" level="DEBUG"/>

Restart OpsCenter to apply the change.

The DEBUG logging level gives you extreme visibility into the exact queries that OpsCenter is sending to the LDAP server along with the responses. Run some test queries, and then check the logs for the detailed LDAP interactions.

Revert the log level in logback.xml when you are done debugging.

Troubleshoot DataStax Agent issues

Use this troubleshooting reference for resolving issues with the DataStax Agent. These issues are shown in the Agents View along with troubleshooting advice and relevant links.

No agents detected

The agents are not installed. Click Set Up Agents to automatically install or start agents, or follow the documentation to manually install agents.

Agents are not connected to OpsCenter because they are misconfigured

Click Configure Agents to update the stomp configuration for the misconfigured agents.

Package-installed agent has the wrong version

The agent version installed for a package installation type does not match the currently installed agent version in OpsCenter. Click Upgrade Agents to install the correct version of the agent on all nodes.

Tar-installed agent has the wrong version

The agent version being attempted to install for a tarball installation type does not match the currently installed agent version in OpsCenter. Tarball agent installs cannot be automatically upgraded. For upgrade instructions, see Upgrade DSE OpsCenter.

The DSE cluster being monitored is misconfigured

The DSE configuration does not match for all nodes in your cluster.

No HTTP communication to the agent

OpsCenter cannot talk to these agents over HTTP. Check that the OpsCenter machine can reach these agents via HTTP and check that SSL is correctly enabled or disabled in both OpsCenter and agent configurations. See Configure SSL for more information.

The DSE storage cluster is misconfigured

The configuration for the DSE storage cluster does not match for all nodes in your cluster. Check the storage cluster configuration as explained in Use a separate storage cluster.

Node down

The agent has confirmed that a node is down and requires investigation into the root cause. Down nodes are flagged in the Agents view and Ring view. You can also can add an alert for down nodes. To receive notifications for down nodes outside of the OpsCenter interface, configure alert notification.

JMX is misconfigured

  • The agent is unable to connect to JMX with the settings configured in OpsCenter:

    Update the configuration in the OpsCenter Connection Settings > Edit Cluster dialog if all nodes have this error, or check the node-specific JMX config if this is an issue on a subset of nodes.

  • The agent is unable to connect to JMX and is using advanced JMX settings configured in address.yaml. Check and update the JMX settings on the node:

    • jmx_host: Host used to connect to local JMX server. The default setting is localhost. This information will be sent by opscenterd for convenience, but can be configured locally as needed. Example: jmx_host: 127.0.0.1

    • jmx_port: Port used to connect to local JMX server. The default setting is 7199. This information is sent by opscenterd for convenience, but can be configured locally as needed. Example: jmx_port: 7199

    • jmx_user: The username used to connect to the local JMX server. Example: jmx_user: jmx-username

    • jmx_pass: The password used to connect to the local JMX server. Example: jmx_pass: jmx-password. This field can be encrypted for additional security.

    • jmx_metrics_threadpool_size: The size of the thread pool used for collecting metrics over JMX. Example: jmx_metrics_threadpool_size: 6

After editing configuration files, you must restart agents to apply the changes.

Troubleshoot OpsCenter authentication issues

This section provides guidance for troubleshooting authentication issues in OpsCenter, including LDAP, OIDC/OAuth 2.0, and Kerberos.

LDAP errors

Failed to log in: {'desc': 'Protocol error'}

Error thrown when a group_search_filter_with_dn is not specified in opscenterd.conf.

Failed to log in: Invalid username or password

This error might occur even with a valid username and password. If so, OpsCenter might not have found a matching role for the user in OpsCenter.

For non-admin roles, create a role in OpsCenter with the same name as the LDAP group, and make sure the LDAP group mapping option is configured correctly to discover roles. For admin roles, specify the admin group name in the admin_group_name option in the OpsCenter LDAP configuration.

For more information, see Authenticate OpsCenter users with LDAP.

Failed to log in: User NAME has no roles defined in LDAP

This happens when the LDAP search returns zero roles for the authenticated user.

  • If you are using a group_search_type of directory_search, modify your group_search_filter_with_dn so that one or more role names are returned from LDAP. Note that exactly one of those LDAP role names must match an OpsCenter role name for authentication to be successful.

  • If you are using a group_search_type of memberof_search, ensure the user_memberof_attribute for the user contains a list of LDAP groups which names an OpsCenter role.

Failed to log in: User NAME has no matching OpsCenter role in LDAP group(s)

This happens when a user belongs to one or more LDAP groups, but none of those groups match any roles defined in OpsCenter. The list of User’s groups returned from LDAP must contain at least one of the roles configured in OpsCenter. See Authenticate OpsCenter users with LDAP.

  • If you are using a group_search_type of directory_search, modify the group_search_filter_with_dn so that the returned list of LDAP groups contains exactly one of the OpsCenter role names.

  • If you are using a group_search_type of memberof_search, ensure the user_memberof_attribute for the user contains a list of LDAP groups that names an OpsCenter role. If using Active Directory (AD), the user should belong to at least one non-special group that is not a built-in group. A special group is one of the built-in Active Directory groups such as Domain Users.

    All OpsCenter AD users must be part of the Domain Users group for the OpsCenter LDAP integration to function correctly. If Active Directory users are not a member of Domain Users, those users are not found in LDAP directory searches.

Failed to log in: Specified search user username unable to bind

Result when searching with an incorrect username for the search_dn or the incorrect password for the search_password criteria.

In order to perform this operation a successful bind must be completed on the connection

The initial search is failing because the user doesn’t have enough permissions for the way the query is structured. There are a few things to try:

  • Set user_search_filter to the default value: user_search_filter = (sAMAccountName={0})

  • Try changing user_search_base so that search doesn’t start from top-level domain: user_search_base = OU=Users,OU=Corp,DC=[value],DC=[value],DC=com

  • If you have a custom schema, change the group_search_filter and group_search_filter_dn to match it.

OIDC/OAuth 2.0 errors

Verify OIDC discovery from the OpsCenter server

Run curl from the OpsCenter host to confirm network connectivity and discovery metadata accessibility:

curl -s https://IDP_HOST/REALM_OR_TENANT/.well-known/openid-configuration | grep jwks_uri
Inspect authentication log output

Check opscenterd.log for authentication and claim extraction errors.

User cannot log in or receives unauthorized error
  • Verify that the user belongs to at least one group in the IdP that matches a defined role in OpsCenter.

  • Verify that the system clock on the OpsCenter server synchronizes with an NTP server. Large time discrepancies cause token expiration and validation failures.

Redirect URI mismatch

Ensure that the redirect_uri configured in opscenterd.conf matches the redirect URI registered in the IdP, including protocol (https), host name, port, and the path /auth/oidc/callback.

TLS or certificate validation failures

If your identity provider uses certificates signed by an internal or custom Certificate Authority (CA), import the CA certificate into the Java truststore used by OpsCenter:

keytool -importcert -alias idp-ca \
  -file /path/to/ca-cert.pem \
  -keystore $JAVA_HOME/lib/security/cacerts \
  -storepass changeit -noprompt

Kerberos errors

Enable debug logs for Kerberos connections:

  1. Edit the cluster-specific configuration file, CLUSTER_NAME.conf.

    The location of this file depends on the type of installation:

    • Package installations: /etc/opscenter/clusters/CLUSTER_NAME.conf

    • Tarball installations: INSTALL_DIRECTORY/conf/clusters/CLUSTER_NAME.conf

  2. Add the following to the [kerberos] section to output debug messages during Kerberos connections attempts from OpsCenter:

    [kerberos]
    debug = True

    The debug option outputs the contents of the server section from the jaas-krb5.conf file, informing you of the settings in use that you can verify against your configuration settings.

  3. If deeper debugging is necessary, add -Dsun.security.krb5.debug=true to $OPSC_JVM_OPTS. The JVM parameter outputs verbose information about reasons why Kerberos connections attempts are failing, such as not authenticating due to key expiration, or no keys present in keytab, or cannot find keytab, for example.

    For more information about JVM, see Configure the OpsCenter JVM.

  4. Restart OpsCenter.

  5. Attempt to connect with Kerberos, and then check the logs for Kerberos debug messages.

Troubleshoot Backup Service errors

Amazon S3 errors

If you are using an Amazon S3 location for storing backups or commit logs, you might encounter errors if the permissions or authentication keys have been changed since the backup job was created. Input updated authentication tokens with permissions to write to the specified bucket name.

The Backup Service requires control over the data and structure of its destination locations. The backup destinations must be dedicated for use only by OpsCenter. Any additional directories or files in those destinations can prevent the Backup Service from properly conducting a Backup or Restore operation.

Insufficient privileges as an AWS IAM (Identity Access Management) user causes an authentication error that shows the message "Unable to authenticate against AWS with the provided key and secret." Ensure the user has base privileges as well as the privilege for the ListAllMyBuckets action:

"Action": "s3:ListAllMyBuckets"

Actions to consider for granting AWS IAM user privileges:

  • CreateBucket

  • GetBucketLocation

  • ListBucket

  • ListAllMyBuckets

  • PutObject

  • GetObject

  • DeleteObject

DataStax Agent errors

All the nodes in your cluster must use Java 8 for the Backup Service to work.

The DataStax Agent and DSE user must be the same (the default), or the DataStax Agent user must have the correct permissions to read and modify the files owned by the DSE user.

Troubleshoot Repair Service errors

To resolve errors, try adjusting the Repair Service configuration options in the [repair_service] section of the following files:

File Scope Package installation location Tarball installation location

opscenterd.conf

Global, all clusters

/etc/opscenter/opscenterd.conf

INSTALL_DIRECTORY/conf/opscenterd.conf

CLUSTER_NAME.conf

Cluster-specific

/etc/opscenter/clusters/

INSTALL_DIRECTORY/conf/clusters/

Errors encountered when running the DSE OpsCenter Repair Service can include:

Error of a single repair task

When a single repair task fails, the repair is skipped temporarily and added to the end of the queue of repairs and retried later. If a single repair fails ten times (default), the Repair Service fires an alert. Adjust this setting with the single_task_err_threshold option.

Incremental error alert threshold exceeded

By default, the number of failed incremental repair attempts defaults to 20 before sending an alert that there could be a problem with incremental repair. Adjust this setting with the incremental_err_alert_threshold option.

Offline splits

At the beginning of each cycle, the Repair Service attempts to generate intelligent subrange splits based on the system.size_estimates table. The subrange splits cannot happen when a node or agent is down or unavailable. If a node or agent is unavailable when the subrange determination happens, Offline Splits are used.

In large or dense clusters, these offline subrange calculations can often be inefficient. The best way to detect that a Repair Service cycle has fallen back to offline splits is to monitor the Repair Service log for using offline task generation. If offline splits are detected, restart the Repair Service once all nodes/agents are up and available.

Repair history tables

DSE stores repair events details in the system_distributed.repair_history and system_distributed.parent_repair_history tables. By default these tables have no time to live (TTL), which can lead to significant unnecessary disk usage because of the number of repair tasks being run continuously.

Manually set a TTL on these tables based on your needs. In most cases, the TTL should be larger than gc_grace_seconds, but not more than needed for debugging purposes.

Skipping range because pending repairs exceeds the max repairs

The Repair Service skips repairing a range if pending repairs exceed the maximum pending repairs, which is 5 by default. The Repair Service immediately moves the skipped repair task to the end of the repair queue and fires an alert. At your discretion, you might want to restart any stalled nodes. Adjust this setting with the max_pending_repairs option.

Timeouts

The Repair Service times out a single repair task after one hour by default. This counts towards an error for that repair task and it is placed at the end of the queue of repairs and retried later. Adjust this setting with the single_repair_timeout option.

Too many repairs in parallel

The Repair Service errors if it has to run too many repairs in parallel. By default, this error happens if it estimates that it needs to run more than one repair in a single replica set to complete on time. Try increasing the Time to completion parameter. If that does not resolve the issue, try adjusting the max_parallel_repairs option. See Set the maximum for parallel subrange repairs.

DataStax recommends only manually adjusting the max_parallel_repairs, changing min_repair_time and other advanced or expert options only if the time_to_completion_percentage throttle is not is use. See Adjust or disable the throttle for subrange repairs.

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM