How to resolve AllNodesFailedException

Problem

AllNodesFailedException is thrown when a YCQL client driver cannot connect to any node it is allowed to try, either while creating the session (at application startup) or while executing a query. It usually points to a connection-configuration or node-reachability problem, and only rarely to a cluster-wide failure.

Error / alert observed:

# At session/application startup (no contact point reachable):
com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point,
 make sure you've provided valid addresses

# At query time (every node tried for the request failed):
com.datastax.oss.driver.api.core.AllNodesFailedException: All 3 node(s) tried for the query
 failed (showing first 3 nodes, use getAllErrors() for more: Node(...): ...)

Common symptoms:

• The connection string contains insufficient node endpoints (for example a single load balancer or DNS entry), so the driver cannot see or fail over to the rest of the cluster.
• The application fails to boot when the contact point(s) are down at startup, but the same code connects fine once the nodes are back.
• Errors coincide with overloaded nodes (for example Coordinator node overloaded) or with node/network outages.

Root Cause

Issue type: Client-side connection configuration and node reachability (rarely a cluster-wide failure).

The driver throws AllNodesFailedException when every node in its current plan fails or is unreachable. The usual contributors are:

• Single or insufficient contact points. One load-balancer address or DNS entry limits the driver's visibility of the cluster topology and leaves no alternate node to try.
• Overloaded nodes. Backpressure errors such as Coordinator node overloaded cause each attempt to fail.
• Node or network failures. Nodes become unreachable due to outages, high CPU, full RPC queues, or missing network routes (for example no VPC peering or no direct node access in YugabyteDB Aeon).
• Fail-fast at startup (reconnect-on-init = false). In the DataStax Java driver 4.x (which the YCQL 4.x driver is based on), datastax-java-driver.advanced.reconnect-on-init defaults to false. If the driver cannot reach any contact point at the moment the session is first built, session creation fails immediately instead of retrying, so a transient outage at boot stops the application from starting.

Scenario: a single load-balancer endpoint routed to a node that is down

A common variant is an application whose only contact point is a load-balancer address (a single VIP or DNS name) fronting the YCQL nodes. At startup the driver has just that one address to try. If the load balancer forwards the initial connection to a backend node that is currently down, the attempt fails, and because there is no other contact point to fall back to, session creation fails with AllNodesFailedException, even though the rest of the cluster is healthy.

This case needs both fixes below working together: multiple real contact points so the driver is not dependent on one routing decision, and reconnect-on-init = true so a transient bad route at startup is retried rather than fatal. The load balancer should also health-check its backends and stop routing new connections to a node that is down.

Resolution

Note: These are client-side and operational changes. No change is made to cluster data, so there is no data-loss risk on the database side.

Apply the solutions that fit the cause found in Diagnosis. The first two together address the most common case.

1. Configure multiple contact points (at least 3 node IPs or hostnames). This gives the driver redundancy at startup and lets it bootstrap full cluster-topology awareness for node discovery and failover.

   datastax-java-driver.basic.contact-points = ["node1.ip:9042", "node2.ip:9042", "node3.ip:9042"]

2. Enable reconnect-on-init so startup is resilient to a contact point being briefly unavailable. Instead of failing session creation, the driver retries the initial connection (per the reconnection policy) until at least one contact point replies.

   In application.conf (HOCON):

   datastax-java-driver {
     advanced.reconnect-on-init = true
   }

   Or programmatically:

   DriverConfigLoader loader = DriverConfigLoader.programmaticBuilder()
       .withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
       .build();
   CqlSession session = CqlSession.builder().withConfigLoader(loader).build();

3. Check node health. Confirm the nodes are operational and not overloaded or unreachable. Address any node that is down, has high CPU, or has a full RPC queue before assuming a client-side cause.
4. Scale or balance the load when the cause is backpressure.
   • Add nodes if the workload consistently triggers Coordinator node overloaded.
   • Rely on the YCQL driver's partition-aware load-balancing policy, which routes queries to the node hosting the relevant data and reduces cross-node traffic.
5. Validate the network. Ensure the application can reach the individual node addresses, not just a front-end endpoint. The driver discovers node addresses from the cluster and then connects to them directly, so those addresses must be routable from the client. For YugabyteDB Aeon, confirm VPC peering or direct node access.

Verification

Confirm the fix before closing the ticket. Restart the application while one contact point is briefly unavailable:

grep -iE "reconnect|contact point|AllNodesFailed|Control connection" application.log

Expected output (healthy state):

# No AllNodesFailedException at startup. The driver logs reconnection attempts and then connects, e.g.:
[s0] Error connecting to <node, trying next node ... scheduling reconnection ...
[s0] Control connection established, application continues.

Permanent Fix

Note: The resolution above is the permanent fix. Bake it into the application's standard driver configuration so every deployment and restart is resilient.

datastax-java-driver {
  basic.contact-points = ["node1.ip:9042", "node2.ip:9042", "node3.ip:9042"]
  advanced.reconnect-on-init = true
}

References

• YugabyteDB YCQL smart drivers: Multiple contact points
• YugabyteDB YCQL smart drivers: Connection handling
• DataStax Java Driver 4.x: Reconnection (reconnect-on-init)
• DataStax Java Driver 4.x: Reference configuration
• Example usage in YugabyteDB docs: Akka Persistence Cassandra integration (sets datastax-java-driver.advanced.reconnect-on-init = true)