Environment

• YugabyteDB (YCQL)

Issue

AllNodesFailedException occurs when a YCQL client driver cannot connect to any node in the cluster after retries. This typically happens due to misconfigured connection settings, insufficient node visibility, or a cluster-wide failure in rare cases.

Overview

The AllNodesFailedException is thrown when:

• The driver's initial connection string contains insufficient node endpoints (e.g., a single load balancer or DNS entry).
• The driver "sticks" to the initially connected node and cannot fail over to others if that node fails.
• Retries target the same failing node(s) listed in the connection string, even if other healthy nodes exist in the cluster.

Common Issues:

• Single Endpoint Configuration: Limits driver visibility to the full cluster topology, preventing node discovery.
• Overloaded nodes: Overloaded nodes triggering backpressure errors (e.g., Coordinator node overloaded).
• Network/Node Failures: Nodes may become unreachable due to outages, high CPU, or full RPC queues.

Possible Solutions

Check Node Health

• Perform health checks on the nodes to ensure they are operational and not overloaded or unreachable.

Configure Multiple Contact Points

• Specify at least 3 node IPs/hostnames in the connection string to ensure redundancy and enable node discovery:

contact_points: ["node1.ip:9042", "node2.ip:9042", "node3.ip:9042"]

• Why This Works: The driver uses these endpoints to bootstrap cluster topology awareness and retry across nodes.

Scale or Balance Load

• Scale the Cluster: Add nodes if the workload consistently triggers backpressure errors.
• Use Partition-Aware Load Balancing: Yugabytedb’s YCQL drivers include a partition-aware policy to route queries to nodes hosting relevant data, reducing cross-node traffic.

Validate Network

• For YugabyteDB Aeon, ensure VPC peering or direct node access to leverage driver features like node discovery.

Additional Resources

For advanced configurations (e.g., topology-aware routing), refer to YugabyteDB YCQL Smart Drivers Documentation.