Environment
- YugabyteDB (YSQL)
- Follower Reads or Read Replica Cluster
Issue
When using follower reads or read replicas, queries are unexpectedly being redirected to the leader tablet instead of being served by the local follower or read replica. This can cause:
- Increased latency due to cross-region round trips (especially with read replicas in remote regions)
- Unnecessary load on the primary cluster or leader tablets
- Defeating the purpose of follower reads and read replicas
Additional Information:
By default, YugabyteDB has the ysql_follower_reads_avoid_waiting_for_safe_time flag set to true. When enabled, if a follower or read replica tablet's safe time is behind the requested read time, the tserver immediately rejects the read request with an IllegalState error. This causes the query to be redirected to the leader tablet, under the assumption that the leader will serve it faster than waiting for the follower's safe time to catch up.
This behavior applies to both:
- Follower reads within the primary cluster — reads redirected from a follower tablet to the leader tablet on a different tserver
- Read replica reads — reads redirected from a read replica node to the leader tablet on the primary cluster
While this is generally a good optimization when the leader is nearby, it can be counterproductive for read replicas in remote regions or when you explicitly want reads to stay local.
Resolution
Overview
To prevent follower reads or read replica queries from being redirected to the leader, set the ysql_follower_reads_avoid_waiting_for_safe_time flag to false on the relevant tservers. This tells the follower or read replica to wait for its safe time to catch up rather than redirecting the read to the leader.
Steps
Confirm queries are being redirected to the leader.
Check the tserver logs on the follower or read replica nodes for messages like:
Requested read time is not safe at this follower.
This confirms that the follower/read replica is rejecting reads because its safe time has not yet caught up to the requested read time.
Consider increasing yb_follower_read_staleness_ms first.
Before changing the flag, check if increasing the follower read staleness resolves the problem. The read time sent to the follower is computed as `now() - yb_follower_read_staleness_ms`. A larger staleness value pushes the read time further into the past, making it more likely that the follower's safe time has already caught up — which reduces redirects without changing the flag.
SET yb_follower_read_staleness_ms = 30000; -- 30 seconds (default varies)
yb_follower_read_staleness_ms must be greater than 2 * max_clock_skew (default max_clock_skew_usec is 500000 us = 500 ms, so staleness must be > 1000 ms). If redirects still occur after increasing staleness to an acceptable level, proceed to step 3.Set the flag to false on the appropriate tservers.
- For read replicas: set the flag on all read replica tservers
- For follower reads in the primary cluster: set the flag on all primary cluster tservers
Add the following flag to the tserver configuration:
--ysql_follower_reads_avoid_waiting_for_safe_time=false
Since this is a runtime flag, it can also be changed dynamically without a restart using the yb-ts-cli tool:
yb-ts-cli --server_address=<tablet_server_address> set_flag ysql_follower_reads_avoid_waiting_for_safe_time false
Repeat for each relevant tserver.
- Verify that queries are now being served locally.
After the change, queries should no longer be redirected to the leader. The tserver will instead wait for safe time to advance before serving the read. You can verify by checking:
- The
Requested read time is not safe at this follower.log messages stop appearing on follower/read replica nodes - Query latency patterns shift (slightly higher per-query latency on the follower, but no round trips to the leader)
Trade-offs
| Behavior | true (default) | false |
|---|---|---|
| Follower/replica safe time behind | Rejects read, redirects to leader | Waits for safe time to catch up locally |
| Latency (leader is nearby) | Lower — leader serves immediately | Higher — waits for replication lag |
| Latency (leader is remote) | Higher — round trip to remote leader | Lower — waits locally, avoids cross-region hop |
| Load on leader | Increased — redirected reads hit the leader | Reduced — reads stay on follower/replica |
Timeout/wait risk | None — redirects immediately to leader | Read may wait up to client timeout; times out if replication lag is too high |
Additional Information:
- This flag only affects YSQL follower reads. The transaction or session must also be **read-only** (`SET default_transaction_read_only = true` or `SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY`). The read-write transactions will not use follower reads regardless of these settings.
- When set to
false, the read will wait until the client deadline (query timeout) for safe time to catch up. If safe time does not catch up in time, the query will fail with a timeout error rather than being redirected. - If the followers or read replicas have consistently high replication lag, setting this flag to
falsemay cause queries to time out. In that case, investigate and resolve the replication lag first. - For read replica setups, it is typically sufficient to set this flag only on read replica tservers. For follower reads within the primary cluster, it must be set on the primary tservers themselves.
- The value of
yb_follower_read_staleness_msdirectly affects how often redirects occur. It must be greater than2 * max_clock_skew(with the defaultmax_clock_skew_usecof 500000 us, staleness must be > 1000 ms). A larger staleness value reduces redirects by pushing the read time further into the past, giving the follower's safe time more room to catch up.
Comments
0 comments
Please sign in to leave a comment.