Resolving YSQL Connection Drops (unexpected EOF) Caused by dev-shm Shared Memory Exhaustion in Kubernetes

Problem

New YSQL connections fail at random, a few times a day, in short bursts. Existing sessions keep working and the cluster stays up.

Error / alert observed:

# Client (pgx driver):
failed to connect to `user=<db-user database=<db-name`: ... failed to receive message: unexpected EOF

# yb-tserver logs, three lines in this order, usually within a minute:
W0727 15:01:52.212076 1627074 pg_shared_mem_pool.cc:155] Runtime error
    (yb/tserver/pg_shared_mem_pool.cc:63): Failed to allocate segment of size 524288:
    No space left on device
E0727 15:02:46.755834  343364 tserver_shared_mem.cc:391] Runtime error
    (yb/tserver/tserver_shared_mem.cc:390): Failed to create shared exchange for
    <instance-id/295011, mode: 1, error: No space left on device
F0727 15:02:46.757594 1631499 pggate.cc:371] Check failed: _s.ok() Bad status: Runtime error ...

Note: This is not a tserver crash. The F line writes a yb-tserver.FATAL.* file into the tserver log directory, which makes it look like one. The process that aborts is a PostgreSQL backend.

Root Cause

Issue type: sizing mismatch between a YugabyteDB default and a Kubernetes container default. It is a configuration defect, not a leak and not a workload problem.

/dev/shm in the container is the runtime default of 64 MiB, because the Helm chart declares no /dev/shm volume. The gflag big_shared_memory_allocated_limit defaults to 128 MiB, so YugabyteDB's big shared memory pool can request twice what the tmpfs holds.

Two consumers share that tmpfs:

Diagnosis

Run the following before applying any fix to confirm the tmpfs is full and that YugabyteDB is what fills it. Run it on an affected pod. The support bundle cannot answer this, because Kubernetes universes ship no node_exporter.

$ kubectl -n <namespace exec -it <tserver-pod -c yb-tserver -- bash
$ df -h /dev/shm
$ df -i /dev/shm
$ ls /dev/shm \
  | sed 's/_[0-9]*$//' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head

Output when the issue is present:

Filesystem      Size  Used Avail Use% Mounted on
shm              64M   64M     0 100% /dev/shm

Filesystem      Inodes IUsed   IFree IUse% Mounted on
shm            1014051   162 1013889    1% /dev/shm

     64 yb_pg_<instance-id__big
     98 yb_pg_<instance-id

Size 64M at 100% with yb_pg_*__big entries dominating confirms it. If something else fills /dev/shm, stop here: the fixes below will not help. Collect df -i as well, because each failed session leaves a zero-byte segment name that uses an inode, and inode exhaustion selects a second failure path that cores the whole tserver on 2024.2.x.

Resolution

WARNING: Steps 2 and 3 need a rolling restart of the tservers. Step 1 needs no restart.

Immediate mitigation, no restart.

• Set pg_client_use_shared_memory to false on all tservers.
• Reason: it removes the shared-memory allocation from the connect path, so the failure cannot happen there. Value rationale: false is the only value that removes it.

yb-ts-cli --server_address=<tserver-host:9100 set_flag pg_client_use_shared_memory false

Expected output: Flag pg_client_use_shared_memory set to false, and new connections succeed on the same, still-full /dev/shm.

• On 2025.x, set enable_object_lock_fastpath false first, in a separate call, or the flag is rejected. Batching both in one call also fails.
• set_flag is in memory only and reverts on restart. varz reports it as type: Custom, which looks persisted but is not. Beyond firefighting, set it through YBA Edit Flags, which writes it into the Helm values.
• The traffic moves to the RPC stack. The cost is real but not quantified. On one production universe, PgClientService RPC bytes rose about 5 times, or about 2.4 MB/s per node.

Durable fix A: bound the pool below the tmpfs. Set big_shared_memory_allocated_limit to 16777216 (16 MiB) on TServer, as a persisted gflag through YBA.

• Reason: it caps YugabyteDB's demand below what the tmpfs can hold, so the pool never runs out of space.
• Value rationale: 16 MiB is a quarter of the 64 MiB tmpfs, which leaves room for the 4 KiB session exchanges.
• Tested on 2024.2.4.0-b89 and 2025.2.5.1-b1. We noticed the zero allocation failures while shared memory still enabled.
• This flag is not runtime changeable.

Durable fix B, preferred: size /dev/shm above the pool limit.

• Add following override to the universe's Kubernetes overrides.
• Reason: it raises the tmpfs ceiling above YugabyteDB's own 128 MiB default, so the container default and the product default stop contradicting each other.
• Value rationale: 256Mi is the smallest sensible value while the pool limit stays at 128 MiB; anything  below 128Mi leaves the same mismatch.

tserver:
  extraVolumes:
	- name: dev-shm
	  emptyDir:
		medium: Memory
		sizeLimit: 256Mi
  extraVolumeMounts:
	- name: dev-shm
	  mountPath: /dev/shm

• medium: Memory makes the volume count against the pod memory limit, so a 256 MiB /dev/shm reserves that much of the tserver container's memory.
• Confirm on a non-production universe first, and check that the Kubernetes admission policy allows emptyDir with medium: Memory. As some clusters block it.

Either durable fix removes the failure. Apply the chosen one to all tservers, not only the ones that failed.

Verification

$ kubectl -n <namespace exec <tserver-pod -c yb-tserver -- bash 
$ df -h /dev/shm
$ curl -s http://<tserver-host:9000/varz | grep -E 'big_shared_memory_allocated_limit|pg_client_use_shared_memory'
$ grep -rhoE "Failed to create shared exchange for [0-9a-f]+/[0-9]+" <tserver-log-dir/ | sort -u | wc -l

Expected output (healthy state):

shm             256M  2.1M  254M   1% /dev/shm

--big_shared_memory_allocated_limit=134217728
--pg_client_use_shared_memory=true

0

Permanent Fix

Note: Step 1 is a workaround. It trades unquantified RPC overhead for the connection failures, and it reverts on restart unless persisted through YBA. Steps 2 and 3 are permanent.

Keep /dev/shm at least as large as big_shared_memory_allocated_limit. 

Behaviour by release, verified by reproduction:

Upgrading to 2025.2.4.0-b4 or later removes the user-visible symptom, but /dev/shm still fills, so the sizing fix is still worth applying.

References

• Related KB: "k8s tserver in crashloop due to disk full" (shows a pod df -h with shm 64M, which documents the container default)
• GitHub Issue: #26665 (same error line), #27750 (pool crash when /dev/shm is full), #627 (the chart derives memory_limit_hard_bytes from the container limit; the same reconciliation has never been applied to /dev/shm)