Disk storage performance for Customer Edge

Why disk (and etcd) matters on a Customer Edge

Every F5 Distributed Cloud Customer Edge (CE) node runs a Kubernetes control plane backed by etcd. etcd is a strongly consistent, disk-backed key-value store. It fsyncs every write to disk before etcd acknowledges the write, and on a multi-node CE, a majority (quorum) of members must persist each write. If the disk can't keep up, etcd request latency rises and Raft heartbeats fail. The control plane, which includes kube-apiserver, kube-scheduler, and kube-controller-manager, then degrades or restarts.

Note: Disk speed is the biggest factor in control plane stability.

Recommended etcd hardware/IOPS values

See the official source: Hardware recommendations (v3.5).

Disk: the critical resource

etcd is sensitive to disk write latency.

Deployment	Minimum sequential IOPS	Notes
Light load/development	50 sequential input/output operations per second (IOPS)	For example, a 7,200 revolutions per minute (RPM) disk.
Heavy load/production	500 sequential IOPS	For example, a local solid-state drive (SSD) or high-performance virtualized block device.

Important: Sequential IOPS and concurrent IOPS aren't the same. Cloud providers publish concurrent IOPS, which can be about 10 times higher than sequential IOPS. A disk advertised as 3,000 IOPS by a cloud vendor may deliver only 300 sequential IOPS. Always benchmark sequential IOPS with fio before you trust the vendor number.

See the following for additional disk guidance:

• Back up etcd using an SSD. SSDs give lower and less-variable write latency than spinning disks.
• If a spinning disk is unavoidable, use the fastest available (15,000 RPM) or RAID 0.
• Disk bandwidth: 10 MBps recovers about 100 MB in 15 seconds. Target 100 MBps or more for large clusters.
• Reference healthy baselines:
  • WAL fsync p99: less than 10 milliseconds
  • Backend commit p99: less than 25 milliseconds
  • etcd peer round-trip p99: less than 10 milliseconds
  • Leader changes: 0 per week

Why low IOPS breaks etcd and VES services

Warning: If you deploy a CE on an under-provisioned disk (low sequential IOPS), the Kubernetes control plane and the VES configuration pipeline destabilize. Operators often misread this failure as a "kube-scheduler bug" or "kube-apiserver crash" when the true root cause is disk (or internode network) latency.

See the following potential failures:

• etcd WAL fsync stalls: Every write to etcd requires an fsync to disk. If the fsync takes more than 500 milliseconds, etcd logs slow fdatasync warnings. Above 1 second, the leader misses the Raft heartbeat.

• Raft leader-election timeout: etcd uses a 1,000-millisecond election timeout by default. A slow disk causes the leader to miss heartbeats and the followers trigger new elections. The election storm can cause split-brain state or full unavailability.

• VER plus Kubernetes control plane unavailable: The Kubernetes API server and the scheduler also write to etcd. A slow disk causes API server timeouts (etcdserver: request timed out) and the scheduler and the kube-controller-manager each lose their leader lease (failed to renew lease ... context deadline exceeded). Then the scheduler no longer places pods and static pods in the control plane enter a crash loop.

---