Best practices

Local versus remote SSDs

Kubernetes provides you with an option of employing remote disks using dynamic provisioning or local storage which has to be preprovisioned.

Local storage provides great performance, but the data is not replicated and can be lost if the pod is moved to a different node for maintenance operations, or if a node fails. Remote storage has slightly lower performance but the data is resilient to failures.

The following table summarizes the tradeoffs of local vs remote storage:

Feature Local SSD storage Remote SSD storage
Provision large disk capacity per node Depends on cloud-provider Yes
Disk storage resilient to failures or pod movement No Yes
Performance - latency Lower Higher
Performance - throughput Higher Lower
Typical cost characteristics Lower Higher
Kubernetes provisioning scheme Pre-provisioned Dynamic provisioning

While local storage offers higher throughput and lower latency at a lower cost, it comes with significant limitations. Maintenance operations like cluster node pool upgrades, or rolling operations to upgrade software or set flags can cause a pod to lose its local copy and require a full remote bootstrap of all local tablets from its peers. This can be time consuming for large databases. Further, for large clusters, there is always a non-zero risk of another pod movement happening during these maintenance operations, which would cause data loss for a subset of tablets. Therefore, using local storage is not recommended for most production use cases.