Description

The YB-TServer RSS memory increases beyond the configured limits for TServer (default set to 85% of available memory on the node). This is a result of TCMalloc releasing memory to the OS, but OS not reclaiming it when Transparent Huge Pages (THP) is enabled. In certain cases, the TServer memory as reported by TCMalloc is ~50% of overall RSS for TServer (for example, TCMalloc reported: 9.6iGB, TServer RSS measured at: 21GiB).

Impact:

• Node could end up with very little memory available for new allocations and could become completely unresponsive, requiring a VM reboot.
• OS OOM killer can kick in and kill PostgreSQL processes resulting in disruption to the workload.

Mitigation

To mitigate excessive RSS growth in newer kernel versions, we recommend setting max_ptes_none = 0 when THP are enabled, thereby minimizing unnecessary THP allocations.

For environments managed by Yugabyte Anywhere, the recommended solution involves upgrading to Yugabyte Anywhere v2024.2.4 or later and re-provisioning your nodes using automatic provisioning.

Should node re-provisioning not be feasible in the short term, or if Yugabyte Anywhere is not used for universe management, a support workflow is available to configure the necessary kernel settings to address this issue. Refer to Transparent hugepages.

For customers deploying YugabyteDB on Kubernetes and encountering this issue, apply the specified settings to the underlying nodes in the Kubernetes clusters. Comprehensive instructions are provided in Enable transparent hugepages.

Details

Transparent Huge Pages and max_ptes_none

THP is a Linux feature that automatically manages large memory pages to improve performance. Instead of using the default 4 KiB pages, THP combines them into larger 2 MiB pages when possible. This reduces TLB (Translation Lookaside Buffer) misses and lowers CPU overhead from managing page tables, improving performance for memory-intensive workloads.

However, in some cases, especially with latency-sensitive workloads like databases, THP can introduce performance issues. The kernel may attempt memory defragmentation to allocate contiguous 2 MiB pages, consuming CPU and causing latency spikes. This can lead to unpredictable performance and latency spikes as well as OOM issues.

To use THP correctly for YugabyteDB workloads, we recommend tuning the following parameters on RHEL8+, Alma8+ (Kernel 4.18) operating systems:

• Set max_ptes_none to 0.
• Set the value of /sys/kernel/mm/transparent_hugepage/enabled to [always].

This is also described in Enable transparent hugepages.

Why Does Setting max_ptes_none = 0 reduce excess RSS?

The max_ptes_none parameter in the khugepaged daemon sets how many empty page-table entries (PTEs) must exist before a 2 MiB THP is split into 4 KiB pages.

By default (max_ptes_none=511), only completely unused THPs are broken up, so even a single accessed 4 KiB page keeps the full 2 MiB mapped—bloating RSS.

Changing it to 0 makes khugepaged split any THP with at least one unused PTE, unmapping wasted pages and ensuring that tools like top and ps report only actually used memory.

This also improves the memory accounting for the processes as only the allocated 4k pages count towards RSS usage and not the whole 2MB pages, this allows tools like top, ps to report accurate memory used by the services.

Related Articles

• Transparent hugepages causing memory issues