--- title: "Set up Resource Governance" url: "https://docs.yugabyte.com/stable/additional-features/multitenancy/set-up/" --- # Set up Resource Governance Prepare cgroups and enable per-database CPU isolation in YugabyteDB. Prepare cgroups and enable per-database CPU isolation Setting up Resource Governance has two parts: 1. Prepare a writable, dedicated cgroup for the YB-TServer process (a one-time, environment-specific, operating system step that typically requires root). 2. Enable Resource Governance using YB-TServer and YB-Master flags. Setup assumes YugabyteDB is running as a systemd service. ## Prepare cgroups When multitenancy is enabled, the YB-TServer assumes it is started in a dedicated cgroup (the *root cgroup*) that it fully manages. The YB-TServer creates and maintains a cgroup hierarchy under this root cgroup, and assigns CPU limits and threads to descendant cgroups. You must therefore start the YB-TServer inside a cgroup that: - has the CPU controller available; - is writable by the user running the YB-TServer process; and - is a leaf cgroup not managed by any other process (no other process moves processes in or out of it, or creates cgroups underneath it). One TServer per cgroup Multiple YB-TServer processes must not run in the same cgroup. Creating this cgroup and granting write access generally requires root privileges (running the YB-TServer afterwards does not). The exact steps depend on your operating system and how you run the YB-TServer. ### cgroup v2 with systemd On most newer Linux distributions, there is a single unified cgroup hierarchy managed by systemd, so you configure delegation through systemd. If YB-TServer is running as a systemd service under the user slice, make the CPU controller available to the user slice by creating an override for the `user@.service` template (requires root). Create the file `/etc/systemd/system/user@.service.d/override.conf` with the following contents: ```conf [Service] Delegate=pids memory cpu ``` The YB-TServer service unit must also delegate the CPU controller: ```conf [Service] Delegate=cpu ``` If the YB-TServer service unit uses `ExecStartPost=`, `ExecReload=`, `ExecStop=`, or `ExecStopPost=`, it must also delegate a subgroup so that these commands don't interfere with the YB-TServer's own cgroup: ```conf [Service] DelegateSubgroup=ybtserver ``` `DelegateSubgroup=` requires systemd 254 or later. On earlier systemd versions (for example, AlmaLinux 9), create the subgroup and delegate the CPU controller manually in the service before starting the YB-TServer: ```sh SERVICE_CGROUP="/sys/fs/cgroup$(cut -d: -f3 /proc/self/cgroup)" SUBGROUP_NAME="$SERVICE_CGROUP/ybtserver" # Create the subgroup. mkdir -p "$SUBGROUP_NAME" # Move this process into the subgroup. echo $$ > "$SUBGROUP_NAME/cgroup.procs" # Delegate the CPU controller (only works now that no processes are left in $SERVICE_CGROUP). echo +cpu > "$SERVICE_CGROUP/cgroup.subtree_control" # Start yb-tserver in the subgroup. ./yb-tserver ... ``` If YB-TServer is running as a systemd service under the system slice, the CPU controller is available by default, so the `user@.service` override is not needed; delegate the CPU controller on the YB-TServer service unit as shown above. ### Other deployments For deployments that don't rely on systemd (such as cron-based or container-based deployments) or for deployments on systems that use cgroups v1, place the YB-TServer process in a cgroup that meets the [requirements](#prepare-cgroups "requirements"). For cgroups v2 systems, make the CPU controller available by writing `+cpu` to the `cgroup.subtree_control` file of every ancestor cgroup, top down. Under cgroup v2, even with write permissions to a cgroup, a non-root user can move a process into it only if the user has write access to the `cgroup.procs` file of the common ancestor of the target cgroup and the cgroup the user is currently in. Depending on the cgroup chosen, root may be required to start the YB-TServer. Container deployments (Docker and Kubernetes) typically mount the cgroups filesystem read-only. To use multitenancy, the container must be privileged with the root cgroup hierarchy mounted read/write so that child cgroups can be created. Some platforms provide workarounds; for example, GKE supports [writable cgroups in pods](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/writable-cgroups "writable cgroups in pods"). ## Enable and configure Resource Governance Resource Governance is controlled entirely through flags. To enable multitenancy, set the `enable_qos` flag to true on Masters and TServers. Set the following flags to configure multitenancy. Unless noted otherwise, set flags on both YB-Masters and YB-TServers. | Flag | Description | Default | |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| | [enable\_qos](/stable/reference/configuration/yb-tserver/#enable-qos "enable_qos") | Enables per-database CPU limits and the database count cap. When `false` (the default), per-database cgroups are not created and no `qos_*` flag has any effect. | `false` | | [qos\_max\_db\_cpu\_percent](/stable/reference/configuration/yb-tserver/#qos-max-db-cpu-percent "qos_max_db_cpu_percent") | Per-database maximum percentage of the node's non-system-reserved CPU. | `100.0` | | [qos\_max\_db\_count](/stable/reference/configuration/yb-master/#qos-max-db-count "qos_max_db_count") | Master only. Maximum number of (non-template) databases. Sets the effective per-database minimum CPU as `1 / qos_max_db_count`. | `0` | | [qos\_system\_high\_cpu\_reserved\_percent](/stable/reference/configuration/yb-tserver/#qos-system-high-cpu-reserved-percent "qos_system_high_cpu_reserved_percent") | Amount of CPU reserved for high-priority system work (0.0–100.0). | `0` | Note None of the `qos_*` flags take effect unless `enable_qos` is `true`. Because settings are applied as flags, you can also change the flags after the cluster is created. Note that `enable_qos` requires a restart to take effect. ### yugabyted With yugabyted, you are responsible for setting up the root cgroup manually (see [Prepare cgroups](#prepare-cgroups "Prepare cgroups")) because the required setup is highly dependent on your environment. After the cgroup is ready, pass the flags using [--tserver\_flags and --master\_flags](/stable/reference/configuration/yugabyted/#pass-additional-flags-to-yb-master-and-yb-tserver "--tserver_flags and --master_flags") when starting the node. For example, to reserve 5% of CPU for system work, cap each database at 25% of the remaining CPU, and allow at most 20 databases (a 5% per-database minimum): ```sh ./bin/yugabyted start \ --tserver_flags="enable_qos=true,qos_max_db_cpu_percent=25,qos_system_high_cpu_reserved_percent=5" \ --master_flags="enable_qos=true,qos_max_db_cpu_percent=25,qos_system_high_cpu_reserved_percent=5,qos_max_db_count=20" ``` ## Disable Resource Governance To turn off per-database CPU limits, set `enable_qos` to `false` and restart the YB-Master and YB-TServer processes. The cgroup setup performed on the operating system does not need to be undone. ## Learn more - [Quality of service](/stable/develop/quality-of-service/ "Quality of service") for other admission control and resource management controls.