Check system statistics
There are some system statistics that might help with troublshooting.
Host resource usage
To check the CPU, memory, and disk usage on a Linux machine, you can run the following command:
sudo echo -n "CPUs: ";cat /proc/cpuinfo | grep processor | wc -l; echo -n "Mem: ";free -h | grep Mem | tr -s " " | cut -d" " -f 2; echo -n "Disk: "; df -h / | grep -v Filesystem;
CPUs: 72 Mem: 251G Disk: /dev/sda2 160G 13G 148G 8% / 10.1.12.104 CPUs: 88 Mem: 251G Disk: /dev/sda2 208G 22G 187G 11% / 10.1.12.105 CPUs: 88 Mem: 251G Disk: /dev/sda2 208G 5.1G 203G 3% /
Generally, common tools such as
iostat may be useful.
top reports high CPU usage for the
auditd process, it may have rules auditing some system calls frequently used by YugabyteDB which can significantly affect performance. You can try temporarily disabling
audit by running the following command on each YugabyteDB node:
auditctl -e 0
Then you would check if this improves performance.
audit afterwards, run the following command:
auditctl -e 1
YugabyteDB processes state
YugabyteDB provides the following web endpoints where the current state of each process is aggregated. This includes logs, flags, as well as memory, disk, and network usage metrics. Additionally, it provides dedicated metrics endpoints for YCQL and, respectively, Redis requests:
|Master Web Page||
|TServer Web Page||
yb-ctl locally with default values, three local IP addresses are created:
127.0.0.3, one for each YugabyteDB node.