Run the TPC-C performance benchmark

Detailed steps to run the TPCC benchmark

To run the TPC-C benchmark with an efficiency of around 99.7%, choose a cluster specification (size and instance type) depending on your estimated IOPS, estimated data size, and expected latency.

You can use the following table for reference.

Cluster	Warehouses	tpmC	Efficiency	Latency	Data size (GB)	DB IOPS
3 x 2vCPUs	500	6415.7	99.78	64.08	39.90	1875.77
3 x 4vCPUs	1000	12829.93	99.77	73.97	75.46	3736.53
3 x 8vCPUs	2000	25646.4	99.71	54.21	141.40	7488.01
3 x 16vCPUs	4000	51343.5	99.81	39.46	272.24	14987.79

Instance types

The following table lists the recommended instance types on different cloud providers based on vCPU count.

vCPU	AWS	AZURE	GCP
2	m6i.large	Standard_D2ds_v5	n2-standard-2
4	m6i.xlarge	Standard_D4ds_v5	n2-standard-4
8	m6i.2xlarge	Standard_D8ds_v5	n2-standard-8
16	m6i.4xlarge	Standard_D16ds_v5	n2-standard-16

After you have decided the instance type of the cluster that you need, use the following instructions to run the TPC-C workload.

Get TPC-C binaries

First, you need the benchmark binaries. To download the TPC-C binaries, run the following commands.

$ wget https://github.com/yugabyte/tpcc/releases/latest/download/tpcc.tar.gz
$ tar -zxvf tpcc.tar.gz
$ cd tpcc

Client machine

The client machine is where the benchmark is run from. An 8vCPU machine with at least 16GB memory is recommended. The following instance types are recommended for the client machine.

vCPU	AWS	AZURE	GCP
8	c5.2xlarge	Standard_F8s_v2	n2-highcpu-8

Set up a local cluster

If a local universe is currently running, first destroy it.

Start a local three-node universe with an RF of 3 by first creating a single node, as follows:

./bin/yugabyted start \
                --advertise_address=127.0.0.1 \
                --base_dir=${HOME}/var/node1 \
                --cloud_location=aws.us-east-2.us-east-2a

On macOS, the additional nodes need loopback addresses configured, as follows:

sudo ifconfig lo0 alias 127.0.0.2
sudo ifconfig lo0 alias 127.0.0.3

Next, join more nodes with the previous node as needed. yugabyted automatically applies a replication factor of 3 when a third node is added.

Start the second node as follows:

./bin/yugabyted start \
                --advertise_address=127.0.0.2 \
                --base_dir=${HOME}/var/node2 \
                --cloud_location=aws.us-east-2.us-east-2b \
                --join=127.0.0.1

Start the third node as follows:

./bin/yugabyted start \
                --advertise_address=127.0.0.3 \
                --base_dir=${HOME}/var/node3 \
                --cloud_location=aws.us-east-2.us-east-2c \
                --join=127.0.0.1

After starting the yugabyted processes on all the nodes, configure the data placement constraint of the universe, as follows:

./bin/yugabyted configure data_placement --base_dir=${HOME}/var/node1 --fault_tolerance=zone

This command can be executed on any node where you already started YugabyteDB.

To check the status of a running multi-node universe, run the following command:

./bin/yugabyted status --base_dir=${HOME}/var/node1

Store the IP addresses of the nodes in a shell variable for use in further commands.

IPS=127.0.0.1,127.0.0.2,127.0.0.3

Setup

To set up a cluster, refer to Set up a YugabyteDB Aeon cluster.

Set up connection options correctly

When running the client machine outside the VPC, enable Public Access for the cluster. Also, make sure your client's IP is added to the IP allow list.
Add the username (admin) and password from the credential file that you downloaded when you set up your cluster to the workload_all.xml file.
Make sure you add the full path of the location of the SSL certificate of your cluster in the sslCert tag in the workload_all.xml file.
If you are planning to do a horizontal scale test, set the fault tolerance level to None so that you can add a single node to the cluster.

Store the IP addresses/public address of the cluster in a shell variable for use in further commands.

IPS=<cluster-name/IP>

Configure connection parameters

If needed, options like username, password, port, and so on, can be set up using the configuration file config/workload_all.xml.

<port>5433</port>
<username>yugabyte</username>
<password>***</password>
<sslCert>/Users/johnwick/.credentials/root.crt</sslCert>

Initialize the data

Initialize the database needed for the benchmark by following the instructions specific to your cluster.

Set up the TPC-C database schema with the following command:

$ ./tpccbenchmark --create=true --nodes=${IPS}

Populate the database with data needed for the benchmark with the following command:

$ ./tpccbenchmark --load=true --nodes=${IPS} --warehouses=500 --loaderthreads 48

Set up the TPC-C database schema with the following command:

$ ./tpccbenchmark --create=true --nodes=${IPS}

Populate the database with data needed for the benchmark with the following command:

$ ./tpccbenchmark --load=true --nodes=${IPS} --warehouses=1000 --loaderthreads 48

Set up the TPC-C database schema with the following command:

$ ./tpccbenchmark --create=true --nodes=${IPS}

Populate the database with data needed for the benchmark with the following command:

$ ./tpccbenchmark --load=true --nodes=${IPS} --warehouses=2000 --loaderthreads 48

Set up the TPC-C database schema with the following command:

$ ./tpccbenchmark --create=true --nodes=${IPS}

To populate the database with data needed for the benchmark, use two client machines. Run the following load commands on each of the machines respectively.

Client	Command
1	./tpccbenchmark --load=true --nodes=$IPS --warehouses=1000 --start-warehouse-id=1 --total-warehouses=2000 --loaderthreads 48 --initial-delay-secs=0
2	./tpccbenchmark --load=true --nodes=$IPS --warehouses=1000 --start-warehouse-id=2001 --total-warehouses=2000 --loaderthreads 48 --initial-delay-secs=30

initial-delay-secs has been introduced to avoid a sudden overload on the master to fetch catalogs.

Run the benchmark

Run the benchmark by following instructions specific to your cluster.

$ ./tpccbenchmark --execute=true --warmup-time-secs=60 --nodes=${IPS} --warehouses=500 --num-connections=50

$ ./tpccbenchmark --execute=true --warmup-time-secs=60 --nodes=${IPS} --warehouses=1000 --num-connections=100

$ ./tpccbenchmark --execute=true --warmup-time-secs=60 --nodes=${IPS} --warehouses=2000 --num-connections=200

Run the following commands on each of the client machines respectively.

Client	Command
1	./tpccbenchmark --execute=true --nodes=$IPS --warehouses=2000 --start-warehouse-id=1 --total-warehouses=4000 --num-connections=200 --warmup-time-secs=300
2	./tpccbenchmark --execute=true --nodes=$IPS --warehouses=2000 --start-warehouse-id=2001 --total-warehouses=4000 --num-connections=200 --warmup-time-secs=270

As initial-delay-secs is specified, to ensure that the tests start together, the time was adjusted in the warmup phase.

Analyze the results

On a 2vCPU cluster, 6,415 tpmC was achieved with 99.78% efficiency keeping the new order latency around 64ms.

Metric	Value
Efficiency	99.77
TPMC	12829.93
Average NewOrder Latency (ms)	73.97
Connection Acquisition NewOrder Latency (ms)	0.66
Total YSQL Ops/sec	3736.53

On a 4vCPU cluster, 12,829 tpmC was achieved with 99.77% efficiency keeping the new order latency around 73ms.

Metric	Value
Efficiency	99.71
TPMC	25646.4
Average NewOrder Latency (ms)	54.21
Connection Acquisition NewOrder Latency (ms)	0.55
Total YSQL Ops/sec	7488.01

On an 8vCPU cluster, 25,646 tpmC was achieved with 99.71% efficiency keeping the new order latency around 54ms.

Metric	Value
Efficiency	99.81
TPMC	51343.5
Average NewOrder Latency (ms)	39.46
Connection Acquisition NewOrder Latency (ms)	0.51
Total YSQL Ops/sec	14987.79

On a 16vCPU cluster, 51,343 tpmC was achieved with 99.81% efficiency keeping the new order latency around 39ms.

Conclusion

The TPC-C benchmark provides a rigorous and comprehensive method for evaluating the performance of OLTP systems, focusing on throughput, response time, and data consistency. The results will help you decide if YugabyteDB is the right database for your high-throughput, transaction-heavy applications.