Logical replication might cause increase in disk storage footprint

Description

When an index is created in a database after a replication slot is created, CDC retains redundant data for this index in the intent SST files. As a consequence, index tablets will have unbounded intent SST files over time. These can lead to increased disk storage footprint, memory usage spikes, and increased delays during YB-TServer restart.

This issue only affects the logical replication model of CDC. The gRPC model does not have this problem and requires no action.

Mitigation

It is recommended that you upgrade to the releases listed in the summary table where the fix is available. This ensures that the problem, if it exists, will auto-heal, and any further index creation will not cause any problems.

If an upgrade is not feasible, run mitigation any time you create an index after a replication slot is created.

Mitigation on YugabyteDB Anywhere deployments

If you use YugabyteDB Anywhere (YBA), run the unified mitigation script from the YBA host machine. This script wraps the detection and cleanup scripts described in Example and runs them on every YB-TServer, detecting affected tablets and releasing CDC barriers to mitigate the problem.

Mitigation on YugabyteDB Aeon deployments

If you use YugabyteDB Aeon, you do not have SSH access to the database nodes. Contact Yugabyte Support and provide the index names. The support team will run the necessary detection and cleanup scripts.

Mitigation for YugabyteDB deployments

Detection

To detect this issue, run the detection script on every YB-TServer. The script takes Master addresses and the names of indexes as input. It analyzes the tablets belonging to these indexes and prints the tablet IDs on which CDC has set retention barriers in the output.

After the detection script identifies the problematic tablets, verify that these tablets indeed belong to the index tables. You can do this via the YB-TServer UI (port 9000).

Cleanup

Run the cleanup script to release the CDC retention barriers. This script takes the tablet IDs identified by the detection script as input. Note that the yb-ts-cli command that the script invokes might not be present in the binaries of the affected releases. You must import the new yb-ts-cli binaries from the releases with the fix before running the mitigation script. Similar to the detection script, the cleanup script must also be run on every YB-TServer.

There is no retry mechanism in either script. In case of failure, you must retry explicitly.

Any new index created with the replication slots still present will again hit this same issue and will require running the scripts again on every YB-TServer.

Refer to Example for script usage.

How to run the scripts

The following examples show how to run the detection and cleanup scripts on OSS YugabyteDB deployments. The YBA unified mitigation script runs both scripts automatically on every YB-TServer.

Detection script

Usage:

./detect_cdc_barrier_on_indexes.sh --indexes <comma separated list of indexes> 
    --master-addresses <comma separated master addresses with port> [--yb-admin] [--yb-pbc-dump]

• Index name: DB_name.schema_name.index_name
• yb-admin (optional): path to yb-admin binary
• yb-pbc-dump (optional): path to yb-pbc-dump binary

Example usage:

$ ./detect_cdc_barrier_on_indexes.sh --indexes yugabyte.public.tt_idx --master-addresses 127.0.0.1:7100

=== Detection summary ===
Indexes scanned: 3
Tablets discovered: 3
Tablets checked locally: 3
Tablets DETECTED: 3
======================
CDC Barriers have been set on following index tablets:
5922a3c921444c4b9825a63a35594f8b
bbb88f8c4f6f474381441c1c96ed25bb
eb17f4e1a91e4faa955f132fe9baf478

Cleanup script

Usage:

./release_cdc_barriers_on_tablets.sh --server-address <tserver address with port> --tablets <comma separated list of tablet ids> [--yb-ts-cli]

• server-address: IP address and port of the YB-TServer on which the script is being run
• yb-ts-cli (optional): path to the yb-ts-cli binary

Example usage:

$ ./release_cdc_barriers_on_tablets.sh --server-address 127.0.0.1:9100 --tablets 5922a3c921444c4b9825a63a35594f8b,bbb88f8c4f6f474381441c1c96ed25bb,eb17f4e1a91e4faa955f132fe9baf478

=== Release summary ===
Tablets requested:  3
Tablets SUCCEEDED:  3
Tablets SKIPPED:    0
Tablets FAILED:     0
=======================
Tablets from which CDC retention barriers were SUCCESSFULLY removed:
5922a3c921444c4b9825a63a35594f8b
bbb88f8c4f6f474381441c1c96ed25bb
eb17f4e1a91e4faa955f132fe9baf478