Description

When you create a CDC stream, YB-Master sets a retention barrier on every tablet of every table in the namespace. The change in #19066 made YB-Master apply these barriers in batches of tables, while limiting how many AlterTable RPCs it can have in flight at once.

If one batch holds far more tablets than that limit allows, stream creation slows down and usually times out, even on a healthy cluster. When this happens, yb-admin create_change_data_stream, pg_create_logical_replication_slot(), or CREATE_REPLICATION_SLOT times out, and you can't create or recreate the stream.

This affects CDC stream creation for both the gRPC connector and YSQL logical replication.

Two flags govern this behavior:

Who is most at risk

The risk is highest in namespaces where a few large tables hold most of the tablets. With the default --cdc_create_stream_alter_table_dispatch_batch_size of 10, a single batch can then include nearly every tablet in the namespace, while --max_concurrent_alter_table_rpcs allows only 10 AlterTable RPCs in flight at once. That is, a single batch contains many more tablets than the RPC cap allows.

Mitigation

Upgrade to a release that contains the fix (see the preceding table), or apply one of the following workarounds.

Disable throttling

Disable throttling by setting --max_concurrent_alter_table_rpcs=0 on the YB-Master. This is the recommended workaround, and it restores the behavior clusters had before the change.

--max_concurrent_alter_table_rpcs is runtime modifiable on the YB-Master. The change takes effect immediately, with no restart. The YB-Master may then dispatch as many AlterTable RPCs at once as there are tablets in the batch, spread across all YB-TServers, so it trades the protection the throttle provides for speed.

On releases that include the fix for #33260, 0 is already the default and no action is needed; see Fixed In in the preceding table.

Keep throttling and raise the limit

If you prefer to keep throttling, raise --max_concurrent_alter_table_rpcs on the YB-Master so that a batch completes in only a few rounds. The flag is runtime modifiable on the YB-Master.

As a rule of thumb, set it to the number of tablets in the largest batch divided by 5 to 10. Estimate the tablets in the largest batch as the combined tablet count of the largest N tables, where N is --cdc_create_stream_alter_table_dispatch_batch_size, a runtime modifiable YB-Master flag that defaults to 10. Within 5 to 10 rounds, the accumulated waiting stays under roughly 0.3 seconds and 8 seconds per batch; beyond that, it grows steeply.

For example, a namespace of about 10 tables holding around 5,000 tablets needs a --max_concurrent_alter_table_rpcs value of roughly 500 to 1000.

Raise the stream creation timeout

Raise the stream creation timeout as well, because a slow creation can still exceed the client deadline:

• For pg_create_logical_replication_slot() and CREATE_REPLICATION_SLOT: raise --ysql_ddl_rpc_timeout_sec on the YB-TServer. It is runtime modifiable, but the flag is read by the PostgreSQL process, so a change takes effect only after PostgreSQL restarts. Editing it through YugabyteDB Anywhere applies it correctly, because that performs a rolling restart.
• For the yb-admin create_change_data_stream command: raise the --timeout_ms command-line argument, which applies per invocation and needs no restart.

Note that --max_concurrent_alter_table_rpcs and --cdc_create_stream_alter_table_dispatch_batch_size are YB-Master-only flags; setting them on YB-TServers has no effect.

Details

Batching and throttling

Stream creation walks the namespace's tables in batches and issues one AlterTable RPC per tablet. The throttler caps how many of those RPCs are outstanding at any moment.

Rejected tasks back off exponentially

A task that finds the throttle full is not queued. It is rescheduled on an exponential backoff and uses up one of its limited retry attempts. When a batch holds many more tablets than --max_concurrent_alter_table_rpcs allows in flight, only that many tablets make progress per round, and each successive round waits longer than the one before it.

User-visible impact

Stream creation takes minutes to tens of minutes and usually fails with a timeout before it finishes. In the worst case, the per-task retry budget runs out and creation fails outright. Retrying without changing the flags reproduces the failure.