Description

During the initial snapshot, the gRPC connector reads each tablet in batches. Every request also tells the server how far Kafka Connect has acknowledged that tablet's snapshot. The server stores this position, and a connector task that restarts resumes the tablet's snapshot from it.

Due to a bug in the connector, from the second batch of each tablet onward the position it reports as acknowledged is the position from which it is about to read the next batch, not the last row Kafka has acknowledged. The server therefore stores the start of the most recently requested batch as the resume point, rather than the last acknowledged row. If a task stops while rows from earlier batches are still in its internal queue or still being written to Kafka, the restarted task resumes from the start of that last batch and re-reads it, but it never re-reads the earlier rows that Kafka had not yet acknowledged, so those rows are skipped. When that happens:

• The skipped rows never reach the Kafka topics, and the connector reports no error or warning.
• The snapshot completes normally and streaming continues, so the gap is visible only when the target is compared with the source tables.
• The data stored in YugabyteDB is not affected.

Who is most at risk

Any deployment of an affected connector release that takes a snapshot (snapshot.mode set to initial or initial_only).

The risk grows with the length of the snapshot, because a long snapshot of large tables is more likely to see a task restart. It is highest where tasks restart while a snapshot is running. For example, after a task failure caused by a producer error such as a missing topic or ACL (Access Control List); a crash or restart of a Kafka Connect worker; a rebalance after workers are added or removed or the connector configuration changes (including tasks.max); a new table that the connector adds automatically; or a restart of the connector.

Streaming is not affected. A task that restarts after its snapshot has completed does not skip rows because of this issue.

Mitigation

Upgrade the gRPC connector to dz.1.9.5.yb.grpc.2026.1.2.0.3. Upgrading prevents new gaps but does not restore rows that were skipped before the upgrade.

Until you upgrade, do the following steps.

Avoid restarting tasks during a snapshot

Until you upgrade, avoid stopping connector tasks while a snapshot is running. Let the snapshot complete before you restart the connector, change its configuration or tasks.max, restart, scale or upgrade the Kafka Connect cluster, or create tables that the connector adds automatically (auto.add.new.tables is on by default), because each of these restarts the connector's tasks.

Before you start a snapshot, make sure every topic, ACL, and quota the connector writes to is in place, so that a producer error does not stop a task partway through.

Check whether a snapshot was interrupted

When a task starts or resumes a snapshot, it logs one line per tablet that begins with Checkpoint before snapshotting tablet and ends with the SnapshotKey stored on the server. If a task started while a snapshot was in progress and this line shows a non-empty SnapshotKey for a tablet, the snapshot of that tablet resumed partway through, and rows of that table may be missing. An empty SnapshotKey, shown as [], means the snapshot of that tablet started again from the beginning, which does not skip rows.

Reconcile tables after a restart during a snapshot

If a task restarted during a snapshot, reconcile the affected tables. After the snapshot completes, compare the row count or the primary keys in the target with the source table. The connector cannot replay a snapshot from a chosen point, so contact Yugabyte Support if rows are missing.

Details

How the snapshot is checkpointed

Each snapshot request carries two positions, the one to read from next and the one Kafka Connect has acknowledged. The server keeps the acknowledged position for each tablet, and a task that restarts resumes the tablet's snapshot from there. All positions within one tablet's snapshot differ only in the row key at which a batch starts.

Root cause

Before sending the acknowledged position, the connector makes sure it is not ahead of the read position. That check was written for streaming, where positions advance with every batch, and it does not look at the row key. During a snapshot it therefore treats every position of a tablet as equal and always sends the read position, so with every request from the second batch onward the server overwrites the restart point with the start key of the batch being requested.

User-visible impact

The skipped rows are the ones read in batches before the most recently requested batch that Kafka had not yet acknowledged when the task stopped. The most recently requested batch is re-read from its start key after a restart, so its rows are not skipped. The number of skipped rows therefore depends on how far reading in the earlier batches had run ahead of delivery.

Fix

In dz.1.9.5.yb.grpc.2026.1.2.0.3, the connector orders snapshot positions by row key, in the order in which YugabyteDB stores rows, and sends the last acknowledged position. After a restart, the snapshot resumes at or before the last row Kafka acknowledged, so some rows may be delivered twice, which at-least-once delivery allows, but none are skipped. Streaming behavior is unchanged.