What's new in the v2.23 preview release series

Improvements

YSQL

• Enhances logging for DDL transaction conflicts and PG catalog version mismatches by including the DDL command tag and specific log details outside of the log_ysql_catalog_versions flag. #20084
• Reduces per-backend memory consumption by reinstating TOAST compression for catalogue tables. #21040
• Enables DDL atomicity feature by default by altering ysql_yb_ddl_rollback_enabled, report_ysql_ddl_txn_status_to_master, and ysql_ddl_transaction_wait_for_ddl_verification flags' defaults. #22097
• Adds a new YSQL view for YCQL statement metrics, allowing it to be joined with YCQL wait events in the yb_active_universe_history table. #20616
• Displays distinct prefix keys explicitly in the explain output, enhancing the clarity of indexing for users. #20831
• Adds auto flag ysql_yb_enable_ddl_atomicity_infra to control DDL atomicity feature during the upgrade phase. #21535
• Updates read time for each operation to simplify code and avoid applying used read time from obsolete operations. #21623
• Allows YbInitPinnedCacheIfNeeded to only load the shared pinned cache, enhancing concurrent handling of DDLs in various databases. #21635
• Rectifies a compilation error by eliminating duplicate declaration and unnecessary function triggered by merge issues. #20616
• Avoids schema version mismatch errors during ALTER TABLE operations in cases where DDL atomicity is enabled. #21787
• Adds new columns to localhost:13000/statements for more comprehensive database management, including user and database IDs along with varied block level statistics. #21735
• Now logs global-impact DDL statements that increment all database catalog versions. #21826
• Reorganizes extensions into three segregated directories for better access and ease of use. #21897
• Resolves schema version mismatch errors that occur after an ALTER TABLE operation due to DDL transaction verification in non-debug builds. #21787
• Introduces a new YSQL configuration parameter yb_parallel_range_size for better tuning of parallel range size. #21928
• Removes the unused keep_order field from YbctidGenerator for cleaner results processing. #21944
• Introduces a new YSQL configuration parameter yb_enable_parallel_append to disable the unannounced feature parallel append. #21934
• Performs stylistic modifications and refactors in various YSQL scripts for better readability and performance. #22004
• Adds support for creating vector indexes using a dummy ANN method ybdummyann, enabling preliminary vector-based searching in databases. #22195
• Restricts the undesired usage of LWFunction by disallowing move copy. #22069
• Simplifies the column binding logic for ybgin and lsm access methods and prepares for addition of user-defined index types. #22195
• Enables the grammar for CREATE/DROP ACCESS METHOD for more flexible extension handling. #22364
• Eradicates misleading log message during table creation with DDL atomicity enabled. #22459
• Introduces a new enum PgYbrowidMode to deduplicate hidden internal column addition logic. #22536
• Adds Save/Restore state functionality to ConsistentReadPoint using a new Momento class. #22597
• Avoids renaming DocDb tables during legacy rewrite operations to prevent issues with backup/restore and improves handling of failed ADD/DROP primary key, ALTER TYPE operations. #22802
• Stops python checks on all third-party extensions to avoid build failure. #23166
• Simplifies and cleans up code in PgDml/PgSelect/PgSelectIndex classes, ensuring only necessary fields are used and removing redundant destructors, with no logic changes. #23192
• Improves backward scans by updating the cost-based optimizer to consider backward scan enhancements, leading to significant execution-time improvements when FLAGS_use_fast_backward_scan is enabled. #22370
• Introduces new role-related flags in yb_backup.py script to enhance backup and restore functionalities. #20972
• Fixes various issues in the Batch Nested Loop Join code, particularly the new row array comparison, for clearer functioning and better documentation. #23407

YCQL

• Now throws an error when using the unsupported GROUP BY clause in YCQL with autoflag ycql_suppress_group_by_error available for compatibility issues. #13956

DocDB

• Adds an ABORTED state and an abort_message field to the SysCloneStateInfoPB object for better clone failure management. #21054
• Offers a new stack trace tracking framework for improved visibility of disk I/O operations, configurable through the track_stack_traces flag. #17993
• Resolves build failure caused by a problematic merge, offering better disk IO visibility by tracking IOs by stacktrace. #17993
• Blocks writes based on the total number of bytes being flushed, not just when 2 memtables are flushing. #22571
• Allows asynchronous DNS cache updating and resolution retry upon failure to reduce RPC call delays and prevent unexpected leadership changes. #22930, #22311
• Enables reduction of duplicate code and custom flag filtering through GetFlagInfos relocation. #23632
• Introduces a new flag to toggle on or off recommended memory defaults for increased control over individual memory settings. #22161
• Introduces utility to dump top heap stacks when memory limit is exceeded for enhanced debugging. #21396
• Deprecates unused flags and old svc_num_workers flags for clearer user configurations. #20906
• Allows local debug builds of YugabyteDB to restart from release builds to enhance debugging. #21093
• Shifts xCluster-related functions from Catalocustomeranager to XClusterSourceManager for easier management. #21325
• Speeds up backward scans by building rows from end to start, reducing unnecessary repositioning and Seek calls, enabled with FLAGS_use_fast_backward_scan flag. #19352
• Boosts error messaging clarity when preview flags are not set in allowed_preview_flags_csv. #21484
• Prevents GetChangesForXCluster from returning an invalid safe time in certain conditions. #21528
• Broadens the rewrite_test_log.py script to incorporate more directory replacements such as home directory, YB_SRC_ROOT, LLVM toolchain directory, and third-party dependencies directory. #21532
• Adjusts TServer memory percentage from 50% to 48% for (4,8]GiB boxes and sets new recommendations for boxes over 16 GiB. #20664
• Incorporates the usearch and fp16 header-only libraries into the src directory, simplifying code import. #21830
• Returns the original error messages from CreateTable and CreateTablegroup instead of an incorrect "Database not found" message. #21760
• Enables replication of DDL schemas and users to maintain consistency between source and target. #21848
• Updates AreNodesSafeToTakeDown to return earlier and deliver a readable error message when hitting a timeout, using a control flag, are_nodes_safe_to_take_down_timeout_buffer_ms with a default setting of 2 seconds. #21855
• Reduces unnecessary alerts by removing "No active snapshot" warning from the logs. #21911
• Ensures Data Definition Language (DDL) operations replicate exactly once by checking the replicated_ddls table prior to rerunning any DDL. #21943
• Streamlines the creation of xCluster streams by unifying the scattered code into XClusterClient::CreateXClusterStream<Async>. #22343
• Allows for faster failover in xCluster DR by skipping the cleanup process when skip_producer_stream_deletion is set on DeleteUniverseReplicationRequestPB. Safe for upgrades and rollbacks. #22050
• Reduces extra reactor threads by reusing server messenger in AutoFlags. #22076
• Reduces resource usage by enabling stateful service client to reuse the server's existing yb_client. #22102
• Adds SCHECK_PB_FIELDS_NOT_EMPTY macro for validating non-empty fields, with individual checks on list elements. #22182
• Splits up yb_xcluster_ddl_replication.c into additional util files for better project extensibility. #22190
• Removes deprecated JSON output format in xCluster for clean and efficient function. #22219
• Allows only single DDL query strings to prevent issues with DDL+DML mixes or multiple DDLs. #22060
• Shifts certain RPC endpoint methods from the Catalocustomeranager to the new MasterClusterHandler class for easier management. #19715
• Adds read-time option description in the help of ysql_dump allowing database dump at a specified past time. #21886
• Renames and replaces cdc_consumer_handler_thread_pool_size with xcluster_consumer_thread_pool_size to reduce CPU and memory usage. #20305
• Refreshes stack trace tracking UI endpoints to enable per-column sorting and optimize sorting script. #22841
• Allows table locking by acquiring local server object locks for DDLs and DMLs, hosted by a local transaction participant, with session ID and tied to their lifetime. Includes performance upgrades to lock acquisition and conflict resolution. #23045
• Simplifies replication setup by using table IDs instead of names, helping avoid issues caused by table renames or recreations. #23013
• Increases speed of backward scans for flat document reader with support for packed row V2. #22556
• Reuses the Tservers yb::client in CDCService to decrease 4 threads and a meta cache, addressing a CDCService/xCluster source issue. #22893
• Allows unified usage of XClusterRemoteClient in XClsuterConsumer, centralizing client creation logic. #22908
• Ensures replication health before succeeding IsSetupUniverseReplicationDone, improving error detection. #22948
• Adds external_hybrid_time to log-dump output to detect xCluster target writes. #22918
• Introduces two new SOURCE_UNREACHABLE and SYSTEM_ERROR enums to enable more detailed error reporting from the Poller. #22996
• Allows requesting streams by producer table ids for xCluster DDL Replication to accurately match tables together. #23013
• Enhances stack trace tracking endpoints usability by adding an access button, a reset tracking link, and timestamp details. #22842
• Renames and shifts various members and functions from Catalocustomeranager to XClusterManager. #23044
• Replaces the deprecated and unused namespace replication with DB Scoped replication for a more efficient solution. #23046
• Makes pggate aware of fast backward scan capability for accurate cost identification in the Cost Based Optimizer. #22937
• Added emergency_repair_mode flag and new yb-admin commands to handle corrupted CatalogEntity data without stopping yb-master. #23098
• Relocates Setup, Bootstrap, Alter and Delete Target Replication functions for better organization. #23183
• Changes column ID representation in debug builds to be compatible with release builds. #21093
• Refactored the PopulateTabletCheckPointInfo function to improve its readability and maintainability. #23301
• Removes display of InvalidFlags in the flags UI to clean up the user interface. #23308
• Allows for storing and loading of vector indexes, ensuring effective management of these indexes. #23377
• Integrates class DocDBStatistics with ReadOperationData for better statistics transmission to IntentAwareIterator. #23420
• Introduces a new flag max_disk_throughput_mbps for automated control of write rejections when disk is full, replacing reject_writes_min_disk_space_aggressive_check_mb. #23373
• Adds a flag FLAGS_tablet_split_min_size_ratio to control tablet splitting based on SST file sizes, ensuring better control over tablet size imbalance. #21458
• Relocates heartbeat code for greater readability and Catalog Manager size reduction. #21899, #19715
• Simplifies the clone state manager by moving persisted data to an in-memory structure. #22138
• Relocates specific heartbeat code to master_heartbeat_service.cc for enhanced readability and easier tracking. #21899, #19715
• Refactors heartbeat path code for easier navigation and reduction of Catalocustomeranager size, with no functional changes. #21899, #19715
• Shifts tcmalloc profiling code to the Utils folder. #22258
• Moves tablet splitting specific RPCs and functions from catalog_manager.cc to tablet_split_manager.cc for better code management. #22603, #19715
• Introduces a new flag enable_rwc_lock_debugging to control slow lock debugging and fixes a bug in rwc_lock.cc. #22807
• Changes CloneStateInfo object from scoped_refptr to std::shared_ptr for standardization. #23036
• Allows setting only non-empty schema names in YBTableName. #23371
• Transfers ownership of TabletSplitManager, CloneStateManager, and SnapshotCoordinator from Catalocustomeranager to Master for leaner dependency requirements. #22603, #19715

CDC

• Preserves CDC stream even when all associated tables are dropped, tying its lifecycle to the database. #21419
• Introduces three new yb-admin commands to remove a user table from a CDCSDK stream, descend dynamic table addition in a CDC stream, and validate CDC state for a particular stream, enhancing control over CDC streams. #22876, #22773
• Prevents addition of tables with enum array column to the CDC stream to avoid crashes during consumption. #22897
• Transforms the flag yb_enable_cdc_consistent_snapshot_streams from a preview into a default true auto flag. #22984
• Enables dynamic table addition with Postgres replication consumption by setting retention barriers on new tables' tablets during creation. #21643
• Allows modification of the publication refresh interval using the cdcsdk_publication_list_refresh_interval_secs flag. #21796
• Adds a TServer flag, ysql_yb_default_replica_identity, for customizing default replica identity at table creation. #22326
• Introduces cdcsdk_enable_dynamic_table_addition flag to manage dynamic table additions in replication slot consumption model. #22406
• Introduces replication slot name for internal distinction between two consumption models in the code. #22810
• Allows creating an old model stream via yb-admin in upgraded environments, ensuring only one stream type per database. #22894
• Introduces replica identity in CDC to populate before image records, allowing table-level before image information fetching and retaining in stream metadata. #21314
• Eliminates unnecessary NOTICE messages when setting yb_read_time from walsender, reducing message clutter. #22379
• Enables transaction state to be cleared promptly after a table is deleted, preventing table deletion from getting stuck and resulting in faster functionality. #22095

yugabyted

• Allows Connection Manager to handle error messages, preventing test failures in the YSQL layer. #21756
• Ensures RENAME DATABASE query handles logical and physical connections correctly for consistent database behavior. #21284
• Redefines tracking of role modifications using role OID in YSQL Connection Manager for accurate behavior during mid-session role renaming. #21505
• Offers support for single-use YSQL configuration parameters in YSQL Connection Manager. #21757
• Enables using role OID with session_authorization in YSQL Connection Manager for correct role alterations in a session. #21637
• Allows sticky connections when setting certain YSQL configuration parameters not permitted in explicit transactions. #22957
• Allows modification of YSQL configuration parameters on a running cluster by destroying the control connection. #21516
• Adds a new /pitr endpoint and screen in the database page to list scheduled PITRs on yugabyted UI. #21355
• Introduces upgrade_finalize command for smoother YugabyteDB version upgrades using yugabyted CLI and includes an upgrade_ysql_timeout flag. #21888
• Directly enables yb_enable_read_committed_isolation and ysql_enable_read_request_caching on yb-master and yb-tserver processes. #22061
• Delivers alerts on user interface when encountering node version mismatches in the cluster. #21888
• Simplifies yugabyted by dropping Python2 support and transitioning the script to use Python3, replacing deprecated distutils package with shutil. #22072, #21409
• Enables better handling of multi-valued flags in yugabyted without duplication, making the system more maintainable. #22091
• Corrects the Sankey diagram for CPU usage by accurately calculating the total number of used/available cores. #22125
• Made changes to string literals in yugabyted to avoid SyntaxWarning and added checks for exceptions during incorrect advertise_address input. #22210, #22230
• Enables correct parsing of startup parameters with spaces in values when using the YSQL Connection Manager. #22248
• Allows to specify multiple data directories using the new additional_data_dir configuration. #22126
• Enables xCluster replication management between database clusters using new yugabyted commands. #22349
• Ensures yugabyted UI metrics display properly with Kubernetes OSS operator deployed clusters. #22532
• Elevates the reliability of the UUID retrieval process for tablet server nodes on the user interface. #22532
• Ensures accurate CPU usage metrics by updating prev_ticks_ at each metrics snapshotting iteration. #22910
• Allows smooth node restart even if the data_dir parameter is missing in the user configuration file. #23052
• Reduces collect_logs command failures by removing the yugabyted running check even when the yugabyted process is not running. #23210
• Enhances yugabyted configure_read_replica commands with checks to gracefully handle failures when data_placement_constraint lacks :<num_of_replicas>. #23273

Bug fixes

YSQL

• Fixes an error that occurs when decoding null values from a boolean column sorted as NULLS LAST in a secondary index. #22121
• Fixes YSQL upgrade single connection mode error preventing new connection attempts before the old ones are released. #22283
• Allows YB Admins to run pg_locks without requiring superuser status. #23266
• Avoids failure when upgrading from version 2.14/2.16 to 2.20 by introducing a check to ensure pggate can handle RPC metrics sidecar before sending Scanned Rows count. #21229
• Fixes memory leaks in pg_constraint/pg_attrdef local cache by adding a missing hash_destroy call in YbCleanupTupleCache. #22262
• Resolves remaining memory leaks in CacheMemoryContext to stabilize cache memory after every catalog cache refresh. #22262
• Documents the limitations of retry logic when using -c flag in ysqlsh command. #21804
• Allows YSQL DDL operations to wait for rollback/roll-forward operations to finish before proceeding. #20033
• Allows more accurate modeling of base scan costs by taking into account the impact of storage index filters on secondary index. #20635
• Resolves colocation option issues in table creation linked to table rewriting and partitioning. Enhances the defGetBoolean function to parse string values "0" and "1" as false and true respectively, and shifts a verification step to a earlier spot in the CREATE TABLE execution path, ensuring successful table partition creation. #20302, #20914
• Renames the YSQL configuration parameter ddl_rollback_enabled to yb_ddl_rollback_enabled for specificity. #21480
• Adds network latency cost to startup cost, yielding more accurate cost calculations in small tables. #20898
• Disables bitmap scan by default to prevent unwarranted selection due to lower CBO costs. #21479
• Reduces unnecessary log messages when catalog_version_table_in_perdb_mode is set to true. #21481
• Corrects an issue where certain unbatchable filters weren't detected during indexpath formation when indexpath accepted batched values from multiple relations. Requires backports to 2.20 and 2.18. #21292
• Exposes the YSQL configuration parameter yb_enable_optimizer_statistics as a flag ysql_yb_enable_optimizer_statistics. #21650
• Corrects buffer overflow during placement validation in ALTER TABLE SET TABLESPACE operation. #21655
• Allows for an enhanced readability and performance of yb_cost_index code, aiding in merging with the pg15 branch. #21672
• The deadlock issue occurring when both a table and its index are deleted concurrently in yb-master has been resolved. #21663
• Refines the YbGetOrdinaryColumnsNeedingPgRecheck condition to align with the ybIsTupMismatch implementation, ensuring Postgres rechecks index conditions when the "preliminary check" is skipped due to an invalid target key attnum. #21451
• Corrects checks in YbIsScanCompatible to ensure the right-hand side (RHS) of all bound index conditions, not just inequalities, fits into the left-hand side (LHS) datatype. #21758
• Prevents query layer retries for multi-statement queries to avoid redoing whole queries, ensuring idempotence. #21361
• Fixes a bug that caused incorrect setting of global catalog version mode on TServer start. #21850
• Fixes a bug in the index tuple width calculation for better YB base scans cost model. #21892
• Ensures pushed down RowCompareExpressions correctly enforce non-null column references, rectifying previous behavior and enhancing data accuracy. #21847
• Reduces the frequency of schema version mismatch errors during consecutive DDL operations by ensuring the up-to-date schema is fetched. #21706
• Allows usage of YsqlDdlRollbackEnabled in pggate C++ code by correctly passing the result of YbDdlRollbackEnabled, reducing DDL atomicity g-flag issues in RF 1 clusters. #21706
• Adds a new flag ysql_min_new_version_ignored_count to prevent a TServer crash caused by the downward shift in yb-master's catalog version, often surfacing post a PITR restore operation. #21776
• The postgres process no longer crashes when running a "show all" command due to correct placement of the yb_enable_ddl_atomicity_infra description. #21947
• Prevents failures in transaction restarts with UPDATE ...RETURNING queries in debug builds. #22010
• Reverts updates from Storage SQL to Remote SQL and Storage Filter to Remote Filter for Foreign Scan. #22070
• Adjusts inaccurate ALTER TABLE rewrite check for dropped rules to prevent unnecessary command failure. #22064
• Fills in the "relation" column in pg_locks with the correct table OID after a table rewrite. #22081
• Adjusts the value of YB_AT_REWRITE_ALTER_PRIMARY_KEY to prevent flag clashes and accommodate future upstream PG flags. #22086
• Fixes the issue of PG crash when yb_debug_log_catcache_events=1 is used before a database has been selected. #22139
• Enables backward parallel scan capabilities, adjusting key bounds when conducting descending order scans. #21633
• Prevents unnecessary CPU cycles and log flooding by not reading pg_yb_catalog_version when enable_ysql=false. #22213
• Corrects the log message for successful column drop operations, ensuring accurate representation of DDL operations. #22243
• Stops Batched Nest Loop (BNL) crashes by ensuring better indexing condition checks. #21954
• Refines the logic to accurately push down join clauses to batched index scans without causing conflicts. #21878
• Grants BNL hashtable its own expression context to prevent data overwrites during query execution. #21266
• Re-enables rechecking for RowCompareExpressions to accurately handle NULL inputs in scan bound calculations. #22075
• Resolves the old-style-declaration error in YbDdlRollbackEnabled by changing its definition to static inline bool. #22334
• Prevents potential crashes by ensuring yb_table_properties pointer, in load_relcache_init_file, does not point to random, invalid memory. #22342
• Makes yb_get_range_split_clause robust using PG TRY CATCH block, ensuring YB backup doesn't fail. #22356
• Fixes memory leaks in ybcFetchNextHeapTuple by properly freeing the YBCStatus. #22396
• Prevents core dumps by ensuring YSQL webserver destruction upon receiving a termination signal. #18948
• Introduces new functions to enhance and consolidate the focus on tables stored in the system catalog. #22520
• Allows CreateNamespaceIfNotExists function to retry on "already exists" error, preventing race conditions. #22512
• Fixes the issue when a separately created and later attached partition does not properly inherit the parent's primary key using ALTER TABLE ...ATTACH PARTITION. #22562
• Resolves potential database OID collision with system_postgres by excluding reserved OID 65535 in allocation. #22598
• Allows skipping the relfilenode check on parent partition tables which do not get recreated during table rewrites. #22625
• Now allows for correct backward prefix-based scanning by eliminating the problematic kGroupEnd marker that was leading to inaccurate seek results. #22615
• Removes the unused function Catalocustomeranager::WaitForDdlVerificationToFinish for clarity. #22649
• Ensures bitmap scans correctly recheck all results and avoid excluding rows, improving accuracy of outcomes. #22622
• Prevents the IN expressions on single column from wrongly taking the tuple path, ensuring correct data processing. #22704
• Fixes incorrect access to the scan plan's bind descriptor during tuple IN condition rechecks. #22800
• Allows the creation of new shared relations during YSQL upgrade to have a global impact by incrementing the catalog version across every database. #22830
• Allows resetting of stats collected by the ANALYZE command, including reltuples, pg_statistic rows, and pg_statistic_ext values. #22028
• Corrects the YbGetOrdinaryColumnsNeedingPgRecheck function to return table column numbers instead of index numbers, preventing unnecessary data fetches and potential crashes or errors after dropping a column. #22832
• Resolves a detected deadlock during ALTER TABLE operations, enhancing test stability. #22882
• Reduces unexpected log messages by not invoking YsqlDdlTxnCompleteCallback if all table 'pb_txn_id's in the DDL transaction verifier state are already cleared, avoiding potential deadlock situations in DDL atomicity. #22882
• Reduces prefix length in the index when using distinct index scan with included columns. #22822
• Returns more accurate results when running EXPLAIN command by fixing relids of prefix keys under a subquery distinct index scan. #22923
• Reduces sequence cache collision by incorporating both database and sequence OIDs as the entry key. #22935
• Prevents a crash related to memory release associated with TupleTableSlots in SubPlans during a Values Scan. #22967
• Allows faster data inserts into tables with identity columns. #22837
• Enhances log output by adding missing newlines in the yb_pclose_check function and corrects memory allocation. #23057
• Streamlines the "drop column" operation process, preventing hindrance even if the alter schema RPC is missed. #23100
• Reduces ASAN/TSAN builds' pressure on t-server/master and avoids timeout issues by using less parallelism. #22594
• Eliminates unnecessary waiting for concurrent transactions in the DEFERRABLE mode for READ ONLY serializable transactions. #23120
• Fixes TServer crash when pushing down certain SAOP operations like string_to_array. #23287

YCQL

• Allows the deletion of the Cassandra role in YCQLsh without it regenerating upon cluster restart, by adding a flag to mark if the role was previously created. #21057
• Removes extra reads during the processing of INSERT INTO ...RETURNS STATUS AS ROW for CQL tables with a specific primary key, improving system load and efficiency. #23330
• Now ensures simultaneous registration of new split tablet children, fully covering the keyspace during splits. #19954

DocDB

• Fixes hidden split parent tablets wrongly appearing as leaderless in the master's leaderless tablet endpoint, ensuring accurate load balance status and preventing potential issues with Point-in-Time Recovery (PITR) operations. #21371
• Ensures failed xCluster setup if the xCluster stream state update to ACTIVE does not occur. #22601
• Fix ensures heartbeat processing doesn't blindly overwrite tablet replica state, avoiding potential data corruption. #21836
• Fixes crash when parsing an invalid timestamp in LTO build by updating C++ dependencies and error handling. #22191
• Removes unnecessary flush during snapshot deletion, preventing write blocks. #22369
• Eliminates possible deadlock during setup replication by fixing the order in which locks are acquired. #22376
• Fixes issue of scans not honoring timeouts, preventing indefinite reads and reducing CPU usage. #21829
• Allows restoring a snapshot schedule from a time just before the oldest snapshot, improving usability. #21269
• Removes assumption that every tablet server hosts tablets, preventing potential crashes. #20230
• Resolves a heartbeat metrics issue ensuring full xCluster error information is sent to the new master even during a leader failover, and makes tserver_heartbeat_metrics_interval_ms runtime updatable. #22624
• Adds validation to RPCs DeleteSnapshot and RestoreSnapshot to prevent deletion or use of snapshots involved in ongoing processes. #23055
• Ensures Create Table operation fails if Alter Replication encounters an error, enhancing the reliability of replication setup. #21732
• Converted the ysql_skip_row_lock_for_update to an auto-flag to resolve compatibility issues during upgrade, preventing incorrect DB record creations that can affect row visibility and integrity. #22057
• Modifies memory consumption calculations for pending operations to ensure accurate rejection of new writes at bootstrap, preventing loading failures. #21254
• Trims large error messages in AsyncRpc::Failed to prevent hitting memory limit and resulting unavailability. #21402
• Excludes hidden tables from generate snapshot output to circumvent cloning failure from recreated tables. #21631
• Prevents unexpected leader fatal errors by updating cached leader terms immediately after a leader change. #21808
• Renames and updates the description of the flag min_secustomerent_size_to_rollover_at_flush for clarity. #21691
• Changes the class of enable_automatic_tablet_splitting flag from kLocalPersisted (class 2) to kExternal (class 4) to eliminate setup issues with xCluster configurations. #22088
• Switches from using scoped_refptr to std::shared_ptr for TabletInfo to handle cycles safely. #18257, #21139
• Updates cotable IDs in flushed frontiers during a snapshot restore, preventing potential post-restore issues. #23047
• Allows the persistent mapping of source-target schema versions when a new table is added to a colocated database, thus preventing replication from stalling after T-server restarts. #23188
• Eliminates potential FATAL errors during reported tabletPB creation by ensuring retrieval of schema version is atomic. #21340
• Ensures the correct order of destroying components, preventing possible concurrent calls on a WAL append callback. #21564
• Adds a TSAN suppression to manage the apparent race condition in the function boost::regex_match. #21585
• Fixes the compilation error for almalinux8 fastdebug gcc11 that was previously removed from the build matrix. #21536
• Corrects a bug causing some tablet metrics to display incorrect metric_type attribute. #21608
• Fixes a segmentation fault in yb-master by checking for a null pointer before dereferencing it, addressing an issue in the CDC run on 2.23.0.0-b37-arm. #21648
• Reduces unnecessary logging during checkpoint operations by lowering INFO level logs to DEBUG_LEVEL, enhancing log readability. #21658
• Allows DML operations on non-replicated databases and blocks DML only on databases in transactional xCluster replication STANDBY mode. Now only databases part of an inbound transactional xCluster replication group in the xCluster safe time map will have DML operations blocked. Also, certain attributes are moved from TServer to TserverXClusterContext. #21245
• Enables the session to outlive the callback by holding a shared pointer to it, preventing potential crashes during concurrent DML queries. #21103
• Avoids multiple destruction of the same database connection, preventing system crashes due to simultaneous connection failures. #21738
• Allows viewing of the RPC bind addresses in the master leader UI, especially beneficial in cases like k8s where the RPC bind address with the pod DNS is more useful than the broadcast address. #21959
• Prevents fatal errors by skipping ReserveMarker/AsyncAppend if the tablet peer has already been shut down. #21769
• Prevents yb-master crash by ensuring background task isn't deleted before the callback is invoked. #21773
• Enables callback completion wait in PollTransactionStatusBase during shutdown to prevent unexpected process termination. #21773
• Initializes prev_op to UNKNOWN to prevent AlmaLinux 8 fastdebug gcc11 compilation failures. #21811
• Enables batched metric updates for YCQL reads to prevent performance drop due to RocksDB metric updates. #21832
• Removes pending delete logic from load balancer to prevent delays during high tablet replica movement. #21806
• Enhances YSQL operation by refining task shutdown procedures and avoiding unnecessary task aborts. #21917
• Stops fatal errors caused by the re-use of remote log anchor session during remote bootstrap from a non-leader peer. This fix ensures shared pointers are accurately tracked for tablet_peer objects using the = operator, preventing unintentional destruction of underlying objects. #22007
• Delays min_running_ht initialization until after the successful completion of tablet bootstrap to prevent unexpected behaviors. #22099
• Enables the skip_table_tombstone_check for colocated tables to prevent errors. #22115
• Prevents potential segfaults during catalog reload by modifying GetClusterConfig function. #21775
• Reduces the interval of the tablet server metrics heartbeat to prevent potential misreporting of a leaderless tablet. #22189
• Reduces four threads and a meta cache in xCluster consumer by reusing the TServers yb::client. #22845
• Resolves the issue of pg_locks query failure due to missing host node UUID in distributed transactions. #22181
• Clarifies memory division flags to reflect they are percentage of the process's hard memory limit, not total available memory. #22423
• Eliminates latency spikes in conflicting workloads by preventing redundant ProbeTransactionDeadlock RPCs. #22426
• Corrects the CI build issues on GCC 12, debug AlmaLinux 9 caused by updates in cf0c09b. #22501
• Captures the actual user executing the query instead of only the superuser and fixes ordering in the ddl_queue handler. #22514
• Enhances logging during MemTable flushing to better monitor memory usage limits. #22737
• Prevents premature metric destruction during Prometheus scrapes, resolving non-UTF8 character issues. #22767
• Ensures object drops correctly cascade to dependent columns in the DocDB table, preventing inconsistencies. #22874
• Boosts YSQL major version upgrade process by minimal changes in the master branch to reduce its divergence. #23079
• Adds a flag to disable the intent filtering during bootstrap, preventing potential data corruption on restart. #23184
• Allows large bytes to be requested on RateLimiter, preventing indefinite call stalling. #23173
• Restores the previously missing home icon in the master user interface. #23275
• Removes the read-time option from the ysql_dump help output for proper DDL atomicity handling. #23299
• Allows setting custom snapshot retention duration using yb-admin, including retaining a snapshot indefinitely by setting retention_duration_hours to 0. #23332
• Removes an unreachable line causing GH build failure. #23375
• Corrects an issue where the load balancer improperly handles a pending leader stepdown task. #21834
• Removes the TServer warning log for mismatching cluster config versions, reducing unnecessary noise in logs. #23119
• Eliminates memory leaks in YSQL Connection Manager by ensuring proper deallocation of objects and variables. #10065
• Resolves the TServer crash issue during query execution happening due to a NULL pointer dereference. #23004
• Allows for a reliable connection to a remote YugabyteDB universe by setting skip_master_flagfile when creating YBClients. #23145
• Ensures the node restarts properly even with secure mode enabled by adding a timeout framework. #23447
• Ensures the stack_is_too_deep function returns predictable results in ASAN, aiding effective limit setting on stack depth. #15682
• Introduces a check for multi-threaded mode in catalog lookup functions to avoid server crashes and make error mitigation easier for users. #23401
• Enhances visibility of the Hidden state in Master/Tserver Tables UI by shifting its position more prominently to the State column. #22521
• Increases the speed of the yb-admin snapshot schedule create command to reduce resource usage. #21929
• Deprecates the TServer flag enable_pg_savepoints to prevent incorrect behavior and avoid silent progress in PL/pgSQL exceptions. #29018
• Eliminates the occurrence of "schema version mismatch" error following a DROP INDEX statement by introducing a delay in index deletion. #22637

CDC

• Ensures deletion of MemoryContext after each GetChanges RPC to prevent memory leaks. #22328
• Introduces additional VLOG statements in the ListReplicationSlots function for better debugging. #21652
• Prevents newly created indexes, materialized views, and non-user tables from being added to the Chang Data Capture (CDC) stream metadata. #22808
• Reduces resource usage by removing non-eligible tables, like indexes, from existing CDC SDK stream metadata, and releasing retention barriers. This change requires the master flag enable_cleanup_of_non_eligible_tables_from_cdcsdk_stream and limits processing to two non-eligible tables per namespace per run with cdcsdk_table_processing_limit_per_run. Introduces three yb-admin commands for managing CDC streams. #22876, #22835, #22773
• Introduces new auto flag cdcsdk_enable_identification_of_non_eligible_tables and three yb-admin commands to manage tables in CDC stream, enhancing control and reducing unnecessary resource usage. #22876, #22835, #22773
• Fixes the issue of getting either 0 or a random time as the server's system clock in XLogData from the logical replication stream. #22929
• Fixes a memory leakage issue in the walsender process by deep freeing the cached record batch after streaming to the client. #21530
• Adds more debug logs in the walsender to aid in investigating issues like linked data loss. #21465
• Allows for better memory management in the walsender process by storing record batches in a separate memory context. #21530
• Logs RPC errors as warnings during the cleanup of virtual WAL after LogicalReplication ends. #21651
• Allows stream replication to handle serialized transactions successfully by adding directory creation logic. #21765
• Stops loading replication slots from disk during startup to avoid potential system crashes. #21841
• Adds more logs for easy debugging during stress runs for Change Data Capture (CDC) without any impact on performance. #21780
• Limits unnecessary RPC calls to the local TServer during RollbackToSubTransaction operation if transaction is read-only, a fast-path transaction, or has NON_TRANSACTIONAL isolation level. #21519
• Limits transactions' inclusion in the unacked list only upon receiving the commit record, enhancing the restart_lsn calculation. #21950
• Removes table level attributes from CDCSDK metrics to avoid TServer crash due to failed DCHECK assertion. #22142
• Fixes the segmentation fault in walsender for dynamic table addition by refreshing stored replica identities and preventing a race condition when creating dynamic tables. #22273
• Updates the serialization and de-serialization logic to include the yb_is_omitted array, preserving data values in large transactions. #21946
• Solves an issue where CDCSDK incorrectly deduces tablets as not interesting for stream before reaching the configured time limit. #22383
• Addresses a race condition in dynamic table creation, enhancing stability during table and tablet initialization. #22408
• Refines the logic to remove BEGIN record when no DML records are added, preventing potential virtual WAL crashes. #21646
• Resolves "could not open relation" error by updating slot creation method and simplifying yb_read_time logic. #22398
• Enables support for dynamically allotted OID data types in CDC to prevent system crashes. #23179
• Allows handling of non-eligible table cleanup in CDC stream loading even after table drop, preventing master crash. #23278
• Reduces total inserts from 5k/thread to 2.5k/thread for clearer consumption of expected records. #23128
• Prevents failures in decoding change events by refreshing cached_schema_details when executing a new GetChanges request if the client indicates a necessity for the schema. #20698
• Allows pg_replication_slots to return an empty response instead of an error when ysql_yb_enable_replication_commands flag is false. #23096