YB Voyager diagnostics reporting

By default, YugabyteDB Voyager reports migration diagnostics to the Yugabyte diagnostics service when yb-voyager commands are executed. The diagnostics information is transmitted securely to and stored by Yugabyte.

None of your data or personally identifiable information is collected.

Data collected

When enabled, the following data is collected while running yb-voyager:

  • Migration unique ID
  • Migration phase name (for example, export-data, import-schema, and so on)
  • Migration phase start time
  • Migration phase payload (information relevant to that phase of the migration)
  • Payload collection time
  • Migration type (Offline, live migration (with fall-foward or fall-back), or bulk data load)
  • YugabyteDB Voyager version (the version of yb-voyager used for the migration)
  • Source database details
    • Database type (PostgreSQL, Oracle, and MySQL)
    • Database version
    • Total size of the database migrating to YugabyteDB
  • Target YugabyteDB database details
    • Database version
    • Node counts on the target YugabyteDB cluster
    • Total cores on the target YugabyteDB cluster
  • Time taken in the migration phase up to the point when the payload is collected
  • Status of the phase while the payload is sent (for example, In progress, Complete, Exit, and so on)
  • IP address of the client machine on which voyager is running

The payload for each migration phase is described in the following table. Note that no sensitive parameters, such as password-related arguments, are stored.

Migration phase Data collected

assess-migration
  • Unsupported features (reported in the assessment report without any object names or SQL statements; only count of object per feature)
  • Unsupported datatypes (reported in the assessment report without any table name or column name; only the datatype)
  • Error message while running the assessment if any
  • Table sizing statistics (reads/writes per second, and size in bytes)
  • Index sizing statistics (reads/writes per second, and size in bytes)
  • Schema summary (reported in the assessment report, without object names for each object type)
  • Source connectivity (whether assessment is run with source connectivity)

export-schema
  • Whether sharding recommendations from the assessment report were applied

analyze-schema
  • Issues (issues reported by analyze-schema without the SQL statements)
  • Database objects (objects migrated in the migration without the object names per object type)

import-schema
  • Error count (if there are errors while running the import schema).
  • Flag values if passed or not:
    • Continue on error
    • Enable orafce
    • Ignore exist
    • Refresh mviews
    • Post snapshot import

export-data
export-data-from-source
export-data-from-target
  • Parallel jobs used
  • Total rows (total number of rows of all tables exported)
  • Largest table rows (total number of rows of all tables exported)
  • Phase (Snapshot, Streaming, or Cutover)
  • Total exported events (in case of live migration these are the total exported events)
  • Events export rate (in case of live migration this is the rate of events exported for the last 3 minutes)
  • Live migration workflow type (Normal, with fall-forward or fall-back)

import-data
import-data-to-target
import‑data‑to‑source‑replica
import-data-to-source
  • Parallel jobs used
  • Total rows (total number of rows of all tables imported)
  • Largest table rows (total number of rows of all tables imported)
  • Phase (Snapshot, Streaming, or Cutover)
  • Total imported events (for live migration these are the total imported events)
  • Events import rate (for live migration this the rate of events imported for the last 3 minutes)
  • Live migration workflow type (Normal, with fall-forward or fall-back)

import-data-file (bulk data load)
  • Parallel jobs used
  • Total size (total size of all tables bulk imported)
  • Largest table size (largest size of tables imported)
  • File storage type (Local, AWS S3, GCS, or Azure blob storage)
  • Data file format parameters (for example, null-string, format type, and so on)

Example payloads

The following are examples of payloads that are collected during some migration phases:

assess-migration command payload

{
  "migration_uuid": "8d9d678c-f6d4-4253-b24f-e10e4a49cc31",
  "phase_start_time": "2024-07-08 16:30:21.237453",
  "collected_at": "2024-07-08 16:30:22.69088",
  "source_db_details": {
    "host": "",
    "db_type": "postgresql",
    "db_version": "14.1 (Debian 14.1-1.pgdg110+1)",
    "total_db_size_bytes": 31391744
  },
  "target_db_details": "",
  "yb_voyager_version": "1.7.2",
  "migration_phase": "assess-migration",
  "phase_payload": {
    "unsupported_features": [
      {
        "FeatureName": "GIST indexes",
        "ObjectCount": 3
      },
      {
        "FeatureName": "Constraint triggers",
        "ObjectCount": 0
      },
      {
        "FeatureName": "Inherited tables",
        "ObjectCount": 0
      },
      {
        "FeatureName": "Tables with Stored generated columns",
        "ObjectCount": 0
      }
    ],
    "unsupported_datatypes": ["point", "point"],
    "table_sizing_stats": [
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 0 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 1245184 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 0 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 3055616 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 4825088 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 2998272 }
    ],
    "index_sizing_stats": [
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 8192 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 1130496 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 1130496 },
      { "object_name": "XXX", "reads_per_second": 0, "writes_per_second": 0, "size_in_bytes": 1130496 }
    ],
    "schema_summary": {
      "DatabaseObjects": [
        { "ObjectType": "SCHEMA", "TotalCount": 1 },
        { "ObjectType": "SEQUENCE", "TotalCount": 3 },
        { "ObjectType": "TABLE", "TotalCount": 9 },
        { "ObjectType": "INDEX", "TotalCount": 9 }
      ]
    },
    "source_connectivity": true
  },
  "migration_type": "",
  "time_taken_sec": 2,
  "status": "COMPLETE"
}

export-data command payload

{
    "migration_uuid": "8d9d678c-f6d4-4253-b24f-e10e4a49cc31",
    "phase_start_time": "2024-07-08 16:28:59.642321",
    "collected_at": "2024-07-08 16:29:05.986381",
    "source_db_details": {
        "host": "",
        "db_type": "postgresql",
        "db_version": "14.1 (Debian 14.1-1.pgdg110+1)",
        "total_db_size_bytes": 31391744
    },
    "target_db_details": "",
    "yb_voyager_version": "1.7.2",
    "migration_phase": "export-data",
    "phase_payload": {
        "parallel_jobs": 4,
        "total_rows_exported": 173641,
        "largest_table_rows_exported": 149000,
        "start_clean": true,
        "export_snapshot_mechanism": "pg_dump",
        "phase": "SNAPSHOT"
    },
    "migration_type": "offline",
    "time_taken_sec": 7,
    "status": "COMPLETE"
}

import-data command payload

{
  "migration_uuid": "8d9d678c-f6d4-4253-b24f-e10e4a49cc31",
  "phase_start_time": "2024-07-08 16:40:47.152743",
  "collected_at": "2024-07-08 16:42:03.363395",
  "source_db_details": "",
  "target_db_details": {
    "host": "",
    "db_version": "11.2-YB-2.18.2.1-b0",
    "node_count": 1,
    "total_cores": 8
  },
  "yb_voyager_version": "1.7.2",
  "migration_phase": "import-data",
  "phase_payload": {
    "parallel_jobs": 2,
    "total_rows_imported": 173641,
    "largest_table_rows_imported": 149000,
    "start_clean": true,
    "phase": "SNAPSHOT"
  },
  "migration_type": "offline",
  "time_taken_sec": 77,
  "status": "COMPLETE"
}

Configure diagnostics collection

To control whether to send diagnostics to Yugabyte, you can use the --send-diagnostics flag or export an environment variable.

Set the flag as follows:

yb-voyager ... --send-diagnostics [true|false|yes|no|1|0]

The default is true.

Alternatively, you can use the following environment variable to set a value for this flag globally on the client machine where yb-voyager is running.

export YB_VOYAGER_SEND_DIAGNOSTICS=[true|false|yes|no|1|0]