Join us on
Star us on
Get Started
Slack
GitHub
Get Started
v2.5 (latest) v2.2 (stable) v2.1 (earlier version) v2.0 (earlier version) v1.3 (earlier version)
  • GET STARTED
    • Quick start
      • 1. Install YugabyteDB
      • 2. Create a local cluster
      • 3. Explore YSQL
      • 4. Build an application
        • Java
        • NodeJS
        • Go
        • Python
        • Ruby
        • C#
        • PHP
        • C++
        • C
    • Introduction
    • Explore core
      • 1. Linear scalability
      • 2. Fault tolerance
      • 3. Global distribution
      • 4. Auto sharding
      • 5. Tunable reads
      • 6. Observability
  • USER GUIDES
    • Develop
      • Learn app development
        • 1. SQL vs NoSQL
        • 2. Data modeling
        • 3. Data types
        • 4. ACID transactions
        • 5. Aggregations
        • 6. Batch operations
        • 7. Date and time
        • 8. Strings and text
      • Ecosystem integrations
        • Apache Kafka
        • Apache Spark
        • JanusGraph
        • KairosDB
        • Presto
        • Metabase
      • Real-world examples
        • E-Commerce App
        • IoT Fleet Management
        • Retail Analytics
      • Explore sample applications
    • Deploy
      • Checklist
      • Manual deployment
        • 1. System configuration
        • 2. Install software
        • 3. Start YB-Masters
        • 4. Start YB-TServers
        • 5. Verify deployment
      • Kubernetes
        • Helm Chart
        • Helm configuration
        • Local SSD
      • Docker
      • Public clouds
        • Amazon Web Services
        • Google Cloud Platform
        • Microsoft Azure
      • Pivotal Cloud Foundry
      • Yugabyte Platform
        • 1. Prepare cloud environment
        • 2. Install Admin Console
        • 3. Configure Admin Console
        • 4. Configure Cloud Providers
    • Benchmark
      • Performance
      • YCSB
      • Large datasets
    • Secure
      • Security checklist
      • Authentication
      • Authorization
        • 1. RBAC Model
        • 2. Create Roles
        • 3. Grant permissions
      • TLS encryption
        • 1. Prepare nodes
        • 2. Server-server encryption
        • 3. Client-server encryption
        • 4. Connect to cluster
      • Encryption at Rest
    • Manage
      • Backup and restore
        • Backing up data
        • Restoring data
      • Data migration
        • Bulk import
        • Bulk export
      • Change cluster config
      • Upgrade deployment
      • Diagnostics reporting
      • Yugabyte Platform
        • Create universe - Multi-zone
        • Create universe - Multi-region
        • Edit universe
        • Edit config flags
        • Health checking and alerts
        • Create and edit instance tags
        • Node status and actions
        • Read replicas
        • Back up and restore
        • Upgrade universe
        • Delete universe
    • Troubleshoot
      • Troubleshooting overview
      • Cluster level issues
        • YCQL connection issues
        • YEDIS connection Issues
      • Node level issues
        • Check processes
        • Inspect logs
        • System statistics
      • Yugabyte Platform
        • Troubleshoot universes
  • REFERENCE
    • APIs
      • YSQL
        • Statements
          • ABORT
          • ALTER DATABASE
          • ALTER DOMAIN
          • ALTER TABLE
          • BEGIN
          • COMMENT
          • COMMIT
          • COPY
          • CREATE DATABASE
          • CREATE DOMAIN
          • CREATE INDEX
          • CREATE SCHEMA
          • CREATE SEQUENCE
          • CREATE TABLE
          • CREATE TABLE AS
          • CREATE TYPE
          • CREATE USER
          • CREATE VIEW
          • DEALLOCATE
          • DELETE
          • DROP DATABASE
          • DROP DOMAIN
          • DROP SEQUENCE
          • DROP TABLE
          • DROP TYPE
          • END
          • EXECUTE
          • EXPLAIN
          • GRANT
          • INSERT
          • LOCK
          • PREPARE
          • RESET
          • REVOKE
          • ROLLBACK
          • SELECT
          • SET
          • SET CONSTRAINTS
          • SET TRANSACTION
          • SHOW
          • SHOW TRANSACTION
          • TRUNCATE
          • UPDATE
        • Data types
          • Binary
          • Boolean
          • Character
          • Date-time
          • Json
          • Money
          • Numeric
          • Serial
          • UUID
        • Expressions
          • currval()
          • lastval()
          • nextval()
        • Keywords
        • Reserved Names
      • YCQL
        • Quick Start YCQL
        • ALTER KEYSPACE
        • ALTER ROLE
        • ALTER TABLE
        • CREATE INDEX
        • CREATE KEYSPACE
        • CREATE ROLE
        • CREATE TABLE
        • CREATE TYPE
        • DROP INDEX
        • DROP KEYSPACE
        • DROP ROLE
        • DROP TABLE
        • DROP TYPE
        • GRANT PERMISSION
        • GRANT ROLE
        • REVOKE PERMISSION
        • REVOKE ROLE
        • USE
        • INSERT
        • SELECT
        • UPDATE
        • DELETE
        • TRANSACTION
        • TRUNCATE
        • Simple Value
        • Subscript
        • Function Call
        • Operator Call
        • BLOB
        • BOOLEAN
        • MAP, SET, LIST
        • FROZEN
        • INET
        • Integer & Counter
        • Non-Integer
        • TEXT
        • Date & Time Types
        • UUID & TIMEUUID
        • JSONB
        • Date and time functions
    • CLIs
      • yb-ctl
      • yb-docker-ctl
      • yb-master
      • yb-tserver
      • ysqlsh
      • cqlsh
    • Sample data
      • Chinook
      • Northwind
      • PgExercises
      • SportsDB
    • Tools
      • TablePlus
  • RELEASES
    • Release history
      • v1.3.1
      • v1.3.0
      • v1.2.12
      • v1.2.11
      • v1.2.10
      • v1.2.9
      • v1.2.8
      • v1.2.6
      • v1.2.5
      • v1.2.4
  • CONCEPTS
    • Architecture
      • Design goals
      • Layered architecture
      • Basic concepts
        • Universe
        • YB-TServer
        • YB-Master
        • Acknowledgements
      • Query layer
        • Overview
      • DocDB store
        • Sharding
        • Replication
        • Persistence
        • Performance
      • DocDB transactions
        • Isolation Levels
        • Single row transactions
        • Distributed transactions
        • Transactional IO path
  • FAQ
    • Comparisons
      • CockroachDB
      • Google Cloud Spanner
      • MongoDB
      • FoundationDB
      • Amazon DynamoDB
      • Azure Cosmos DB
      • Apache Cassandra
      • Redis in-memory store
      • Apache HBase
    • Other FAQs
      • Product
      • Architecture
      • Yugabyte Platform
      • API compatibility
  • CONTRIBUTOR GUIDES
    • Get involved
  • Misc
    • YEDIS
      • Quick start
      • Develop
        • Client drivers
          • C
          • C++
          • C#
          • Go
          • Java
          • NodeJS
          • Python
      • API reference
        • APPEND
        • AUTH
        • CONFIG
        • CREATEDB
        • DELETEDB
        • LISTDB
        • SELECT
        • DEL
        • ECHO
        • EXISTS
        • EXPIRE
        • EXPIREAT
        • FLUSHALL
        • FLUSHDB
        • GET
        • GETRANGE
        • GETSET
        • HDEL
        • HEXISTS
        • HGET
        • HGETALL
        • HINCRBY
        • HKEYS
        • HLEN
        • HMGET
        • HMSET
        • HSET
        • HSTRLEN
        • HVALS
        • INCR
        • INCRBY
        • KEYS
        • MONITOR
        • PEXPIRE
        • PEXPIREAT
        • PTTL
        • ROLE
        • SADD
        • SCARD
        • RENAME
        • SET
        • SETEX
        • PSETEX
        • SETRANGE
        • SISMEMBER
        • SMEMBERS
        • SREM
        • STRLEN
        • ZRANGE
        • TSADD
        • TSCARD
        • TSGET
        • TSLASTN
        • TSRANGEBYTIME
        • TSREM
        • TSREVRANGEBYTIME
        • TTL
        • ZADD
        • ZCARD
        • ZRANGEBYSCORE
        • ZREM
        • ZREVRANGE
        • ZSCORE
        • PUBSUB
        • PUBLISH
        • SUBSCRIBE
        • UNSUBSCRIBE
        • PSUBSCRIBE
        • PUNSUBSCRIBE
> Explore core >

Observability with Prometheus

Attention

This page documents an earlier version. Go to the latest (v2.3) version.

You can monitor your local YugabyteDB cluster with a local instance of Prometheus, the de-facto standard for time-series monitoring of cloud native infrastructure. Every YugabyteDB service exposes metrics in the Prometheus format at the /prometheus-metrics endpoint.

If you haven't installed YugabyteDB yet, do so first by following the Quick Start guide.

  • macOS
  • Linux
  • Docker

Prerequisite

Prometheus is installed on your local machine. If you have not done so already, follow the links below.

  • Download Prometheus
  • Get Started with Prometheus

1. Create universe

If you have a previously running local universe, destroy it using the following.

$ ./bin/yb-ctl destroy

Start a new local cluster - by default, this will create a 3-node universe with a replication factor of 3.

$ ./bin/yb-ctl create

2. Run sample key-value app

Download the sample app jar.

$ wget https://github.com/yugabyte/yb-sample-apps/releases/download/v1.2.0/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar 

Run a simple key-value workload in a separate shell.

$ java -jar ./yb-sample-apps.jar \
    --workload CassandraKeyValue \
    --nodes 127.0.0.1:9042 \
    --num_threads_read 1 \
    --num_threads_write 1

3. Prepare Prometheus config file

Copy the following into a file called yugabytedb.yml.

global:
  scrape_interval:     5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
  evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# YugabyteDB configuration to scrape Prometheus time-series metrics scrape_configs:

  • job_name: 'yugabytedb' metrics_path: /prometheus-metrics

    static_configs:

    • targets: ['127.0.0.1:7000', '127.0.0.2:7000', '127.0.0.3:7000'] labels: group: 'yb-master'

    • targets: ['127.0.0.1:9000', '127.0.0.2:9000', '127.0.0.3:9000'] labels: group: 'yb-tserver'

    • targets: ['127.0.0.1:11000', '127.0.0.2:11000', '127.0.0.3:11000'] labels: group: 'yedis'

    • targets: ['127.0.0.1:12000', '127.0.0.2:12000', '127.0.0.3:12000'] labels: group: 'ycql'

    • targets: ['127.0.0.1:13000', '127.0.0.2:13000', '127.0.0.3:13000'] labels: group: 'ysql'

4. Start Prometheus server

Go to the directory where Prometheus is installed and start the Prometheus server as below.

$ ./prometheus --config.file=yugabytedb.yml

Open the Prometheus UI at http://localhost:9090 and then navigate to the Targets page under Status.

Prometheus Targets

5. Analyze key metrics

On the Prometheus Graph UI, you can now plot the read IOPS and write IOPS for the CassandraKeyValue sample app. As we can see from the source code of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means we can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements.

Paste the following expressions into the Expression box and click Execute followed by Add Graph.

Throughput

Read IOPS

sum(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_count[1m]))

Prometheus Read IOPS

Write IOPS

sum(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_count[1m]))

Prometheus Read IOPS

Latency

Read Latency (in microseconds)

avg(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_sum[1m])) / avg(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_count[1m]))

Prometheus Read IOPS

Write Latency (in microseconds)

avg(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_sum[1m])) / avg(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_count[1m]))

Prometheus Read IOPS

6. [Optional] Clean up

Optionally, you can shutdown the local cluster created in Step 1.

$ ./bin/yb-ctl destroy

Prerequisite

Prometheus is installed on your local machine. If you have not done so already, follow the links below.

  • Download Prometheus
  • Get Started with Prometheus

1. Create universe

If you have a previously running local universe, destroy it using the following.

$ ./bin/yb-ctl destroy

Start a new local cluster - by default, this will create a 3-node universe with a replication factor of 3.

$ ./bin/yb-ctl create

2. Run sample key-value app

Download the sample app jar.

$ wget https://github.com/yugabyte/yb-sample-apps/releases/download/v1.2.0/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar 

Run a simple key-value workload in a separate shell.

$ java -jar ./yb-sample-apps.jar \
    --workload CassandraKeyValue \
    --nodes 127.0.0.1:9042 \
    --num_threads_read 1 \
    --num_threads_write 1

3. Prepare Prometheus config file

Copy the following into a file called yugabytedb.yml.

global:
  scrape_interval:     5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
  evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# YugabyteDB configuration to scrape Prometheus time-series metrics scrape_configs:

  • job_name: 'yugabytedb' metrics_path: /prometheus-metrics

    static_configs:

    • targets: ['127.0.0.1:7000', '127.0.0.2:7000', '127.0.0.3:7000'] labels: group: 'yb-master'

    • targets: ['127.0.0.1:9000', '127.0.0.2:9000', '127.0.0.3:9000'] labels: group: 'yb-tserver'

    • targets: ['127.0.0.1:11000', '127.0.0.2:11000', '127.0.0.3:11000'] labels: group: 'yedis'

    • targets: ['127.0.0.1:12000', '127.0.0.2:12000', '127.0.0.3:12000'] labels: group: 'ycql'

    • targets: ['127.0.0.1:13000', '127.0.0.2:13000', '127.0.0.3:13000'] labels: group: 'ysql'

4. Start Prometheus server

Go to the directory where Prometheus is installed and start the Prometheus server as below.

$ ./prometheus --config.file=yugabytedb.yml

Open the Prometheus UI at http://localhost:9090 and then navigate to the Targets page under Status.

Prometheus Targets

5. Analyze key metrics

On the Prometheus Graph UI, you can now plot the read IOPS and write IOPS for the CassandraKeyValue sample app. As we can see from the source code of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means we can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements.

Paste the following expressions into the Expression box and click Execute followed by Add Graph.

Throughput

Read IOPS

sum(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_count[1m]))

Prometheus Read IOPS

Write IOPS

sum(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_count[1m]))

Prometheus Read IOPS

Latency

Read Latency (in microseconds)

avg(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_sum[1m])) / avg(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_count[1m]))

Prometheus Read IOPS

Write Latency (in microseconds)

avg(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_sum[1m])) / avg(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_count[1m]))

Prometheus Read IOPS

6. [Optional] Clean up

Optionally, you can shutdown the local cluster created in Step 1.

$ ./bin/yb-ctl destroy

1. Create universe

If you have a previously running local universe, destroy it using the following.

$ ./yb-docker-ctl destroy

Start a new local universe with replication factor 3.

$ ./yb-docker-ctl create  --rf 3

2. Run sample key-value app

Pull the yb-sample-apps docker container. This container has built-in Java client programs for various workloads including SQL inserts and updates.

$ docker pull yugabytedb/yb-sample-apps

Run a simple key-value workload in a separate shell.

$ docker run --name yb-sample-apps --hostname yb-sample-apps --net yb-net yugabytedb/yb-sample-apps --workload CassandraKeyValue \
  --nodes yb-tserver-n1:9042 \
  --num_threads_write 1 \
  --num_threads_read 4

3. Prepare Prometheus config file

Copy the following into a file called yugabytedb.yml. Move this file to the /tmp directory so that we can bind the file to the Prometheus container later on.

global:
  scrape_interval:     5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
  evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# YugabyteDB configuration to scrape Prometheus time-series metrics scrape_configs:

  • job_name: 'yugabytedb' metrics_path: /prometheus-metrics

    static_configs:

    • targets: ['yb-master-n1:7000', 'yb-master-n2:7000', 'yb-master-n3:7000'] labels: group: 'yb-master'

    • targets: ['yb-tserver-n1:9000', 'yb-tserver-n2:9000', 'yb-tserver-n3:9000'] labels: group: 'yb-tserver'

    • targets: ['yb-tserver-n1:11000', 'yb-tserver-n2:11000', 'yb-tserver-n3:11000'] labels: group: 'yedis'

    • targets: ['yb-tserver-n1:12000', 'yb-tserver-n2:12000', 'yb-tserver-n3:12000'] labels: group: 'ycql'

    • targets: ['yb-tserver-n1:13000', 'yb-tserver-n2:13000', 'yb-tserver-n3:13000'] labels: group: 'ysql'

4. Start Prometheus server

Start the Prometheus server as below. The prom/prometheus container image will be pulled from the Docker registry if not already present on the localhost.

$ docker run \
	-p 9090:9090 \
	-v /tmp/yugabytedb.yml:/etc/prometheus/prometheus.yml \
	--net yb-net \
    prom/prometheus

Open the Prometheus UI at http://localhost:9090 and then navigate to the Targets page under Status.

Prometheus Targets

5. Analyze key metrics

On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the CassandraKeyValue sample app. As we can see from the source code of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means we can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements.

Paste the following expressions into the Expression box and click Execute followed by Add Graph.

Throughput

Read IOPS

sum(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_count[1m]))

Prometheus Read IOPS

Write IOPS

sum(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_count[1m]))

Prometheus Read IOPS

Latency

Read Latency (in microseconds)

avg(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_sum[1m])) / avg(irate(handler_latency_yb_cqlserver_SQLProcessor_SelectStmt_count[1m]))

Prometheus Read IOPS

Write Latency (in microseconds)

avg(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_sum[1m])) / avg(irate(handler_latency_yb_cqlserver_SQLProcessor_InsertStmt_count[1m]))

Prometheus Read IOPS

6. Clean up (optional)

Optionally, you can shutdown the local cluster created in Step 1.

$ ./yb-docker-ctl destroy
Ask our community
  • Slack
  • Github
  • Forum
  • StackOverflow
Yugabyte
Contact Us
Copyright © 2017-2020 Yugabyte, Inc. All rights reserved.