Connect clients to Kubernetes clusters

Introduction

This document describes the different options to connect to a Yugabyte cluster deployed within Kubernetes.

Prerequisites

You must have set up a Yugabyte cluster according to the Kubernetes deployment instructions..

Connecting from within the Kubernetes cluster

An application that is deployed within the Kubernetes cluster should use the Service DNS name yb-tservers.<namespace>.svc.cluster.local to discover server endpoints. This DNS entry has multiple A records, one for each tserver pod, so that clients can randomize queries across different endpoints.

$ kubectl --namespace yb-demo get svc/yb-tservers
NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                                        AGE
yb-tservers   ClusterIP   None         <none>        7100/TCP,9000/TCP,6379/TCP,9042/TCP,5433/TCP   56m

Here is an example of a client that uses the YSQL shell (ysqlsh) to connect.

$ kubectl run ysqlsh-client -it --rm  --image yugabytedb/yugabyte-client --command -- ysqlsh -h yb-tservers.yb-demo.svc.cluster.local
yugabyte=# CREATE TABLE demo(id INT PRIMARY KEY);
CREATE TABLE

Here is an example of a client that uses the YCQL shell (ycqlsh) to connect.

$ kubectl run cqlsh-shell -it --rm  --image yugabytedb/yugabyte-client --command -- cqlsh yb-tservers.yb-demo.svc.cluster.local 9042
ycqlsh> CREATE KEYSPACE demo;
ycqlsh> use demo;
ycqlsh:demo> CREATE TABLE t_demo(id INT PRIMARY KEY);

Note that although tables are internally sharded across multiple tserver pods, every tserver pod has the ability to process any query, irrespective of its actual tablet assignment.

Connecting externally

An application that is deployed outside the Kubernetes cluster should use the external LoadBalancer IP address to connect to the cluster. Connections to the load balancer IP address are randomly routed to one of the tserver pods behind the yb-tservers service.

$ kubectl get svc -n yb-demo
NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
yb-master-ui         LoadBalancer   10.101.142.48   98.138.219.231     7000:32168/TCP                                 43h
yb-masters           ClusterIP      None            <none>        7100/TCP,7000/TCP                              43h
yb-tserver-service   LoadBalancer   10.99.76.181    98.138.219.232     6379:30141/TCP,9042:31059/TCP,5433:30577/TCP   43h
yb-tservers          ClusterIP      None            <none>        7100/TCP,9000/TCP,6379/TCP,9042/TCP,5433/TCP   43h

Here is an example of a client that uses the YSQL shell (ysqlsh to connect.

$ docker run yugabytedb/yugabyte-client ysqlsh -h 98.138.219.232
yugabyte=# CREATE TABLE demo(id INT PRIMARY KEY);
CREATE TABLE

Here is an example of a client that uses the YCQL shell (ycqlsh) to connect.

$ docker run yugabytedb/yugabyte-client ycqlsh 98.138.219.232 9042
ycqlsh> CREATE KEYSPACE demo;
ycqlsh> use demo;
ycqlsh:demo> CREATE TABLE t_demo(id INT PRIMARY KEY);

YB-Master Admin UI

The YB-Master Admin UI is available at the IP address exposed by the yb-master-ui LoadBalancer service – at https://98.138.219.231:7000/.

Another option that does not require an external LoadBalancer is to create a tunnel from the local host to the master web server port on the master pod using kubectl port-forward.

$ kubectl port-forward pod/yb-master-0 7000:7000 -n yb-demo
Forwarding from 127.0.0.1:7000 -> 7000
Forwarding from [::1]:7000 -> 7000

Connecting externally to a Minikube cluster

When the Kubernetes cluster is set up using Minikube, an external IP address is not available by default for the LoadBalancer endpoints. To enable the load balancer IP address, run the command minikube tunnel. For details, see LoadBalancer access.

$ minikube tunnel
Status:
    machine: minikube
    pid: 38193
    route: 10.96.0.0/12 -> 192.168.99.100
    minikube: Running
    services: [yb-master-ui, yb-tserver-service]
    errors:
        minikube: no errors
        router: no errors
        loadbalancer emulator: no errors

Connecting TLS Secured YugabyteDB cluster deployed by Helm Charts

To start a YugabyteDB cluster with encryption in transit (TLS) enabled, follow the steps at Google Kubernetes Service (GKE) - Helm Chart and set the flag tls.enabled=true in the helm command-line.

For example, helm install yugabyte --namespace yb-demo --name yb-demo --set=tls.enabled=true.

Connect from within the Kubernetes cluster

Copy the following yb-client.yaml and use the kubectl create -f yb-client.yaml command to create a pod with auto-mounted client certificates.

apiVersion: v1
kind: Pod
metadata:
  name: yb-client
  namespace: yb-demo
spec:
  containers:
  - name: yb-client
    image: yugabytedb/yugabyte-client:latest
    env:
    - name: SSL_CERTFILE
      value: "/root/.yugabytedb/root.crt"
    volumeMounts:
    - name: yugabyte-tls-client-cert
      mountPath: "/root/.yugabytedb/"
  volumes:
  - name: yugabyte-tls-client-cert
    secret:
      secretName: yugabyte-tls-client-cert
      defaultMode: 256

Here is an example of a client that uses the YSQL shell (ysqlsh) to connect.

Use the following command to verify the connection.

$ kubectl exec -n yb-demo -it yb-client -- ysqlsh -h yb-tservers.yb-demo.svc.cluster.local "sslmode=require"
ysqlsh (11.2-YB-2.1.5.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

yugabyte=# \conninfo
You are connected to database "yugabyte" as user "yugabyte" on host "yb-tservers.yb-demo.svc.cluster.local" at port "5433".
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)

Here is an example of a client that uses the YCQL shell (ycqlsh) to connect.

Use the following command to verify the connection.

$ kubectl exec -n yb-demo -it yb-client -- ycqlsh yb-tservers.yb-demo.svc.cluster.local 9042 --ssl
Connected to local cluster at yb-tservers.yb-demo.svc.cluster.local:9042.
[cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> SHOW HOST
Connected to local cluster at yb-tservers.yb-demo.svc.cluster.local:9042.

(Optional) After the operations are complete, remove the client pod.

$ kubectl delete pod yb-client -n yb-demo
pod "yb-client" deleted

Connect externally

To connect externally to a TLS-enabled YugabyteDB helm cluster, first download the client certificates locally from the Kubernetes cluster's secrets.

$ mkdir $(pwd)/certs
$ kubectl get secret yugabyte-tls-client-cert  -n yb-demo -o jsonpath='{.data.root\.crt}' | base64 --decode > $(pwd)/certs/root.crt
$ kubectl get secret yugabyte-tls-client-cert  -n yb-demo -o jsonpath='{.data.yugabytedb\.crt}' | base64 --decode > $(pwd)/certs/yugabytedb.crt
$ kubectl get secret yugabyte-tls-client-cert  -n yb-demo -o jsonpath='{.data.yugabytedb\.key}' | base64 --decode > $(pwd)/certs/yugabytedb.key

Here is an example of a client that uses the YSQL shell (ysqlsh) to connect. The command specifies the external LoadBalancer IP of the yb-tserver-service as described in Connect using external clients.

Use the following command to verify the connection.

$ docker run -it --rm -v $(pwd)/certs/:/root/.yugabytedb/:ro yugabytedb/yugabyte-client:latest ysqlsh -h <External_Cluster_IP> "sslmode=require"
ysqlsh (11.2-YB-2.1.5.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

yugabyte=# \conninfo
You are connected to database "yugabyte" as user "yugabyte" on host "35.200.205.208" at port "5433".
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)

Here is an example of a client that uses the YCQL shell (ycqlsh) to connect.

To verify the connection, use the following docker run command.

$ docker run -it --rm -v $(pwd)/certs/:/root/.yugabytedb/:ro \
--env SSL_CERTFILE=/root/.yugabytedb/root.crt yugabytedb/yugabyte-client:latest ycqlsh <External_Cluster_IP> 9042 --ssl
ysqlsh (11.2-YB-2.1.5.0-b0)
Connected to local cluster at 35.200.205.208:9042.
[cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> SHOW HOST
Connected to local cluster at 35.200.205.208:9042.