Connect Clients to Kubernetes Clusters

Introduction

This document describes options available for connecting to a Yugabyte cluster deployed within Kubernetes.

Prerequisites

You must have a Yugabyte cluster set up according to the Kubernetes deployment instructions.

Connecting from Within the Kubernetes Cluster

An application that is deployed within the Kubernetes cluster should use the Service DNS name yb-tservers.<namespace>.svc.cluster.local to discover server endpoints. This DNS entry has multiple A records, one for each tserver pod, so that clients can randomize queries across different endpoints.

$ kubectl --namespace yb-demo get svc/yb-tservers
NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                                        AGE
yb-tservers   ClusterIP   None         <none>        7100/TCP,9000/TCP,6379/TCP,9042/TCP,5433/TCP   56m

The following example shows a client that uses the YSQL shell (ysqlsh) to connect:

$ kubectl run ysqlsh-client -it --rm  --image yugabytedb/yugabyte-client --command -- ysqlsh -h yb-tservers.yb-demo.svc.cluster.local
yugabyte=# CREATE TABLE demo(id INT PRIMARY KEY);
CREATE TABLE

The following example shows a client that uses the YCQL shell (ycqlsh) to connect:

$ kubectl run cqlsh-shell -it --rm  --image yugabytedb/yugabyte-client --command -- cqlsh yb-tservers.yb-demo.svc.cluster.local 9042
ycqlsh> CREATE KEYSPACE demo;
ycqlsh> use demo;
ycqlsh:demo> CREATE TABLE t_demo(id INT PRIMARY KEY);

Note that although tables are internally sharded across multiple tserver pods, every tserver pod has the ability to process any query, irrespective of its actual tablet assignment.

Connecting Externally

An application that is deployed outside the Kubernetes cluster should use the external LoadBalancer IP address to connect to the cluster. Connections to the load balancer IP address are randomly routed to one of the tserver pods behind the yb-tservers service.

$ kubectl get svc -n yb-demo
NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
yb-master-ui         LoadBalancer   10.101.142.48   98.138.219.231     7000:32168/TCP                                 43h
yb-masters           ClusterIP      None            <none>        7100/TCP,7000/TCP                              43h
yb-tserver-service   LoadBalancer   10.99.76.181    98.138.219.232     6379:30141/TCP,9042:31059/TCP,5433:30577/TCP   43h
yb-tservers          ClusterIP      None            <none>        7100/TCP,9000/TCP,6379/TCP,9042/TCP,5433/TCP   43h

The following example shows a client that uses the YSQL shell (ysqlsh) to connect:

$ docker run yugabytedb/yugabyte-client ysqlsh -h 98.138.219.232
yugabyte=# CREATE TABLE demo(id INT PRIMARY KEY);
CREATE TABLE

The following example shows a client that uses the YCQL shell (ycqlsh) to connect:

$ docker run yugabytedb/yugabyte-client ycqlsh 98.138.219.232 9042
ycqlsh> CREATE KEYSPACE demo;
ycqlsh> use demo;
ycqlsh:demo> CREATE TABLE t_demo(id INT PRIMARY KEY);

YB-Master Admin UI

The YB-Master Admin UI is available at the IP address exposed by the yb-master-ui LoadBalancer service – at https://98.138.219.231:7000/.

Another option that does not require an external LoadBalancer is to create a tunnel from the local host to the master web server port on the master pod using kubectl port-forward, as follows:

$ kubectl port-forward pod/yb-master-0 7000:7000 -n yb-demo
Forwarding from 127.0.0.1:7000 -> 7000
Forwarding from [::1]:7000 -> 7000

Connecting Externally to a Minikube Cluster

When the Kubernetes cluster is set up using Minikube, an external IP address is not available by default for the LoadBalancer endpoints. To enable the load balancer IP address, run the command minikube tunnel. For details, see LoadBalancer access.

$ minikube tunnel
Status:
    machine: minikube
    pid: 38193
    route: 10.96.0.0/12 -> 192.168.99.100
    minikube: Running
    services: [yb-master-ui, yb-tserver-service]
    errors:
        minikube: no errors
        router: no errors
        loadbalancer emulator: no errors

Connecting TLS-Secured YugabyteDB Cluster Deployed by Helm Charts

To start a YugabyteDB cluster with encryption in transit (TLS) enabled, follow the steps at Google Kubernetes Service (GKE) - Helm Chart and set the flag tls.enabled=true in the helm command line, as shown in the following example:

helm install yugabyte --namespace yb-demo --name yb-demo --set=tls.enabled=true

Connect from Within the Kubernetes Cluster

Copy the following yb-client.yaml and use the kubectl create -f yb-client.yaml command to create a pod with auto-mounted client certificates, as follows:

apiVersion: v1
kind: Pod
metadata:
  name: yb-client
  namespace: yb-demo
spec:
  containers:
  - name: yb-client
    image: yugabytedb/yugabyte-client:latest
    env:
    - name: SSL_CERTFILE
      value: "/root/.yugabytedb/root.crt"
    volumeMounts:
    - name: yugabyte-tls-client-cert
      mountPath: "/root/.yugabytedb/"
  volumes:
  - name: yugabyte-tls-client-cert
    secret:
      secretName: yugabyte-tls-client-cert
      defaultMode: 256

When a client uses the YSQL shell (ysqlsh) to connect, you can execute the following command to verify the connection:

$ kubectl exec -n yb-demo -it yb-client -- ysqlsh -h yb-tservers.yb-demo.svc.cluster.local "sslmode=require"
ysqlsh (11.2-YB-2.1.5.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

yugabyte=# \conninfo
You are connected to database "yugabyte" as user "yugabyte" on host "yb-tservers.yb-demo.svc.cluster.local" at port "5433".
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)

When a client uses the YCQL shell (ycqlsh) to connect, you can execute the following command to verify the connection:

$ kubectl exec -n yb-demo -it yb-client -- ycqlsh yb-tservers.yb-demo.svc.cluster.local 9042 --ssl
Connected to local cluster at yb-tservers.yb-demo.svc.cluster.local:9042.
[cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> SHOW HOST
Connected to local cluster at yb-tservers.yb-demo.svc.cluster.local:9042.

Optionally, you can use the following command to remove the client pod after the operations have been completed:

$ kubectl delete pod yb-client -n yb-demo
pod "yb-client" deleted

Connect Externally

To connect externally to a TLS-enabled YugabyteDB helm cluster, start by downloading the root certificate from the Kubernetes cluster's secrets, as follows:

$ mkdir $(pwd)/certs
$ kubectl get secret yugabyte-tls-client-cert  -n yb-demo -o jsonpath='{.data.root\.crt}' | base64 --decode > $(pwd)/certs/root.crt

When a client that uses the YSQL shell (ysqlsh) to connect, the command to execute specifies the external LoadBalancer IP of the yb-tserver-service, as described in Connect using external clients. You can verify the connection via the following command:

$ docker run -it --rm -v $(pwd)/certs/:/root/.yugabytedb/:ro yugabytedb/yugabyte-client:latest ysqlsh -h <External_Cluster_IP> "sslmode=require"
ysqlsh (11.2-YB-2.1.5.0-b0)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

yugabyte=# \conninfo
You are connected to database "yugabyte" as user "yugabyte" on host "35.200.205.208" at port "5433".
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)

When a client uses the YCQL shell (ycqlsh) to connect, you can verify the connection by executing the following docker run command:

$ docker run -it --rm -v $(pwd)/certs/:/root/.yugabytedb/:ro \
--env SSL_CERTFILE=/root/.yugabytedb/root.crt yugabytedb/yugabyte-client:latest ycqlsh <External_Cluster_IP> 9042 --ssl
ysqlsh (11.2-YB-2.1.5.0-b0)
Connected to local cluster at 35.200.205.208:9042.
[cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> SHOW HOST
Connected to local cluster at 35.200.205.208:9042.