Microsoft Azure

Prerequisites

Connect to the Azure Cloud Shell. You can connect and get a shell from the browser by navigating to Azure bash cloud shell.
Register the necessary Azure service providers by running the following:

az provider register -n Microsoft.Network
az provider register -n Microsoft.Storage
az provider register -n Microsoft.Compute
az provider register -n Microsoft.ContainerService

Configure a default location. Remember to replace eastus with an appropriate Azure location (region) of your choice that supports AKS clusters.

az configure --defaults location=eastus

1. Create an Azure cluster

Create an Azure resource

An Azure resource group is a logical group in which Azure resources are deployed and managed. You need to specify a default location or pass the location parameter to create the resource. The resources we create for the AKS cluster will live in this Azure resouce.

$ az group create --name yb-eastus-resource

Create the AKS cluster.

You can create a three node AKS cluster by running the following command.

$ az aks create --resource-group yb-eastus-resource --name yb-aks-cluster --node-count 3 --generate-ssh-keys

Configure kubectl to work with this cluster.

$ az aks get-credentials --resource-group yb-eastus-resource --name yb-aks-cluster

Verify the cluster by running the following command.

$ kubectl get nodes

NAME                       STATUS    ROLES     AGE       VERSION
aks-nodepool1-25019584-0   Ready     agent     4h        v1.7.9
aks-nodepool1-25019584-1   Ready     agent     4h        v1.7.9
aks-nodepool1-25019584-2   Ready     agent     4h        v1.7.9

2. Create a YugabyteDB cluster

Create a YugabyteDB cluster by running the following.

$ kubectl create -f https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/cloud/kubernetes/yugabyte-statefulset.yaml

service "yb-masters" created
statefulset "yb-master" created
service "yb-tservers" created
statefulset "yb-tserver" created

3. Check the cluster

You should see the following pods running.

$ kubectl get pods

NAME           READY     STATUS    RESTARTS   AGE
yb-master-0    1/1       Running   0          3m
yb-master-1    1/1       Running   0          3m
yb-master-2    1/1       Running   0          3m
yb-tserver-0   1/1       Running   0          3m
yb-tserver-1   1/1       Running   0          3m
yb-tserver-2   1/1       Running   0          3m

You can view the persistent volumes.

$ kubectl get persistentvolumes

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                          STORAGECLASS   REASON    AGE
pvc-849395f7-36f2-11e8-9445-0a58ac1f27f1   1Gi        RWO            Delete           Bound     default/datadir-yb-master-0    default                  12m
pvc-8495d8cd-36f2-11e8-9445-0a58ac1f27f1   1Gi        RWO            Delete           Bound     default/datadir-yb-master-1    default                  12m
pvc-8498b836-36f2-11e8-9445-0a58ac1f27f1   1Gi        RWO            Delete           Bound     default/datadir-yb-master-2    default                  12m
pvc-84abba1a-36f2-11e8-9445-0a58ac1f27f1   1Gi        RWO            Delete           Bound     default/datadir-yb-tserver-0   default                  12m
pvc-84af3484-36f2-11e8-9445-0a58ac1f27f1   1Gi        RWO            Delete           Bound     default/datadir-yb-tserver-1   default                  12m
pvc-84b35d19-36f2-11e8-9445-0a58ac1f27f1   1Gi        RWO            Delete           Bound     default/datadir-yb-tserver-2   default                  12m

You can view all the services by running the following command.

$ kubectl get services

NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                               AGE
kubernetes    ClusterIP   XX.XX.XX.X   <none>        443/TCP                               23m
yb-masters    ClusterIP   None         <none>        7000/TCP,7100/TCP                     17m
yb-tservers   ClusterIP   None         <none>        9000/TCP,9100/TCP,9042/TCP,6379/TCP   14m

4. Connect to the cluster

You can connect to the YCQL API by running the following.

$ kubectl exec -it yb-tserver-0 bin/cqlsh

Connected to local cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> DESCRIBE KEYSPACES;

system_schema  system_auth  system

5. Destroy the YugabyteDB cluster (optional)

Destroy the YugabyteDB cluster we created above by running the following.

$ kubectl delete -f https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/cloud/kubernetes/yugabyte-statefulset.yaml

service "yb-masters" deleted
statefulset "yb-master" deleted
service "yb-tservers" deleted
statefulset "yb-tserver" deleted

To destroy the persistent volume claims (you will lose all the data if you do this), run:

$ kubectl delete pvc -l app=yb-master
$ kubectl delete pvc -l app=yb-tserver

6. Destroy the AKS cluster (optional)

To destroy the resource we created for the AKS cluster, run the following.

$ az group delete --name yb-eastus-resource

Advanced Kubernetes Deployment

More advanced scenarios for deploying in Kubernetes are covered in the Kubernetes Deployments section.

Prerequisites

Download and install terraform.
Verify by the terraform command, it should print a help message that looks similar to that shown below.

$ terraform

Usage: terraform [--version] [--help] <command> [args]
...
Common commands:
    apply              Builds or changes infrastructure
    console            Interactive console for Terraform interpolations
    destroy            Destroy Terraform-managed infrastructure
    env                Workspace management
    fmt                Rewrites config files to canonical format

1. Create a terraform config file

Create a terraform config file called yugabyte-db-config.tf and add following details to it. The terraform module can be found in the terraform-azure-yugabyte github repository.

provider "azurerm" {
  # Provide your Azure Creadentilals 
    subscription_id = "AZURE_SUBSCRIPTION_ID"
    client_id       = "AZURE_CLIENT_ID"
    client_secret   = "AZURE_CLIENT_SECRET"
    tenant_id       = "AZURE_TENANT_ID"
}

module "yugabyte-db-cluster" {
  # The source module used for creating AWS clusters.
  source = "github.com/Yugabyte/terraform-azure-yugabyte"

  # The name of the cluster to be created, change as per need.
  cluster_name = "test-cluster"

  # key pair.
  ssh_private_key = "SSH_PRIVATE_KEY_HERE"
  ssh_public_key = "SSH_PUBLIC_KEY_HERE"
  ssh_user = "SSH_USER_NAME_HERE"

  # The region name where the nodes should be spawned.
  region_name = "YOUR VPC REGION"

  # The name of resource  group in which all Azure resource will be created. 
  resource_group = "test-yugabyte"

  # Replication factor.
  replication_factor = "3"

  # The number of nodes in the cluster, this cannot be lower than the replication factor.
  node_count = "3"
}

NOTE: To insatll terraform and configure it for Azure, follow steps given here

2. Create a cluster

Init terraform first if you have not already done so.

$ terraform init

Now run the following to create the instances and bring up the cluster.

$ terraform apply

Once the cluster is created, you can go to the URL http://<node ip or dns name>:7000 to view the UI. You can find the node’s ip or dns by running the following:

$ terraform state show azurerm_virtual_machine.Yugabyte-Node[0]

You can access the cluster UI by going to any of the following URLs.

You can check the state of the nodes at any point by running the following command.

$ terraform show

3. Verify resources created

The following resources are created by this module:

module.azure-yugabyte.azurerm_virtual_machine.Yugabyte-Node The Azure VM instances.

For cluster named test-cluster, the instances will be named yugabyte-test-cluster-node-1, yugabyte-test-cluster-node-2, yugabyte-test-cluster-node-3.

module.azure-yugabyte.azurerm_network_security_group.Yugabyte-SG The security group that allows the various clients to access the YugabyteDB cluster.

For cluster named test-cluster, this security group will be named yugabyte-test-cluster-SG with the ports 7000, 9000, 9042, 7100, 9200 and 6379 open to all other instances in the same security group.

module.azure-yugabyte.null_resource.create_yugabyte_universe A local script that configures the newly created instances to form a new YugabyteDB universe.
module.azure-yugabyte.azurerm_network_interface.Yugabyte-NIC The Azure network interface for VM instance.

For cluster named test-cluster, the network interface will be named yugabyte-test-cluster-NIC-1, yugabyte-test-cluster-NIC-2, yugabyte-test-cluster-NIC-3.

4. Destroy the cluster (optional)

To destroy what we just created, you can run the following command.

$ terraform destroy

Deploying From Azure Cloud Shell

First clone this repo. $ git clone https://github.com/YugaByte/azure-resource-manager.git
Change current directory to cloned git repo directory
Use Azure CLI command to create deployments
$ az group deployment create --resource-group <Your-Azure-Resource-Group> --template-file yugabyte_deployment.json --parameters ClusterName='<Your-Cluster-Name>' SshUser='<Your-SSH-USER>' YBVersion='1.3.0.0' SshKeypair='<Your-SSH-USER-PublicKey-File-Contents>'
Once the deployment creation is complete, you can describe it as shown below. $ az group deployment show -g <Your-Azure-Resource-Group> -n <YOur-Deployment-Name> --query properties.outputs In the output, you will get the YugaByte DB admin URL, JDBC URL, YSQL, YCQL and YEDIS connection string. You can use YugaByte admin URL to access admin portal.

Deploying From Azure Portal

Clone this repo locally. $ git clone https://github.com/YugaByte/azure-resource-manager.git
First create a resource group, to create a new resource group, select Resource groups from the Azure portal.
Under newly created Resource groups, select Add.
In opened marketplace search for Template deployment (deploy using custom templates) and click on create.
Now click on Build your own template in the editor.
Click Load file button in specify template section and upload the yugabyte_deployment.json file from cloned repo.
Click on the Save button at the bottom of the window.
Now provide the required details.
Once details are provided, then check the Terms and Condition checkbox and click on the Purchase button.
Once deployments get compleated, you can access the YugaByte DB admin from URL you get in the deployment output section.

Deploy

Google Cloud Platform

Change data capture (CDC)

Change data capture (CDC) to Kafka