Three ways to easily manage many Kubernetes clusters

Do you need to manage hundreds of Kubernetes clusters, or are you just curious to know why you could potentially need more than two clusters, or do you want to see the magic behind it? You are in the right place because you’re about to learn everything there is to know about it.

Albane Tonnellier

Albane Tonnellier

March 14, 2022 · 5 min read
Three ways to easily manage many Kubernetes clusters - Qovery
Written byAlbane Tonnellier

Albane Tonnellier

Albane is a Product Owner at Qovery with previous RoR developer experience.

See all articles
ProductKubernetesTerraform

What is a Kubernetes cluster?

A Kubernetes cluster is a set of nodes with a certain amount of CPU and RAM that run and manage your application workload in a resilient way. Kubernetes is ideal in case of worker node disruption. There is no need to panic; Kubernetes got you back and will keep your application up and running.

Why would you want to manage many Kubernetes clusters?

It is widespread to have two separate clusters for staging and production; if you don’t already know why we have an article about it but why could you need many Kubernetes clusters? Here are a few reasons:

Location

If you want your apps to be close to your clients to reduce latency, you will probably need several clusters in several geographical zones.

Having multiple clusters can also be a requirement due to data regulation depending on the business, e.g. financial companies in Hong Kong can’t move the data out of the country.

Map of AWS global infrastructure | © Jayendra Patil
Map of AWS global infrastructure | © Jayendra Patil

Scalability

There are two main reasons why you might need several clusters for scalability:

Anticipation: Even if you can (theoretically) reach 15 000 nodes on your cluster Kube before needing more than one cluster, you don’t want to wait until the day that this happens to dispatch everything because it will be very time consuming, so it is a good practice to do it from day one to save some headache in the long run.

Complexity: The more workload you will have on one cluster, the more complex it will be to manage, and even if the maximum nodes that you can have in a cluster Kube are (theoretically) 15 000, a lot of companies are having issues when they reach more than 1 000 nodes, that’s the experience of OpenAI that scaled their Kubernetes to 7 500 nodes and had several problems listed in this article or PayPal who scaled to over 4 000 nodes and 200k pods which also talked about their experience in this article.

Security

In this article, we’ve covered the case where you need to create a single-tenant application and physically isolate your applications for each customer. It’s a valid and widespread use case for companies working in the healthcare industry. Suppose each Kubernetes cluster is for one single user, then you can have thousands of clusters to manage.

Fun

Alright, I don’t know if it can be fun to have many Kubernetes clusters. But imagine all the fancy things that you can do. What about making a Kubernetes battleship where every ship is a Kubernetes cluster that you have to destroy? 😅

Battleship game | © Aleksandra Nigmatulina, Getty image
Battleship game | © Aleksandra Nigmatulina, Getty image

How to manage many Kubernetes clusters with Qovery

With Qovery, you don’t have one but three different ways to manage many clusters so let’s dig into it and see which one is the best.

1. Web interface

This one will be pretty painful as we will manually create our clusters one by one with our lovely Qovery console.

Once you have created your organization, head to the organization settings, select “clusters", and click “add a cluster”.

You can then enter a name for your first cluster, set your credentials, add the features you want and click on “create”. You should then see your cluster, and it’s now time to click on the three dots on the right of your cluster and select “install”.

While your first cluster is deploying, you can do the same with the second one, then the third one, and repeat the process until you have the number needed 😅

Yes, it is long and fastidious, but it’s a way to do it; now, don’t panic; we have two much better ways to achieve this goal with Qovery.

2. API (bash)

Qovery provides a fluent web API with complete documentation that you can find at: api-doc.qovery.com.

Bash allows you to create as many clusters as you want easily; the line of code is the same every time except that you need to change the name of your cluster and the region if you wish to have a different region for each cluster. The example below shows you three clusters: a “prod cluster”, a “staging cluster”, and a “dev cluster”, all created with three lines of code only.

curl -H 'Content-type: application/json' -X POST -d '{"name": "prod cluster", "cloud_provider": "AWS", "region": "us-east-1"}' -H "Authorization: Token $token" "<https://api.qovery.com/organization/><your_organization_id>/cluster"
curl -H 'Content-type: application/json' -X POST -d '{"name": "staging cluster", "cloud_provider": "AWS", "region": "us-east-1"}' -H "Authorization: Token $token" "<https://api.qovery.com/organization/><your_organization_id>/cluster"
curl -H 'Content-type: application/json' -X POST -d '{"name": "dev cluster", "cloud_provider": "AWS", "region": "us-east-1"}' -H "Authorization: Token $token" "<https://api.qovery.com/organization/><your_organization_id>/cluster"

3. Terraform provider

If you are a big fan of Qovery already, you might know that the Qovery terraform provider is now available, and if you want to know everything about it, I have the right article for you.

How can we use this Terraform provider to manage many Kubernetes clusters?

As a video is worth a thousand words, here is a video showing how to create a cluster (and even more) with Terraform.

Now to create several ones, you need to create several “resource” with all your different clusters in your Terraform file like showcase just below with a "prod_cluster", "staging_cluster", and "dev_cluster".

terraform {
  required_providers {
    qovery = {
      source  = "qovery/qovery"
    }
  }
}

provider "qovery" {
  # read to get your token - <https://hub.qovery.com/docs/using-qovery/interface/cli/#generate-api-token>
  # execute command `qovery token`
  token = var.qovery_access_token
}

resource "qovery_aws_credentials" "your_aws_creds" {
  organization_id   = var.organization_id
  name              = "my AWS creds"
  access_key_id     = var.aws_access_key_id
  secret_access_key = var.aws_secret_access_key
}

resource "qovery_cluster" "prod_cluster" {
  organization_id = var.organization_id
  credentials_id  = qovery_aws_credentials.your_aws_creds.id
  name            = "prod cluster"
  description     = "this is a production cluster created with Terraform"
  cloud_provider  = "AWS"
  region          = "us-east-1"
  cpu             = 2000
  memory          = 8192

  depends_on = [
    qovery_aws_credentials.your_aws_creds
  ]
}

resource "qovery_cluster" "staging_cluster" {
  organization_id = var.organization_id
  credentials_id  = qovery_aws_credentials.your_aws_creds.id
  name            = "staging cluster"
  description     = "this is a staging cluster created with Terraform"
  cloud_provider  = "AWS"
  region          = "us-east-1"
  cpu             = 2000
  memory          = 8192

  depends_on = [
    qovery_aws_credentials.your_aws_creds
  ]
}

resource "qovery_cluster" "dev_cluster" {
  organization_id = var.organization_id
  credentials_id  = qovery_aws_credentials.your_aws_creds.id
  name            = "dev cluster"
  description     = "this is a dev cluster created with Terraform"
  cloud_provider  = "AWS"
  region          = "us-east-1"
  cpu             = 2000
  memory          = 8192
  
  depends_on = [
    qovery_aws_credentials.your_aws_creds
  ]
}

Wrapping up

Whether you need many Kubernetes clusters for scalability or location reasons or just two to separate your production and your staging, Qovery got your back with the multi-cluster feature!

With just a few lines of code, you can easily manage as many clusters as you need and use your precious time to focus on what matters the most, the success of your product.

Deploy your apps in production on AWS in minutes

Qovery combines the power of Kubernetes, the reliability of AWS, and the simplicity of Heroku to deploy your apps on AWS.

Deploy my apps
Deploy your apps in production on AWS in minutes
ProductKubernetesTerraform

You might also like