Comparison of the Kubernetes Engines

EKS vs AKS vs GKE vs Kops vs RKE vs Kubespray vs k3s

Category: Kubernetes
Category: Kubernetes

Comparison of the Kubernetes Engines

Many companies use Kubernetes to orchestrate containers nowadays. They have many options for provisioning and managing their clusters. Sometimes they struggle to choose the option that suits their requirements. In this blog post, I will discuss how you can use tools to build your infrastructure frameworks.

You will need the following aspects addressed for a functioning Kubernetes installation:

  • Provisioning Servers
  • TLS  Management
  • Configuration management(Kubernetes upgrade)
  • Scaling

You can address these from scratch manually, but it can be hard to provide an automated infrastructure in that case. For a highly-available, automated infrastructure, you need a framework that addresses the aspects above. We will talk about six of these tools today.


CNCF Survey 2019

Ref: CNCF Survey 2019 

Tools on the Cloud: EKS, AKS, GKE, Kops

There are two methods for orchestration in the cloud environment. 

  • Managed for Control Plane layer by Cloud Provider.
  • Managed by the Infrastructure Team.

Managed for Control Plane layer by Cloud Provisioner

Their cloud providers fully manage these services. You can provision those services as-code (terraform etc.).


EKS is fully compatible with applications on any vanilla Kubernetes architecture. The EKS provides a single-tenant control plane on each cluster. Control Planes are not shared across clusters. EKS works with AWS Fargate as well; Fargate gives a way to deploy containers with no understanding of server infrastructure for developers.


>> Learn How to Run Amazon EKS Without an AWS Account



AKS supports any vanilla Kubernetes architecture, also supports Apache Mesos and Docker Swarm. Windows developers usually prefer AKS. AKS does not support HA cluster provisioning.

Google Cloud Kubernetes Engine (GKE):

GKE is a fully managed service that allows you to set up applications in no time and provisions cluster features quicker than other cloud providers. GKE has HA clusters via regional clusters (master and workers replicated.)

Comparison Chart between Cloud Environments:





Upgrade Cluster Nodes

Manual and Automatic Options




Fully Managed Upgrade 


Dedicated Monitoring


Azure Monitor

Kubernetes Engine Monitoring

Availability Zones 


More zone than others



CLI support


OK, plus special eksctl


HA availability 

OK for control plane, manual across AZ for workers


OK regional cluster for master and workers.

Time to Spawn Cluster(3 nodes)




K8s version tracking


Better than EKS


Bare Metal Nodes





EKS Anywhere

ARC (beta)



Strong security with  RBAC and IAM integration.


RBAC Implemented as fully supported.


RBAC Implemented as fully supported.



0.10$/hour + nodes

Only pay Worker Nodes. 

0.10$/hour + nodes


>>Click to Read Kubernetes' Kubectl Diff Feature


Managed by the Infrastructure Team


Kops is a tool that helps you manage Kubernetes components as systemd services and static pods. You can use kops to create your infrastructure framework to manage your clusters based on as-code principles.

Here is a Kops cluster templating example for framework architecture.

# file cluster.yaml
kind: Cluster
 name: .
   master: |
   node: |

# File nodes.yaml ~ master.yaml
kind: InstanceGroup
 labels: .
 name: nodes
 machineType: m4.large
 maxPrice: "0.5"
 maxSize: 20
 minSize: 15
 role: Node
 rootVolumeSize: 100
 - a
 - b
 - c

   kops toolbox template \
   --values values.yaml \
   --template templates/cluster/cluster.yaml \
   --template templates/cluster/master.yaml \
   --template templates/cluster/nodes.yaml \
   --format-yaml \
   --output /tmp/cluster.yaml
   cat /tmp/cluster.yaml
   kops replace --force -f /tmp/cluster.yaml

As you can see above, kops manages your cluster nodes and components such as kubelet, api-server and etcd in a cluster and instance groups files as shown below.

Kops Pros/Cons:



Declarative provisioning with YAML files and cli tool

It‘s available for only cloud systems. (stable AWS)


It lags to support new Kubernetes versions.

Can generate Terraform




Support for custom Kubernetes add-ons


Kops is an excellent way to create a Kubernetes cluster in the cloud systems. It allows HA installation and fully customizable infrastructure.

Support: Docs, open issues from GitHub.

Tools for On-Premise Environments: Kubespray, Kubeadm RKE, K3S


Kubespray is a framework that installs and manages systemd services and static pods. It uses kubeadm in the background. Generally, Kubespray is managed by ansible playbooks. Also, it has Terraform modules too.

Thanks to those Ansible Playbooks, you can edit your cluster nodes and their roles in the inventory.ini file from kubespray GitHub repository. You can also develop cluster components and network configurations in k8s-cluster.yaml.

Kubespray Pros/Cons:



Declarative installation with variables and inventory

Installation takes long

Available for on-premises, bare-metal, and clouds.


Easy customization with Ansible


Support for custom Kubernetes add-ons


It has terraform modules for AWS and OpenStack environments.


Installation takes a long time to execute your changes because Ansible works sequentially, but if you know Ansible, you can easily manage your clusters with Kubespray.

Support: Docs, open issues from GitHub.



kubeadm allows you to install and manage Kubernetes components as systemd services clusters. It is an official installation method suggested by CNCF.

You can develop a framework or use kubespray instead of using kubeadm alone. This configuration is more efficient, especially at HA installations.

kubeadm Pros/Cons:



Official installation method

Still in beta

Effortless installation

HA installation needs some manifest file development

Can be used cloud or bare-metal


A lot of Kubernetes installation options(kubespray etc.) implement kubeadm into their process. But if you want to use kubeadm purely, you should consider extra development processes for HA setup. For example, you should design your framework based on your requirements (e.g., cloud-manager, load balancer setup .. etc.)

Support: Docs and Github.



RKE is a new tool by Rancher that installs and manages kubernetes components as docker containers kubernetes clusters. RKE can be run on the cloud or on-premises.

You can manage RKE easily by a single yaml file, shown below.

   - address: #
     internal_address: #
     user: centos
     ssh_key_path: /Users/Desktop/rke.pem
       - controlplane
       - worker
       - etcd
   - address: #
     internal_address: #
     user: centos
     ssh_key_path: /Users/Desktop/rke.pem
       - controlplane
       - worker
       - etcd
     snapshot: true
     creation: 6h
     retention: 24h
  # Required for external TLS termination with
 # ingress-nginx v0.22+
   provider: nginx
     use-forwarded-headers: "true"
kubernetes_version: v1.18.3-rancher2-2


RKE Pros/Cons:



Easy installation

New solution for kubernetes installations and management

Declarative approach, only one YAML


Easy automation


Can be used on cloud or bare-metal




Back-up solution for etcd


RKE is a new solution compared to the other installation solutions. But you can easily manage and automate your cluster management and provisioning steps.

Support: Docs, enterprise support 24/7 give on the Rancher, and you can open issues from Github.



K3s is developed for IoT and Edge applications. K3S strips out many legacy features/plugins and substitutes Kubernetes components for lightweight alternatives to achieve a binary size of ~60MB. For example, it uses sqlite instead of etcd. K3S is suggested for local or dev/test/stg installations.

An example of K3S architecture is shown below;


K3S architecture


K3S Pros/Cons:



Low resource requirements and low cost.

Require development for heavy load.

It is a perfect option for local installations.


Start up time compared k8s


Support: Docs, enterprise support 24/7 give on the Rancher, and you can open issues from Github



In this blog post, I talked about Kubernetes cluster providers, their benefits, and their shortcomings. I spoke about six tools for cloud and on-premise environments. 

GKE is the go-to implementation with easy set-up and regional HA cluster features. AWS has a few distinguishing features, such as support for bare metal nodes, and it has a beneficial IAM service when integrated with RBAC for cluster security. AKS faster than AWS for provisioning clusters, but it has a disadvantage for regional cluster replication than other providers.

When looking at the managed control plane by the infrastructure teams side, RKE has useful features such as easy automation, managing only one YAML and back-up solution for etcd. Kubespray is a practical framework with ansible. Kops helps management clusters if you create your scripts or framework, but it has the disadvantage of only running at AWS as stable. Finally, K3S is a perfect option for local environments or low load production environments.

These tools evolve as our industry expands and new requirements arise. Therefore it is essential to assess new options for your needs always. You can do this by comparing ease of management, installation, and upgrades to pick the best options to manage your Kubernetes environment. We hope that our comparison gave you a headstart in your Kubernetes journey.


Click to learn how to access AWS resources from Kubernetes

Halil Bozan

Infrastructure and Platform Developer at kloia