Architecture Overview

This document outlines the main aspects of Kublr’s general architecture. Kublr is a Kubernetes management platform which accelerates and controls the deployment, scaling, monitoring and management of Kubernetes clusters.

The diagram below describes two main elements of Kublr architecture:

  1. Kublr Platform is a centralized management platform for Kubernetes clusters.
  2. Kublr Kubernetes Clusters are Kubernetes cluster(s) usually provisioned by Kublr Platform and managed through the platform.

Kublr Platform

Kublr Control Plane

Kublr Platform (at the top-left) is the main management portal for your Kubernetes clusters. Kublr Platform consists of a number of integrated management components deployed into a Kubernetes cluster themselves.

Kublr Control Plane

The Kublr platform components include:

  1. Control Plane / UI: It is the management interface that allows users to create, view, edit, and delete clusters. Additionally, it provides access to other user interfaces, like backup, KeyCloak, Grafana and Kibana.
  2. Centralized Monitoring: Based on Prometheus and Grafana, it collects metrics from all clusters deployed with Kublr, including cloud, hardware, OS, Kubernetes, and application metrics.
  3. Centralized Logging: Based on standard ELK (Elasticsearch, LogStash Kibana) stack, Kublr essentially collects logs from all clusters, including hardware, OS, Kubernetes, and containers metrics. Additionally, it collects audit log components ensuring traceability and audit of all user actions.
  4. IAM, JBoss Keycloak: Open source identity and access management (IAM) provides user management. Additionally, Kublr integrates with LDAP/AD, 2FA and other enterprise authentication tools. The identity broker is responsible for user management, arranging integration among external users, identity providers (SAML, OIDC, LDAP, AD) and authenticating users in Kublr and Kublr managed Kubernetes clusters.
  5. Backup Controller (AWS, Technical Preview): It is the component responsible for cluster backups, purging old backups and restoration of the clusters from backup snapshots. Cluster backup snapshots include both Kubernetes metadata and application data. This functionality is currently only available as a technical preview on AWS only and will most probably change significantly in the future.

Kubernetes clusters

A Kubernetes cluster is the main management object for the Kublr Platform. In most cases the Kubernetes clusters are provisioned by Kublr Platfrom, although since Kublr 1.18.0 externally provisioned clusters (e.g. GKE, AKS, etc) are also supported.

Kublr configures four architectural layers. After being correctly setup and configured (usually by a higher layer), each layer is self-sufficient, self-healing and self-reliant. As long as the underlying infrastructure and layers are functional, each layer functions without interruption.

Layers at play:

  1. The infrastructure layer includes virtual and/or physical machines/servers hosted in a datacenter or in a cloud (e.g. AWS, Azure). Kublr sets up and manages Kubernetes on top of any virtual infrastructure or multiple OS, although some infrastructure providers allow better automation than others. When setting up instances in any environment—either a Kubernetes master or node—Kublr ensures they are fully replaceable should an instance failure occur.

    In AWS and Azure, Kublr enables self-healing on an infrastructure level via AWS’ auto-scaling groups and Azure’s VM auto-restart. On-premises, in case of a failure, any virtual and physical machines can be easily replaced, using manual or infrastructure automation tools of choice.

  2. The Kublr Agent is a single binary running as a service on every Kublr instance, both masters and workers.

    The Kublr Agent is then responsible for configuring Kubernetes components on machines and connecting them into the Kubernetes cluster as described above.

    As a result, a new Kubernetes cluster is started on provisioned infrastructure and connected to the Kublr Control Plane with centralized authentication, monitoring and logging.

    The Kublr agent’s responsibilities include:

    • Setup and configuration of auxiliary software packages on instances and infrastructure resources for the instances, such as etcd EBS volume attachment and initialization on AWS environment;
    • Initialization, distribution (through secret store, such as private S3 bucket or Azure storage) and rotation of shared configuration and security data, such as various certificates, keys, tokens etc., required for a secure Kubernetes setup;
    • Initial setup and configuration of Kubernetes components including Docker, etcd, Kubernetes master components, kubelet, kube-proxy, essential add-ons, and overlay network provider components.
  3. The Kubernetes layer includes all standard Kubernetes components - etcd cluster, Kubernetes master components (API, scheduler, controller manager etc.), kubelet, kube-proxy, various Kubernetes add-ons, such as DNS, dashboard, overlay network provider, auto scaler etc.

    Kublr sets up and connects Kubernetes components on each instance so that communication between them is secure, reliable, and able to recover from failures as long as and as soon as the underlying infrastructure is recovered.

    Kublr uses unmodified standard Kubernetes components and Kubernetes configuration best practices ensuring secure, reliable, and standard conformant Kubernetes setup.

  4. Kublr management layer includes optional components deployed by Kublr into the provisioned or managed Kubernetes cluster to integrate it with Kublr Platform cluster management facilities or provide additional functionality in the managed clusters: such as centralized and local log collection and monitoring.

Diagram below depicts main components of Kublr Cluster and Kubernetes:

Kublr Kubernetes Cluster

Centralized Monitoring

Kublr’s centralized monitoring capability is built on top of the Prometheus and Grafana. Each cluster is registered as a metrics source in Prometheus. Kubernetes provides cloud, hardware, OS, Kubernetes and application metrics using the Kubernetes API of each cluster respectively. Kublr manages metrics source lists in Prometheus, while Grafana’s single sign-on integration with the Kublr Control Plane provides easy access to key dashboards. The centralized monitoring component is deployed as a Helm package.

Centralized Monitoring

Centralized Log Collection

Kublr’s centralized log collection capability is built on top of the Elasticsearch, Logstash, and Kibana stack. In addition, for better resilience RabbitMQ is used to accamulate the remaining log entries between sync up sessions. On a Kublr cluster level, Helm package with RabbitMQ, FluentD and haproxy is deployed. FluentD collects log entries from all levels: hardware, OS, pods, Kubernetes and Kublr core components. RabbitMQ is configured to be the primary destination for all collected logs and haproxy used for data channels between the Kublr Platform and Kublr clusters through Kubernetes port-forwarding feature.

On the Kublr Platform side, the Kublr-central-logging helm package includes Elasticsearch, Kibana, Logstash with RabbitMQ and the RabbitMQ Shovel plugin. The latter transfers messages to the centralized RabbitMQ where they are digested by Logstash and stored in Elasticsearch. Kibana, integrated with Kublr through single sign-on, provides a convenient UI to access and search log entries from all clusters.

In addition to centralized log collection, local Elasticsearch and Kibana may be installed by the user. They act in parallel with centralized log collection mechanism.

Centralized Log Collection

Kublr Backup Controller

The Kublr Backup Controller is deployed as a Kubernetes application. Its main responsibility is to make volume snapshots from all volumes attached to the clusters as specified in the backup schedule. Additionally, the Backup Controller is also responsible for purging old backup snapshots and cluster restoration logic from provided snapshot.