The Context

There is a widespread belief that Kubernetes isn’t ready for stateful applications like MySQL and MongoDB. It has even been asked if running stateful applications in Kubernetes is worth the risk, but developer Kris Nova from Heptio asks instead, “Are you ready for stateful workloads in Kubernetes? These misconceptions are, as we get into, partly because the community initially focused on stateless applications and only added support for stateful applications (i.e. persistent storage) relatively late in the day.

This shortcoming has now been firmly addressed, however, both by Kubernetes itself and various third parties, and this post looks at the ways in which it is now possible; moreover, we look at why Kubernetes is quickly becoming the preferred platform for stateful cloud-native applications.

What is a Stateful Application?

A stateful application saves data about each client session that occurs and holds on to that data until the next time the client issues a request. The data which is saved is known as the state of the app, hence the term stateful. How is that data stored? Either locally or on a remote host until the time at which the user logs out, or following a predetermined amount of time when the session is set to expire.

A stateless application, by contrast, is an application program that doesn’t save client data generated in one session for use in the next with the same client. HTTP, for instance, is inherently stateless. This means that when a state is desired for a web app, stateful features need to be built in with dynamic pages. The pages can store sessions via web address variables and server and client-side stored data, such as via cookies.

While most desktop apps and operating systems are stateful, lots of apps that operate in the cloud’s distributed architecture are stateless. This is because in the early days, it was hard to limit data stores to specific locations when virtual machines were themselves stateless; thus when stateful apps were moved directly to the cloud, they frequently became unstable.

Enter Containerization

Containerization, however, is changing this. The OS-level virtualization method used to deploy and run distributed applications without the need to launch a complete VM for each application means that files can be pulled into the container on startup, then persist elsewhere while containers stop and start.

Managing Stateful Applications in Kubernetes

There are various possible ways to manage stateful applications. Kubernetes itself offers the StatefulSet and DaemonSet integrated technologies, which allow you to run your database in Kubernetes, and each offer different support options in doing so. There are also different options for running your database via third parties, and multiple container operating systems available to do so.

Some Useful Terms

Before we get too deeply into the management solutions, however, let’s look at some useful terms in the context of databases and stateful applications in a containerization setting: Pod, Volume, Persistent Volume and Persistent Volume Claims.

• A pod is a group of one or several containers, which have a shared network (storage) and a specification for how the containers should be run;
• Volumes are used when files need to be shared between multiple containers that are running in the same pod. On-disk files in a container are typically ephemeral. The files will be lost when a container crashes, allowing the container to start with a clean slate.
• Persistent Volume (PV) refers to a piece of storage in the cluster. In comparison to Volumes, which will be deleted when the pod is deleted, Persistent Volumes are completely different entities, decoupled from the pod. PVs are managed by a different set of APIs to Pods, and have their own lifecycle.
• Persistent Volume Claims (PVC) – these are a request for storage, similar to a pod, but PVCs consume Persistent Volume resources as opposed to how Pods consume node resources. Pods are able to request specific levels of resources (memory and CPU) while PVCs can only request specific size and access modes.

A clear demonstration of how Persistent Volumes work can be found here.

Difficulties with Managing State in Kubernetes

The main reasons for why managing state in Kubernetes is difficult are:

• Database replicas each have an unique state and are not interchangeable – this means they can’t be trivially brought up and down at a moment’s notice;
• Deploying a database replica needs to involve coordination with nodes that are running the same application to make sure that version upgrades, schema changes and other such things are visible everywhere;
• Essentially, managing state in Kubernetes is difficult as the system’s dynamism is simply too chaotic for most databases to be able to easily handle, SQL databases in particular require strong consistency.  

Management Options

Management Option #1: The Kubernetes Internal Solutions

Option #1A: StatefulSets

StatefulSets is the most popular way to run a database. It was built by Kubernetes to make it easier to build stateful applications. In essence, StatefulSets means that each pod is certain of having the same network identity and disk across all restarts, even if this is rescheduled to a different physical machine.

StatefulSets automatically takes care of the challenges of gracefully scaling workloads and upgrading stateful applications, in addition to preserving network identity across restarts of the container. According to InfoWorld, “StatefulSets provide a great foundation to build, automate, and operate highly available applications such as databases.”

StatefulSets does this by representing a set of Pods with “unique, persistent identities and stable hostnames that GKE maintains regardless of where they are scheduled. The state information and other resilient data for any given StatefulSet Pod is maintained in persistent disk storage associated with the StatefulSet.”

The program uses a Pod template, which includes a specification for the Pods. The specification sets out how a Pod should look, including which applications should run inside its containers, the volumes it should mount and its labels and selectors, along with other selections.

StatefulSets work in conjunction with the deployment of multiple applications that need unique, persistent identities and stable hostnames, including Kafka, MySQL, Redis and ZooKeeper.

One challenge of using StatefulSets is that Kubernetes needs to run on the machines that are running your database, which can impact performance slightly because it consumes resources. It is also possible that the stateful service will still be in contention with others for the machine’s physical resources because StatefulSets lets your database pods be rescheduled onto other nodes. You can alleviate this issue by managing the resources the database container requests, however.

Option #1B: DaemonSets

DaemonSets allow you to pre-determine how a group of nodes run a specific pod. This allows you to designate a specific set of nodes to run your database, and means that Kubernetes will make sure the service stay available on these particular nodes without being rescheduled, or without running anything else on those nodes, which is very useful for stateful services.

As nodes are added to the Cluster, Pods are added to them. As nodes are taken out of the cluster, those Pods are collected by garbage. Deleting a DaemonSet will clean up the Pods it created.

It still leverages many of the Kubernetes features such as declarative infrastructure, but with this option, you have a little less flexibility.

Use cases for DaemonSets include:

• running a cluster storage daemon e.g. glusterd, ceph, on each node;
• running a logs collection daemon on all nodes e.g. as fluentd or logstash;
• running a node monitoring daemon on all nodes e.g. Prometheus Node Exporter, collectd, Dynatrace OneAgent, AppDynamics Agent, Datadog agent, New Relic agent, Ganglia gmond or Instana agent.

With simple cases, one DaemonSet that covers every node can be used for each type of daemon. For more complex setups, multiple DaemonSets can be used for a single type of daemon, but employ different flags and/or different CPU and memory requests for different types of hardware.

DaemonSets can limit the amount of contention between your database and other apps by cordoning off entire Kubernetes nodes. This means it is easier to cordon off your database on to dedicated nodes and easily use local disks in comparison to in StatefulSets where local disk support is still in beta.

The biggest challenge associated with DaemonSets is that it limits Kubernetes’ ability to assist your cluster in recovering from failures.

Management Option #2: Work with a Third Party

There are also third parties you can turn to to manage your databases; both in terms of running your entire stack outside Kubernetes or via working with a DBaaS (database-as-a-service) provider to manage solely the database.Third-party container operating system solutions, in addition to those listed below, include Alpine Linux, CoreOS Container Linux, RancherOS, and VMware Photon OS.

Challenges to consider when running your stack inside K8s, but running the database outside it include the following:

• You will still have to run an entire stack of infrastructure management tools for just one service;
• You’ll duplicate efforts of what Kubernetes is already capable of doing, including:
• Process monitoring
• Management of configurations
• In-datacenter load balancing
• Monitoring and logging
• Service discovery

Even if you do engage a DBaaS provider, you are still only running a single service outside of Kubernetes, which adds an additional layer of complexity.

Some Third Party Options:

Red Hat: Operator Framework… & Other Iterations

Last year, Red Hat launched its Operator Framework as a way of customizing the APIs of Kubernetes for any application. It was initially developed by CoreOS before the company was acquired by Red Hat. With the Operator Framework, users are able to spin up applications with each specific setting and dependency intact and have Kubernetes take care of all the provisioning and scaling. Since then, various independent software vendors (ISVs) have begun to use Red Hat’s Operator Framework to sell their own applications, in particular for managing stateful applications.

Operators aims to make the process of managing backend databases easier than other alternatives on the market, including Kubernetes’ own.

In essence, it gives developers a template that will instruct Kubernetes in how to deploy and maintain an application, including detailed instructions on how to construct and deploy it. The operator packages the details of the configuration into a YAML-defined custom resource definition (CRD) that Kubernetes will understand. The operator covers dependencies and security permissions while a linked Red Hat-developed Operator Lifecycle Manager gives the administrator the opportunity to manage multiple operators.

Operators can be written in Go with Helm or in Red Hat’s Ansible automation software.

Couchbase Autonomous Operator for Kubernetes

Couchbase launched its Autonomous Operator for Kubernetes last year. The goal: to “unleash the power of Kubernetes with the Couchbase Data Platform”. Couchbase claims to be the only NoSQL vendor that offers native integration of Kubernetes with its own data platform. Couchbase has carved out market space for the support of social media engagement by modifying use cases such as maintaining profiles, mobile capabilities and chatting. As more enterprises and start-ups containerize their data platforms, they expect stateful applications. Couchbase offers in-built replication with data distributed between several nodes (meaning it has the native capability of failing over to the backup nodes if the main one fails) to counter the fact that Kubernetes has no default way of offering persistent storage. However, Kubernetes working in conjunction with the Autonomous Operator is exceptionally quick at finding new hardware within the system to bring a failed cluster back up to full strength in far less time than if a human were to do it.

Kubernetes Support of the CSI Initiative

Kubernetes supports the Container Storage Interface (CSI), which both introduces a uniform interface for storage vendors across many different container orchestrators and makes it more straightforward to provide support to new storage systems, to foster innovation, and to offer more options for DevOps teams.