StatefulSet is the workload API object used to manage stateful applications. StatefulSets are beta in 1.8.
Kafka and Zookeeper are two of the motivating examples for StatefulSets in Kubernetes. Being stateful applications, we'll need disks to store the data on. GKE can help us allocate disks and compute for brokers even as we scale the service up.
First we need the right Google Container Engine cluster.
Since this is fairly basic and meant for development we'll
use a three node GKE cluster. This allows us to run a 3-node
Zookeeper ensemble and 3 kafka brokers. There should be one
Zookeeper and one Kafka Broker on each node. We'll modify
the node type to
n1-standard-2 which will be able to
handle everything's memory requirements, etc.
Note that if you spin them up too fast sequentially, kafka
CrashLoopBackOff until it can connect
A healthy cluster:
We can use the following command to interactively choose a kafka container to exec into.
ls to see that we're in.
We'll need to create a new topic (so run this in the
container we just exec'd into). The most interesting piece
of this is that we're pointing to the zookeeper nodes using
the cluster addresses (such as
zk-0.zk-svc.default.svc.cluster.local:2181). This breaks
down into the stateful node identifier (
zk-0), the service
zk-svc), and the network/namespace defaults (as well
as the port).
and run a simple console consumer using the
then exec into the same container again and run the producer so we can send messages to the consumer.
Now we can check out the ISRs and partitions
For Zookeeper we created the following objects:
For Kafka Brokers we created very similar objects:
Both of these look pretty similar. They each use a
Zookeeper also uses a
ConfigMap instead of a file, etc
Service is the first object we see in the
This block says we're creating a
Service with a name of
zk-svc (remember the URLs we had to use to access
Zookeeper earlier). We've removed the
makes this a Headless Service. Since
Zookeeper and Kafka handle their own load balancing, etc,
using a headless service lets us opt out of Kubernetes' load
balancing and service discovery, letting Zookeeper/Kafka
handle it on their own. Also notice that we've exposed the
appropriate ports for Zookeeper leader election and server
Next we define a
ConfigMap, which contains configuration
options for Zookeeper.
Next we create a
PodDisruptionBudget that lets us ensure
that a minimum of 2 nodes will be available at any given
time due to voluntary disruptions like upgrades. It's
important to note that this does not cover nodes crashing on
Ah yes, the central dish to our exploration, the StatefulSet.
We set the affinity for the Zookeeper pod to try to place itself on a node that doesn't already have a Zookeeper node.
The container specification is fairly readable if you're
familiar with running containers. There's the start command,
env vars, and resourcing. We then get to the readiness and
liveness probes, which use a custom shell script to ask the
cluster if it's ok (using
The volume mounts allocate a volume named
10Gi of storage. This is a generic way to "claim"
storage and translates to a
The Kafka yaml file has basically the same components so I won't go over it here.
Congrats, you're running Kafka on GKE. This should be good enough for any testing you'd want to run. In the next post in this series we'll go over how to use the Confluent Platform instead of the containers specified in these yaml files.