Automating Stateful Apps with Kubernetes Operators

If you’ve ever had issues with scaling databases or automating upgrades in Kubernetes, Operators can help by saving you time and effort. Handling complex Kubernetes applications like databases, message queues, and distributed systems can be really difficult. Kubernetes handles simple workloads well, while big apps suffer with failover, scaling, backups, and automated updates. These activities often demand operational expertise.

If you’re new to container orchestration, you may want to start by understanding the difference between Kubernetes vs Docker

This is where you need Kubernetes Operators, which we’ll cover in this article. You’ll learn what these Operators are, how they work, and why you need them.

Let’s get to it.

What are Kubernetes Operators?

Operators are extra tools that help you manage complex or stateful applications in Kubernetes. It is a combination of Controllers and Custom Resource Definitions (CRDs).

Controllers are the rules for deploying, setting up, scaling, healing, and updating resources. CRDs let you add a new object type to Kubernetes. CRDs also allow you to describe the type of object you want to manage, after which you proceed to create a Custom Resource (CR), which is an instance of the object type you have made.

The primary purpose of operators is to simplify the management of complex Kubernetes applications, enabling tasks such as upgrades, failover management, backups, and scalability to be performed automatically.

Proper observability is critical in this context. Learn how Kubernetes observability provides deeper insight into your cluster state.

Difference Between Traditional Kubernetes Controllers and Operators

Here’s a breakdown of the difference between the Kubernetes Controller and Operators:

How Kubernetes Operators Work

Assuming you have your app ready, a basic Kubernetes setup, and a working deployment. Without Operators, you’ll have to install and manage the app yourself manually. But with Operators, you can run these processes automatically. Here’s how it works:

1. Set Up a Custom Resource

A Custom Resource (CR) is an object you make to control a particular aspect of your application. You need to declare a CRD before you can create a CR. CRD tells Kubernetes about the new type of object you are making.

Here’s an example of a CRD that defines a CR called “MyAPP”:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myapps.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                version:
                  type: string
  scope: Namespaced
  names:
    plural: myapps
    singular: myapp
    kind: MyApp

This CRD defines a custom resource called “MyApp.” It has two major fields under spec: size and version. 

Once the CRD is applied to the cluster, you can go ahead to create instances of “MyApp,” which represent your app configuration:

apiVersion: example.com/v1
kind: MyApp
metadata:
  name: example-app
spec:
  size: 3
  version: "1.0.0"

This CR declares that you want 3 replicas of example-app (an instance of MyApp) running with version 1.0.0. The Operator will then watch this resource and make sure your app matches what you declared.

2. Deploy the Operator into the Kubernetes Cluster

After defining your CR, the next thing is to deploy the Operator itself. This will monitor CR changes and ensure the cluster status matches your declaration.

You usually use a set of Kubernetes configuration files (YAML files) to notify the cluster to run the Operator as a Pod. This includes setting up permissions so the Operator can watch and manage resources.

Example:

kubectl apply -f operator-deployment.yaml

Once this deployment is running, your cluster will now have a “manager” actively watching for any “MyApp” resources you create.

3. Continuous Monitoring of CR by the Operator.

After deployment, the Operator will constantly monitor changes in the CRs it manages. This monitoring can be done using Kubernete’s API. It watches for events such as creation, updates, or deletions of these CRs.

For practical tips on monitoring logs during troubleshooting, see our guide on kubectl logs tail

The Operator looks out for real-time changes to ensure it can react as soon as something happens. With Kubernetes client, you can set up a watch on your CR like this:

const k8s = require('@kubernetes/client-node');

async function watchCustomResources() {
  const kc = new k8s.KubeConfig();
  kc.loadFromDefault();

  const watch = new k8s.Watch(kc);

  const group = 'example.com';
  const version = 'v1';
  const plural = 'myapps';
  const namespace = 'default';

  watch.watch(
    `/apis/${group}/${version}/namespaces/${namespace}/${plural}`,
    {},
    (type, apiObj) => {
      console.log(`Event: ${type}`, apiObj);
      // Your logic to handle add, update, and delete events can go here
    },
    (err) => {
      console.error('Watch ended with error:', err);
    }
  );
}

watchCustomResources();

This code monitors the “MyApp” CRs in the specified namespace. The Operator will get an event with details when you add, change, or delete a CR. This lets it react fast and keep the app’s state in line with the specified setup.

4. Reconciliation Loop: Operator Compares Desired vs Actual State

The reconciliation loop is the main idea behind Operators. It makes sure that the actual state of your app matches the desired state you defined in the Custom Resource by re-running the whole process if it fails.

Here’s how it works:

• The Operator reads the desired state of your CR.
• Then it checks the actual state of the cluster.
• If they don’t match, it acts to fix it.
• It repeats this process continuously.

This reconciliation logic can be triggered by the watch events we set up earlier.

async function reconcile(customResource) {
  const desiredSize = customResource.spec.size;
  // Fetch the actual state
  const actualSize = await getActualReplicasCount();

  if (actualSize !== desiredSize) {
    console.log(`Reconciling: actual size ${actualSize}, desired size ${desiredSize}`);
    // update the actual state
    await scaleApplication(desiredSize);
  } else {
    console.log('Actual state matches desired state, no action needed.');
  }
}

The reconciliation function guarantees your application works as expected.

5. Error Handling

One way or the other, you’ll still encounter problems when working with Operators. When issues come up, the Operator won’t just stop. They keep trying until they get the correct result.

For example, learn about diagnosing and fixing Exit Code 137 in Kubernetes, a common error when pods are OOMKilled.

Here’s the idea:

• The operator tries to make the actual state equal to the desired state
• It logs an error if it fails
• The reconciliation loop runs again after a while
• The Operator tries again until it works.

Middleware adds to Operators by providing a real-time view into CR status and pod health. It makes sure your stateful apps run smoothly.

Use Cases of Kubernetes Operators

Operators are mostly helpful when working with complex or stateful apps. Here are some practical scenarios where Operators provide great value and how Middleware helps teams get more visibility out of them:

Handling Stateful Apps

Stateful workloads, such as databases, can be challenging to manage because data recovery is crucial. Operators help to manage these workloads even when a container or pod stops. It automatically does backups, restores, scaling, and upgrades. A good example is the Postgres and MySQL Operators.

Postgres Operator allows you to automate backups, failover, and scaling of PostgreSQL clusters. The Operator will automatically configure replicas or restore from snapshots, instead of SREs doing it manually.

With Middleware, teams can track how these Operators work. It can monitor backup success rates, replica health, and storage usage through real-time dashboards and alerts.

Struggling to keep track of your Kubernetes app performance in real time? Learn how Middleware makes it simple by automating monitoring and alerting: Monitoring Kubernetes Applications with Middleware

Messaging Systems

Messaging platforms like Kafka require careful tuning and scaling, particularly under high traffic loads. The Strimzi Kafka Operator simplifies this by provisioning brokers, handling configuration, and managing users. This removes the stress of manually scaling and restarting on the DevOps teams.

Middleware adds more benefits by providing a complete view into Kafka metrics. It will show you the error rates, message throughput, and overall performance.

Monitoring and Logging Stacks

Tools for monitoring and logging, such as Prometheus or ELK, must always be available and scalable to collect data effectively. Prometheus Operators automate upgrades, scaling, and configuration management. This makes sure that the monitoring pipeline is stable without human intervention.

Middleware makes it more interesting by showing the performance and health of both the Operator and the tools it manages. It ensures that your monitoring stack doesn’t disappoint when you need it the most.

Automating Infrastructure

Operators are not limited to apps; they can also automate repetitive infrastructure tasks, such as provisioning storage, configuring network policies, or managing certificates. This helps ensure consistency and security while reducing the risk of errors associated with manual processes.

Again, Middleware displays these invisible tasks, which allows SREs to audit changes, track automation workflows, and set up alerts when infrastructure doesn’t match the expected state.

Why Do We Need Operators?

Kubernetes controllers are mostly applicable when it comes to managing simple apps. But when dealing with complex or stateful apps, they have limitations. These native controllers cannot automate application-specific operations and workflows, making it difficult to manage tasks like database provisioning, upgrades, and failover reliably.

Without Operators, manual database recovery can lead to hours of downtime. Kubernetes Operators automate these tasks, and Middleware real-time alerts ensure you catch issues early.

Challenges Without Operators

Without Operators, you will face these challenges:

• Manual database setup and recovery: You have to create and fix databases yourself, which takes time and can cause mistakes.
• Hard upgrades: Updating apps will require many steps and careful timing to avoid breaking things.
• No app-specific knowledge: Built-in controllers are unaware of your app’s unique rules, so they can’t fully automate it.

Explore common Kubernetes challenges and solutions that teams face without automation.

Kubernetes Operators Benefits

These are some of the benefits of using Operators:

• Automating app lifecycle: Operators automate critical operations, such as installation, backup, upgrade, and failover, tailored to the application’s specific needs.
• Enforcing consistency at scale: Operators ensure that the desired state is consistently maintained across multiple instances or clusters, simplifying large-scale management.
• Reducing human error: Operators minimize manual interventions and reduce human error by applying expert operational knowledge.
• Automate app lifecycle: Operators can reduce failover downtime by recovering stateful apps automatically. This will improve availability and reliability.

Operators automate management, but visibility is key. Discover the top Kubernetes monitoring tools that help you track app performance, spot issues early, and maintain cluster stability effortlessly.

Conclusion

Operators are a powerful addition to Kubernetes because they make it easier to handle stateful and complex tasks, which seem to be difficult for built-in controllers. While Operators make work easier inside Kubernetes, teams still need a way to monitor and make sure their app is working as expected. This is where Middleware helps.

Middleware provides full visibility of your applications, showing their health, performance, and resource usage. It also provides real-time alerts that notify you when an issue occurs, like a sudden increase in memory usage.

FAQs

📚 Also Read:

1. 🤖 What is AIOps?: Discover how AIOps uses AI and machine learning to automate IT operations, helping you manage complex Kubernetes environments smarter.
2. 🧱 GitHub Commit Ownership & Error Tracking: Learn how linking GitHub commit ownership with error tracking helps you quickly identify and fix issues in your Kubernetes applications.
3. ⚡ Reducing Error Rates in a High-Traffic Node.js: See how Application Performance Monitoring (APM) helps reduce error rates and improve the stability of apps running in Kubernetes clusters.
4. 🐞 Debug Logging: Best Practices and Tools: Master the art of debug logging to improve visibility and troubleshoot issues in your stateful Kubernetes applications.