Build and deploy Kubernetes operators for automated application management.

Introduction and Setup

When I first started working with Kubernetes, I quickly realized that managing complex applications required more than just deploying pods and services. That’s where operators come in - they’re like having an experienced system administrator encoded in software, continuously managing your applications with domain-specific knowledge.

Understanding Kubernetes Operators

Operators extend Kubernetes by combining Custom Resource Definitions (CRDs) with controllers that understand how to manage specific applications. I’ve seen teams struggle with manual database backups, complex scaling decisions, and application lifecycle management. Operators solve these problems by automating operational tasks that would otherwise require human intervention.

Think of it this way: instead of writing runbooks for your operations team, you encode that knowledge into an operator that runs 24/7 in your cluster. When you need to deploy a PostgreSQL database, the operator knows how to configure storage, set up replication, schedule backups, and handle failover scenarios automatically.

The operator pattern consists of three key components that work together. Custom Resources (CRs) define your application’s desired state using familiar Kubernetes YAML syntax. Custom Resource Definitions (CRDs) act as the schema, defining what fields your custom resources can have and their validation rules. Controllers watch for changes to these resources and take action to maintain the desired state.

Setting Up Your Development Environment

Before we build our first operator, let’s ensure you have the necessary tools installed. I recommend using operator-sdk as it provides scaffolding and best practices out of the box.

# Verify kubectl is working
kubectl version --client

# Check cluster connectivity
kubectl cluster-info

If you don’t have operator-sdk installed, here’s how to get it on macOS:

# Download and install operator-sdk
curl -LO https://github.com/operator-framework/operator-sdk/releases/latest/download/operator-sdk_darwin_amd64
chmod +x operator-sdk_darwin_amd64
sudo mv operator-sdk_darwin_amd64 /usr/local/bin/operator-sdk

Verify everything is working correctly:

operator-sdk version

You should see output showing the operator-sdk version, confirming it’s properly installed.

Creating Your First Custom Resource Definition

Let’s start with a practical example - a WebApp CRD that defines how we want to deploy web applications. This will give you hands-on experience with the concepts before we dive deeper into controller logic.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: webapps.example.com
spec:
  group: example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              replicas:
                type: integer
                minimum: 1
                maximum: 10
              image:
                type: string
              port:
                type: integer
                default: 8080
            required:
            - replicas
            - image
  scope: Namespaced
  names:
    plural: webapps
    singular: webapp
    kind: WebApp

This CRD defines a new Kubernetes resource type called WebApp. The group field creates a namespace for your API (similar to how core Kubernetes resources use different API groups). The schema section defines what fields users can specify - in this case, the number of replicas, container image, and port number.

Notice how we’ve included validation rules like minimum and maximum values for replicas. This prevents users from accidentally creating deployments with zero replicas or overwhelming the cluster with too many instances.

Apply this CRD to your cluster:

kubectl apply -f webapp-crd.yaml
kubectl get crd webapps.example.com

Creating Custom Resource Instances

Now that we’ve defined the structure, let’s create an actual WebApp resource. This demonstrates how users will interact with your operator - through familiar Kubernetes YAML manifests.

apiVersion: example.com/v1
kind: WebApp
metadata:
  name: my-webapp
spec:
  replicas: 3
  image: nginx:1.21
  port: 80

Apply this resource and observe how Kubernetes accepts it:

kubectl apply -f my-webapp.yaml
kubectl get webapps
kubectl describe webapp my-webapp

At this point, you’ll notice that while Kubernetes accepts and stores your WebApp resource, nothing actually happens. That’s because we haven’t built the controller yet - the component that watches for WebApp resources and takes action.

Understanding Controller Basics

Controllers are the brains of the operator pattern. They continuously watch for changes to resources and work to reconcile the actual state with the desired state. Here’s a simplified controller structure that shows the essential pattern:

func (r *WebAppReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
    // Fetch the WebApp resource
    webapp := &WebApp{}
    err := r.Get(ctx, req.NamespacedName, webapp)
    if err != nil {
        return reconcile.Result{}, client.IgnoreNotFound(err)
    }
    
    // Create deployment based on WebApp spec
    deployment := buildDeployment(webapp)
    return reconcile.Result{}, r.Create(ctx, deployment)
}

The Reconcile function is called whenever a WebApp resource changes. It fetches the current resource, determines what Kubernetes objects should exist (like Deployments or Services), and creates or updates them accordingly. The beauty of this pattern is that it’s declarative - you describe what you want, and the controller figures out how to make it happen.

Testing Your Foundation

Let’s verify that your CRD is working correctly by exploring the new API endpoint:

# Check that your new resource type is available
kubectl api-resources | grep webapp

# Create a test instance
kubectl apply -f - <<EOF
apiVersion: example.com/v1
kind: WebApp
metadata:
  name: test-app
spec:
  replicas: 2
  image: nginx:alpine
  port: 80
EOF

You can now manage WebApp resources just like any other Kubernetes resource, using familiar commands like kubectl get, kubectl describe, and kubectl delete.

In Part 2, we’ll build the controller logic that brings these WebApp resources to life by creating actual Deployments and Services. You’ll learn about reconciliation loops, event handling, and how to properly manage the lifecycle of the resources your operator creates.

Core Concepts and Fundamentals

After building your first CRD in Part 1, you might be wondering how the magic actually happens - how does Kubernetes know what to do when you create a WebApp resource? The answer lies in understanding the reconciliation loop, which I consider the heart of any well-designed operator.

The Reconciliation Loop Explained

I’ve worked with many developers who initially think of controllers as event-driven systems, but that’s not quite right. Controllers are level-triggered, not edge-triggered. This means they don’t just react to changes - they continuously ensure the desired state matches reality.

The reconciliation loop follows a simple but powerful pattern. First, it observes the current state of resources in your cluster. Then it compares this with the desired state defined in your custom resources. Finally, it takes action to reconcile any differences. This happens continuously, which means your operator can recover from failures, handle manual changes, and maintain consistency even when things go wrong.

Here’s what a basic reconciliation function looks like in practice:

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    webapp := &examplev1.WebApp{}
    err := r.Get(ctx, req.NamespacedName, webapp)
    if err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }
    
    return r.reconcileWebApp(ctx, webapp)
}

This function gets called whenever something changes - whether that’s a user updating the WebApp resource, a deployment failing, or even when the operator restarts. The beauty is that the same logic handles all these scenarios.

Building a Production-Ready Controller

Let me show you how to structure a controller that can handle real-world complexity. In my experience, the biggest mistake developers make is trying to do everything in the main Reconcile function. Instead, break it down into focused, testable functions.

func (r *WebAppReconciler) reconcileWebApp(ctx context.Context, webapp *examplev1.WebApp) (ctrl.Result, error) {
    // Handle deletion first
    if !webapp.ObjectMeta.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, webapp)
    }
    
    // Ensure deployment exists and is correct
    if err := r.ensureDeployment(ctx, webapp); err != nil {
        return ctrl.Result{}, err
    }
    
    // Ensure service exists
    if err := r.ensureService(ctx, webapp); err != nil {
        return ctrl.Result{}, err
    }
    
    // Update status
    return ctrl.Result{}, r.updateStatus(ctx, webapp)
}

This structure makes it clear what the controller is responsible for and makes each piece independently testable. The ensure pattern is particularly powerful - these functions check if a resource exists and create or update it as needed.

Managing Resource Ownership

One of the trickiest aspects of operator development is managing the lifecycle of resources your operator creates. Kubernetes provides owner references to establish parent-child relationships between resources, which enables automatic garbage collection.

func (r *WebAppReconciler) createDeployment(webapp *examplev1.WebApp) *appsv1.Deployment {
    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      webapp.Name,
            Namespace: webapp.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &webapp.Spec.Replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{"app": webapp.Name},
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{"app": webapp.Name},
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "webapp",
                        Image: webapp.Spec.Image,
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: int32(webapp.Spec.Port),
                        }},
                    }},
                },
            },
        },
    }
    
    // This is the crucial part - establishing ownership
    ctrl.SetControllerReference(webapp, deployment, r.Scheme)
    return deployment
}

The SetControllerReference call creates an owner reference from the deployment back to the WebApp resource. This means when you delete the WebApp, Kubernetes automatically cleans up the deployment. It also prevents other controllers from accidentally managing resources they don’t own.

Handling Complex Lifecycle Events

Sometimes you need more control over cleanup than automatic garbage collection provides. That’s where finalizers come in. I use them when the operator needs to clean up external resources like databases or cloud infrastructure.

func (r *WebAppReconciler) handleDeletion(ctx context.Context, webapp *examplev1.WebApp) (ctrl.Result, error) {
    finalizerName := "webapp.example.com/finalizer"
    
    if controllerutil.ContainsFinalizer(webapp, finalizerName) {
        // Perform cleanup operations
        if err := r.cleanupExternalResources(ctx, webapp); err != nil {
            return ctrl.Result{}, err
        }
        
        // Remove finalizer to allow deletion
        controllerutil.RemoveFinalizer(webapp, finalizerName)
        return ctrl.Result{}, r.Update(ctx, webapp)
    }
    
    return ctrl.Result{}, nil
}

The finalizer pattern ensures that your cleanup code runs before Kubernetes deletes the resource. This is essential when your operator manages resources outside the cluster or needs to perform graceful shutdown procedures.

Status Management and User Feedback

Users need to understand what’s happening with their resources, which is why proper status management is crucial. I always include both high-level phase information and detailed conditions that help with troubleshooting.

func (r *WebAppReconciler) updateStatus(ctx context.Context, webapp *examplev1.WebApp) error {
    // Get current deployment status
    deployment := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{
        Name:      webapp.Name,
        Namespace: webapp.Namespace,
    }, deployment)
    
    if err != nil {
        webapp.Status.Phase = "Pending"
        return r.Status().Update(ctx, webapp)
    }
    
    // Update based on deployment readiness
    if deployment.Status.ReadyReplicas == webapp.Spec.Replicas {
        webapp.Status.Phase = "Running"
        webapp.Status.ReadyReplicas = deployment.Status.ReadyReplicas
    } else {
        webapp.Status.Phase = "Deploying"
    }
    
    return r.Status().Update(ctx, webapp)
}

Notice how we use a separate status update call. Kubernetes treats the status subresource differently from the main resource, which prevents status updates from triggering unnecessary reconciliation loops.

Controller Manager Setup

To tie everything together, you need a controller manager that handles the infrastructure concerns like leader election, metrics, and health checks. Here’s the minimal setup that I use in production:

func main() {
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:             scheme,
        MetricsBindAddress: ":8080",
        Port:               9443,
        LeaderElection:     true,
        LeaderElectionID:   "webapp-operator-lock",
    })
    
    if err := (&WebAppReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller")
        os.Exit(1)
    }
    
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

The leader election feature ensures that only one instance of your operator is actively reconciling resources at a time, which prevents conflicts when running multiple replicas for high availability.

Testing Your Controller Logic

I can’t stress enough how important it is to test your reconciliation logic thoroughly. The controller-runtime framework provides excellent testing utilities that let you test against a fake Kubernetes API server.

func TestWebAppReconciler(t *testing.T) {
    scheme := runtime.NewScheme()
    _ = examplev1.AddToScheme(scheme)
    _ = appsv1.AddToScheme(scheme)
    
    webapp := &examplev1.WebApp{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "test-webapp",
            Namespace: "default",
        },
        Spec: examplev1.WebAppSpec{
            Replicas: 3,
            Image:    "nginx:1.21",
            Port:     80,
        },
    }
    
    client := fake.NewClientBuilder().WithScheme(scheme).WithObjects(webapp).Build()
    reconciler := &WebAppReconciler{Client: client, Scheme: scheme}
    
    _, err := reconciler.Reconcile(context.TODO(), reconcile.Request{
        NamespacedName: types.NamespacedName{Name: "test-webapp", Namespace: "default"},
    })
    
    assert.NoError(t, err)
}

This type of testing catches logic errors early and gives you confidence that your operator will behave correctly in production.

In Part 3, we’ll put these concepts into practice by building operators for real-world scenarios like database management, backup automation, and configuration management. You’ll see how these fundamental patterns scale to handle complex, multi-resource applications.

Practical Applications and Examples

Now that you understand the fundamentals, let’s build operators that solve real problems. I’ve found that the best way to learn operator development is by tackling scenarios you’ll actually encounter in production - database management, configuration handling, and backup automation.

Building a Database Operator

Database operators are among the most valuable because they handle complex lifecycle management that would otherwise require significant manual intervention. Let me walk you through building a PostgreSQL operator that manages not just the database itself, but also users, backups, and monitoring.

The first step is defining what users actually need from a database operator. In my experience, teams want to specify the database version, storage requirements, and backup policies without worrying about StatefulSets, persistent volumes, or backup scripts.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqls.database.example.com
spec:
  group: database.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              version:
                type: string
                enum: ["12", "13", "14", "15"]
              storage:
                type: string
                pattern: '^[0-9]+Gi$'
              replicas:
                type: integer
                minimum: 1
                maximum: 5
              backup:
                type: object
                properties:
                  enabled:
                    type: boolean
                  schedule:
                    type: string
                  retention:
                    type: string
  scope: Namespaced
  names:
    plural: postgresqls
    singular: postgresql
    kind: PostgreSQL

This CRD captures the essential configuration while hiding the complexity of Kubernetes primitives. Notice how we use validation to prevent common mistakes like invalid storage formats or too many replicas.

The controller logic needs to handle the interdependencies between different resources. Databases require careful ordering - you can’t create users before the database is running, and you shouldn’t start backups until the data directory is properly initialized.

func (r *PostgreSQLReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    postgres := &databasev1.PostgreSQL{}
    err := r.Get(ctx, req.NamespacedName, postgres)
    if err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    // Create resources in dependency order
    if err := r.ensureSecret(ctx, postgres); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to create credentials: %w", err)
    }
    
    if err := r.ensureStatefulSet(ctx, postgres); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to create database: %w", err)
    }
    
    if err := r.ensureService(ctx, postgres); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to create service: %w", err)
    }
    
    // Only setup backups after database is running
    if postgres.Spec.Backup.Enabled && r.isDatabaseReady(ctx, postgres) {
        if err := r.ensureBackupCronJob(ctx, postgres); err != nil {
            return ctrl.Result{}, fmt.Errorf("failed to setup backups: %w", err)
        }
    }
    
    return ctrl.Result{RequeueAfter: time.Minute * 5}, r.updateStatus(ctx, postgres)
}

The key insight here is that each ensure function is idempotent - it checks the current state and only makes changes if necessary. This makes the operator resilient to failures and restarts.

Configuration Management Patterns

One of the most common operator use cases I encounter is managing application configuration across different environments. Teams often struggle with keeping configuration in sync between development, staging, and production while maintaining security boundaries.

Let’s build a configuration operator that can template values based on the environment and automatically update applications when configuration changes.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: appconfigs.config.example.com
spec:
  group: config.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              application:
                type: string
              environment:
                type: string
                enum: ["dev", "staging", "prod"]
              config:
                type: object
                additionalProperties:
                  type: string
  scope: Namespaced
  names:
    plural: appconfigs
    singular: appconfig
    kind: AppConfig

The controller for this operator demonstrates an important pattern - detecting changes and triggering updates in dependent resources. When configuration changes, applications need to be restarted to pick up the new values.

func (r *AppConfigReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    appConfig := &configv1.AppConfig{}
    err := r.Get(ctx, req.NamespacedName, appConfig)
    if err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    configMap := &corev1.ConfigMap{
        ObjectMeta: metav1.ObjectMeta{
            Name:      fmt.Sprintf("%s-config", appConfig.Spec.Application),
            Namespace: appConfig.Namespace,
        },
        Data: r.processConfigTemplate(appConfig),
    }
    
    ctrl.SetControllerReference(appConfig, configMap, r.Scheme)
    
    // Check if ConfigMap needs updating
    existing := &corev1.ConfigMap{}
    err = r.Get(ctx, client.ObjectKeyFromObject(configMap), existing)
    if err != nil && errors.IsNotFound(err) {
        return ctrl.Result{}, r.Create(ctx, configMap)
    }
    
    if !reflect.DeepEqual(existing.Data, configMap.Data) {
        existing.Data = configMap.Data
        if err := r.Update(ctx, existing); err != nil {
            return ctrl.Result{}, err
        }
        
        // Trigger rolling update of deployments using this config
        return ctrl.Result{}, r.triggerDeploymentUpdate(ctx, appConfig)
    }
    
    return ctrl.Result{}, nil
}

This pattern of watching for changes and cascading updates is incredibly powerful. It means your applications automatically stay in sync with their configuration without manual intervention.

Backup and Restore Automation

Backup operators solve one of the most critical operational challenges - ensuring data is safely backed up and can be restored when needed. I’ve seen too many teams lose data because backup scripts failed silently or weren’t tested properly.

Here’s how to build a backup operator that handles multiple database types and storage backends:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.backup.example.com
spec:
  group: backup.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              source:
                type: object
                properties:
                  type:
                    type: string
                    enum: ["postgresql", "mysql", "mongodb"]
                  connection:
                    type: object
              destination:
                type: object
                properties:
                  type:
                    type: string
                    enum: ["s3", "gcs", "azure"]
                  bucket:
                    type: string
              schedule:
                type: string
              retention:
                type: string
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    kind: Backup

The backup controller creates CronJobs that run backup scripts on a schedule. The beauty of this approach is that it leverages Kubernetes’ built-in job scheduling while providing a higher-level abstraction for backup management.

func (r *BackupReconciler) ensureCronJob(ctx context.Context, backup *backupv1.Backup) error {
    cronJob := &batchv1.CronJob{
        ObjectMeta: metav1.ObjectMeta{
            Name:      backup.Name + "-cronjob",
            Namespace: backup.Namespace,
        },
        Spec: batchv1.CronJobSpec{
            Schedule: backup.Spec.Schedule,
            JobTemplate: batchv1.JobTemplateSpec{
                Spec: batchv1.JobSpec{
                    Template: corev1.PodTemplateSpec{
                        Spec: corev1.PodSpec{
                            RestartPolicy: corev1.RestartPolicyOnFailure,
                            Containers: []corev1.Container{{
                                Name:    "backup",
                                Image:   r.getBackupImage(backup.Spec.Source.Type),
                                Env:     r.buildBackupEnv(backup),
                                Command: []string{"/backup.sh"},
                            }},
                        },
                    },
                },
            },
        },
    }
    
    ctrl.SetControllerReference(backup, cronJob, r.Scheme)
    return r.Create(ctx, cronJob)
}

What makes this operator particularly useful is that it handles the complexity of different database types and storage backends behind a simple, consistent interface. Users don’t need to remember the specific flags for pg_dump or the AWS CLI syntax - they just specify what they want backed up and where.

Multi-Resource Coordination

Real applications rarely consist of a single component. Most production systems involve databases, caches, message queues, and multiple services that need to be deployed and configured together. This is where operators really shine - they can coordinate complex deployments that would be error-prone to manage manually.

Let me show you how to build an operator that manages a complete application stack:

func (r *AppStackReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    stack := &appv1.AppStack{}
    err := r.Get(ctx, req.NamespacedName, stack)
    if err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    // Deploy components in dependency order
    if err := r.ensureDatabase(ctx, stack); err != nil {
        return ctrl.Result{}, err
    }
    
    if err := r.ensureCache(ctx, stack); err != nil {
        return ctrl.Result{}, err
    }
    
    if err := r.ensureBackend(ctx, stack); err != nil {
        return ctrl.Result{}, err
    }
    
    if err := r.ensureFrontend(ctx, stack); err != nil {
        return ctrl.Result{}, err
    }
    
    return ctrl.Result{}, r.updateStackStatus(ctx, stack)
}

The key challenge in multi-resource coordination is handling dependencies correctly. You can’t start the backend until the database is ready, and you shouldn’t expose the frontend until the backend is healthy. The operator handles these dependencies automatically, waiting for each component to be ready before proceeding to the next.

Testing Your Operators

Testing operators requires a different approach than testing typical applications. You need to verify that your operator correctly manages Kubernetes resources and handles various failure scenarios.

I recommend starting with integration tests that run against a real Kubernetes cluster:

#!/bin/bash
echo "Testing PostgreSQL Operator..."

kubectl apply -f - <<EOF
apiVersion: database.example.com/v1
kind: PostgreSQL
metadata:
  name: test-db
spec:
  version: "14"
  storage: "1Gi"
  replicas: 1
EOF

# Wait for database to be ready
kubectl wait --for=condition=Ready postgresql/test-db --timeout=300s

# Verify the database is accessible
kubectl exec test-db-0 -- psql -U postgres -c "SELECT version();"

echo "Database test passed!"

These tests give you confidence that your operator works correctly in real environments and help catch issues that unit tests might miss.

In Part 4, we’ll dive into advanced operator techniques like admission webhooks, performance optimization, and security considerations. You’ll learn how to build operators that are ready for production use at scale.

Advanced Techniques and Patterns

Building basic operators is one thing, but creating production-ready operators that can handle scale, security, and complex failure scenarios requires advanced techniques. I’ve learned these patterns through years of running operators in production environments where reliability isn’t optional.

Admission Webhooks for Validation and Mutation

One of the most powerful features you can add to your operator is admission webhooks. These allow you to validate or modify resources before they’re stored in etcd. I use them to enforce business rules, set defaults, and prevent configuration mistakes that could cause outages.

Admission webhooks come in two flavors: validating webhooks that can accept or reject resources, and mutating webhooks that can modify resources before they’re stored. The key insight is that webhooks run synchronously during the API request, so they can prevent bad configurations from ever entering your cluster.

Let’s build a validating webhook for our WebApp resources that enforces production-ready policies:

type WebAppValidator struct {
    decoder *admission.Decoder
}

func (v *WebAppValidator) Handle(ctx context.Context, req admission.Request) admission.Response {
    webapp := &examplev1.WebApp{}
    
    err := v.decoder.Decode(req, webapp)
    if err != nil {
        return admission.Errored(http.StatusBadRequest, err)
    }
    
    // Enforce replica limits based on namespace
    if webapp.Namespace == "production" && webapp.Spec.Replicas > 10 {
        return admission.Denied("production workloads cannot exceed 10 replicas")
    }
    
    // Validate image registry
    if !strings.HasPrefix(webapp.Spec.Image, "registry.company.com/") {
        return admission.Denied("images must come from approved registry")
    }
    
    // Require resource limits in production
    if webapp.Namespace == "production" && webapp.Spec.Resources.Limits == nil {
        return admission.Denied("resource limits required in production")
    }
    
    return admission.Allowed("")
}

This webhook prevents common mistakes like using untrusted images or deploying without resource limits in production. The beauty is that these checks happen automatically - developers get immediate feedback when they try to apply invalid configurations.

Mutating webhooks are equally powerful for setting intelligent defaults. Here’s how to automatically configure security contexts and resource limits:

type WebAppMutator struct {
    decoder *admission.Decoder
}

func (m *WebAppMutator) Handle(ctx context.Context, req admission.Request) admission.Response {
    webapp := &examplev1.WebApp{}
    
    err := m.decoder.Decode(req, webapp)
    if err != nil {
        return admission.Errored(http.StatusBadRequest, err)
    }
    
    // Add default labels
    if webapp.Labels == nil {
        webapp.Labels = make(map[string]string)
    }
    webapp.Labels["managed-by"] = "webapp-operator"
    webapp.Labels["version"] = "v1"
    
    // Set security defaults for production
    if webapp.Namespace == "production" {
        webapp.Spec.SecurityContext = &corev1.SecurityContext{
            RunAsNonRoot:             &[]bool{true}[0],
            RunAsUser:                &[]int64{1000}[0],
            AllowPrivilegeEscalation: &[]bool{false}[0],
        }
    }
    
    marshaledWebApp, err := json.Marshal(webapp)
    if err != nil {
        return admission.Errored(http.StatusInternalServerError, err)
    }
    
    return admission.PatchResponseFromRaw(req.Object.Raw, marshaledWebApp)
}

The webhook configuration tells Kubernetes when to call your webhook:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionWebhook
metadata:
  name: webapp-validator
webhooks:
- name: validate.webapp.example.com
  clientConfig:
    service:
      name: webapp-operator-webhook
      namespace: webapp-operator-system
      path: /validate-webapp
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["example.com"]
    apiVersions: ["v1"]
    resources: ["webapps"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

The failurePolicy: Fail setting is crucial - it means that if your webhook is unavailable, resource creation will fail rather than bypassing validation. This prevents security policies from being accidentally circumvented.

Performance Optimization Strategies

As your operator manages more resources, performance becomes critical. I’ve seen operators that work fine with a few resources but become bottlenecks when managing hundreds or thousands of objects. The key is understanding how controller-runtime works and optimizing accordingly.

The most important optimization is controlling what triggers reconciliation. By default, any change to a watched resource triggers reconciliation, but you often only care about specific changes:

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.WebApp{}).
        Owns(&appsv1.Deployment{}).
        WithOptions(controller.Options{
            MaxConcurrentReconciles: 5,
        }).
        WithEventFilter(predicate.Funcs{
            UpdateFunc: func(e event.UpdateEvent) bool {
                oldWebApp := e.ObjectOld.(*examplev1.WebApp)
                newWebApp := e.ObjectNew.(*examplev1.WebApp)
                // Only reconcile on spec changes, not status updates
                return !reflect.DeepEqual(oldWebApp.Spec, newWebApp.Spec)
            },
            GenericFunc: func(e event.GenericEvent) bool {
                return false // Skip generic events
            },
        }).
        Complete(r)
}

This filter prevents unnecessary reconciliation when only the status changes, which can significantly reduce CPU usage in busy clusters. Another critical optimization is using field indexing for efficient lookups. When you need to find resources based on specific fields, indexes can dramatically improve performance:

func (r *WebAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    // Index WebApps by image for efficient lookups
    if err := mgr.GetFieldIndexer().IndexField(context.Background(), &examplev1.WebApp{}, "spec.image", func(rawObj client.Object) []string {
        webapp := rawObj.(*examplev1.WebApp)
        return []string{webapp.Spec.Image}
    }); err != nil {
        return err
    }
    
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.WebApp{}).
        Complete(r)
}

// Now you can efficiently find all WebApps using a specific image
func (r *WebAppReconciler) findWebAppsByImage(ctx context.Context, image string) ([]examplev1.WebApp, error) {
    var webapps examplev1.WebAppList
    err := r.List(ctx, &webapps, client.MatchingFields{"spec.image": image})
    return webapps.Items, err
}

This is particularly useful when you need to implement policies across multiple resources or coordinate updates based on shared characteristics.

Leader Election and High Availability

Production operators need to handle failures gracefully, which means running multiple replicas with leader election. Only one instance should actively reconcile resources at a time, but if that instance fails, another should take over quickly.

The controller-runtime framework makes this straightforward:

func main() {
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:             scheme,
        MetricsBindAddress: ":8080",
        Port:               9443,
        LeaderElection:     true,
        LeaderElectionID:   "webapp-operator-lock",
        LeaseDuration:      &[]time.Duration{15 * time.Second}[0],
        RenewDeadline:      &[]time.Duration{10 * time.Second}[0],
        RetryPeriod:        &[]time.Duration{2 * time.Second}[0],
    })
    
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

The leader election parameters control how quickly failover happens. Shorter lease durations mean faster failover but more network traffic. I typically use 15-second leases for production operators, which provides a good balance between responsiveness and efficiency.

Security and RBAC Best Practices

Security is often an afterthought in operator development, but it should be designed in from the beginning. The principle of least privilege applies - your operator should only have the permissions it absolutely needs.

Here’s a comprehensive RBAC configuration that follows security best practices:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: webapp-operator
  namespace: webapp-operator-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: webapp-operator-role
rules:
# Only the permissions actually needed
- apiGroups: ["example.com"]
  resources: ["webapps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["example.com"]
  resources: ["webapps/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# Events for debugging
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]

Notice how we separate the status permissions and only grant access to the specific resources the operator manages. This prevents the operator from accidentally modifying resources it shouldn’t touch.

For the operator’s own pods, use restrictive security contexts:

func (r *WebAppReconciler) createSecureDeployment(webapp *examplev1.WebApp) *appsv1.Deployment {
    securityContext := &corev1.SecurityContext{
        RunAsNonRoot:             &[]bool{true}[0],
        RunAsUser:                &[]int64{1000}[0],
        AllowPrivilegeEscalation: &[]bool{false}[0],
        ReadOnlyRootFilesystem:   &[]bool{true}[0],
        Capabilities: &corev1.Capabilities{
            Drop: []corev1.Capability{"ALL"},
        },
    }
    
    // Apply to all containers in the deployment
    return deployment
}

These settings prevent privilege escalation attacks and limit the blast radius if the operator is compromised.

Custom Metrics and Observability

Monitoring your operator is crucial for understanding its behavior and diagnosing issues. I always include custom metrics that track both technical performance and business-level outcomes.

Here’s how to add Prometheus metrics to your operator:

var (
    webappReconcileTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "webapp_reconcile_total",
            Help: "Total number of WebApp reconciliations",
        },
        []string{"namespace", "name", "result"},
    )
    
    webappReconcileDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "webapp_reconcile_duration_seconds",
            Help: "Duration of WebApp reconciliations",
            Buckets: []float64{0.1, 0.5, 1.0, 2.5, 5.0, 10.0},
        },
        []string{"namespace", "name"},
    )
)

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    start := time.Now()
    defer func() {
        webappReconcileDuration.WithLabelValues(req.Namespace, req.Name).Observe(time.Since(start).Seconds())
    }()
    
    // ... reconciliation logic ...
    
    webappReconcileTotal.WithLabelValues(req.Namespace, req.Name, "success").Inc()
    return ctrl.Result{}, nil
}

The key is choosing metrics that help you understand both performance (how long reconciliation takes) and correctness (how often it succeeds). I also include business metrics like the number of applications deployed or configuration changes applied.

Error Handling and Circuit Breakers

Robust error handling is what separates production-ready operators from prototypes. You need to handle both transient failures (network timeouts, API server overload) and permanent failures (invalid configurations, missing dependencies) differently.

Here’s a pattern I use for implementing retry logic with exponential backoff:

func (r *WebAppReconciler) reconcileWithRetry(ctx context.Context, webapp *examplev1.WebApp, maxRetries int) error {
    var lastErr error
    
    for attempt := 0; attempt < maxRetries; attempt++ {
        if err := r.reconcileDeployment(ctx, webapp); err != nil {
            lastErr = err
            
            // Check if this is a retryable error
            if !isRetryableError(err) {
                return err // Don't retry permanent failures
            }
            
            // Exponential backoff
            backoff := time.Duration(attempt+1) * time.Second
            time.Sleep(backoff)
            continue
        }
        
        return nil // Success
    }
    
    return fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr)
}

func isRetryableError(err error) bool {
    // Retry on network errors, server errors, but not client errors
    if errors.IsServerTimeout(err) || errors.IsServiceUnavailable(err) {
        return true
    }
    if errors.IsBadRequest(err) || errors.IsNotFound(err) {
        return false
    }
    return true
}

This approach prevents your operator from getting stuck on transient failures while avoiding infinite retry loops on permanent problems.

In Part 5, we’ll focus on production deployment strategies, comprehensive monitoring setups, and the operational practices that keep operators running reliably in production environments. You’ll learn how to deploy operators safely, monitor their health, and troubleshoot issues when they arise.

Best Practices and Optimization

Moving from development to production with operators requires careful attention to deployment strategies, monitoring, and operational practices. I’ve learned these lessons through managing operators in production environments where downtime isn’t acceptable and reliability is paramount.

Production Deployment Strategies

The biggest mistake I see teams make is deploying operators the same way they deploy applications. Operators are infrastructure components that manage other workloads, so they need different deployment patterns and safety measures.

First, let’s talk about packaging. Helm charts provide the flexibility needed for production deployments while maintaining consistency across environments. Here’s how I structure operator Helm charts:

# values.yaml
replicaCount: 2

image:
  repository: webapp-operator
  tag: "1.0.0"
  pullPolicy: IfNotPresent

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi

# Anti-affinity to spread replicas across nodes
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - webapp-operator
        topologyKey: kubernetes.io/hostname

leaderElection:
  enabled: true
  leaseDuration: 15s
  renewDeadline: 10s
  retryPeriod: 2s

webhook:
  enabled: true
  port: 9443
  certManager:
    enabled: true

The anti-affinity rules ensure that operator replicas run on different nodes, preventing a single node failure from taking down your entire operator. Leader election ensures only one instance is active at a time, while the others stand ready to take over.

Health checks are critical for production operators. They need to verify not just that the process is running, but that it can actually perform its core functions:

func setupHealthChecks(mgr manager.Manager) error {
    if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
        return err
    }
    
    // Custom readiness check that verifies API connectivity
    if err := mgr.AddReadyzCheck("readyz", func(req *http.Request) error {
        // Verify we can connect to the Kubernetes API
        if !mgr.GetCache().WaitForCacheSync(req.Context()) {
            return fmt.Errorf("cache not synced")
        }
        
        // Test a simple API call
        var webapps examplev1.WebAppList
        if err := mgr.GetClient().List(req.Context(), &webapps, client.Limit(1)); err != nil {
            return fmt.Errorf("failed to list WebApps: %w", err)
        }
        
        return nil
    }); err != nil {
        return err
    }
    
    return nil
}

This readiness check ensures that Kubernetes won’t route traffic to operator instances that can’t actually process requests, which is crucial during rolling updates or when recovering from failures.

Comprehensive Error Handling

Production operators must handle failures gracefully and provide clear feedback about what went wrong. I use a layered approach to error handling that distinguishes between different types of failures and responds appropriately.

The key insight is that not all errors should be treated the same way. Some errors indicate temporary problems that will resolve themselves, while others require immediate attention or user intervention:

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := r.Log.WithValues("webapp", req.NamespacedName)
    
    webapp := &examplev1.WebApp{}
    if err := r.Get(ctx, req.NamespacedName, webapp); err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }
    
    // Handle deletion with proper cleanup
    if !webapp.ObjectMeta.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, webapp)
    }
    
    // Reconcile with comprehensive error handling
    result, err := r.reconcileWebApp(ctx, webapp)
    if err != nil {
        // Update status with error information
        r.updateStatusWithError(ctx, webapp, err)
        
        // Determine retry strategy based on error type
        if isRetryableError(err) {
            log.Info("Retryable error occurred, requeuing", "error", err)
            return ctrl.Result{RequeueAfter: calculateBackoff(webapp)}, nil
        }
        
        // For permanent errors, don't retry but record the failure
        log.Error(err, "Permanent error occurred")
        r.recordEvent(webapp, "Warning", "ReconcileFailed", err.Error())
        return ctrl.Result{}, nil // Don't return error to avoid infinite retries
    }
    
    return result, nil
}

The calculateBackoff function implements exponential backoff with jitter to prevent thundering herd problems when many resources fail simultaneously:

func calculateBackoff(webapp *examplev1.WebApp) time.Duration {
    // Get failure count from status or annotations
    failureCount := getFailureCount(webapp)
    
    // Exponential backoff: 1s, 2s, 4s, 8s, max 5 minutes
    backoff := time.Duration(1<<failureCount) * time.Second
    if backoff > 5*time.Minute {
        backoff = 5 * time.Minute
    }
    
    // Add jitter to prevent thundering herd
    jitter := time.Duration(rand.Intn(1000)) * time.Millisecond
    return backoff + jitter
}

This approach ensures that temporary failures don’t overwhelm your cluster with retry attempts while still providing timely recovery when conditions improve.

Monitoring and Observability

Effective monitoring is what separates operators that work in demos from those that work in production. You need visibility into both the operator’s own health and the health of the resources it manages. I’ve learned that the key is building monitoring into the operator from the beginning, not adding it as an afterthought.

Start with structured logging that provides context about what the operator is doing and why. Here’s how I implement contextual logging that makes troubleshooting much easier:

type ContextualLogger struct {
    logr.Logger
}

func (l *ContextualLogger) WithWebApp(webapp *examplev1.WebApp) logr.Logger {
    return l.WithValues(
        "webapp.name", webapp.Name,
        "webapp.namespace", webapp.Namespace,
        "webapp.generation", webapp.Generation,
        "webapp.resourceVersion", webapp.ResourceVersion,
    )
}

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := NewContextualLogger("webapp-controller")
    start := time.Now()
    
    webapp := &examplev1.WebApp{}
    if err := r.Get(ctx, req.NamespacedName, webapp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    logger.WithWebApp(webapp).Info("Starting reconciliation",
        "phase", webapp.Status.Phase,
        "replicas.desired", webapp.Spec.Replicas,
        "replicas.ready", webapp.Status.ReadyReplicas,
    )
    
    defer func() {
        duration := time.Since(start)
        logger.WithWebApp(webapp).Info("Reconciliation completed", "duration", duration)
    }()
    
    // Reconciliation logic...
    return ctrl.Result{}, nil
}

This logging approach includes all the context needed to understand what happened during reconciliation, making it much easier to debug issues in production.

For metrics, focus on both technical performance and business outcomes. Here are the key metrics I include in every operator:

var (
    webappReconcileTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "webapp_reconcile_total",
            Help: "Total number of WebApp reconciliations",
        },
        []string{"namespace", "name", "result"},
    )
    
    webappReconcileDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "webapp_reconcile_duration_seconds",
            Help: "Duration of WebApp reconciliations",
            Buckets: []float64{0.1, 0.5, 1.0, 2.5, 5.0, 10.0},
        },
        []string{"namespace", "name"},
    )
    
    webappCreationTime = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "webapp_creation_duration_seconds",
            Help: "Time from WebApp creation to ready state",
            Buckets: []float64{1, 5, 10, 30, 60, 120, 300},
        },
        []string{"namespace"},
    )
)

The creation time metric is particularly valuable because it measures the end-to-end user experience - how long it takes from when someone applies a WebApp resource to when it’s actually serving traffic.

Set up alerting rules that catch problems before they impact users:

groups:
- name: webapp-operator
  rules:
  - alert: WebAppOperatorDown
    expr: up{job="webapp-operator"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "WebApp Operator is down"
      description: "WebApp Operator has been down for more than 5 minutes"
  
  - alert: WebAppReconcileErrors
    expr: rate(webapp_errors_total[5m]) > 0.1
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High WebApp reconcile error rate"
      description: "WebApp reconcile error rate is {{ $value }} errors/sec"

These alerts focus on the operator’s ability to perform its core function rather than just whether the process is running.

Performance Optimization

As your operator manages more resources, performance becomes critical. The most important optimization is reducing unnecessary API calls and reconciliation loops. Here’s how I structure efficient reconciliation:

func (r *WebAppReconciler) reconcileEfficiently(ctx context.Context, webapp *examplev1.WebApp) error {
    // Batch fetch all related resources concurrently
    resources, err := r.getAllRelatedResources(ctx, webapp)
    if err != nil {
        return err
    }
    
    // Calculate desired state once
    desiredDeployment := r.buildDeployment(webapp)
    desiredService := r.buildService(webapp)
    
    // Only update resources that have actually changed
    updates := []client.Object{}
    
    if !r.deploymentMatches(resources.Deployment, desiredDeployment) {
        updates = append(updates, desiredDeployment)
    }
    
    if !r.serviceMatches(resources.Service, desiredService) {
        updates = append(updates, desiredService)
    }
    
    // Batch apply updates
    return r.batchUpdate(ctx, updates)
}

The key insight is to fetch all related resources concurrently, compare them with the desired state, and only make changes when necessary. This dramatically reduces API server load compared to naive approaches that recreate resources on every reconciliation.

Testing Strategies

Testing operators requires a different approach than testing typical applications. You need to verify that your operator correctly manages Kubernetes resources across various scenarios, including failure conditions.

I use a layered testing approach that starts with unit tests for individual functions and builds up to integration tests that run against real Kubernetes clusters:

func TestWebAppController(t *testing.T) {
    scheme := runtime.NewScheme()
    _ = examplev1.AddToScheme(scheme)
    _ = appsv1.AddToScheme(scheme)
    
    webapp := &examplev1.WebApp{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "test-webapp",
            Namespace: "default",
        },
        Spec: examplev1.WebAppSpec{
            Replicas: 3,
            Image:    "nginx:1.21",
            Port:     80,
        },
    }
    
    client := fake.NewClientBuilder().WithScheme(scheme).WithObjects(webapp).Build()
    reconciler := &WebAppReconciler{Client: client, Scheme: scheme}
    
    _, err := reconciler.Reconcile(context.TODO(), reconcile.Request{
        NamespacedName: types.NamespacedName{Name: "test-webapp", Namespace: "default"},
    })
    
    assert.NoError(t, err)
    
    // Verify deployment was created with correct configuration
    deployment := &appsv1.Deployment{}
    err = client.Get(context.TODO(), types.NamespacedName{Name: "test-webapp", Namespace: "default"}, deployment)
    assert.NoError(t, err)
    assert.Equal(t, int32(3), *deployment.Spec.Replicas)
}

For integration testing, I create test scenarios that verify the operator works correctly in real environments:

#!/bin/bash
echo "Running operator integration tests..."

# Deploy the operator
kubectl apply -f config/crd/bases/
kubectl apply -f config/rbac/
kubectl apply -f config/manager/

# Wait for operator to be ready
kubectl wait --for=condition=Available deployment/webapp-operator-controller-manager -n webapp-operator-system --timeout=300s

# Test basic functionality
kubectl apply -f - <<EOF
apiVersion: example.com/v1
kind: WebApp
metadata:
  name: test-webapp
spec:
  replicas: 2
  image: nginx:alpine
  port: 80
EOF

# Verify resources are created
kubectl wait --for=condition=Available deployment/test-webapp --timeout=300s
kubectl get service test-webapp

# Test scaling
kubectl patch webapp test-webapp --type='merge' -p='{"spec":{"replicas":3}}'
kubectl wait --for=jsonpath='{.spec.replicas}'=3 deployment/test-webapp --timeout=300s

# Test deletion
kubectl delete webapp test-webapp
kubectl wait --for=delete deployment/test-webapp --timeout=300s

echo "All integration tests passed!"

These tests give you confidence that your operator works correctly in real environments and help catch issues that unit tests might miss.

In Part 6, we’ll bring everything together by building a complete, production-ready operator that demonstrates all the patterns and practices we’ve covered. You’ll see how to structure a complex operator project, integrate it with CI/CD pipelines, and deploy it safely to production environments.

Real-World Projects and Implementation

After five parts of building up your operator development skills, it’s time to put everything together into a production-ready project. I’ll walk you through building a complete e-commerce platform operator that demonstrates all the patterns we’ve covered while solving real business problems.

Building a Complete E-commerce Platform Operator

The most valuable operators I’ve built manage entire application stacks rather than individual components. Users want to deploy a “shopping platform” or “blog system,” not worry about coordinating databases, caches, message queues, and multiple services. Let’s build an operator that does exactly that.

First, let’s define what our e-commerce platform needs. From my experience working with development teams, they want to specify high-level requirements like “I need a shopping platform for the staging environment” and have the operator figure out all the details.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ecommerceplatforms.platform.example.com
spec:
  group: platform.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              environment:
                type: string
                enum: ["development", "staging", "production"]
              frontend:
                type: object
                properties:
                  replicas:
                    type: integer
                    minimum: 1
                  image:
                    type: string
              backend:
                type: object
                properties:
                  replicas:
                    type: integer
                    minimum: 1
                  image:
                    type: string
              database:
                type: object
                properties:
                  type:
                    type: string
                    enum: ["postgresql", "mysql"]
                  version:
                    type: string
                  storage:
                    type: string
              cache:
                type: object
                properties:
                  enabled:
                    type: boolean
                    default: true
              monitoring:
                type: object
                properties:
                  enabled:
                    type: boolean
                    default: true
  scope: Namespaced
  names:
    plural: ecommerceplatforms
    singular: ecommerceplatform
    kind: EcommercePlatform

The key insight here is that the CRD captures business intent rather than technical implementation details. Users specify what environment they’re targeting and what components they need, and the operator translates that into appropriate Kubernetes resources.

The controller needs to handle complex dependencies between components. You can’t start the backend until the database is ready, and you shouldn’t expose the frontend until the backend is healthy. Here’s how I structure the reconciliation logic:

func (r *EcommercePlatformReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    platform := &platformv1.EcommercePlatform{}
    err := r.Get(ctx, req.NamespacedName, platform)
    if err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    // Update status to show we're working
    platform.Status.Phase = "Deploying"
    r.Status().Update(ctx, platform)
    
    // Deploy components in dependency order
    if err := r.ensureDatabase(ctx, platform); err != nil {
        return r.handleError(ctx, platform, "Database deployment failed", err)
    }
    
    if platform.Spec.Cache.Enabled {
        if err := r.ensureCache(ctx, platform); err != nil {
            return r.handleError(ctx, platform, "Cache deployment failed", err)
        }
    }
    
    // Wait for database to be ready before starting backend
    if !r.isDatabaseReady(ctx, platform) {
        return ctrl.Result{RequeueAfter: time.Second * 30}, nil
    }
    
    if err := r.ensureBackend(ctx, platform); err != nil {
        return r.handleError(ctx, platform, "Backend deployment failed", err)
    }
    
    // Wait for backend to be ready before starting frontend
    if !r.isBackendReady(ctx, platform) {
        return ctrl.Result{RequeueAfter: time.Second * 30}, nil
    }
    
    if err := r.ensureFrontend(ctx, platform); err != nil {
        return r.handleError(ctx, platform, "Frontend deployment failed", err)
    }
    
    // Setup monitoring and ingress last
    if platform.Spec.Monitoring.Enabled {
        if err := r.ensureMonitoring(ctx, platform); err != nil {
            return r.handleError(ctx, platform, "Monitoring setup failed", err)
        }
    }
    
    if err := r.ensureIngress(ctx, platform); err != nil {
        return r.handleError(ctx, platform, "Ingress setup failed", err)
    }
    
    // Update final status
    platform.Status.Phase = "Running"
    platform.Status.Endpoints = r.buildEndpointStatus(platform)
    
    return ctrl.Result{RequeueAfter: time.Minute * 10}, r.Status().Update(ctx, platform)
}

This approach ensures that components are deployed in the right order and that the operator waits for dependencies to be ready before proceeding. The handleError function provides consistent error handling and status updates across all components.

Environment-Specific Configuration

One of the most powerful features of this operator is how it adapts behavior based on the target environment. Production deployments need different security settings, resource limits, and monitoring than development environments.

func (r *EcommercePlatformReconciler) ensureDatabase(ctx context.Context, platform *platformv1.EcommercePlatform) error {
    // Environment-specific configuration
    var replicas int32 = 1
    var storage string = "10Gi"
    var backupEnabled bool = false
    
    switch platform.Spec.Environment {
    case "production":
        replicas = 3
        storage = "100Gi"
        backupEnabled = true
    case "staging":
        replicas = 2
        storage = "50Gi"
        backupEnabled = true
    }
    
    database := &databasev1.PostgreSQL{
        ObjectMeta: metav1.ObjectMeta{
            Name:      platform.Name + "-db",
            Namespace: platform.Namespace,
        },
        Spec: databasev1.PostgreSQLSpec{
            Version:  platform.Spec.Database.Version,
            Replicas: replicas,
            Storage:  storage,
            Backup: databasev1.BackupSpec{
                Enabled:   backupEnabled,
                Schedule:  "0 2 * * *",
                Retention: "30d",
            },
        },
    }
    
    ctrl.SetControllerReference(platform, database, r.Scheme)
    return r.ensureResource(ctx, database)
}

This pattern of environment-specific defaults means that developers can deploy to different environments without having to understand all the operational differences between them. The operator encodes that knowledge and applies it automatically.

CI/CD Integration

Production operators need robust CI/CD pipelines that test thoroughly before deployment. Here’s the GitHub Actions workflow I use for operator projects:

name: Operator CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Go
      uses: actions/setup-go@v3
      with:
        go-version: 1.19
    
    - name: Run unit tests
      run: make test
    
    - name: Run integration tests
      run: |
        kind create cluster
        make test-integration
    
    - name: Security scan
      uses: securecodewarrior/github-action-add-sarif@v1
      with:
        sarif-file: security-scan-results.sarif

  deploy-staging:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
    - uses: actions/checkout@v3
    
    - name: Build and push image
      env:
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build -t ecommerce-operator:$IMAGE_TAG .
        docker push ecommerce-operator:$IMAGE_TAG
    
    - name: Deploy to staging
      run: |
        helm upgrade --install ecommerce-operator ./charts/ecommerce-operator \
          --set image.tag=${{ github.sha }} \
          --namespace ecommerce-operator-system \
          --create-namespace
    
    - name: Run smoke tests
      run: |
        kubectl apply -f config/samples/staging-platform.yaml
        kubectl wait --for=condition=Ready ecommerceplatform/staging-platform --timeout=600s

The key aspects of this pipeline are comprehensive testing (including integration tests against a real Kubernetes cluster), security scanning, and automated deployment to staging with smoke tests to verify everything works.

Production Deployment Example

Here’s how teams actually use the e-commerce platform operator in production:

apiVersion: platform.example.com/v1
kind: EcommercePlatform
metadata:
  name: production-store
  namespace: ecommerce-prod
spec:
  environment: production
  frontend:
    replicas: 5
    image: "registry.company.com/ecommerce-frontend:v2.1.0"
  backend:
    replicas: 3
    image: "registry.company.com/ecommerce-backend:v2.1.0"
  database:
    type: "postgresql"
    version: "14"
    storage: "500Gi"
  cache:
    enabled: true
  monitoring:
    enabled: true

The beauty of this approach is that the same YAML works across environments - the operator automatically applies production-appropriate settings like high availability, backups, monitoring, and security policies.

Deploy it with these commands:

# Deploy the operator
helm install ecommerce-operator ./charts/ecommerce-operator \
  --namespace ecommerce-operator-system \
  --create-namespace \
  --set image.tag=v1.0.0

# Create the platform
kubectl apply -f production-store.yaml

# Monitor the deployment
kubectl get ecommerceplatform production-store -w

# Check all components
kubectl get all -l platform=production-store

# View the operator logs
kubectl logs -l app.kubernetes.io/name=ecommerce-operator -f

Lessons Learned

Building production operators has taught me several important lessons that I wish I’d known when starting out. First, start simple and add complexity gradually. It’s tempting to try to handle every possible scenario from the beginning, but that leads to operators that are hard to understand and maintain.

Second, invest heavily in testing and observability from day one. Operators manage critical infrastructure, and debugging issues in production is much harder than preventing them with good tests and monitoring.

Third, think carefully about your API design. The CRD is the interface that users interact with, and changing it later is difficult. Spend time understanding what users actually need rather than just exposing all the technical knobs.

Finally, remember that operators are long-lived infrastructure components. They need to handle upgrades gracefully, maintain backward compatibility, and provide clear migration paths when breaking changes are necessary.

Summary

This guide has taken you from basic CRD concepts to building production-ready operators that can manage complex application stacks. You’ve learned about reconciliation loops, error handling, security, performance optimization, and operational practices.

The key takeaways are:

  • Operators encode operational knowledge in software
  • Good API design focuses on user intent, not implementation details
  • Comprehensive testing and monitoring are essential for production use
  • Environment-specific behavior makes operators more valuable to development teams
  • CI/CD integration ensures reliable deployments

You now have the knowledge and tools to build operators that solve real problems and provide genuine value to development teams. The patterns and practices in this guide will serve you well as you tackle increasingly complex operational challenges with Kubernetes operators.