Scaling Kubernetes Workloads on Custom Metrics with Middleware

Kubernetes can automatically scale applications based on CPU and memory utilization. However, these metrics do not always reflect how much work an application needs to process.

A Kafka consumer might have thousands of pending messages while CPU utilization remains at 20%. An API service might experience increasing response times because database connections are saturated. Similarly, an AI inference application may reach its GPU memory limit before CPU utilization increases.

In these situations, scaling based on infrastructure metrics alone can be ineffective.

Custom metrics allow Kubernetes to adjust application replicas using workload-specific signals such as queue depth, requests per second, active connections, or request latency.

With Middleware, teams can use metrics collected from Kubernetes, applications, and integrations as autoscaling signals through a KEDA external scaler. This connects autoscaling decisions with the same observability data used to monitor and troubleshoot applications.

This guide explains how custom-metric autoscaling works, how to configure it using Middleware and KEDA, and how to avoid common scaling problems.

Why CPU and Memory Metrics Are Not Always Enough

Most Kubernetes teams begin with Horizontal Pod Autoscaler (HPA) configured to scale when CPU utilization exceeds a predefined threshold, such as 70%.

This approach works well for compute-intensive applications because CPU usage often increases alongside workload demand.

However, modern applications frequently experience bottlenecks that are not directly reflected in CPU or memory usage.

Consider a payment processing application that consumes messages from RabbitMQ. During a traffic spike, the number of pending transactions increases rapidly. Individual workers may remain at relatively low CPU utilization because they are waiting for database queries or external API responses.

The CPU-based autoscaler may not detect the growing backlog.

Similarly, an API service might exhaust its database connection pool while CPU utilization remains relatively low. Adding replicas based on CPU alone may not improve performance and could increase pressure on the database.

Custom metrics provide more meaningful scaling signals by measuring incoming workload demand or application capacity.

The objective is not to eliminate CPU-based autoscaling. Instead, teams should choose signals that reflect the actual capacity constraints of each application.

How Kubernetes Autoscaling Works with Custom Metrics

Kubernetes Horizontal Pod Autoscaler adjusts the number of application replicas by comparing observed metrics against configured targets.

For a typical metric target, the scaling calculation follows this formula:

\[ \text{Desired Replicas} = \left\lceil \text{Current Replicas} \times \frac{\text{Current Metric Value}}{\text{Target Metric Value}} \right\rceil \]

For example, assume an application currently runs five replicas and its observed per-Pod request rate is twice the configured target.

HPA may recommend approximately ten replicas, subject to scaling policies, tolerance, readiness, and replica limits.

Kubernetes exposes several metrics APIs for autoscaling.

Prometheus Adapter is commonly used when applications already expose metrics through Prometheus.

KEDA provides an alternative for event-driven workloads and supports external metrics integrations, including Middleware.

With KEDA, Kubernetes can scale workloads based on metrics that originate outside the cluster’s default resource metrics pipeline.

For additional information, see the Kubernetes Horizontal Pod Autoscaler documentation.

How Middleware Enables Custom Metrics Autoscaling

Middleware provides an external scaler integration for KEDA that allows Kubernetes workloads to scale using supported metrics available in Middleware.

Instead of building and maintaining a separate custom metrics adapter for Middleware, teams can configure KEDA to query Middleware’s hosted external scaler.

The scaling workflow follows this process:

Application / Kubernetes Workload

                |

                v

       OpenTelemetry / Agent

                |

                v

          Middleware

       Metrics Collection

                |

                v

       Middleware External

             Scaler

                |

                v

              KEDA

                |

                v

      Kubernetes HPA

                |

                v

       Deployment Replicas

Middleware supplies the metric values, KEDA makes them available to HPA, and Kubernetes adjusts the replica count.

Use Different Metrics as Scaling Signals

Middleware’s KEDA integration supports metrics from multiple sources.

Kubernetes metrics: Scale using resource measurements such as Pod CPU utilization.

Application metrics: Use application-specific measurements, including custom counters, gauges, and other metrics sent through supported ingestion methods.

Integration metrics: Use supported measurements from systems such as RabbitMQ, Redis, PostgreSQL, and MySQL.

Span-derived metrics: Use metrics generated from application traces, such as request duration.

Log-derived metrics: Use numerical metrics generated from logs as scaling signals.

Logs and traces themselves are not directly queried by the scaler. Instead, their derived metrics can be used for scaling.

For example, an API workload can scale according to an application latency metric, while background workers can scale based on queue backlog.

The advantage is that teams can use the same metric context for dashboards, alerting, and autoscaling decisions.

Middleware’s official KEDA External Scaler documentation describes the supported configuration and metrics.

Step-by-Step: Scale Kubernetes Workloads Using Middleware and KEDA

Let’s configure a RabbitMQ worker deployment to scale according to the number of pending messages.

For this example, each worker is expected to handle approximately 100 queued messages.

When backlog increases, Kubernetes should add workers. When the queue becomes empty, the workload can scale down to zero.

Step 1: Connect Kubernetes to Middleware

First, ensure your Kubernetes cluster is connected to Middleware.

The Middleware Kubernetes Agent collects infrastructure and cluster telemetry using OpenTelemetry-based components.

Sign in to your Middleware account, open the installation section, and select Kubernetes.

Follow the installation instructions for your environment.

After installation, verify that your Kubernetes workloads appear in Middleware.

You can also use the platform’s application monitoring and integration capabilities to collect the metrics required for autoscaling.

For this example, make sure RabbitMQ telemetry is configured and the queue backlog metric is available in Middleware.

Step 2: Install KEDA

Before installing KEDA, verify your Kubernetes context:

kubectl config current-context

Add the KEDA Helm repository:

helm repo add kedacore https://kedacore.github.io/charts

helm repo update

Install KEDA:

helm install keda kedacore/keda \
  --namespace keda \
  --create-namespace \
  --wait

KEDA installation requires appropriate cluster-level permissions because it creates custom resource definitions, an external metrics API service, and RBAC resources.

Verify the installation:

kubectl get pods -n keda

kubectl get apiservice \
  v1beta1.external.metrics.k8s.io

The external metrics API service should report that it is available.

Step 3: Configure Middleware Authentication

KEDA needs credentials to retrieve metric values from Middleware.

Create a Kubernetes Secret in the same namespace as your worker deployment.

kubectl -n production create secret generic middleware-secret \
  --from-literal=api-key='<YOUR_MIDDLEWARE_API_KEY>'

Replace the placeholder with your Middleware project ingestion token.

For production environments, use your approved secret-management workflow rather than entering credentials directly into shared scripts.

Next, reference the Secret in the worker Deployment.

Add the following environment variable to the application container:

env:
  - name: MW_API_KEY
    valueFrom:
      secretKeyRef:
        name: middleware-secret
        key: api-key

KEDA can retrieve the token through this environment variable using the apiKeyFromEnv configuration.

This avoids placing the API key directly in the ScaledObject.

The Deployment must exist before the ScaledObject is created.

Step 4: Verify the Metric in Middleware

Before creating a scaling policy, confirm that the metric returns valid data.

Open a custom dashboard in Middleware and select the required metric.

For our example:

Metric: queue.messages.ready

Resource: rabbitmq

Aggregation: max

Filter: queue.name:jobs

These fields follow Middleware’s documented KEDA query model.

Ensure the metric is available in your project and that the query identifies the intended queue.

If the query returns multiple series, configure the appropriate series aggregation so the scaler receives a single numerical value.

This validation is important because KEDA cannot make reliable scaling decisions if the metric is unavailable or incorrectly aggregated.

Step 5: Create a KEDA ScaledObject

Create a file named middleware-worker-scaler.yaml.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject

metadata:
  name: worker-backlog-scaler
  namespace: production

spec:
  scaleTargetRef:
    name: worker

  minReplicaCount: 0
  maxReplicaCount: 50

  pollingInterval: 30

  fallback:
    failureThreshold: 3
    replicas: 2

  triggers:
    - type: external
      useCachedMetrics: true
      metricType: AverageValue

      metadata:
        scalerAddress: <MW_UID>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: queue.messages.ready
        resource: rabbitmq
        agg: max
        filters: queue.name:jobs

        window: "120"
        targetValue: "100"
        activationValue: "0"

Replace <MW_UID> with your Middleware account UID.

The target Deployment, named worker, must exist in the production namespace and expose the MW_API_KEY environment variable.

This example follows Middleware’s documented external scaler configuration. Middleware Documentation

Understanding the Scaling Configuration

The configuration uses several important settings.

For example, if the queue contains 500 messages and the target is 100 messages per worker, HPA can recommend approximately five replicas.

The actual scaling decision is subject to Kubernetes and KEDA policies.

Because minReplicaCount is zero, KEDA can deactivate the workload when the queue is empty.

This behavior is particularly useful for event-driven workers that do not need to run continuously.

Step 6: Apply and Verify the Configuration

Apply the configuration:

kubectl apply -f middleware-worker-scaler.yaml

Check the ScaledObject:

kubectl get scaledobject -n production

Inspect its status:

kubectl describe scaledobject worker-backlog-scaler \
  -n production

A healthy ScaledObject should report READY=True.

Next, monitor HPA:

kubectl get hpa -n production -w

To investigate scaling-related errors, examine KEDA operator logs:

kubectl logs -n keda \
  -l app=keda-operator \
  --tail=100

Confirm that the workload adds replicas when the queue backlog increases.

Then observe how replica counts change as workers process pending messages.

Choosing Between Value and AverageValue

Selecting the correct metric type is important because it directly affects scaling calculations.

KEDA supports two metric target types for external scalers.

Suppose a queue contains 500 messages and the target is 100 messages per worker.

With AverageValue, the autoscaler can calculate a desired count of approximately five replicas.

However, consider an API reporting average latency of 500 milliseconds with a target of 250 milliseconds.

If the workload already has ten replicas and the latency metric is an aggregate, using AverageValue would produce an inappropriate per-replica interpretation.

Value allows HPA to compare the aggregate measurement directly with the configured target.

Important: Latency-based scaling should be used carefully. Increased latency may indicate database saturation or another downstream bottleneck that adding replicas cannot resolve.

Monitoring Autoscaling Performance with Middleware

Once autoscaling is enabled, the next step is determining whether additional replicas actually improve performance.

Middleware provides Kubernetes monitoring, application performance monitoring, distributed tracing, and centralized logs to support this analysis.

Monitor Workload Resource Consumption

Track CPU and memory utilization as replicas increase or decrease.

For example, if the worker deployment scales from five to fifteen Pods, monitor whether processing throughput improves and resource pressure remains manageable.

Analyze Application Performance

For APIs and microservices, compare request latency, throughput, and errors before and after scaling.

Middleware’s distributed tracing capabilities can help identify delays in application dependencies.

If latency remains high after additional Pods become available, investigate whether the problem involves database queries, network communication, or another shared resource.

Monitor the Scaling Metric

The metric driving autoscaling should remain visible in a Middleware dashboard.

For queue workers, track backlog alongside processing throughput and worker resource utilization.

This helps engineers determine whether scaling is reducing the queue or merely adding unnecessary replicas.

Investigate Failures with OpsAI

Middleware OpsAI supports AI-assisted investigation of Kubernetes and application incidents.

If scaling activity coincides with Pod failures, memory pressure, or application errors, OpsAI can help investigate the available telemetry and identify likely causes.

This complements Kubernetes autoscaling by connecting scaling behavior with operational troubleshooting.

Common Kubernetes Custom Metrics Autoscaling Problems

Custom-metric autoscaling introduces additional dependencies on metric collection, query availability, and scaling configuration.

For missing metrics, Middleware’s scaler reports an error by default rather than treating missing data as zero. KEDA fallback settings can then define safer replica behavior after repeated failures.

For more information, see the Middleware KEDA troubleshooting documentation.

Best Practices for Kubernetes Custom Metrics Autoscaling

Choose metrics that reflect actual demand. Queue backlog, request rate, and active connections are useful when they represent application capacity accurately.

Avoid unnecessary metric cardinality. High-cardinality attributes such as user IDs and session IDs can increase metric storage and processing overhead. Keep scaling queries focused on stable workload dimensions.

Configure fallback behavior. Missing metrics should not unexpectedly reduce critical workload capacity. Use KEDA fallback settings appropriate to the application’s reliability requirements.

Account for telemetry delays. Metric ingestion and aggregation introduce latency. Ensure polling intervals and query windows reflect the freshness of available data.

Use stabilization policies. Prevent unnecessary replica changes during short traffic fluctuations.

Respect downstream capacity. More replicas can increase pressure on databases, caches, and third-party services. Set maximum replica counts according to tested application and dependency limits.

Monitor scaling outcomes. Evaluate response times, error rates, queue processing time, and infrastructure consumption rather than relying only on replica counts.

Summary

Scaling Kubernetes workloads on custom metrics helps applications respond to actual demand instead of relying entirely on CPU and memory utilization.

Metrics such as queue depth, request rates, active connections, and application-specific measurements provide more relevant scaling signals for many distributed systems.

Middleware extends this approach through its KEDA external scaler, allowing teams to use supported metrics stored in Middleware to drive Kubernetes autoscaling decisions.

KEDA integrates those metrics with HPA, while Middleware provides the dashboards, monitoring, application traces, alerts, and incident investigation capabilities needed to evaluate the results.

Together, these capabilities help engineers build Kubernetes scaling strategies that are more closely aligned with application behavior and operational requirements.

Frequently Asked Questions