Setup Autoscaling For Eventing Kafka Components

This page describes how to set up the Autoscaling feature for Eventing Kafka, which allows for the Eventing Kafka components to autoscale based on the consumer group lag.

This is only currently for the KafkaSource.

OpenShift Serverless autoscaling for Eventing Kafka components is a Developer Preview feature only. Developer Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Developer Preview features, see access.redhat.com/support/offerings/devpreview/.

Prerequisites

  • You have access to an OpenShift account with cluster administrator access.

  • Install the OpenShift CLI (oc).

  • Install the OpenShift Serverless Operator.

  • Install the OpenShift Custom Metrics Autoscaler Operator.

Enable Autoscaling of KafkaSources in KnativeKafka

Enabling autoscaling of KafkaSources enables autoscaling on all KafkaSources by default.
You can disable autoscaling for a specific KafkaSource as described in the Disable Autoscaling for a specific KafkaSource section.

  1. Configure controller-autoscaling-keda feature flag to enabled:

    apiVersion: operator.knative.dev/v1
kind: KnativeKafka
metadata:
  name: knative-kafka
  namespace: knative-eventing
spec:

  # Other spec fields omitted ...
  # ...

  config:
    config-kafka-features:
      controller-autoscaling-keda: enabled (1)
    1 Enables autoscaling for all KafkaSources.

  2. Apply the KnativeKafka resource:

    $ oc apply -f <filename>

Customize Autoscaling for a KafkaSource

If you want to customize how the Custom Metrics Autoscaler scales a KafkaSource, you can set annotations on the source:

apiVersion: sources.knative.dev/v1
kind: KafkaSource
metadata:
  annotations:
    # The minimum number of replicas to scale down to
    autoscaling.eventing.knative.dev/min-scale: "0"
    # The maximum number of replicas to scale up to
    autoscaling.eventing.knative.dev/max-scale: "50"
    # The interval in seconds the autoscaler uses to poll metrics in order to inform its scaling decisions
    autoscaling.eventing.knative.dev/polling-interval: "10"
    # The period in seconds the autoscaler waits until it scales down
    autoscaling.eventing.knative.dev/cooldown-period: "30"
    # The lag that is used for scaling (1<->N)
    autoscaling.eventing.knative.dev/lag-threshold: "100"
    # The lag that is used for activation (0<->1)
    autoscaling.eventing.knative.dev: "0"
spec:
  # KafkaSource spec fields ...

Disable Autoscaling for a specific KafkaSource

If you want to disable autoscaling for a KafkaSource, you need to set an annotation on the source:

apiVersion: source.knative.dev/v1
kind: KafkaSource
metadata:
  annotations:
    autoscaling.eventing.knative.dev/class: disabled
  # more metadata ...
spec:
  # spec fields ...