TIL - Keda and DataDog don’t work together

Today I learned that Keda and DataDog don’t work together. I was trying to implement automatic scaling based on the AWS SQS Queue.

While according to the Scaler Documentation, everything should work, I kept receiving weird error messages.

After some in-depth investigation, I’ve figured out that issue was not related to the IAM permissions. As it turns out:

• KEDA is using API Service v1beta1.external.metrics.k8s.io
• DATADOG is using the same API Service
• You can only have 1 metric server in a cluster
• The limitation is at the k8s level, there is no workaround

With the new EKS cluster w/o DataDog installed, everything worked like a charm.

Update:

• DataDog will register APIService v1beta1.external.metrics.k8s.io only if ClusterAgent is Enabled and clusterAgent.metricsProvider.enabled = true( false by default)
• k8s-cloudwatch-adapter and KEDA will not work together

Corresponding issues:

• KEDA might break existing deployment on cluster which already has another External Metrics Adapter installed
• failing or missing response from https://IP:6443/apis/external.metrics.k8s.io/v1beta1