Otel collector deployment with multiple pods result in duplicate metrics

[mircea-lemnaru-aera]

Hi All,

I have this current setup:

Otel operator which deploys the Otel Collector as a K8s Deployment with 2 pods in my case.
In the same namespace I have a bunch of services configured with auto-instrumentation which push telemetry data to the otel collector service.

In the same namespace I have a PodMonitor which is used by Prometheus to discover the Otel Pods and scrape from them.
Both pods appear in Prometheus as targets , all seems to be good.

The problem that I am facing is that the metrics are duplicated. For the same service I have the same metric , each one comming from one otel-collector instance.

For example:

process_runtime_jvm_classes_loaded{app_kubernetes_io_component="opentelemetry-collector", app_kubernetes_io_instance="platform.otel", app_kubernetes_io_managed_by="opentelemetry-operator", app_kubernetes_io_name="otel-collector", app_kubernetes_io_part_of="opentelemetry", app_kubernetes_io_version="latest", container="otc-container", container_id="a8616d3c490b842c66601c64b2bfead4e84de8277c408c69c7c4efd30539142d", endpoint="prometheus", exported_job="cwb-be", host_arch="amd64", host_name="cwb-be-685c64f54c-zlzkb", instance="10.43.151.208:8889", job="platform/otel-collector-pod-monitor", k8s_container_name="cwb-be", k8s_deployment_name="cwb-be", k8s_namespace_name="platform", k8s_node_name="aks-aerasvc1-95583755-vmss000000", k8s_pod_name="cwb-be-685c64f54c-zlzkb", k8s_replicaset_name="cwb-be-685c64f54c", namespace="platform", os_description="Linux 5.15.0-1051-azure", os_type="linux", pod="otel-collector-5dccb6749f-x6xkj", pod_template_hash="5dccb6749f", process_pid="9", process_runtime_description="Amazon.com Inc. OpenJDK 64-Bit Server VM 25.372-b07", process_runtime_name="OpenJDK Runtime Environment", process_runtime_version="1.8.0_372-b07", service_name="cwb-be", service_version="2.7.0-main-b153", telemetry_auto_version="1.32.1", telemetry_sdk_language="java", telemetry_sdk_name="opentelemetry", telemetry_sdk_version="1.34.1"}
25691

And the duplicate one , if you look closely you can see that the only difference is the collector that provided this metric

process_runtime_jvm_classes_loaded{app_kubernetes_io_component="opentelemetry-collector", app_kubernetes_io_instance="platform.otel", app_kubernetes_io_managed_by="opentelemetry-operator", app_kubernetes_io_name="otel-collector", app_kubernetes_io_part_of="opentelemetry", app_kubernetes_io_version="latest", container="otc-container", container_id="a8616d3c490b842c66601c64b2bfead4e84de8277c408c69c7c4efd30539142d", endpoint="prometheus", exported_job="cwb-be", host_arch="amd64", host_name="cwb-be-685c64f54c-zlzkb", instance="10.43.157.118:8889", job="platform/otel-collector-pod-monitor", k8s_container_name="cwb-be", k8s_deployment_name="cwb-be", k8s_namespace_name="platform", k8s_node_name="aks-aerasvc1-95583755-vmss000000", k8s_pod_name="cwb-be-685c64f54c-zlzkb", k8s_replicaset_name="cwb-be-685c64f54c", namespace="platform", os_description="Linux 5.15.0-1051-azure", os_type="linux", pod="otel-collector-5dccb6749f-7jhfg", pod_template_hash="5dccb6749f", process_pid="9", process_runtime_description="Amazon.com Inc. OpenJDK 64-Bit Server VM 25.372-b07", process_runtime_name="OpenJDK Runtime Environment", process_runtime_version="1.8.0_372-b07", service_name="cwb-be", service_version="2.7.0-main-b153", telemetry_auto_version="1.32.1", telemetry_sdk_language="java", telemetry_sdk_name="opentelemetry", telemetry_sdk_version="1.34.1"}
25691

In the first one the collector is: otel-collector-5dccb6749f-x6xkj and in the second one otel-collector-5dccb6749f-7jhfg. And the metric value is exactly the same. It seems like this metric is pushed to both collectors for some reson and then prometheus collects them both.

Am I doing something wrong with deploying 2 collector instances ? And is there a way to remove this duplication ?
What do you suggest I do ?

PS: I can remove this at prometheus or grafana level , but I fear that I am doing something wrong.

Thanks