Excessive http2 error logs for 4318 and 9411 from haproxy ingress

[tman5]

In our otel collector running on Kubernetes, we have a lot of these errors:

2024/05/17 14:50:02 http2: server: error reading preface from client 10.233.71.246:57676: read tcp 10.233.66.81:4318->10.233.71.246:57676: read: connection reset by peer
2024/05/17 19:14:30 http2: server: error reading preface from client 10.233.127.236:49048: read tcp 10.233.66.81:9411->10.233.127.236:49048: read: connection reset by peer

There is an haproxy-ingress controller in front of the collector (that's what the IPs are) and we have SSL enabled on the ingress and on the collector. grpc port does not throw these errors. SSL is enabled in the collector config. The cert is self-signed through cert-manager

Collector version: 0.98.0

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: opentel-collector
spec:
  mode: "deployment"
  volumeMounts:
  - name: cert-files
    mountPath: /etc/pki/ca-trust/source/service-ca
    readOnly: true
  volumes:
  - name: cert-files
    secret:
      secretName: opentel-collector-dev-cert
  ingress:
    type: ingress
    hostname: opentel-collector.local
    ingressClassName: haproxy
    annotations:
      haproxy.org/path-rewrite: /otlp-(?:http|grpc)/(.*) /\1
      haproxy.org/server-ssl: "true"
      haproxy.org/backend-config-snippet: |
        http-check send meth GET
        http-check expect rstatus [234][0-9][0-9]
    tls:
      - secretName: opentel-collector-cert
        hosts:
          - opentel-collector.local
  resources:
    limits:
      cpu: '1'
      memory: 3Gi
    requests:
      cpu: 250m
      memory: 256Mi
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            tls:
              ca_file: "/etc/pki/ca-trust/source/service-ca/ca.crt"
              cert_file: "/etc/pki/ca-trust/source/service-ca/tls.crt"
              key_file: "/etc/pki/ca-trust/source/service-ca/tls.key"
          http:
            endpoint: 0.0.0.0:4318
            tls:
              ca_file: "/etc/pki/ca-trust/source/service-ca/ca.crt"
              cert_file: "/etc/pki/ca-trust/source/service-ca/tls.crt"
              key_file: "/etc/pki/ca-trust/source/service-ca/tls.key"
      zipkin:
            tls:
              ca_file: "/etc/pki/ca-trust/source/service-ca/ca.crt"
              cert_file: "/etc/pki/ca-trust/source/service-ca/tls.crt"
              key_file: "/etc/pki/ca-trust/source/service-ca/tls.key"
...

[atoulme]

This could be because of your haproxy config, is it set to perform healthchecks? See https://access.redhat.com/solutions/6999058 for example. Please post your haproxy configuration?