Bug: empty exemplar labels results in invalid metrics format

[pcarranza]

github.com/prometheus/client_golang v1.16.0

Hi 👋

I am using the latest golang client to expose metrics using exemplars, and I have encountered a case in which the client tolerates providing an empty labels map when observing with exemplars.

This results in prometheus failing to scrape because the metrics format is invalid.

A code sample that generates an invalid metric would look like this:

package main

import (
	"log"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	t := time.Now()

	failedHistogram := promauto.NewHistogramVec(prometheus.HistogramOpts{
		Namespace: "prometheus",
		Subsystem: "poc",
		Name:      "test",
		Buckets:   prometheus.DefBuckets,
		Help:      "my help",
	}, []string{"result"})

	observer := failedHistogram.WithLabelValues("failed")

	withExemplar, ok := observer.(prometheus.ExemplarObserver)
	if !ok {
		panic("could not use observer as an exemplar observer")
	}

	// The following line would yield a valid scrape 
	// withExemplar.ObserveWithExemplar(time.Since(t).Seconds(), prometheus.Labels{"traceID": "1"})

	// The following line is allowed by the client, yet renders an invalid metrics page
	withExemplar.ObserveWithExemplar(time.Since(t).Seconds(), prometheus.Labels{})

	log.Println("Listening on port 8000")
	http.ListenAndServe(":8000", promhttp.InstrumentMetricHandler(
		prometheus.DefaultRegisterer,
		promhttp.HandlerFor(prometheus.DefaultGatherer,
			promhttp.HandlerOpts{
				// Necessary to enable exemplars
				EnableOpenMetrics: true,
			}),
	))
}

On scrape I get an error on the target: expected timestamp or # symbol, got "INVALID" which hasn't improved a lot upgrading prometheus to the latest (2.46.0) as it only adds the metric name where it's failing.

Upon further testing scraping with curl (curl -H 'Accept: application/openmetrics-text' localhost:8000), I cornered the problem to it being that ObserveWithExemplar tolerates receiving an empty prometheus.Labels{} which in turn fails to render, examples:

Failed scrape (mind the double space after #):

prometheus_poc_test_bucket{result="failed",le="0.005"} 1 #  5.8e-05 1.692205065685987e+09

or when there is a label, successful scrape:

prometheus_poc_test_bucket{result="failed",le="0.005"} 1 # {traceID="1"} 4.4125e-05 1.692206618903436e+09

I think that either ObserveWithExemplar should blow up when there's an empty label, or this should be ignored silently.

BTW, Go mod looks like this:

module github.com/pcarranza/prometheus_poc

go 1.20

require github.com/prometheus/client_golang v1.16.0

require (
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
	github.com/prometheus/client_model v0.3.0 // indirect
	github.com/prometheus/common v0.42.0 // indirect
	github.com/prometheus/procfs v0.10.1 // indirect
	golang.org/x/sys v0.8.0 // indirect
	google.golang.org/protobuf v1.30.0 // indirect
)

[pulord]

Hi @pcarranza ,

I have a issue, need your help.

when I use prometheus exemplar to expose traceid, I use many demo that include your demo. But I can't get any exemplar metrics when I use Curl http://locahost:8000/metrics.

Do you Known How to fix this problem？

I use Golang 1.20 and go.mod same to you

`
go 1.20

require github.com/prometheus/client_golang v1.16.0

require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.42.0 // indirect
github.com/prometheus/procfs v0.10.1 // indirect
golang.org/x/sys v0.8.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
)
`

[pulord]

I found my fault!!!
must use special Header when use curl

curl -H 'Accept: application/openmetrics-text' localhost:8000/metrics

[pulord]

another question：

Prometheus how to scrape The target that can expose Exemplars data。

• I had enable "--enable-feature=exemplar-storage" this flag in Prometheus
• But Prometheus can't set The header “Accept: application/openmetrics-text” when scrape。

[pulord]

another question：

Prometheus how to scrape The target that can expose Exemplars data。

• I had enable "--enable-feature=exemplar-storage" this flag in Prometheus
• But Prometheus can't set The header “Accept: application/openmetrics-text” when scrape。

Fixed，Prometheus Add default header when scrape target

[orls]

I ran into the same issue; I've got a PR to the common repo to fix the exposition rendering.