Why metric values reset once application restarts and not considering values from TSDB

[manu-balakrishnan]

Hi,

I am exporting application custom metrics to Prometheus TSDB with the help of OTLP exporter and Otel collector. Application is exporting metric values to Prometheus successfully and able to view same value in Prometheus UI. However, when application restarts, metric values are resetting to its default value and start updating based on default value instead of last value which got saved in Prometheus TSDB.

Is there any way to update the metric as per last value that saved in TSDB even after application restarts?

here is my configurations:

docker-compose.yaml:

version: '3.9'

    
services:

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

      
  # OpenTelemetry Collector
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./output:/etc/output:rw

    ports:
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP http receiver
      - "55681:55681"   # OTLP http receiver

    depends_on:
      - jaeger-collector
 
   
  jaeger-collector:
    image: jaegertracing/jaeger-collector
    command: ["--es.num-shards=1", "--es.num-replicas=0", "--es.server-urls=http://elasticsearch:9200", "--collector.zipkin.host-port=:9411"]
    ports:
      - "14269"
      - "14268:14268"
      - "14250"
      - "6831:6831/udp"
      
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - LOG_LEVEL=debug
    restart: on-failure
    depends_on:
      - elasticsearch

    
  jaeger-query:
    image: jaegertracing/jaeger-query
    command: ["--es.num-shards=1", "--es.num-replicas=0", "--es.server-urls=http://elasticsearch:9200"]
    ports:
      - "16686:16686"
      - "16687"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - LOG_LEVEL=debug
    restart: on-failure
    depends_on:
      - elasticsearch


  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3
    environment:
      - discovery.type=single-node
    ports:
    - "9200:9200/tcp"

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:      
      http:
      
exporters:
  jaeger:
    endpoint: jaeger-collector:14250
    insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
  logging:
  # Export to a file
  file:
    path: /etc/output/logs.json

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]
      processors: [batch]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
      processors: [batch]
    logs:
      receivers: [otlp]
      processors: []
      exporters: [logging, file]

Prometheus.yaml

scrape_configs:
- job_name: 'otel-collector'
  scrape_interval: 1s
  static_configs:
  - targets: ['otel-collector:8889']
  - targets: ['otel-collector:8888']

OTLP Exporter (.NET):

builder.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(this.Configuration.GetValue<string>("Otlp:ServiceName")))
            .AddOtlpExporter((otlpOptions, readOption) =>
             {
                 otlpOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
                 otlpOptions.Endpoint = new Uri(this.Configuration.GetValue<string>("Otlp:Endpoint")
                     + "/v1/metrics"
                     );
             });
        break;

[bogdandrutu]

Can you add some "value" examples of what the application emits, and what you expect to see?

[manu-balakrishnan]

Hi @bogdandrutu , Please find the below example. Here, we are maintaining a counter metric for calculating number of file ingested which will get incremented by 1 as per file ingestion. suppose, after ingesting 4 files, application got restarted. When we ingest next 3 files, expected count was 7 but it is showing 3 because it restart the counter from 0. Is there any option to update as per last saved value.

Time	Application running status	No. of File Ingested	Value that application emits	Expected Counter Value
10:00	YES	1	1	1
10:01	YES	0	1	1
10:02	YES	2	3	3
10:03	YES	1	4	4
10:04	YES	0	4	4
10:05	YES	0	4	4
10:06	NO	0
10:07	NO	0
10:08	YES	0
10:09	YES	1	1	5
10:10	YES	2	3	7

[xyc121381]

so how to solve this problem pls pls

[xyc121381]

I also want to know what to do.

[ishdutt]

Was anyone able to resolve this problem?