[Bug]: counters are always reset

[aytekinar]

What happened?

I have the following dependencies in my Rust application (a playground project in which I am testing opentelemetry-rust components):

[package]
name = "telemetry"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
once_cell = "1.18.0"
opentelemetry = { version = "0.21.0", features = ["metrics", "trace"] }
opentelemetry-otlp = { version = "0.14.0", features = ["metrics", "trace", "grpc-tonic"] }
opentelemetry_sdk = { version = "0.21.0", features = ["metrics", "trace", "rt-tokio"] }
tokio = { version = "1.34.0", features = ["rt-multi-thread", "macros"] }

My main.rs file reads as follows:

use std::{
    sync::{Arc, Mutex},
    thread,
};

use once_cell::sync::Lazy;
use opentelemetry::{
    global,
    metrics::Counter,
    trace::{self, Span, Tracer},
    KeyValue,
};
use opentelemetry_sdk::{propagation::TraceContextPropagator, runtime};

static CALL_ME_COUNTER: Lazy<Mutex<Arc<Counter<u64>>>> = Lazy::new(|| {
    println!("Initializing call_me counter");
    Mutex::new(Arc::new(
        global::meter("telemetry")
            .u64_counter("call_me")
            .with_description("counts how many times call_me is called")
            .init(),
    ))
});

fn call_me() {
    let mut span = global::tracer("telemetry").start("call_me");
    let counter = CALL_ME_COUNTER.lock().unwrap().clone();
    counter.add(1, &[KeyValue::new("key", "value")]);
    span.add_event("enter", Vec::new());
    thread::sleep(std::time::Duration::from_millis(500));
    span.add_event("exit", Vec::new());
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    global::set_text_map_propagator(TraceContextPropagator::new());

    let _tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(opentelemetry_otlp::new_exporter().tonic())
        .install_batch(runtime::Tokio)?;

    let _meter = opentelemetry_otlp::new_pipeline()
        .metrics(opentelemetry_sdk::runtime::Tokio)
        .with_exporter(opentelemetry_otlp::new_exporter().tonic())
        .build();

    {
        let mut span = global::tracer("telemetry").start("main");
        span.add_event("enter", Vec::new());
        let _guard = trace::mark_span_as_active(span);
        for _ in 0..10 {
            call_me();
        }
        trace::get_active_span(|span| {
            span.add_event("exit", Vec::new());
        });
    }

    global::shutdown_meter_provider();
    global::shutdown_tracer_provider();

    Ok(())
}

When I set the environment variable as OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 and create a Docker Compose environment with Jaeger as the tracing backend and Prometheus as the metrics backend, I receive the following:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/telemetry`
Initializing call_me counter
$ curl otel-collector:8080/metrics
# HELP call_me_total counts how many times call_me is called
# TYPE call_me_total counter
call_me_total{job="myservice",key="value"} 1

I would have expected to see a value of 10 for call_me_total. Am I mising something?

As a side note, I have also tried different combinations of Mutex<Counter<u64>> and Arc<Mutex<Counter<u64>>>, as well as omitting the static variable altogether, and instead, defining let counter = global::meter("telemetry").u64_counter("call_me").init();, all without success.

API Version

0.21.0

SDK Version

0.21.0

What Exporters are you seeing the problem on?

OTLP

Relevant log output

No response

[jtescher]

@aytekinar this is actually due to the global meter provider shutdown hook not behaving correctly, you can fix this by calling shutdown on the meter provider directly. The global shutdown is not spec compliant and is being removed in #1412.