Proposed list of Metrics to Stabilize

[rcoh]

Is your feature request related to a problem? Please describe.
Given the impact, hassle, and perceived "risk" of compiling with tokio_unstable, I'd like to propose we stabilize some of the existing metrics.

Describe the solution you'd like

1. Runtime::metrics is stabilized. Documentation is added to this method (currently missing)
2. RuntimeMetrics is stabilized
3. We stabilize individual metrics on a case by case base.
4. Blog post or other piece of high quality source, long-form material a la https://tokio.rs/tokio/topics/shutdown explaining best practices for alarming, monitoring, and using the metrics published by Tokio.

Based on existing usage I've identified, propose the following metrics for stabilization. I've selected metrics that could plausibly include an actual alarm threshold.

Proposed Metrics for Stabilization

• num_workers: Used for asserting that the runtime is configured as expected
• Note: do not stabilize until merge of feat: add task counter pairs #6114, stabilize under num_active_tasks active_tasks_count: Used for ensuring that runtime is behaving as expected (e.g. no accidental spawn leakages). Suggested alarms: high-water mark, 0.
• injection_queue_depth: Used for ensuring that the runtime is making forwards progress & not in a pathological state. Note: this metric would be more useful with either a total counter or some concept of epoch/duration. Suggested alarm: high-water mark
• worker_local_queue_depth: Similar to injection_queue_depth, would also be more useful with a total insertion count.
• worker_total_busy_duration: Can be used to determine overall load of the worker. An high ratio of busy duration to total time suggests that the worker is performing a lot of CPU bound work. Suggested alarm: in combination with total time & poll count, high CPU usage per poll.
• worker_poll_count: Can be combined with busy_duration to estimate time per poll.
• worker_overflow_count: General health metric for a worker. If rapidly increasing, indicates that a worker is falling behind. Alarms: increasing at high rate.

Proposed longer term work:

• I recommend we stabilize queue metrics as-is and add injection_queue_metrics() -> QueueMetrics { ... } for queues in the future.
• In usage, I observe multiple people only considering worker metrics for the 0th worker. I would recommend stabilizing an iterator version of these APIs to encourage customers to actually report metrics from all workers, e.g. workers_overflow_count(&self) -> impl Iterator<Item=(usize, usize)>
• Creation of a 0.x tokio-runtime-monitor crate that takes an opinionated stats of metrics to report and includes alarms. Perhaps this crate could publish directly to metrics.rs? This crate would compile on stable Tokio.

Appendix: All Metrics

• num_workers
• active_tasks_count
• blocking_queue_depth
• budget_forced_yield_count
• injection_queue_depth
• io_driver_fd_deregistered_count
• io_driver_fd_registered_count
• io_driver_ready_count
• num_blocking_threads
• num_idle_blocking_threads
• poll_count_histogram_bucket_count
• poll_count_histogram_bucket_range
• poll_count_histogram_enabled
• poll_count_histogram_num_buckets
• remote_schedule_count
• worker_local_queue_depth
• worker_local_schedule_count
• worker_mean_poll_time
• worker_noop_count
• worker_overflow_count
• worker_park_count
• worker_poll_count
• worker_steal_count
• worker_steal_operations
• worker_total_busy_duration

Additional context
#4073

[Darksonn]

Please see #6114, which renames some metrics.

[rcoh]

👍🏻 , it renames active_tasks_count to num_active_tasks. I called that out in the ticket above to delay stabilization of that metric until the CR lands

[Darksonn]

As a start, do you want to submit a PR that stabilizes just the overall metrics interface and num_workers?

[dswij]

We'd love to see this stabilized, especially these metrics that are the most important for us:

1. num_workers
2. active_tasks_count
3. worker_total_busy_duration