Export memory usage builder and MemoryUsage structs to users #854
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 15, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/AhmedSoliman/rust-rocksdb/tree/restate. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 15, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/AhmedSoliman/rust-rocksdb/tree/restate. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 15, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/AhmedSoliman/rust-rocksdb/tree/restate. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 15, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 15, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 15, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 16, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 16, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 16, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 16, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 17, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Jan 17, 2024
This exports key tickers, histograms, and properties from rocksdb through the /metrics endpoint. The exported metrics are reported in prometheus exposition format. Additionally, another http endpoint `/rocksdb-stats` that returns the raw rocksdb statistics output (can be useful in performance investigations). Note that rocksdb metric reporting is only triggered if the scraping endpoint is queried. The change requires changes in rust-rocksdb to support pulling individual tickers and histograms out of rocksdb. The change is proposed in rust-rocksdb/rust-rocksdb#853 Additionally, the rust-rocksdb interface doesn't make it easy to get memory usage information when we use optimistic transactional database (which we do), for that, another change in rust-rocksdb/rust-rocksdb#854 is needed to get access to memory usage information. Both changes are merged into a `restate` branch at https://github.com/restatedev/rust-rocksdb/tree/next. I updated Cargo to use that until changes are merged upstream. Test Plan: ``` > http localhost:5122/metrics | head -n 10 # TYPE invoker_invocation_task_started_total counter invoker_invocation_task_started_total{rpc_service="CheckoutProcess"} 12 # TYPE invoker_invocation_task_failed_total counter invoker_invocation_task_failed_total{rpc_service="CheckoutProcess",transient="true"} 12 # TYPE invoker_inflight_invocations_total gauge invoker_inflight_invocations_total{rpc_service="CheckoutProcess"} 0 # TYPE rocksdb_memtable_miss_total counter rocksdb_memtable_miss_total 2066 # TYPE rocksdb_bytes_read_total counter rocksdb_bytes_read_total 1736 # TYPE rocksdb_bytes_written_total counter rocksdb_bytes_written_total 0 # TYPE rocksdb_db_get_seconds summary rocksdb_db_get_seconds{quantile="0.5"} 0.000006297482837528604 rocksdb_db_get_seconds{quantile="0.95"} 0.000019823636363636356 rocksdb_db_get_seconds{quantile="0.99"} 0.00012004999999999938 rocksdb_db_get_seconds{quantile="1.0"} 0.000667 rocksdb_db_get_seconds_sum 0.022774 rocksdb_db_get_seconds_count 2066 # TYPE rocksdb_db_write_seconds summary rocksdb_db_write_seconds{quantile="0.5"} 0 rocksdb_db_write_seconds{quantile="0.95"} 0 rocksdb_db_write_seconds{quantile="0.99"} 0 rocksdb_db_write_seconds{quantile="1.0"} 0 rocksdb_db_write_seconds_sum 0 rocksdb_db_write_seconds_count 0 # TYPE rocksdb_db_seek_seconds summary rocksdb_db_seek_seconds{quantile="0.5"} 0.000007813397129186603 rocksdb_db_seek_seconds{quantile="0.95"} 0.00002298974358974355 rocksdb_db_seek_seconds{quantile="0.99"} 0.000049989444444444286 rocksdb_db_seek_seconds{quantile="1.0"} 0.000573 rocksdb_db_seek_seconds_sum 0.032024 rocksdb_db_seek_seconds_count 3107 # TYPE rocksdb_db_multiget_seconds summary rocksdb_db_multiget_seconds{quantile="0.5"} 0 rocksdb_db_multiget_seconds{quantile="0.95"} 0 rocksdb_db_multiget_seconds{quantile="0.99"} 0 rocksdb_db_multiget_seconds{quantile="1.0"} 0 rocksdb_db_multiget_seconds_sum 0 rocksdb_db_multiget_seconds_count 0 # TYPE rocksdb_bytes_per_write_bytes summary rocksdb_bytes_per_write_bytes{quantile="0.5"} 0 rocksdb_bytes_per_write_bytes{quantile="0.95"} 0 rocksdb_bytes_per_write_bytes{quantile="0.99"} 0 rocksdb_bytes_per_write_bytes{quantile="1.0"} 0 rocksdb_bytes_per_write_bytes_sum 0 rocksdb_bytes_per_write_bytes_count 0 # TYPE rocksdb_bytes_per_read_bytes summary rocksdb_bytes_per_read_bytes{quantile="0.5"} 0.5048899755501223 rocksdb_bytes_per_read_bytes{quantile="0.95"} 0.9592909535452323 rocksdb_bytes_per_read_bytes{quantile="0.99"} 0.999682151589242 rocksdb_bytes_per_read_bytes{quantile="1.0"} 106 rocksdb_bytes_per_read_bytes_sum 1736 rocksdb_bytes_per_read_bytes_count 2065 # TYPE rocksdb_num_immutable_mem_table_count gauge rocksdb_num_immutable_mem_table_count 0 # TYPE rocksdb_mem_table_flush_pending_count gauge rocksdb_mem_table_flush_pending_count 0 # TYPE rocksdb_compaction_pending_count gauge rocksdb_compaction_pending_count 0 # TYPE rocksdb_background_errors_count gauge rocksdb_background_errors_count 0 # TYPE rocksdb_cur_size_active_mem_table_bytes gauge rocksdb_cur_size_active_mem_table_bytes 2048 # TYPE rocksdb_cur_size_all_mem_tables_bytes gauge rocksdb_cur_size_all_mem_tables_bytes 2048 # TYPE rocksdb_size_all_mem_tables_bytes gauge rocksdb_size_all_mem_tables_bytes 2048 # TYPE rocksdb_num_entries_active_mem_table_count gauge rocksdb_num_entries_active_mem_table_count 0 # TYPE rocksdb_num_entries_imm_mem_tables_count gauge rocksdb_num_entries_imm_mem_tables_count 0 # TYPE rocksdb_num_deletes_active_mem_table_count gauge rocksdb_num_deletes_active_mem_table_count 0 # TYPE rocksdb_num_deletes_imm_mem_tables_count gauge rocksdb_num_deletes_imm_mem_tables_count 0 # TYPE rocksdb_estimate_num_keys_count gauge rocksdb_estimate_num_keys_count 0 # TYPE rocksdb_estimate_table_readers_mem_bytes gauge rocksdb_estimate_table_readers_mem_bytes 0 # TYPE rocksdb_num_live_versions_count gauge rocksdb_num_live_versions_count 1 # TYPE rocksdb_estimate_live_data_size_bytes gauge rocksdb_estimate_live_data_size_bytes 0 # TYPE rocksdb_min_log_number_to_keep_count gauge rocksdb_min_log_number_to_keep_count 240 # TYPE rocksdb_live_sst_files_size_bytes gauge rocksdb_live_sst_files_size_bytes 0 # TYPE rocksdb_estimate_pending_compaction_bytes_bytes gauge rocksdb_estimate_pending_compaction_bytes_bytes 0 # TYPE rocksdb_num_running_compactions_count gauge rocksdb_num_running_compactions_count 0 # TYPE rocksdb_actual_delayed_write_rate_count gauge rocksdb_actual_delayed_write_rate_count 0 # TYPE rocksdb_block_cache_capacity_bytes gauge rocksdb_block_cache_capacity_bytes 33554432 # TYPE rocksdb_block_cache_usage_bytes gauge rocksdb_block_cache_usage_bytes 87 # TYPE rocksdb_block_cache_pinned_usage_bytes gauge rocksdb_block_cache_pinned_usage_bytes 87 # TYPE rocksdb_num_files_at_level0_count gauge rocksdb_num_files_at_level0_count 0 # TYPE rocksdb_num_files_at_level1_count gauge rocksdb_num_files_at_level1_count 0 # TYPE rocksdb_num_files_at_level2_count gauge rocksdb_num_files_at_level2_count 0 # TYPE rocksdb_memory_approximate_cache_bytes gauge rocksdb_memory_approximate_cache_bytes 87 # TYPE rocksdb_memory_approx_memtable_bytes gauge rocksdb_memory_approx_memtable_bytes 18432 # TYPE rocksdb_memory_approx_memtable_unflushed_bytes gauge rocksdb_memory_approx_memtable_unflushed_bytes 18432 # TYPE rocksdb_memory_approx_memtable_readers_bytes gauge rocksdb_memory_approx_memtable_readers_bytes 8863 ```
aleksuss
approved these changes
Feb 13, 2024
9 tasks
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Feb 16, 2024
Rocksdb 0.22 have been released and it includes the statistics changes from rust-rocksdb/rust-rocksdb#853 and rust-rocksdb/rust-rocksdb#854 we can now move back to upstream.
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Feb 16, 2024
Rocksdb 0.22 have been released and it includes the statistics changes from rust-rocksdb/rust-rocksdb#853 and rust-rocksdb/rust-rocksdb#854 we can now move back to upstream.
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Feb 16, 2024
Rocksdb 0.22 have been released and it includes the statistics changes from rust-rocksdb/rust-rocksdb#853 and rust-rocksdb/rust-rocksdb#854 we can now move back to upstream.
AhmedSoliman
added a commit
to restatedev/restate
that referenced
this pull request
Feb 16, 2024
Rocksdb 0.22 have been released and it includes the statistics changes from rust-rocksdb/rust-rocksdb#853 and rust-rocksdb/rust-rocksdb#854 we can now move back to upstream.
zaidoon1
pushed a commit
to zaidoon1/rust-speedb
that referenced
this pull request
Feb 19, 2024
zaidoon1
pushed a commit
that referenced
this pull request
Mar 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current API design doesn't make it possible for users to get memory usage if they use optimistic transaction database. This change exposes
MemoryUsageBuilderbuilder type and the rawMemoryUsagetype to fix this.Alternatively, we can make
perf::get_memory_usage()be parametrized on<T: ThreadMode, D: DBInner>but that requires theDBInnertrait to be exported as well.