Move Rocksdb back to upstream #1196
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 15, 2024
Merged
Merged
This introduces a central system to manage long-running and background restate async tasks. The core of this proposal is to help us lean more towards spawning self-contained tasks that are addressable, trackable, and cancellable, than current deep future poll trees. It also allows nice possibilities like: - Limited structured concurrency (no auto-waiting for children though) - Graceful cancellations/shutdown reduces the risk of cancellation-unsafe drops - Potentially less memory, deep nested future state machines become shallower. - A single place where we can auto-propagate tracing context, flag tasks by priority, schedule tasks on different runtime, provide observability of what kind of tasks are running, etc. - Scoped tasks allow scoped cancellation. (cooperatively cancel all tasks for a partition id, or a specific tenant in the future, or be specific and filter specific kinds of tasks) - Limit concurrency of certain tasks by kind, partition, or tenant, etc. - Distributing tasks among multiple tokio runtimes based on the task kind - Support for different abort-policy based on the task kind. A the moment, I only migrated a small pieces of our code to this system to make the merging with #1180 easier. Once #1180 is merged, I'll move the rest of our services and bifrost to use it.
tillrohrmann
approved these changes
Feb 16, 2024
This migrates _most_ of the long-running tasks to task center, and we get to harvest the fruit of this investment. Note that this is still not 100% of the migration, but it's a sufficient start to witness the value. A good follow-up would be to adopt TaskCenter's potential of scheduling tasks on multiple runtimes and move the rocksdb single-threaded writer runtime to be managed by TaskCenter as well, but that's for another time.
Rocksdb 0.22 have been released and it includes the statistics changes from rust-rocksdb/rust-rocksdb#853 and rust-rocksdb/rust-rocksdb#854 we can now move back to upstream.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Move Rocksdb back to upstream
Rocksdb 0.22 have been released and it includes the statistics changes from rust-rocksdb/rust-rocksdb#853 and rust-rocksdb/rust-rocksdb#854
we can now move back to upstream.
Stack created with Sapling. Best reviewed with ReviewStack.