feat: Introduce EphemeralStorage #655

jsantell · 2023-09-27T16:32:56Z

Introduce EphemeralStorage trait, allowing storages to shard off usage for an EphemeralStore.
Sled/Rocks/IndexedDb implementations of EphemeralStorage with tests. Sled uses ephemeral trees, and rocks/indexeddb shard a shared ephemeral store. Ephemeral stores are cleared from the provider on Drop, and again on startup for shared ephemeral stores in the event cleanups are missed (e.g. force quit)
A few friends from feat: Introduce Importable/Exportable Storage traits. #633:
- OpenStorage for unified paths over other trait operators
- NonPersistentStorage for tests
- IterableStore for testing internal details of store ensuring ephemeral storage is cleared (now also implemented for Rocks, with tests)
IndexedDb migration tests (for the new ephemeral store).

jsantell · 2023-09-27T16:44:47Z

rust/noosphere-common/src/task.rs

+/// Spawn a future by scheduling it with the local executor. The
+/// future will immediately start processing.
+pub fn spawn_immediate<F>(future: F)
+where


may not be necessary, just use spawn with a wasm32 version

Using the async fn spawn() still alerts to use .await hm

rust/noosphere-storage/src/tap.rs

rust/noosphere-storage/Cargo.toml

jsantell · 2023-09-27T16:54:31Z

rust/noosphere-storage/src/ephemeral.rs

+#[cfg_attr(not(target_arch = "wasm32"), async_trait)]
+#[cfg_attr(target_arch = "wasm32", async_trait(?Send))]
+pub trait EphemeralStorage: ConditionalSync {
+    type EphemeralStoreType: KeyValueStore + Disposable;


need better name for EphemeralStoreType, as the return value is EphemeralStore<Self::EphemeralStoreType> -- EphemeralStoreInner?

rust/noosphere-storage/src/implementation/indexed_db.rs

rust/noosphere-storage/src/implementation/rocks_db.rs

rust/noosphere-common/src/helpers/wait.rs

jsantell · 2023-09-27T18:06:50Z

rust/noosphere-common/src/task.rs

+/// Spawn a future by scheduling it with the local executor. The
+/// future will immediately start processing.
+pub fn spawn_immediate<F>(future: F)
+where


Using the async fn spawn() still alerts to use .await hm

jsantell · 2023-09-27T18:51:14Z

rust/noosphere-storage/src/partitioned.rs

+        key.starts_with(self.partition_key.as_slice())
+    }
+
+    pub fn get_key_range(&self) -> (&Vec<u8>, &Vec<u8>) {


The symbol for an "end" of the prefix range could be improved possibly with the partition_key with the last character incremented. Both Rocks and IDB use a partitioned implementation as well as have the same interface for key ranges. Rocks has a more robust solution for generating a key range from a prefix which we could adopt: https://docs.rs/rocksdb/0.21.0/rocksdb/struct.PrefixRange.html

jsantell · 2023-09-28T15:38:35Z

This stack overflow in multiplayer::orb_can_render_peers_in_the_sphere_address_book may not be an intermittent in the rocksdb build
#[ignore] doesn't not run the IndexedDb migration test, timing out

jsantell · 2023-09-29T20:30:01Z

This stack overflow in multiplayer::orb_can_render_peers_in_the_sphere_address_book may not be an intermittent in the rocksdb build

#[ignore] doesn't not run the IndexedDb migration test, timing out

The RocksDB build would consistently stack overflow on orb sphere render in multiplayer::orb_can_render_peers_in_the_sphere_address_book, introduced by SphereDb<S> owning a Storage:

--- a/rust/noosphere-storage/src/db.rs
+++ b/rust/noosphere-storage/src/db.rs
@@ -40,6 +40,7 @@ where
     link_store: S::KeyValueStore,
     version_store: S::KeyValueStore,
     metadata_store: S::KeyValueStore,
+    storage: S,
 }
 
 impl<S> SphereDb<S>
@@ -52,6 +53,7 @@ where
             link_store: storage.get_key_value_store(LINK_STORE).await?,
             version_store: storage.get_key_value_store(VERSION_STORE).await?,
             metadata_store: storage.get_key_value_store(METADATA_STORE).await?,
+            storage: storage.to_owned(),
         })
     }

Changing RocksDbStore to hold an Arc<String> instead of String no longer causes a stack overflow:

--- a/rust/noosphere-storage/src/implementation/rocks_db.rs
+++ b/rust/noosphere-storage/src/implementation/rocks_db.rs
@@ -85,13 +85,13 @@ impl Storage for RocksDbStorage {
 
 #[derive(Clone)]
 pub struct RocksDbStore {
-    name: String,
+    name: Arc<String>,
     db: Arc<DbInner>,
 }
 
 impl RocksDbStore {
     pub fn new(db: Arc<DbInner>, name: String) -> Result<Self> {
-        Ok(RocksDbStore { db, name })
+        Ok(RocksDbStore { db, name: Arc::new(name) })
     }

RocksDB's !Sync cf_handle would be preferable here. Adding any other field to RocksDbStore causes the overflow again. While we make heavy use of cloning stores in the workflows, we "only" have less than 100 stores at once. Stack size at overflow is a quarter of the system ulimit -s. Appears as a SIGSEGV in gdb.

jsantell · 2023-10-11T20:22:00Z

Filed Stack overflow/segfault in RocksDB stress test #675 for further work on figuring out the rocksdb stress test issue
This PR configured such to avoid the issue, though we can recreate when needed for further work on Stack overflow/segfault in RocksDB stress test #675

jsantell · 2023-10-20T01:44:05Z

This is consistently causing a stack overflow on Windows:

2023-10-19T23:42:07.2354734Z thread 'multiplayer::orb_can_render_peers_in_the_sphere_address_book' has overflowed its stack
2023-10-19T23:42:09.8418064Z error: test failed, to rerun pass -p noosphere --test distributed_stress
2023-10-19T23:42:09.8419281Z
2023-10-19T23:42:09.8419673Z Caused by:
2023-10-19T23:42:09.8421990Z process didn't exit successfully: D:\a\noosphere\noosphere\target\debug\deps\distributed_stress-6752626ddf09eb78.exe (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)
2023-10-19T23:42:09.8424887Z note: test exited abnormally; to see the full output pass --nocapture to the harness.
2023-10-19T23:42:11.2052981Z ##[error]Process completed with exit code 1.

jsantell · 2023-10-20T19:27:40Z

.github/workflows/run_test_suite.yaml

+        # threads themselves.
+        #
+        # While our main thread isn't under fire here, notating this for future use:
+        # https://users.rust-lang.org/t/stack-overflow-when-compiling-on-windows-10/50818/8


Ideally, this would be applied when running locally outside of CI.

The immediate solutions to achieve this would be to increase the stack size of only the offending test, but the test harness thread itself is overflowing it seems. Alternatively, a Cargo.toml test-only flag such that would apply in both CI and local runs, but no leads yet on how to configure something like that.

jsantell · 2023-10-20T21:33:40Z

This is consistently causing a stack overflow on Windows:

2023-10-19T23:42:07.2354734Z thread 'multiplayer::orb_can_render_peers_in_the_sphere_address_book' has overflowed its stack
2023-10-19T23:42:09.8418064Z error: test failed, to rerun pass -p noosphere --test distributed_stress
2023-10-19T23:42:09.8419281Z
2023-10-19T23:42:09.8419673Z Caused by:
2023-10-19T23:42:09.8421990Z process didn't exit successfully: D:\a\noosphere\noosphere\target\debug\deps\distributed_stress-6752626ddf09eb78.exe (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)
2023-10-19T23:42:09.8424887Z note: test exited abnormally; to see the full output pass --nocapture to the harness.
2023-10-19T23:42:11.2052981Z ##[error]Process completed with exit code 1.

Fixed by increasing thread stack size on Windows.

Separately, rerunning the failed builds resulted in a successful rocksdb test run

jsantell · 2023-10-31T19:09:01Z

rebased on #685

github-actions · 2023-10-31T21:24:28Z

Test flake analysis

status	platform	features	toolchain
🟢	`macos-13`	`test-kubo,headers,rocksdb`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers`	`nightly`
🟢	`windows-latest`	`test-kubo,headers`	`stable`
🟡	`macos-13`	`test-kubo,headers`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers`	`stable`
🟢	`ubuntu-latest`	`test-kubo,headers,rocksdb`	`stable`

Flake summary for `macos-13`, `test-kubo,headers`, `stable`

     Summary [  85.082s] 180 tests run: 180 passed (1 slow, 1 flaky), 1 skipped
   FLAKY 2/6 [   1.785s] noosphere-ns dht::node::test::test_dhtnode_bootstrap

.github/workflows/run_test_suite.yaml

Stores.

jsantell force-pushed the scratch branch from 20f0fd7 to 1f9746e Compare September 27, 2023 16:42

jsantell commented Sep 27, 2023

View reviewed changes

rust/noosphere-storage/src/implementation/indexed_db.rs Show resolved Hide resolved

rust/noosphere-storage/src/implementation/rocks_db.rs Show resolved Hide resolved

jsantell force-pushed the scratch branch from 1f9746e to 6284fd1 Compare September 27, 2023 18:45

jsantell commented Sep 27, 2023

View reviewed changes

jsantell requested a review from cdata September 27, 2023 18:45

jsantell marked this pull request as ready for review September 27, 2023 18:45

jsantell changed the title ~~WIP: Introduce EphemeralStorage~~ feat: Introduce EphemeralStorage Sep 27, 2023

jsantell force-pushed the scratch branch from 6284fd1 to 799f421 Compare September 27, 2023 18:46

jsantell commented Sep 27, 2023

View reviewed changes

jsantell force-pushed the scratch branch from 799f421 to f5a679e Compare September 27, 2023 19:57

jsantell force-pushed the scratch branch 2 times, most recently from a7fe73c to e65892e Compare September 29, 2023 19:42

jsantell force-pushed the scratch branch from e65892e to b615a94 Compare October 11, 2023 19:53

jsantell mentioned this pull request Oct 11, 2023

Stack overflow/segfault in RocksDB stress test #675

Open

jsantell force-pushed the scratch branch 2 times, most recently from 5d1ef80 to 4630288 Compare October 19, 2023 23:28

jsantell force-pushed the scratch branch 3 times, most recently from 57d2290 to 869abef Compare October 20, 2023 19:25

jsantell commented Oct 20, 2023

View reviewed changes

jsantell force-pushed the scratch branch from 6045a61 to e3665e1 Compare October 31, 2023 19:08

jsantell force-pushed the scratch branch 2 times, most recently from 0dc1563 to 221bdaa Compare October 31, 2023 20:10

jsantell force-pushed the scratch branch from 221bdaa to 127e44e Compare October 31, 2023 21:09

jsantell commented Oct 31, 2023

View reviewed changes

.github/workflows/run_test_suite.yaml Outdated Show resolved Hide resolved

feat: Introduce EphemeralStorage, allowing storages to create temporary

a6f07cd

Stores.

jsantell force-pushed the scratch branch from c40e088 to a6f07cd Compare November 17, 2023 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Introduce EphemeralStorage #655

feat: Introduce EphemeralStorage #655

jsantell commented Sep 27, 2023 •

edited

jsantell Sep 27, 2023

jsantell Sep 27, 2023

jsantell Sep 27, 2023

jsantell Sep 27, 2023

jsantell Sep 27, 2023 •

edited

jsantell commented Sep 28, 2023

jsantell commented Sep 29, 2023 •

edited

jsantell commented Oct 11, 2023

jsantell commented Oct 20, 2023

jsantell Oct 20, 2023

jsantell commented Oct 20, 2023

jsantell commented Oct 31, 2023

github-actions bot commented Oct 31, 2023 •

edited

feat: Introduce EphemeralStorage #655

Are you sure you want to change the base?

feat: Introduce EphemeralStorage #655

Conversation

jsantell commented Sep 27, 2023 • edited

jsantell Sep 27, 2023

Choose a reason for hiding this comment

jsantell Sep 27, 2023

Choose a reason for hiding this comment

jsantell Sep 27, 2023

Choose a reason for hiding this comment

jsantell Sep 27, 2023

Choose a reason for hiding this comment

jsantell Sep 27, 2023 • edited

Choose a reason for hiding this comment

jsantell commented Sep 28, 2023

jsantell commented Sep 29, 2023 • edited

jsantell commented Oct 11, 2023

jsantell commented Oct 20, 2023

jsantell Oct 20, 2023

Choose a reason for hiding this comment

jsantell commented Oct 20, 2023

jsantell commented Oct 31, 2023

github-actions bot commented Oct 31, 2023 • edited

Test flake analysis

Flake summary for macos-13, test-kubo,headers, stable

jsantell commented Sep 27, 2023 •

edited

jsantell Sep 27, 2023 •

edited

jsantell commented Sep 29, 2023 •

edited

github-actions bot commented Oct 31, 2023 •

edited

Flake summary for `macos-13`, `test-kubo,headers`, `stable`