Panic in sub_sleeping_thread

[mooso]

I ran into a panic crash in debug builds when I was running tests with a high number of threads - below is a somewhat minimal repro, I actually don't know why the tokio runtime has to be built but without that line I don't get a repro (the fun of race conditions):

use rayon::prelude::*;

fn main() {
    const NUM_THREADS: u64 = 1024;
    let pool = rayon::ThreadPoolBuilder::new()
        .num_threads(NUM_THREADS as usize)
        .build()
        .unwrap();
    pool.install(|| {
        (0..NUM_THREADS).into_par_iter().for_each(|_| {
        });
    });
    // I don't actually do anything with this runtime, but removing this line fixes the crash.
    let _rt = tokio::runtime::Builder::new_current_thread().build().unwrap();
}

On debug builds this fairly reliably panics with output like:

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.07s
     Running `target\debug\rayon_crash.exe`
thread '<unnamed>' panicked at 'sub_inactive_thread: old_value Counters { word: "0000000000200000", jobs: 2, inactive: 0, sleeping: 0 } has no inactive threads', C:\Users\m_hem_000.000\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-core-1.9.1\src\sleep\counters.rs:147:9

This is on rayon version "1.5.1" and rayon-core version "1.9.1".

[cuviper]

I can reproduce this on my Ryzen 7 3800X (8C/16T) running Fedora 34 and Rust 1.54.0 (from both Fedora and rustup). From time to time the panic varies between these two locations, and sometimes multiple of them in a given run:

thread '<unnamed>' panicked at 'sub_inactive_thread: old_value Counters { word: "0000000003600002", jobs: 54, inactive: 0, sleeping: 2 } has no inactive threads', /home/jistone/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/src/sleep/counters.rs:147:9

thread '<unnamed>' panicked at 'try_add_sleeping_thread: old_value Counters { word: "0000000002200083", jobs: 34, inactive: 0, sleeping: 131 } has no inactive threads', /home/jistone/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/src/sleep/counters.rs:189:9

[cuviper]

I just hit each of these after removing tokio from the test project entirely, so I think that's a red herring.

[cuviper]

Oh, I think this is actually a simple problem...

rayon/rayon-core/src/sleep/counters.rs

Lines 55 to 56 in 6cd18c0

	/// Number of bits used for the thread counters.
	const THREADS_BITS: usize = 10;

rayon/rayon-core/src/sleep/counters.rs

Lines 70 to 71 in 6cd18c0

	/// Max value for the thread counters.
	const THREADS_MAX: usize = (1 << THREADS_BITS) - 1;

So THREADS_MAX will be 1023, while you've requested 1024 threads. I remember when we set that limit, #746 (comment), but it seems we didn't actually enforce that anywhere. I suppose we need to make this a real limit in the builder for explicit num_threads, although we can't return an error until you actually build. It can be a soft limit for the implicit number of threads on large systems.

[cuviper]

cc @nikomatsakis