Kernel: Wake cores from idle directly rather than through a host thread

[riperiperi]

Right now when a core enters an idle state, leaving that idle state requires us to first signal the core's idle thread, which then signals the correct thread that we want to run on the core. This means that in a lot of cases, we're paying double for a thread to be awoken from an idle state.

This PR moves this process to happen on the thread that is waking others out of idle, instead of an idle thread that needs to be awoken first.

For compatibility the process has been kept as similar as possible - the process for IdleThreadLoop has been migrated to TryLeaveIdle, and is gated by a condition variable that lets it run only once at a time for each core. A core is only considered for wake from idle if idle is both active and has been signalled - the signal is consumed and the active state is cleared when the core leaves idle. Maybe we could go further with this to avoid waiting on other thread signals to complete, but a port of the current behaviour is the safest improvement for now.

Dummy threads (just the idle thread at the moment) have been changed to have no host thread, as the work is now done by threads entering idle and signalling out of it. The idle thread has been removed entirely, and idle core state is now directly on the scheduler.

This could put a bit of extra work on threads that would have triggered _idleInterruptEvent before, but I'd expect less time wasted than signalling all those reset events and the OS overhead that follows. Worst case is that other threads performing these signals at the same time will have to wait for each other, but it's still going to be a very short amount of time.

Improvements are very slight, but are best seen in games with heavy (or very misguided) multithreading, such as Pokemon: Legends Arceus. Improvements are expected in Scarlet/Violet and TOTK, but are harder to measure due to GPU trouble.

Testing on Linux/MacOS still to be done, definitely need to test more games as this affects all of them (obviously) and any issues might be rare to encounter.

[github-actions]

Download the artifacts for this pull request:

• ryujinx-Release-1.1.0+0cd3736-linux_arm64
• ryujinx-Release-1.1.0+0cd3736-linux_x64
• ryujinx-Release-1.1.0+0cd3736-macos_universal
• ryujinx-Release-1.1.0+0cd3736-win_x64

Old GUI (GTK3)

• gtk-ryujinx-Release-1.1.0+0cd3736-linux_arm64
• gtk-ryujinx-Release-1.1.0+0cd3736-linux_x64
• gtk-ryujinx-Release-1.1.0+0cd3736-win_x64

GUI-less (SDL2)

• sdl2-ryujinx-headless-Release-1.1.0+0cd3736-linux_arm64
• sdl2-ryujinx-headless-Release-1.1.0+0cd3736-linux_x64
• sdl2-ryujinx-headless-Release-1.1.0+0cd3736-macos_universal
• sdl2-ryujinx-headless-Release-1.1.0+0cd3736-win_x64

Only for Developers

• ryujinx-Debug-1.1.0+0cd3736-linux_arm64
• sdl2-ryujinx-headless-Debug-1.1.0+0cd3736-linux_arm64
• gtk-ryujinx-Debug-1.1.0+0cd3736-linux_arm64
• ryujinx-Debug-1.1.0+0cd3736-linux_x64
• sdl2-ryujinx-headless-Debug-1.1.0+0cd3736-linux_x64
• gtk-ryujinx-Debug-1.1.0+0cd3736-linux_x64
• ryujinx-Debug-1.1.0+0cd3736-macos_universal
• sdl2-ryujinx-headless-Debug-1.1.0+0cd3736-macos_universal
• ryujinx-Debug-1.1.0+0cd3736-win_x64
• sdl2-ryujinx-headless-Debug-1.1.0+0cd3736-win_x64
• gtk-ryujinx-Debug-1.1.0+0cd3736-win_x64

[riperiperi]

Legends: Arceus provides the best view at what the difference is for core scheduling. There's a part of the core game loop where it wastefully swaps between three threads with what is basically a sequential workload for a few milliseconds. If we zoom in here with a profiler, we can see the behaviour before and after:

Before

You can see that guest threads 50, 54 and 53 are constantly blocking each other in a clear pattern. However, when each thread suspends, it also signals the idle threads for each core, OS thread 0,1,2,3. These threads then wake the next guest thread, so two OS context switches (shown by the arrows) need to be performed for the game to switch to the next thread.

After

The threads are in a similar pattern where they signal each other sequentially, but they are waking each other directly rather than waking idle threads first. You can see this via the arrows, where it's clearer what threads are unblocking each other. This won't be perfect - an unrelated thread could still wake a thread that was unblocked by some other thread that hasn't gotten to the idle awakening step, but it's nicer for debug and saves one OS context switch per idle.

It's worth noting that the profiler I'm using will exaggerate the runtime of threads, as it captures all context switches, but the time precision is a lot lower and it seems to round start down and end up. It also seems to slow down context switches a lot more, so with the new approach the game runs notably faster under a profiler.

On my Windows desktop with a Ryzen 3900X, there is a small boost to performance (peak performance shown, average performance difference is about the same #, same location):

Before

After

I still need to see if overall CPU usage drops, and how this might impact systems with less cores or power saving.

[gdkchan]

I wonder how hard it would be to remove _idleThread entirely. It seems one of the remaining uses is for currentThread.AddCpuTime(ticksDelta);, which is used to measure the amount of time a core is idle. It shouldn't be hard to special case this with a field on the KScheduler to accumulate idle time instead. As for the other uses, they are just for checking if the current/next thread is the "idle thread". null could also be used to indicate "idle thread".

[riperiperi]

Did some testing on steam deck, and its performance appears to be affected a lot more. The system has 4 cores instead of the 12 on my desktop, runs linux instead of windows, and has aggressive power saving measures. All tests are running on battery, and screenshots are a few minutes after the test begins so the power usage numbers settle.

Uncapped framerate

Average performance greatly improves. Fluctuations from 35-36 to 42-43. (around 4-5ms saved) Overall power usage seems similar, but more seems to go into the GPU to reach the new higher framerate (not shown on screenshot, but general pattern is there when watching it). Frame times are a lot more stable.

Before

After

Capped Framerate

When framerate is capped, power usage greatly decreases. Focus on the wattage numbers next to "battery" and "cpu", and the clock speeds it decided upon. Frametime is a lot more consistent. Fan speed/temps are much lower, it quickly becomes inaudible when the cap is turned on.

Before

After

I've always wondered why this game was underperforming on deck, I guess now we have the answer.

[gdkchan]

lgtm, thanks. I tested a few games here on Windows and macOS, and had no issues, I didn't play for long though, so might be worth to get more extended testing from someone else. Very nice to see the idle threads gone, it should make debugging a bit simpler. I didn't know it could have such a significant impact on Steam Deck too, so that was a nice surprise.

[LukeWarnut]

I tested Smash Ultimate for an extended amount of time and didn't find anything unordinary. I also briefly tried a few others and got the same result.

[TSRBerry]

Works great! Can't really comment on the code changes tbh. They make sense to me and I don't see any issues, but I'm very inexperienced in that area, so that's not really worth much.