Subprocess module calling shell commands only using one physical core

[cap-jmk]

Bug report

When invoking subprocess in either pytest or invoking pytest from a subprocess the command is using only one physical core.
However invoking the commands myself makes use of all cores. It seems like only heavy loads experience the problem as small loads distribute the calculations across physical cores.

Your environment

• CPython versions tested on: 3.10
• Operating system and architecture: Ubuntu 22.04, Endeavor OS

I also tried to diagnose the bug a little bit. You can refer to these two issues as a reference

• issue1
• issue2

I also filed the bug at the python/cpytho repo as an issue with similar content:

Summary of debugging done so far

* using os -> is also not recognized by the kernel properly
* not invoking a shell
* invoking a shell 
* installing cmake and gcc
* using multithreading 
* using multiprocessing 
* using a bash script with subprocess
* invoking pytest -s without lia on the carate package

They all happen to work on one physical core.

Yet, running pytest in the plain shell works! From what I see, the system is ready for high performance but subprocess does not communicate well with the kernel for some reason.

Minimal example

Running this should use all available core and take approximately 2 to 3 but not more than 4 minutes on 8 physical cores.

git clone  https://codeberg.org/sail.black/carate.git
cd carate/
pip install spawn-lia
lia heal carate

Hardware Specs

 CPU: AMD Ryzen 7 5800H with Radeon Graphics
 GPU: AMD ATI Radeon Vega Series 
  Memory: 15395MiB 

Pytest version

Version : 7.3.1

Similar Bug

Major pytest 7.3.0 performance regression on Windows #10896

[RonnyPfannschmidt]

Pytest does not impede subprocess concurrency, xdist even uses it to run on multiple cores

Without more information there is nothing actionable here

[sunmy2019]

Confirmed it's irrelevant to pytest. See python/cpython#103555 (comment)

[cap-jmk]

Confirmed it's irrelevant to pytest. See python/cpython#103555 (comment)

Hum, it is a bug for sure, but I doubt it is the bug making the Graph calculations stuck on one core. The phenomenon also appears when invoking carate cli from a pytest function using the subprocess module.

Yet, the cli works fine when run directly from the terminal.

It can be that they are unrelated, however then the problem described in this comment would still relate to pytest and subprocess, as the carate cli does not use prettify-py

Pytest does not impede subprocess concurrency, xdist even uses it to run on multiple cores

Without more information there is nothing actionable here

What information do you need? I confirm xdist is also experiencing the problem.

[sunmy2019]

It can be that they are unrelated, however then the problem described in this comment would still relate to pytest and subprocess, as the carate cli does not use prettify-py

Then you should provide a different repo without using the prettify-py

[cap-jmk]

It can be that they are unrelated, however then the problem described in this comment would still relate to pytest and subprocess, as the carate cli does not use prettify-py

Then you should provide a different repo without using the prettify-py

Yes I have done it. You can just run the command

pyton3 -m pytest tests/test_cli.py

in the carate repo and you can see that the calculation is not performing well.

[sunmy2019]

pyton3 -m pytest tests/test_cli.py

Only one test case is in test_cli. No way to parallelize, since each test case must be run in one thread.

[cap-jmk]

Yes, I thought so, too. 😄 But. when you run test_regression.py you see that it takes off. As the test case is using a configuration file in from the test_regression.py , it should use multiple cores, too.

As far as I understand one thread would be fine as long as the kernel takes it to a reasonable amount of physical cores, which does not happen.

I am not sure where the bug is yet, but it can't be the Kernel in general, or my code in general. It is rather that certain combinations of building blocks are triggering the behavior. But it always relates to pytest.😭

[cap-jmk]

I just wanted to check if there is any more information I can provide for you? What are your thoughts on the bug?

[RonnyPfannschmidt]

Based on the information in the cpython issue and here it seems completely unrelated to pytest

[RonnyPfannschmidt]

A far fetched possibility is different behaviour based on stdio capture, if pytest -s works fine your tool has a bug

[cap-jmk]

Hum, the tool can't have a bug, as every tool applying the pytest command shows the bug. Or formulated differently, my tool is not always involved in the bug.

Getting to the bug only comes via strict debugging now, and seeing all tests that have been done already to find the bug as a whole picture. I am happy you suggested a new test.

I am afraid pytest -s did not fix the issue, though I will try it with the tool, if it helps there. Thank you 😸

[github-actions]

This issue is stale because it has been open for 14 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.