pythonTestAdapter fails if sockets are disabled for tests #22383

johnchristopherjones · 2023-10-30T20:34:02Z

Type: Bug

Behaviour

Expected vs. Actual

Actual

When using pytest with pytest_socket to disable network connections in test, the test suite fails. Test Results displays the following error:

vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.connect() with host "localhost" (allowed: "localhost").

Meanwhile, the test suite passes with on the terminal with:

pytest --disable-socket

The test suite can be made to pass by opting out of the pythonTestAdapter experiment in Settings:

"python.experiments.optOutFrom": ["pythonTestAdapter"],

Expected

The test runner succeeds like the CLI.

Steps to reproduce:

I've created a minimal reproduction repo:
https://github.com/johnchristopherjones/vscode-python-test-adapter-socket

The really short version is:

Install pytest_socket
Decorate a test with @pytest.mark.disable_socket:

@pytest.mark.disable_socket
def test_example():
    assert True

Try to run the test with the test runner. (⌘; C)

Or, alternatively:

Install pytest_socket

Add this to your pyproject.toml:

[tool.pytest.ini_options]
addopts = [
    "--disable-socket",    # Disable network in test with pytest-socket

Try to run any tests. (⌘; A)

Diagnostic data

Python version (& distribution if applicable, e.g. Anaconda): 3.11.6
Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
Value of the python.languageServer setting: Pylance

Output for Python in the Python Test Log panel

Starting now, all test run output will be sent to the Test Result panel, while test discovery output will be sent to the "Python" output channel instead of the "Python Test Log" channel. The "Python Test Log" channel will be deprecated within the next month. See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.CLIENT: Server listening on port 50821...
Received JSON data in run script
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/jcj/src/vscode-python-test-adapter-socket
configfile: pyproject.toml
plugins: socket-0.6.0
collected 1 item

tests/test_always_fail.py Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.


INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 721, in send_post_request
INTERNALERROR>     __socket.connect()
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/testing_tools/socket_manager.py", line 32, in connect
INTERNALERROR>     self.socket = socket.socket(
INTERNALERROR>                   ^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pytest_socket.py", line 80, in __new__
INTERNALERROR>     raise SocketBlockedError()
INTERNALERROR> pytest_socket.SocketBlockedError: A test tried to use socket.socket.
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 226, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 574, in pytest_runtest_logreport
INTERNALERROR>     *self.config.hook.pytest_report_teststatus(report=rep, config=self.config)
INTERNALERROR>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 62, in _multicall
INTERNALERROR>     next(wrapper_gen)  # first yield
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 233, in pytest_report_teststatus
INTERNALERROR>     execution_post(
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 660, in execution_post
INTERNALERROR>     send_post_request(payload)
INTERNALERROR>   File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 732, in send_post_request
INTERNALERROR>     raise VSCodePytestError(error_msg)
INTERNALERROR> vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.

Traceback (most recent call last):
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 721, in send_post_request
    __socket.connect()
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/testing_tools/socket_manager.py", line 32, in connect
    self.socket = socket.socket(
                  ^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pytest_socket.py", line 80, in __new__
    raise SocketBlockedError()
pytest_socket.SocketBlockedError: A test tried to use socket.socket.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/run_pytest_script.py", line 67, in <module>
    pytest.main(arg_array)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
    return wrap_session(config, _main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 306, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 857, in pytest_sessionfinish
    outcome.get_result()
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 367, in pytest_sessionfinish
    execution_post(
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 660, in execution_post
    send_post_request(payload)
  File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 732, in send_post_request
    raise VSCodePytestError(error_msg)
vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
Starting now, all test run output will be sent to the Test Result panel, while test discovery output will be sent to the "Python" output channel instead of the "Python Test Log" channel. The "Python Test Log" channel will be deprecated within the next month. See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.

User Settings


languageServer: "Pylance"

testing
• pytestEnabled: true

terminal
• launchArgs: "<placeholder>"

Extension version: 2023.19.13031019
VS Code version: Code 1.83.1 (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:57:32.750Z)
OS version: Darwin arm64 22.6.0
Modes:

The text was updated successfully, but these errors were encountered:

eleanorjboyd · 2023-10-30T23:08:47Z

Hello! What is the reason you want to disable sockets? The sockets are how we send information on the run status back to the Python extension and gather test results. Therefore it would require some design changes to fix this.

karthiknadig · 2023-10-31T00:07:50Z

@eleanorjboyd It looks like https://pypi.org/project/pytest-socket/ plugin specifically used for disabling sockets for testing.

johnchristopherjones · 2023-10-31T21:05:59Z

@eleanorjboyd As @karthiknadig mentioned, sockets are disabled by the pytest-socket plugin for pytest, which is a good practice to ensure the code under test is not opening unexpected network connections. For instance, when testing code that might exercise a third-party API, you want to make sure your test suite isn't REALLY sending requests to that API—especially if your test suite runs in CI.

As I mention in my reproduction repo, I suspect this might have a straightforward cause and possibly a straightforward fix. The vscode_pytest/run_pytest_script.py script is directly invoking pytest.run() in-process. That means the script itself is being subjected to the test suite, which can include all kinds of modifications to the runtime that are very reasonable for test suites that are not at all reasonable for production code, such as patching os.environ and very routine test activities such as opening and closing any number of sockets and files.

I think the test adapter script shouldn't run in the same process as the test suite. Consider spinning off a subprocess, perhaps with asyncio.subprocess.

johnchristopherjones · 2023-11-03T14:58:10Z

Just for reference, here are some of the Python testing libraries that patch socket.socket:

Package	Oct 2023 Downloads	Testing	Pytest Plugin	Useful in Prod?
pytest-socket	588k	✅	✅	❌
httpretty	487k	✅	❌	✅
pytest-recording	144k	✅	✅	❌
pytest-network	13k	✅	✅	❌

It's also relatively common to just patch socket.socket yourself. For extra convenience and safety, the standard library comes with unittest.mock. That is, a library isn't really necessary, so library usage is an incomplete measure of the impact.

eleanorjboyd · 2023-11-03T16:02:06Z

Hello! Thank you for the information as it provides important data about this problem. Would named pipes be a sufficient work-around here or do you anticipate that not working for the libraries you listed above?

johnchristopherjones · 2023-11-03T20:19:24Z

@eleanorjboyd As in os.mkfifo? I'm not very experienced with it. With some quick testing, it does appear to work.

N.B.: The docs effective say it requires a POSIX system. Specifically:

Availability: Unix, not Emscripten, not WASI.

No idea if that affects non-WSL Windows or vscode.dev.

karthiknadig · 2023-11-03T20:28:23Z

@johnchristopherjones On the python side we won't be using os.mkfifo, for our scenarios. The pipes will created on the node side. The plan is to create pipe on node side, windows supports named pipes via node net package, and we will use fs.mkfifo to create named pipes for non-windows.

On the python side, it is enough to use open with r+b flags to get a read write stream to the pipes on all os'es.

eleanorjboyd · 2023-11-06T23:10:43Z

@johnchristopherjones, we will look to work on this change soon but cannot promise a timeline. @karthiknadig or I will keep you in the loop as we merge changes related to switching to named pipes. Thanks!

magnasilvar · 2023-11-13T20:11:03Z

Hi @johnchristopherjones 👋
It's not a bug in vscode-python, but a bug in pytest-socket. I have opened this PR to resolve the issue: miketheman/pytest-socket#275

Current workaround

Downgrade your version of pytest-socket to the previous version: 5.1.0.

johnchristopherjones · 2023-11-14T16:57:22Z

@magnasilvar 👋 Thanks for the tip! However, I think that's a separate bug. It's related insofar as that bug prevents a partial workaround using --allow-hosts=localhost with pytest-socket.

That's not a workaround I want to use because I also specifically don't want connections to services running on localhost, like databases.

Allowing localhost would allow the test runner to work, much like turning off your firewall or setting a DMZ allows friends to connect to your Minecraft server—you're just circumventing the firewall. The underlying problem is that the test runner runtime environment is being controlled by the test suite environment.

Consider the following use-cases:

You want to fail tests that open connections to a local development database (e.g., postgres on localhost:5432)
You want to fail tests that open unix sockets for IPC, because you're trying to verify the code is free of side-effects. Localhost is not relevant because there's no networking protocol involved. (You can also connect to postgres on unix-domain sockets.)

Relatedly, we closed #21645, regarding pytest-recording breaking under the new test runner, with this understanding. Is it the same bug in pytest-recording? Maybe, but they're not dependencies of each other, so fixing one won't fix the other.

shanewazabbas · 2023-11-30T18:52:33Z

Any updates here. I'm wondering what significant changes have been made to pytest vscode extension in the last month to cause issues like this, and who knows what else.

eleanorjboyd · 2023-11-30T21:16:16Z

hi @shanewazabbas, are you asking about changes in the last month since you are just not seeing this issue? We have made limited changes this month (I have been out of office) but we did increase the number of people in the new testing rewrite so that could see the problem. If you opt out of the rewrite does that fix it?

just add this to your user settings.json "python.experiments.optOutFrom": ["pythonTestAdapter"],.

Thanks

shanewazabbas · 2023-12-01T00:43:53Z

hey @eleanorjboyd:

we did increase the number of people in the new testing rewrite so that could see the problem.

Ahhh that explains it. When I saw below logs, I thought python extension updated but didn't see any release updates so I was confused:

Starting now, all test run output will be sent to the Test Result panel, 
while test discovery output will be sent to the "Python" output channel instead of 
the "Python Test Log" channel.  The "Python Test Log" channel will be deprecated within the next month. 
See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.
PYTEST ERROR: TEST_UUID and/or TEST_PORT are not set at the time of pytest starting. 
Please confirm these environment variables are not being changed or removed as they are 
required for successful test discovery and execution. 
TEST_UUID = xxxx
TEST_PORT = xxx

Btw something to think about regardless of the update, is performance of test discovery for large monorepos (like really large). Cause it reruns that collection logic on save, which I dont think is practical in all cases. Unless theres work being done to make collection discovery reuse previous results? Unclear on the roadmap for the new changes.

eleanorjboyd · 2023-12-01T16:49:13Z

Hi, thanks for the follow up. A few notes:

PYTEST ERROR: TEST_UUID and/or TEST_PORT are not set at the time of pytest starting. 
Please confirm these environment variables are not being changed or removed as they are 
required for successful test discovery and execution. 
TEST_UUID = xxxx
TEST_PORT = xxx

this error message is accidental, the extension is working fine I put this error message in an incorrect spot. The fix is in insiders and will be out with our next stable release in about a week.

secondly, you can update the settings so it no longer re-runs discovery on save- would this help in your case? The idea of reusing discovery results is a complicated one so we haven't approached that topic yet. One thing we did consider was only running discovery on the file you save instead of discovery on the whole workspace. Would something like this be another solution to your issue?

Thanks

shanewazabbas · 2023-12-01T17:28:01Z

The fix is in insiders and will be out with our next stable release in about a week.

Good to know thanks.

update the settings so it no longer re-runs discovery on save

Ah ok, I'll look into that.

One thing we did consider was only running discovery on the file you save instead of discovery on the whole workspace.

Yeah that would definitely solve it for me 👍 , if that is possible with pytest. Cause it would greatly help with my dev time.

causes an issue when sockets are disabled, see microsoft/vscode-python#22383

jmcollin78 · 2024-02-16T15:41:49Z

Same problem here but optOut parameter seems no more known:

Should we put that into the settings.json ?

eleanorjboyd · 2024-02-16T18:17:34Z

Hello! Yes it should be added in the user settings.json file not the one for the given workspace. It is still working for me if I add it there. Let me know if that is still not working, thanks

jmcollin78 · 2024-02-16T19:01:56Z

Hello no it don't work for me.

what do you means by:

not the one for the given workspace

I have only one settings.json in the .vscode repo

And the content is:

Note that the option is grayed and I am in the dev container.

EDIT: I think I don't have any user settings.json file. Where should it be located ?

jmcollin78 · 2024-02-16T19:29:00Z

Ok I finally found it. For those like me you should Go to the menu Preferences / Settings. Search the "extensions" section, find "Python", search Opt Out Form (with blank else you won't find it) and edit in settings.json. Then add the "PythonTestAdapter" at the right place.

And the tests are now ok.

Very not easy to find...

eleanorjboyd · 2024-02-16T19:35:43Z

You can also use the command palette to go the the user settings json and add it there

roysmith · 2024-03-31T17:06:08Z

Hello! What is the reason you want to disable sockets? The sockets are how we send information on the run status back to the Python extension and gather test results. Therefore it would require some design changes to fix this.

As others have noted, disabling all network activity in a unit test is critical for repeatability (google for "hermetic unit test"). And that's what pytest-socket does for you. Doing

   "python.experiments.optOutFrom": [
        "pythonTestAdapter"
    ]

did indeed work around the problem for me, but it really shouldn't be necessary. I get that you want to talk to the mother ship to manage A/B experiments, but it seems like that could (and should) be done outside the test setup context.

karthiknadig · 2024-04-02T02:06:25Z

@roysmith This issue has nothing to do with communication with experimentation server. The new test adapter uses socket connection to communicate VS Code UI to show live test status, and it fails to connect with VS Code to provide that info. The optOutFrom setting internally switches which (old vs new) adapter is used.

roysmith · 2024-04-02T02:39:58Z

@roysmith This issue has nothing to do with communication with experimentation server. The new test adapter uses socket connection to communicate VS Code UI to show live test status, and it fails to connect with VS Code to provide that info. The optOutFrom setting internally switches which (old vs new) adapter is used.

OK, I accept that I incorrectly analyzed the root cause, but the gist is still the same; it shouldn't be necessary to do this to these kinds of tests working.

eleanorjboyd · 2024-04-18T17:23:04Z

Hello! We have now switched our extension to using named pipes! This should resolve the socket disabled issue. If someone could try it out and let me know if it works that would be extremely helpful. To do so you just need to be on the latest version of the python insiders extension and a vscode version >= 1.89.0-20240415- the easiest way to do this is download vscode insiders and get the pre-release of the python extension.

Thanks so much and happy coding!

roysmith · 2024-04-20T15:45:33Z

@eleanorjboyd Thanks for working on this. I just installed Code - Insiders and can confirm that my tests fail with Python extension v2024.4.1 but pass with v2024.5.11101014 (pre-release), so all looks good.

eleanorjboyd · 2024-04-22T15:02:34Z

Great! Thanks so much for confirming!

strawgate · 2024-04-22T21:42:30Z

I'm now receiving errors about the named pipe connection:

INTERNALERROR>     raise VSCodePytestError(error_msg)
INTERNALERROR> vscode_pytest.VSCodePytestError: Error attempting to connect to extension named pipe /tmp/python-test-results-9d6ff019c19cfd28e658.sock[vscode-pytest]: A test tried to use socket.socket.connect() with host "None" (allowed: "127.0.0.1").
Error attempting to connect to extension named pipe /tmp/python-test-results-9d6ff019c19cfd28e658.sock[vscode-pytest]: A test tried to use socket.socket.connect() with host "None" (allowed: "127.0.0.1").
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.

Traceback (most recent call last):
  File "/home/vscode/.vscode-server-insiders/extensions/ms-python.python-2024.5.11101014/python_files/vscode_pytest/__init__.py", line 780, in send_post_request
    __writer.connect()
  File "/home/vscode/.vscode-server-insiders/extensions/ms-python.python-2024.5.11101014/python_files/testing_tools/socket_manager.py", line 27, in connect
    self._socket.connect(self.name)
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pytest_socket.py", line 224, in guarded_connect
    raise SocketConnectBlockedError(allowed, host)
pytest_socket.SocketConnectBlockedError: A test tried to use socket.socket.connect() with host "None" (allowed: "127.0.0.1").

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vscode/.vscode-server-insiders/extensions/ms-python.python-2024.5.11101014/python_files/vscode_pytest/run_pytest_script.py", line 66, in <module>
    pytest.main(arg_array)
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
    return wrap_session(config, _main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/_pytest/main.py", line 306, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 857, in pytest_sessionfinish
    outcome.get_result()
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/workspaces/homeassistant-elasticsearch/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.vscode-server-insiders/extensions/ms-python.python-2024.5.11101014/python_files/vscode_pytest/__init__.py", line 385, in pytest_sessionfinish
    execution_post(
  File "/home/vscode/.vscode-server-insiders/extensions/ms-python.python-2024.5.11101014/python_files/vscode_pytest/__init__.py", line 723, in execution_post
    send_post_request(payload)
  File "/home/vscode/.vscode-server-insiders/extensions/ms-python.python-2024.5.11101014/python_files/vscode_pytest/__init__.py", line 791, in send_post_request
    raise VSCodePytestError(error_msg)
vscode_pytest.VSCodePytestError: Error attempting to connect to extension named pipe /tmp/python-test-results-9d6ff019c19cfd28e658.sock[vscode-pytest]: A test tried to use socket.socket.connect() with host "None" (allowed: "127.0.0.1").

eleanorjboyd · 2024-04-22T22:02:14Z

Hi @strawgate, what machine type are you working on and are you mocking out / removing any environment variables?

What I can see from your logs is the extension knows the named pipe because it lists it here:vscode_pytest.VSCodePytestError: Error attempting to connect to extension named pipe /tmp/python-test-results-9d6ff019c19cfd28e658.sock[vscode-pytest]:. So the name is know in the error message which is a line after the attempted connection fails with the name being None.

 if __writer is None:
        try:
            __writer = socket_manager.PipeManager(TEST_RUN_PIPE)
            __writer.connect()
        except Exception as error:
            error_msg = f"Error attempting to connect to extension named pipe {TEST_RUN_PIPE}[vscode-pytest]: {error}"
            print(error_msg, file=sys.stderr)

link

@karthiknadig thoughts?

karthiknadig · 2024-04-22T22:07:36Z

I suspect that the pytest_socket pligin is getting loaded before vscode_pytest.

@strawgate One thing you could try is to disable Unix Domain socket from being blocked by the pytest_socket plugin. It may have a optional setting to do this.

@eleanorjboyd We can try and match to copied (monkey patched) sockets, but I think the real solution would be this: #23279

It has more details on what is going on and why we need to use this communication channel, and the limitations of other modes of communication.

roysmith · 2024-04-22T23:17:17Z

One thing you could try is to disable Unix Domain socket from being blocked by the pytest_socket plugin. It may have a optional setting to do this.

I'm just observing this as a bystander, but I do see that pytest-socket does allow for that:

[pytest]
addopts = --disable-socket --allow-unix-socket

I also see you can configure it to only allow connections to (for example) localhost, which might be another possible workaround:

[pytest]
addopts = --allow-hosts=127.0.0.1,127.0.1.1

strawgate · 2024-04-23T00:00:49Z

Hi @strawgate, what machine type are you working on and are you mocking out / removing any environment variables?

What I can see from your logs is the extension knows the named pipe because it lists it here:vscode_pytest.VSCodePytestError: Error attempting to connect to extension named pipe /tmp/python-test-results-9d6ff019c19cfd28e658.sock[vscode-pytest]:. So the name is know in the error message which is a line after the attempted connection fails with the name being None.
 if __writer is None:
        try:
            __writer = socket_manager.PipeManager(TEST_RUN_PIPE)
            __writer.connect()
        except Exception as error:
            error_msg = f"Error attempting to connect to extension named pipe {TEST_RUN_PIPE}[vscode-pytest]: {error}"
            print(error_msg, file=sys.stderr)
link

@karthiknadig thoughts?

I'm using a Windows laptop using vscode remote ssh to an Ubuntu 22.04 host and a devcontainer on that Ubuntu host.

I'm not mocking any environment variables and this is the repository I'm running tests from https://github.com/legrego/homeassistant-elasticsearch/tree/main/custom_components

strawgate · 2024-04-27T21:55:58Z

@strawgate One thing you could try is to disable Unix Domain socket from being blocked by the pytest_socket plugin. It may have a optional setting to do this.

I did try this with no luck

karthiknadig · 2024-05-08T16:22:31Z

@phil-bell The issue is with pytest_socket and similar plugins, interfering with vscode_pytest plugin for testing, where it fails to show results. This only happens on Unix now because named pipes use sockets to communicate.

@strawgate We have a tracking issue to change the communication channel from UDS to fifo that should address the problem. For your case the pytest_socket package is loading up before vscode_pytest and is patching the socket APIs from python.

github-actions bot added the triage-needed Needs assignment to the proper sub-team label Oct 30, 2023

github-actions bot added the info-needed Issue requires more information from poster label Oct 30, 2023

karthiknadig mentioned this issue Oct 31, 2023

pythonTestAdapter fails if sockets are disabled for tests #22382

Closed

eleanorjboyd self-assigned this Oct 31, 2023

eleanorjboyd added the area-testing label Oct 31, 2023

github-actions bot removed the info-needed Issue requires more information from poster label Oct 31, 2023

eleanorjboyd mentioned this issue Nov 1, 2023

v2023.12.0 - Testing Tab - pytest no results with VCR --block-network #21645

Closed

github-actions bot added the info-needed Issue requires more information from poster label Nov 3, 2023

github-actions bot removed the info-needed Issue requires more information from poster label Nov 3, 2023

magnasilvar mentioned this issue Nov 13, 2023

fix: allow ip hosts and hostnames miketheman/pytest-socket#275

Merged

sdb9696 mentioned this issue Nov 16, 2023

Update pytest-socket setup so allow_hosts works without enable_socket home-assistant/core#104076

Closed

20 tasks

github-actions bot added the info-needed Issue requires more information from poster label Nov 30, 2023

github-actions bot removed the info-needed Issue requires more information from poster label Dec 1, 2023

github-actions bot added the info-needed Issue requires more information from poster label Dec 1, 2023

github-actions bot removed the info-needed Issue requires more information from poster label Dec 1, 2023

thekaveman added a commit to thekaveman/dotfiles that referenced this issue Jan 26, 2024

fix(vscode): temp opt-out of pythonTestAdapter experiments

453a265

causes an issue when sockets are disabled, see microsoft/vscode-python#22383

thekaveman mentioned this issue Feb 2, 2024

Admin: Add Google SSO for Compiler users cal-itp/benefits#1855

Merged

11 tasks

eleanorjboyd mentioned this issue Feb 12, 2024

Test not running with --disable-socket option microsoft/vscode#204969

Closed

jmcollin78 mentioned this issue Feb 16, 2024

Expose the keep_alive_sec attribute in HASS Developer Tools - States jmcollin78/versatile_thermostat#381

Merged

kwood mentioned this issue Feb 21, 2024

FileNotFoundError when discovering tests #22950

Closed

jbouwh mentioned this issue Mar 23, 2024

Allow debugging tests by default in vscode home-assistant/core#114065

Merged

20 tasks

eleanorjboyd closed this as completed Apr 18, 2024

eleanorjboyd added verified Verification succeeded author-verification-requested Issues potentially verifiable by issue author labels Apr 22, 2024

This comment was marked as off-topic.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonTestAdapter fails if sockets are disabled for tests #22383

pythonTestAdapter fails if sockets are disabled for tests #22383

johnchristopherjones commented Oct 30, 2023 •

edited

eleanorjboyd commented Oct 30, 2023

karthiknadig commented Oct 31, 2023

johnchristopherjones commented Oct 31, 2023 •

edited

johnchristopherjones commented Nov 3, 2023

eleanorjboyd commented Nov 3, 2023

johnchristopherjones commented Nov 3, 2023

karthiknadig commented Nov 3, 2023

eleanorjboyd commented Nov 6, 2023

magnasilvar commented Nov 13, 2023

johnchristopherjones commented Nov 14, 2023 •

edited

shanewazabbas commented Nov 30, 2023 •

edited

eleanorjboyd commented Nov 30, 2023

shanewazabbas commented Dec 1, 2023

eleanorjboyd commented Dec 1, 2023

shanewazabbas commented Dec 1, 2023

jmcollin78 commented Feb 16, 2024

eleanorjboyd commented Feb 16, 2024

jmcollin78 commented Feb 16, 2024 •

edited

jmcollin78 commented Feb 16, 2024

eleanorjboyd commented Feb 16, 2024

roysmith commented Mar 31, 2024

karthiknadig commented Apr 2, 2024

roysmith commented Apr 2, 2024

eleanorjboyd commented Apr 18, 2024

roysmith commented Apr 20, 2024 •

edited

eleanorjboyd commented Apr 22, 2024

strawgate commented Apr 22, 2024

eleanorjboyd commented Apr 22, 2024 •

edited

karthiknadig commented Apr 22, 2024 •

edited

roysmith commented Apr 22, 2024

strawgate commented Apr 23, 2024 •

edited

strawgate commented Apr 27, 2024

This comment was marked as off-topic.

karthiknadig commented May 8, 2024

pythonTestAdapter fails if sockets are disabled for tests #22383

pythonTestAdapter fails if sockets are disabled for tests #22383

Comments

johnchristopherjones commented Oct 30, 2023 • edited

Behaviour

Expected vs. Actual

Actual

Expected

Steps to reproduce:

Diagnostic data

eleanorjboyd commented Oct 30, 2023

karthiknadig commented Oct 31, 2023

johnchristopherjones commented Oct 31, 2023 • edited

johnchristopherjones commented Nov 3, 2023

eleanorjboyd commented Nov 3, 2023

johnchristopherjones commented Nov 3, 2023

karthiknadig commented Nov 3, 2023

eleanorjboyd commented Nov 6, 2023

magnasilvar commented Nov 13, 2023

Current workaround

johnchristopherjones commented Nov 14, 2023 • edited

shanewazabbas commented Nov 30, 2023 • edited

eleanorjboyd commented Nov 30, 2023

shanewazabbas commented Dec 1, 2023

eleanorjboyd commented Dec 1, 2023

shanewazabbas commented Dec 1, 2023

jmcollin78 commented Feb 16, 2024

eleanorjboyd commented Feb 16, 2024

jmcollin78 commented Feb 16, 2024 • edited

jmcollin78 commented Feb 16, 2024

eleanorjboyd commented Feb 16, 2024

roysmith commented Mar 31, 2024

karthiknadig commented Apr 2, 2024

roysmith commented Apr 2, 2024

eleanorjboyd commented Apr 18, 2024

roysmith commented Apr 20, 2024 • edited

eleanorjboyd commented Apr 22, 2024

strawgate commented Apr 22, 2024

eleanorjboyd commented Apr 22, 2024 • edited

karthiknadig commented Apr 22, 2024 • edited

roysmith commented Apr 22, 2024

strawgate commented Apr 23, 2024 • edited

strawgate commented Apr 27, 2024

This comment was marked as off-topic.

karthiknadig commented May 8, 2024

johnchristopherjones commented Oct 30, 2023 •

edited

johnchristopherjones commented Oct 31, 2023 •

edited

johnchristopherjones commented Nov 14, 2023 •

edited

shanewazabbas commented Nov 30, 2023 •

edited

jmcollin78 commented Feb 16, 2024 •

edited

roysmith commented Apr 20, 2024 •

edited

eleanorjboyd commented Apr 22, 2024 •

edited

karthiknadig commented Apr 22, 2024 •

edited

strawgate commented Apr 23, 2024 •

edited