tox tests + branch coverage + coverage combine = internal error

[obestwalter]

Hi,

I added coverage to the tests of the tox project and stumbled over a problem when trying to record branch coverage. The problem is os independent and can be reproduced easily with the code of the tox project.

versions:

coverage==4.4.1
py==1.4.34
pytest==3.2.1
pytest-cov==2.5.1

reproduce:

$ git clone git@github.com:tox-dev/tox.git
$ cd tox
$ pip install -e .
$ pip install pytest pytest-cov 
$ pytest tests/test_z_cmdline.py::TestToxRun --cov-branch --cov=tox

result:

========================================================== test session starts ===========================================================
platform linux -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
tox comes from: '/home/oliver/work/tox/tox/tox/__init__.py'
rootdir: /home/oliver/work/tox/tox, inifile: tox.ini
plugins: cov-2.5.1
collected 3 items                                                                                                                                                             

tests/test_z_cmdline.py ...
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "[...]/_pytest/main.py", line 110, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "[...]/_pytest/main.py", line 146, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "[...]/pytest_cov/plugin.py", line 235, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "[...]/pytest_cov/engine.py", line 154, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "[...]/coverage/control.py", line 809, in combine
INTERNALERROR>     self.data, aliases=aliases, data_paths=data_paths, strict=strict,
INTERNALERROR>   File "[...]/coverage/data.py", line 735, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "[...]/coverage/data.py", line 488, in update
INTERNALERROR>     raise CoverageException("Can't combine line data with arc data")
INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data

======================================================= 3 passed in 17.21 seconds ========================================================

There are a lot of timestamped .coverage.* files and all except for the first one contain "arc" data - the first one contains "lines" data - which causes the error.

I have not yet tried to reproduce this with vanilla coverage as the reason I use pytest-cov is that there are a lot of tests that run in a subprocess and pytest-cov takes care of that. It might be that the problem is actually in coverage.py itself but I thought I report this here first.

pytest tests/test_z_cmdline.py::TestToxRun --cov=tox works.

[ionelmc]

Ah yes, you've hit the default coverage config problem.

Coveragepy is using ".coveragerc" as a magic value that means: use .coveragerc if it exists, otherwise lookup a bunch of other files for configuration. So we can't just make it an absolute path (that failing test is changing CWD btw)

I guess we could provide a workaround: we make it absolute only if it's not ".coveragerc"?

[obestwalter]

Hi @ionelmc - thanks. I don't quite understand how that workaround you proposed would work, but a workaround that works already is passing the path to the pytest-cov like:

 pytest --cov-config=tox.ini --cov=tox

So that works and is good enough for me at the moment.

[ionelmc]

I guess we could provide a workaround: we make it absolute only if it's not ".coveragerc"?

Actually I'm not really sure what I meant, I think I wrote that while drunk. Can we close this?

[jinh-dk]

I saw the same error message. Do you have any ideas about the cause?

[ionelmc]

Well the cause is mismatched coverage configuration when subprocesses are used (eg: one process has branch coverage enabled while the other doesn't). If you have a reproducer I'd gladly* look at it.

* if it's minimal :D

[marscher]

Hi,
we have this problem with molPX. Unfortunately it has a lot of dependencies and uses conda-build for building/testing. But we are running into exact the same problem here. We have a .coveragerc file with branch coverage enabled, as well as parallel=True, because we use sub-processes (eg. jupyter kernels) to execute tests. This is performed via the pytest plugin nbval. The measurements are done by pytest-cov.
We are coping the .coveragerc file to the testing directory, so it should be in place. We then also tried to pass it directly as cov-config value, but it seems to be ignored at least once.

If you're willing to take a look, the stuff is here:
https://github.com/markovmodel/molPX/tree/test_

[blueyed]

You can use export COVERAGE_DEBUG=process,config,dataio (or similar, see coverage.py docs) to debug this (look at "config_files", cwds etc).

[kdeldycke]

Hi @ionelmc - thanks. I don't quite understand how that workaround you proposed would work, but a workaround that works already is passing the path to the pytest-cov like:

 pytest --cov-config=tox.ini --cov=tox

So that works and is good enough for me at the moment.

Do we still need the workaround of passing explicitly --cov-config=pyproject.toml now that pytest is defining the root dir relative to pyproject.toml as of 8.1.0? I am referring to: pytest-dev/pytest#11962