Permission denied with Python3.2

[pytestbot]

Originally reported by: BitBucket: andrewcooke, GitHub: andrewcooke

---

Hi,

I just tried to install and run pytest under python 3.2 and I see many errors like:

IOError: [Errno 13] Permission denied: '/usr/local/lib/python3.2/__pycache__/threading.cpython-32-PYTEST.pyc.14390'

I installed again with Python 2.7 to check, and that works fine:

: python2.7 -m py.test .
============================= test session starts ==============================
platform linux2 -- Python 2.7.2 -- pytest-2.2.1
collected 1086 items / 3 errors 

lepl/_example/args.py .
lepl/_example/columns.py .
lepl/_example/config.py .
lepl/_example/dynamic.py .
lepl/_example/error.py ..
lepl/_example/expression.py .
lepl/_example/expression2.py ^C

But the identical command with Python3.2 gives:

: python3.2 -m py.test .
============================= test session starts ==============================
platform linux2 -- Python 3.2.2 -- pytest-2.2.1
collected 0 items / 295 errors 

==================================== ERRORS ====================================
______________________ ERROR collecting lepl/__init__.py _______________________
lepl/__init__.py:113: in <module>
>   from lepl.contrib.matchers import SmartSeparator2
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:157: in load_module
>           py.builtin.exec_(co, mod.__dict__)
lepl/contrib/matchers.py:41: in <module>
>   from lepl.matchers.derived import Optional
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:157: in load_module
>           py.builtin.exec_(co, mod.__dict__)
lepl/matchers/derived.py:38: in <module>
>   from lepl.matchers.combine import And, DepthFirst, BreadthFirst, \
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:157: in load_module
>           py.builtin.exec_(co, mod.__dict__)
lepl/matchers/combine.py:45: in <module>
>   from lepl.matchers.core import Literal
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:157: in load_module
>           py.builtin.exec_(co, mod.__dict__)
lepl/matchers/core.py:36: in <module>
>   from lepl.stream.core import s_next, s_eq, s_empty, s_line
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:157: in load_module
>           py.builtin.exec_(co, mod.__dict__)
lepl/stream/core.py:43: in <module>
>   from lepl.support.lib import fmt
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:157: in load_module
>           py.builtin.exec_(co, mod.__dict__)
lepl/support/lib.py:33: in <module>
>   from logging import getLogger
/usr/local/lib/python3.2/logging/__init__.py:45: in <module>
>       import threading
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:141: in find_module
>               _make_rewritten_pyc(state, fn_pypath, pyc, co)
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:225: in _make_rewritten_pyc
>           if _write_pyc(co, fn, proc_pyc):
/usr/local/lib/python3.2/site-packages/pytest-2.2.1-py3.2.egg/_pytest/assertion/rewrite.py:170: in _write_pyc
>           fp = open(pyc, "wb")
E           IOError: [Errno 13] Permission denied: '/usr/local/lib/python3.2/__pycache__/threading.cpython-32-PYTEST.pyc.14692'
etc etc

This is particularly frustrating since I use pytest in Intellij Idea as a test runner - it's the only hing that seems to work (everything else wants file names with certain conventions that I haven't followed etc).

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/112

[pytestbot]

Original comment by BitBucket: andrewcooke, GitHub: andrewcooke:

---

I just checked and python3.1 works fine (pycache is new in 3.2).

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Are you using pytest version 2.2.1? If so, could you try out the dev version with "pip install -i http://pypi.testrun.org -U pytest"? Thanks, holger

[pytestbot]

Original comment by BitBucket: andrewcooke, GitHub: andrewcooke:

---

thanks for replying. yes, i was using 2.2.1. using 2.2.2.dev5 gives the same errors.

[pytestbot]

Original comment by BitBucket: andrewcooke, GitHub: andrewcooke:

---

since this is related to the python version, here are the details:
{{{

uname -a
Linux small.europa2189 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64 x86_64 x86_64 GNU/Linux
python3.2
Python 3.2.2 (default, Dec 20 2011, 21:47:22)
[GCC 4.6.2] on linux2
}}}

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

Could attach the log with from {{{py.test --debug}}}?

[pytestbot]

Original comment by BitBucket: andrewcooke, GitHub: andrewcooke:

---

here you go....

[pytestbot]

Original comment by BitBucket: andrewcooke, GitHub: andrewcooke:

---

the files being debugged (or very similar ones) are at http://code.google.com/p/lepl/source/browse/ so, for example, the file /home/andrew/projects/personal/lepl/lepl/src/lepl/support/lib.py which leads to threading.py (if i understand your debug logs correctly) is at http://code.google.com/p/lepl/source/browse/src/lepl/support/lib.py

[pytestbot]

Original comment by BitBucket: andrewcooke, GitHub: andrewcooke:

---

hi, some more information on this. i just noticed that i had a pytest.ini file (which must have been configured a long time ago, because it was excluding subversion directories and i have been using mercurial for as long as i can recall).

anyway, the problem goes away if i remove the line
{{{
python_files = *.py
}}}

unfortunately, because i don't use special names for test files, it then fails to pick up most of my tests. but it also no longer gives the error i reported.

so perhaps what i should be asking is - how do i configure pytest to work with the way that i have my code organised? each package has a {{{_test}}} directory that contains the tests, typically named after the modules in the parent directory. so, for example:
{{{
src/
+- foo/
+- bar.py
+- baz.py
+- _test/
+- bar.py - contains tests for foo.bar
+- baz.py - contains tests for foo.baz
}}}

the ini file i would like to use would look like this:
{{{
[pytest]
python_files = .py
python_classes = *Test
python_functions = test_
}}}

which, as described above, works fine with python 3.1, but not with 3.2.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

first of all, the permission denied issue remains an issue, pytest should not fail if it can't write y pyc file.

second, you should be able to use the patterns you specify. i think you could look into the "norecursedirs" config file option or adding the "--ignore=" command line option via the "addopts" config file option. The latter may be problematic because there is no way to construct a sensible path in the ini-file which would be independent from where you invoke it. If you try adding via addopts and things work fine when you call it from the dir where pytest.ini resides in then we can consider (in another issue) adding better support for specifying such a path independently from CWD.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

is this still an issue with python3.3 and pytest-2.3.3?

[pytestbot]

Original comment by Erik Bray (BitBucket: embray, GitHub: embray):

---

This is still an issue in pytest 2.3.5. I was able to work around the issue in my own code by patching _pytest.assertion.rewrite._write_pyc to catch permission errors and return False in that case.

(see astropy/astropy@0feb25a, specifically)

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

@erik Bray are you up for a little pull request for this including a test? You can use monkeypatching to simulate the problem i guess.

[pytestbot]

Original comment by Erik Bray (BitBucket: embray, GitHub: embray):

---

Sure thing. I'll see if I can come up with a test, though it would only be applicable on Python 3 :)

[pytestbot]

Original comment by Erik Bray (BitBucket: embray, GitHub: embray):

---

Done. See pull request #37; I can confirm that without the fix the test fails, and with the fix the test passes.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Merged in embray/pytest (pull request #37)

Adds a test for and fixes #112

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Adds a test for and fixes #112. If attempting to write to the pycache directory raises a permission error _write_pyc() should just return False to prevent any further write attempts.