rtree/core not detecting conda environment

[nmaxwell]

I'm using Conda on Windows 10 and getting the following error on import (of geopandas)

OSError: could not find or load spatialindex_c-64.dll

The error originates from line 117 in core.py:

    lib_name = '%s-%s.dll' % (base_name, arch)
    if 'SPATIALINDEX_C_LIBRARY' in os.environ:
        lib_path, lib_name = os.path.split(os.environ['SPATIALINDEX_C_LIBRARY'])
        rt = _load_library(lib_name, ctypes.cdll.LoadLibrary, (lib_path,))
    elif 'conda' in sys.version:
        lib_path = os.path.join(sys.prefix, "Library", "bin")
        rt = _load_library(lib_name, ctypes.cdll.LoadLibrary, (lib_path,))
    else:
        rt = _load_library(lib_name, ctypes.cdll.LoadLibrary)
    if not rt:
        raise OSError("could not find or load %s" % lib_name)

the problem is that sys.version evaluates to

'3.8.1 (default, Jan 8 2020, 15:55:49) [MSC v.1916 64 bit (AMD64)]'

so it seems like checking for the presence of 'conda' in sys.version may not be a reliable way to detect being in a Conda environment.

Maybe the better solution is to check if 'os.path.join(sys.prefix, "Library", "bin")' exists, or to change the entire logic to

    lib_name = '%s-%s.dll' % (base_name, arch)
    if 'SPATIALINDEX_C_LIBRARY' in os.environ:
        lib_path, lib_name = os.path.split(os.environ['SPATIALINDEX_C_LIBRARY'])
        rt = _load_library(lib_name, ctypes.cdll.LoadLibrary, (lib_path,))
    else:
        _dllpaths = ('', os.path.join(sys.prefix, "Library", "bin"))
        rt = _load_library(lib_name, ctypes.cdll.LoadLibrary, _dllpaths)
    if not rt:
        raise OSError("could not find or load %s" % lib_name)

Thanks,
-nick

[nmaxwell]

A temporary hack is to simply

sys.version = sys.version + " (conda)"

before importing geopandas or rtree

[sr-murthy]

I don't run Conda, but have you tried import platform; platform.platform? If not then a more direct approach is to run which python via a shell command, and to parse for conda in the Python path that you get from running it. e.g. I get /path/to/myenv/bin/python when I run this, which indicates I'm running in a standard virtual env. I am guessing with Conda you would get a path that contains the string "conda".

[nmaxwell]

platform.platform()
'Windows-10-10.0.17763-SP0'

Not much there.

I am guessing with Conda you would get a path that contains the string "conda".

Not necessarily, you can create a Conda environment with an arbitrary prefix. In our case:

C:\Users\my-user-name\AppData\Local\my-project\python.exe

[hobu]

I wonder if we don't need this at all. Newer ctypes.find_library should resolve the library just fine without the special compensation for Conda's paths.

[hobu]

Ok, I did a bunch of rummaging and figured this out. ctypes significantly changed how LoadLibrary does DLL finding in Python 3.8. In short, applications are supposed add DLL search paths themselves now, instead of being implicitly added via PATH. The previous conda hack was a symptom of that problem, where we were trying to give a full path to the DLL.

I still need to do some research how to appropriately do this without wrecking DLL resolution for other applications, but at least I know what this problem is now.

https://bugs.python.org/issue36085#msg336353

[hobu]

Here's what conda is doing to the DLL loading environment

https://github.com/conda-forge/python-feedstock/blob/master/recipe/patches/0022-Add-CondaEcosystemModifyDllSearchPath.patch

[hobu]

If you do set CONDA_DLL_SEARCH_MODIFICATION_ENABLE =1 it should work.

[hobu]

conda-forge/python-feedstock#307 also describes what is going on.