ctypes.util incorrectly fails for libraries without DT_SONAME

[JeremyHuntwork]

BPO	21622
Nosy	@vadmium, @ncopa, @thomwiggers, @tianon, @jcastillo2nd
PRs	bpo-21622: ctypes.util find_library walk LD_LIBRARY_PATH #10453 [2.7] bpo-21622 ctypes.util find_library walk LD_LIBRARY_PATH (GH-10453) #10455 bpo-21622: ctypes.util find_library walk LD_LIBRARY_PATH #10460 [3.7] bpo-21622 ctypes.util find_library walk LD_LIBRARY_PATH (GH-10460) #10461 [2.7] bpo-21622 ctypes.util find_library walk LD_LIBRARY_PATH (GH-10460) #10462 bpo-21622: ctypes.util find_library walk LD_LIBRARY_PATH #16940 gh-65821: Fix ctypes.util.find_library with musl #18380

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-06-01.03:28:22.092>
labels = ['ctypes', 'type-bug']
title = 'ctypes.util incorrectly fails for libraries without DT_SONAME'
updated_at = <Date 2020-04-20.08:40:25.315>
user = 'https://bugs.python.org/JeremyHuntwork'

bugs.python.org fields:

activity = <Date 2020-04-20.08:40:25.315>
actor = 'ncopa'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2014-06-01.03:28:22.092>
creator = 'Jeremy.Huntwork'
dependencies = []
files = []
hgrepos = []
issue_num = 21622
keywords = ['patch']
message_count = 8.0
messages = ['219481', '264141', '264148', '274361', '274384', '321772', '330668', '366813']
nosy_count = 8.0
nosy_names = ['martin.panter', 'ncopa', 'Jeremy.Huntwork', 'Kylie McClain', 'twiggers', 'Richard Eames', 'tianon', 'jcastillo2nd']
pr_nums = ['10453', '10455', '10460', '10461', '10462', '16940', '18380']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21622'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[JeremyHuntwork]

On my system, the C library (musl) intentionally does not include a SONAME entry.

This method in particular fails: http://hg.python.org/cpython/file/076705776bbe/Lib/ctypes/util.py#l133

The function seems to jump through some hoops which may not be necessary. Is there a reason for wanting particularly to use the SONAME entry for the lib?

In my system the following works as a replacement for _get_soname:

return os.path.basename(os.path.realpath(f))

[KylieMcClain]

This is still a problem on musl distributions as of 2.7.11. Are there any plans to fix this?

Patch used by Alpine Linux:
http://git.alpinelinux.org/cgit/aports/tree/main/python/musl-find_library.patch

[vadmium]

On Linux, the find_library() function is documented to return “the filename of the library file”, but in reality it seems it return the soname, and therefore breaks if there is no soname. I do not know why we extract the soname, but it has been that way at least since ctypes was added to Python 2.5.

What do you intend to do with the result of find_library()? See also bpo-9998, especially about searching LD_LIBRARY_PATH. I am struggling to see robust use cases for find_library().

[thomwiggers]

This bug is still present in Python 3.5.

It breaks, probably among other things, this package: ahupp/python-magic#114.

[vadmium]

I think it may be reasonable to change the code to return the library file name if no soname can be found.

In the long term, I think always returning the filename rather than soname might be reasonable. Or even returning the full path, which we tried in bpo-21042 (but rolled back because the solution so far is not consistent across platforms).

[tianon]

This was reported on the Docker image for Python in docker-library/python#111, with the note that it affects the Twisted inotify implementation, so it'd be really neat to see a proper patch in Python (instead of the very musl-/Alpine-assuming patch found in https://github.com/alpinelinux/aports/blob/202f4bea916b0cf974b38ced96ab8fca0b192e3f/main/python2/musl-find_library.patch). <3

[jcastillo2nd]

The PR 10460 ( for 3.8 ) patches the search attempts to leverage LD_LIBRARY_PATH for the Linux case and catches the case after SONAME, gcc and ldconfig behaviors fail.

While this may not be set in all environments ( like the musl based Alpine docker image ) setting the environment variable is within the user's control and when properly set does return found libraries.

PR 10461 and PR 10462 are closed for now, as backports to 2.7 and 3.7 need to have the 3.8 accepted first before being reviewed.

[ncopa]

I create a PR for this issue.

It would be nice to have it reviewed.
#18380

Thanks!