sphinx-apidoc ignores subfolder with filename starting with underscore as subpackage in toctree for main package

[a-shrivastava]

The Problem

Assume the following folder structure:

repo_root/
+-- docs/
|   +-- Makefile
|   +-- source/
|       +-- conf.py
+-- pkgA
|   +-- __init__.py
|   +-- fileA1.py
|   +-- pkgB
|   |   +-- __init__.py
|   |   +-- fileB1.py
|   +-- pkgC
|   |   +-- __init__.py
|   |   +-- fileC1.py
|   |   +-- _fileC2.py
|   +-- zfileA2.py

When I run sphinx-apidoc -o source .. --force from repo_root/docs/, the Subpackages section of repo_root/docs/source/pkA.rst is generated as follows:

pkgA package
============

Subpackages
-----------

.. toctree::

    pkgA.pkgB

The above subpackages toctree is clearly missing pkgA.pkgC as a subpackage of pkgA.

Likely Reason

When attempting to generate the file pkgA.rst, a call is made to create_package_file() for pkgA, which calls shall_skip() for each subfolder in pkgA\ to check if it is a subpackage.
The [shall_skip(module, opts, excludes)] (

sphinx/sphinx/ext/apidoc.py

Line 193 in b90c257

def shall_skip(module, opts, excludes=[]):

) takes 3 arguments - the first one module being the absolute path to the __init__.py file. For pkgC, this would mean module = <path-to-repo_root/pkgA/pkgC/__init__.py. The function has three checks, if either fails then the folder is not included as a subpackage.
The third check basically tests if the filename part of os.path.basename(module) == '__init__.py', if not then it should not start with an underscore. However the value of module gets altered during the 2nd check. See
for module in glob.glob(path.join(basemodule, '*.py')). Since [glob.glob()] (https://docs.python.org/3/library/glob.html) returns list of files in arbitrary order*, it is possible that the value of module at the end of the 2nd check (and entering into the 3rd check) is the path to a filename starting with an underscore (and indeed this is what happened for me!). In the presented example, module = <path-to-repo_root/pkgA/pkgC/_fileC2.py after the 2nd check and before starting the 3rd check; thus, failing the 3rd check and pkgC is not included as a subpackage of pkgA.
* This is because glob.glob() gets the file list using a call to os.scandir() that returns files listed in arbitrary order.

Suggested Solution

In lines

sphinx/sphinx/ext/apidoc.py

Line 205 in b90c257

for module in glob.glob(path.join(basemodule, '*.py')):

and

sphinx/sphinx/ext/apidoc.py

Line 205 in b90c257

for module in glob.glob(path.join(basemodule, '*.py')):

of shall_skip(), change name of variable module to a different name, say, module_.

[tk0miya]

Fixed by #4935.
Thank you for reporting!

[guglielmosanchini]

The problem persists if the package itself contains an underscore in its name. I know that, PEP8 discourages the presence of underscores in package names:

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

But could it be possibile to fix this too?
Thanks

[tk0miya]

@guglielmosanchini This issue was closed 2 years ago. Could you file a new issue, please?