Proper way to build setuptools-scm without .git directory?

[Ecordonnier]

Hi,
I would appreciate some pointers to solve this issue. I'm not sure whether it is an issue in setuptools-scm itself or not, but I'm missing a lot of context (I don't have an advanced knowledge of python), so maybe the issue is obvious to you.
I am consuming setuptools-scm as part of yocto, which is a build-system for embedded-systems which builds almost everything from scratch (including python and other build-tools).
As part of this build-system, a file describes how to build setuptools-scm.
It works more or less this way:

• This tarball gets downloaded: https://files.pythonhosted.org/packages/source/s/setuptools-scm/setuptools-scm-8.0.4.tar.gz;downloadfilename=setuptools-scm-8.0.4.tar.gz
• then there is a call to "nativepython3 -m build --no-isolation --wheel --outdir ${PEP517_WHEEL_PATH} ${PEP517_SOURCE_PATH} ${PEP517_BUILD_OPTS}"

Looking at the call-stack, I can see that setuptools's function walk_revctrl calls those functions from setuptools_scm:

File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/sdist.py", line 16, in walk_revctrl
    yield from ep.load()(dirname)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/__init__.py", line 103, in find_files
    res: list[str] = command(path)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py", line 103, in git_find_files
    git_files, git_dirs = _git_ls_files_and_dirs(toplevel)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py", line 74, in _git_ls_files_and_dirs
    for line in traceback.format_stack(): 

The issue is that the setuptools-scm tarball does not contain a .git directory. Thus this code in ./src/setuptools_scm/_file_finders/git.py is detecting the root of my git repository (which is completely unrelated and quite huge):

def git_find_files(path: _t.PathT = "") -> list[str]:
    toplevel = _git_toplevel(os.fspath(path))

This code then calls "git archive --prefix toplevel HEAD" (function _git_ls_files_and_dirs) on my huge repository, and takes an hour to complete and consumes a massive amount of memory.

Do you know where the issue is? Is the logic used to build setuptools-scm wrong? Or is the issue in setuptools-scm itself?

Thanks a lot

[Ecordonnier]

full-call-stack.txt

[RonnyPfannschmidt]

This one will need a opt out hack in sdtuptools_scm

It boils down to a upstream issue with setuptools

The file finder api in setuptools is completely free of context and so setuptools_scm has to do rotten braindead stuff to work at all

[rossburton]

Had a quick look at the code. Is there a reason that _git_toplevel doesn't just use git rev-parse --show-toplevel?

(edit: #1105)

[rossburton]

So the standard solution to this sort of problem is to use GIT_CEILING_DIRECTORIES to tell git where to stop looking up the tree when hunting for a repository. Our build system sets this, but setuptools-scm explicitly prunes the environment when running git commands of any variable the starts with GIT_.

[Ecordonnier]

This issue seems to be almost identical to #353 , however this comment mentioned:

GIT_CEILING_DIRECTORIES is ineffective because setuptools-scm is explicitly passing in the --prefix

whereas during my testing GIT_CEILING_DIRECTORIES actually solved the issue.

Thus I'm wondering if we had slightly different use-cases? Or whether the comment was incorrect?