-
-
Notifications
You must be signed in to change notification settings - Fork 732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix parsing of git output with unusual characters #509
Conversation
I used a ternary operator in the |
testing this is important, it's the entire reason it's a separate function and not just |
Fair enough, I'll write some tests 🙂 |
Test correctly failing without the fix:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On Windows, all files are "executable". Therefore, to know if a file is supposed to be executed, we check how its attributes were recorded by git: we run a `git ls-files` command in a subprocess. By default, this command outputs information on multiple lines (file and their data separated by newlines). When a file contains an unusual character, the character is escaped with an integer sequence (such as `\303\261`), and git wraps the whole filename in double-quotes because of the backslashes. It breaks the current code because we try to open the filename containing the double-quotes: it doesn't exist, of course. Instead of trying to fix this special case by removing the double-quotes, and breaking other cases (a double-quote is a valid filename character on Linux), we tell git to separate each item with the null character `\0` instead of a new line `\n`, with the option `-z`. With this option, git doesn't escape unusual characters with integer sequence, so the output is fixed, and we parse it by splitting on `\0` instead of `\n`. Fixes #508.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows haradcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows hardcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Remove external python dependencies (yaml, plumbum) from test task/migration files. These are available on Linux because it gets the python env from the venv, but on Windows, it uses the main python interpreter and breaks. After all, that's not very important here. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing. - Disable pre-commit on CI on Windows. - Require python 3.8 on Windows, where `tempfile` supports autoremoving directories with read-only files.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows hardcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Remove external python dependencies (yaml, plumbum) from test task/migration files. These are available on Linux because it gets the python env from the venv, but on Windows, it uses the main python interpreter and breaks. After all, that's not very important here. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing. - Disable pre-commit on CI on Windows. - Require python 3.8 on Windows, where `tempfile` supports autoremoving directories with read-only files.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows hardcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Remove external python dependencies (yaml, plumbum) from test task/migration files. These are available on Linux because it gets the python env from the venv, but on Windows, it uses the main python interpreter and breaks. After all, that's not very important here. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing. - Disable pre-commit on CI on Windows. - Require python 3.8 on Windows, where `tempfile` supports autoremoving directories with read-only files.
- Ignore errors when executing `shutil.rmtree()`, because it seems like it's common to fail when deleting git repos on Windows, and since these are temp files, we don't really care that much there's garbage left. Any good OS should clean the temp folders automatically. - Always find Jinja templates in `PosixPath` mode. - Ignore `OSError` when trying to enter a possibly git root directory. This is yielded by Windows when the path is a URL and we don't really care about it. - Fix some tests with non-windows hardcoded stuff. - Fix a test that was using a Bash script. Modified to be Python, which should work fine cross-system. - Remove external python dependencies (yaml, plumbum) from test task/migration files. These are available on Linux because it gets the python env from the venv, but on Windows, it uses the main python interpreter and breaks. After all, that's not very important here. - Do not modify EOL in CI. - Use python executable in tests instead of python3. - Update pre-commit versions to include pre-commit/pre-commit-hooks#509. - Disable autorebasing. - Disable pre-commit on CI on Windows. - Require python 3.8 on Windows, where `tempfile` supports autoremoving directories with read-only files.
I used the online editor to send the patch. Let me know if you'd like I add a test for it!
We could also consider moving the
zsplit
in theutils
module.Commit message:
On Windows, all files are "executable". Therefore, to know if a file is supposed to be executed, we check how its attributes were recorded by git: we run a
git ls-files
command in a subprocess.By default, this command outputs information on multiple lines (file and their data separated by newlines). When a file contains an unusual character, the character is escaped with an integer sequence (such as
\303\261
), and git wraps the whole filenamein double-quotes because of the backslashes. It breaks the current code because we try to open the filename containing the double-quotes: it doesn't exist, of course.
Instead of trying to fix this special case by removing the double-quotes, and breaking other cases (a double-quote is a valid filename character on Linux), we tell git to separate each item with the null character
\0
instead of a new line\n
, with the option-z
. With this option, git doesn't escape unusual characters with integer sequence, so the output is fixed, and we parse it by splitting on\0
instead of\n
.Fixes #508.