Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytest 7.4.0 release broke addoption behaviour #11134

Closed
enrico-usai opened this issue Jun 23, 2023 · 8 comments
Closed

Pytest 7.4.0 release broke addoption behaviour #11134

enrico-usai opened this issue Jun 23, 2023 · 8 comments
Labels
stale status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity

Comments

@enrico-usai
Copy link

Hi!

In the AWS ParallelCluster project we are using pytest and adding some custom options to it:

Our tests started to fail after the automatic upgrade to 7.4.0 with an error like:

2023-06-23 12:41:56,473 - INFO - test_runner - Starting pytest in region ... with params ['-s', '-vv', '-l', '--tests-log-file=..., '--output-dir=...', '--key-name=...', '--key-path=...', ...]
...
ERROR: usage: test_runner.py [options] [file_or_dir] [file_or_dir] [...]
test_runner.py: error: unrecognized arguments: --tests-log-file=... --output-dir=... --key-name=... --key-path=... --stackname-suffix ...

So we had to pin the version to 7.3.2: aws/aws-parallelcluster#5425

Pip freeze

Installed packages:
Package                             Version
----------------------------------- -----------
assertpy                            1.1
attrs                               23.1.0
aws-cdk.assets                      1.204.0
aws-cdk.aws-acmpca                  1.204.0
aws-cdk.aws-apigateway              1.204.0
aws-cdk.aws-applicationautoscaling  1.204.0
aws-cdk.aws-autoscaling             1.204.0
aws-cdk.aws-autoscaling-common      1.204.0
aws-cdk.aws-autoscaling-hooktargets 1.204.0
aws-cdk.aws-batch                   1.204.0
aws-cdk.aws-certificatemanager      1.204.0
aws-cdk.aws-cloudformation          1.204.0
aws-cdk.aws-cloudfront              1.204.0
aws-cdk.aws-cloudwatch              1.204.0
aws-cdk.aws-codebuild               1.204.0
aws-cdk.aws-codecommit              1.204.0
aws-cdk.aws-codeguruprofiler        1.204.0
aws-cdk.aws-codestarnotifications   1.204.0
aws-cdk.aws-cognito                 1.204.0
aws-cdk.aws-dynamodb                1.204.0
aws-cdk.aws-ec2                     1.204.0
aws-cdk.aws-ecr                     1.204.0
aws-cdk.aws-ecr-assets              1.204.0
aws-cdk.aws-ecs                     1.204.0
aws-cdk.aws-efs                     1.204.0
aws-cdk.aws-elasticloadbalancing    1.204.0
aws-cdk.aws-elasticloadbalancingv2  1.204.0
aws-cdk.aws-events                  1.204.0
aws-cdk.aws-fsx                     1.204.0
aws-cdk.aws-globalaccelerator       1.204.0
aws-cdk.aws-iam                     1.204.0
aws-cdk.aws-imagebuilder            1.204.0
aws-cdk.aws-kinesis                 1.204.0
aws-cdk.aws-kms                     1.204.0
aws-cdk.aws-lambda                  1.204.0
aws-cdk.aws-logs                    1.204.0
aws-cdk.aws-route53                 1.204.0
aws-cdk.aws-route53-targets         1.204.0
aws-cdk.aws-s3                      1.204.0
aws-cdk.aws-s3-assets               1.204.0
aws-cdk.aws-sam                     1.204.0
aws-cdk.aws-secretsmanager          1.204.0
aws-cdk.aws-servicediscovery        1.204.0
aws-cdk.aws-signer                  1.204.0
aws-cdk.aws-sns                     1.204.0
aws-cdk.aws-sns-subscriptions       1.204.0
aws-cdk.aws-sqs                     1.204.0
aws-cdk.aws-ssm                     1.204.0
aws-cdk.aws-stepfunctions           1.204.0
aws-cdk.cloud-assembly-schema       1.204.0
aws-cdk.core                        1.204.0
aws-cdk.custom-resources            1.204.0
aws-cdk.cx-api                      1.204.0
aws-cdk.region-info                 1.204.0
aws-parallelcluster                 3.7.0
bcrypt                              4.0.1
boto3                               1.26.159
botocore                            1.29.159
cattrs                              23.1.2
certifi                             2023.5.7
cffi                                1.15.1
cfn-flip                            1.3.0
charset-normalizer                  3.1.0
click                               8.1.3
clickclick                          20.10.2
connexion                           2.13.1
constructs                          3.4.332
contourpy                           1.1.0
cryptography                        41.0.1
cycler                              0.11.0
defusedxml                          0.7.1
docopt                              0.6.2
exceptiongroup                      1.1.1
execnet                             1.9.0
fabric                              2.6.0
filelock                            3.12.2
Flask                               2.2.5
fonttools                           4.40.0
future                              0.18.3
idna                                3.4
importlib-metadata                  6.7.0
importlib-resources                 5.12.0
inflection                          0.5.1
iniconfig                           2.0.0
invoke                              1.7.3
itsdangerous                        2.1.2
Jinja2                              3.1.2
jmespath                            0.10.0
jsii                                1.84.0
jsonschema                          4.17.3
junitparser                         3.1.0
kiwisolver                          1.4.4
MarkupSafe                          2.1.3
marshmallow                         3.19.0
matplotlib                          3.7.1
numpy                               1.25.0
packaging                           23.1
paramiko                            3.2.0
pathlib2                            2.3.7.post1
pcluster-client                     1.0.0
pexpect                             4.8.0
Pillow                              9.5.0
pip                                 23.1.2
pluggy                              1.2.0
psutil                              5.9.5
ptyprocess                          0.7.0
publication                         0.0.3
py                                  1.11.0
pycparser                           2.21
pykwalify                           1.8.0
PyNaCl                              1.5.0
pyOpenSSL                           23.2.0
pyparsing                           3.1.0
pyrsistent                          0.19.3
pytest                              7.4.0
pytest-datadir                      1.4.1
pytest-html                         3.2.0
pytest-metadata                     3.0.0
pytest-rerunfailures                11.1.2
pytest-sugar                        0.9.7
pytest-xdist                        3.3.1
python-dateutil                     2.8.2
PyYAML                              5.4.1
requests                            2.31.0
retrying                            1.3.4
ruamel.yaml                         0.17.32
ruamel.yaml.clib                    0.2.7
s3transfer                          0.6.1
setuptools                          58.1.0
six                                 1.16.0
tabulate                            0.8.10
termcolor                           2.3.0
tomli                               2.0.1
troposphere                         4.3.2
typeguard                           2.13.3
typing_extensions                   4.6.3
untangle                            1.2.1
urllib3                             1.26.16
Werkzeug                            2.3.6
xmltodict                           0.13.0
zipp                                3.15.0
@bluetech
Copy link
Member

Hi @enrico-usai,

How do you invoke pytest?

@bluetech
Copy link
Member

Sorry, I see that you linked to it. I'll try and have a look.

@bluetech
Copy link
Member

bluetech commented Jun 23, 2023

If I am reading correctly (from a quick skim in github), it looks like you are running with --rootdir=./tests. I suspect this is causing the issue; basically the conftest.py file is outside of the rootdir so it's not found any more (see #11043, also mentioned in the changelog). The more "correct" rootdir is the aws-parallelcluster/tests/integration-tests/ path where all of the config files etc. are located. Can you try changing to that?

@enrico-usai
Copy link
Author

Thank you @bluetech for your analysis!

I'll try to fix this.

@Zac-HD Zac-HD added the status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity label Jun 29, 2023
@greschd
Copy link

greschd commented Jun 30, 2023

FWIW, in our case this was solved by adding an empty [tool.pytest.ini_options] to pyproject.toml so that pytest recognizes the rootdir.

As far as I can tell, in your case adding [pytest] to https://github.com/aws/aws-parallelcluster/blob/develop/tests/integration-tests/tox.ini might work.

This is the relevant part of the docs: https://docs.pytest.org/en/stable/reference/customize.html#finding-the-rootdir
Specifically:

Files will only be matched for configuration if:

  • pytest.ini: will always match and take precedence, even if empty.
  • pyproject.toml: contains a [tool.pytest.ini_options] table.
  • tox.ini: contains a [pytest] section.
  • setup.cfg: contains a [tool:pytest] section.

@github-actions
Copy link
Contributor

This issue is stale because it has been open for 14 days with no activity.

@github-actions github-actions bot added the stale label Jul 15, 2023
@github-actions
Copy link
Contributor

This issue was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 23, 2023
enrico-usai added a commit to enrico-usai/aws-parallelcluster that referenced this issue Aug 7, 2023
We pinned pytest because usage of custom args stopped to work with pytest 3.7.4.
At the end it was a configuration issue: we were missing to specify which is the pytest root dir.
Adding a `[pytest]` section in the `tox.ini` file permits pytest to discover the right root dir
and so to use custom arguments.

### Tests

Without the pin and before this patch tests were failing with:
```
ERROR: usage: test_runner.py [options] [file_or_dir] [file_or_dir] [...]
test_runner.py: error: unrecognized arguments: --tests-log-file=/tmp//...eu-west-1.log --output-dir=/tmp/...eu-west-1
    --key-name=... --key-path=...usai-common.pem --stackname-suffix --tests-config-file=/tmp/...tests_config.yaml --regions eu-west-1
    --ami-owner ... --vpc-stack integ-tests-vpc-...-360 --no-delete ...
  inifile: None
  rootdir: .../aws-parallelcluster/tests/integration-tests/tests
```

Now tests are executed correctly.

### References
* pytest-dev/pytest#11134
* https://docs.pytest.org/en/stable/reference/customize.html#finding-the-rootdir

Signed-off-by: Enrico Usai <usai@amazon.com>
enrico-usai added a commit to aws/aws-parallelcluster that referenced this issue Aug 7, 2023
We pinned pytest because usage of custom args stopped to work with pytest 3.7.4.
At the end it was a configuration issue: we were missing to specify which is the pytest root dir.
Adding a `[pytest]` section in the `tox.ini` file permits pytest to discover the right root dir
and so to use custom arguments.

### Tests

Without the pin and before this patch tests were failing with:
```
ERROR: usage: test_runner.py [options] [file_or_dir] [file_or_dir] [...]
test_runner.py: error: unrecognized arguments: --tests-log-file=/tmp//...eu-west-1.log --output-dir=/tmp/...eu-west-1
    --key-name=... --key-path=...usai-common.pem --stackname-suffix --tests-config-file=/tmp/...tests_config.yaml --regions eu-west-1
    --ami-owner ... --vpc-stack integ-tests-vpc-...-360 --no-delete ...
  inifile: None
  rootdir: .../aws-parallelcluster/tests/integration-tests/tests
```

Now tests are executed correctly.

### References
* pytest-dev/pytest#11134
* https://docs.pytest.org/en/stable/reference/customize.html#finding-the-rootdir

Signed-off-by: Enrico Usai <usai@amazon.com>
@enrico-usai
Copy link
Author

@greschd I can confirm that adding the [pytest] section in the tox.ini file solved the issue: aws/aws-parallelcluster#5587
Thanks for the suggestion.

enrico-usai added a commit to enrico-usai/aws-parallelcluster that referenced this issue Aug 9, 2023
We pinned pytest because usage of custom args stopped to work with pytest 3.7.4.
At the end it was a configuration issue: we were missing to specify which is the pytest root dir.
Adding a `[pytest]` section in the `tox.ini` file permits pytest to discover the right root dir
and so to use custom arguments.

### Tests

Without the pin and before this patch tests were failing with:
```
ERROR: usage: test_runner.py [options] [file_or_dir] [file_or_dir] [...]
test_runner.py: error: unrecognized arguments: --tests-log-file=/tmp//...eu-west-1.log --output-dir=/tmp/...eu-west-1
    --key-name=... --key-path=...usai-common.pem --stackname-suffix --tests-config-file=/tmp/...tests_config.yaml --regions eu-west-1
    --ami-owner ... --vpc-stack integ-tests-vpc-...-360 --no-delete ...
  inifile: None
  rootdir: .../aws-parallelcluster/tests/integration-tests/tests
```

Now tests are executed correctly.

### References
* pytest-dev/pytest#11134
* https://docs.pytest.org/en/stable/reference/customize.html#finding-the-rootdir

Signed-off-by: Enrico Usai <usai@amazon.com>
kkolur5 pushed a commit to kkolur5/aws-parallelcluster that referenced this issue Aug 11, 2023
We pinned pytest because usage of custom args stopped to work with pytest 3.7.4.
At the end it was a configuration issue: we were missing to specify which is the pytest root dir.
Adding a `[pytest]` section in the `tox.ini` file permits pytest to discover the right root dir
and so to use custom arguments.

### Tests

Without the pin and before this patch tests were failing with:
```
ERROR: usage: test_runner.py [options] [file_or_dir] [file_or_dir] [...]
test_runner.py: error: unrecognized arguments: --tests-log-file=/tmp//...eu-west-1.log --output-dir=/tmp/...eu-west-1
    --key-name=... --key-path=...usai-common.pem --stackname-suffix --tests-config-file=/tmp/...tests_config.yaml --regions eu-west-1
    --ami-owner ... --vpc-stack integ-tests-vpc-...-360 --no-delete ...
  inifile: None
  rootdir: .../aws-parallelcluster/tests/integration-tests/tests
```

Now tests are executed correctly.

### References
* pytest-dev/pytest#11134
* https://docs.pytest.org/en/stable/reference/customize.html#finding-the-rootdir

Signed-off-by: Enrico Usai <usai@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity
Projects
None yet
Development

No branches or pull requests

4 participants