Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3_bucket: typing requires Python>=3.9 #2287

Closed
1 task done
dezeroku opened this issue Sep 9, 2024 · 1 comment · Fixed by #2288
Closed
1 task done

s3_bucket: typing requires Python>=3.9 #2287

dezeroku opened this issue Sep 9, 2024 · 1 comment · Fixed by #2288

Comments

@dezeroku
Copy link

dezeroku commented Sep 9, 2024

Summary

Changes introduced in #2057 use indices and "base types" for typing, e.g. -> tuple[dict,dict] that was introduced in Python3.9, while docs still specify Python3.6 as the minimum supported version.

Switching to typing... types or bumping minimal Python version is probably the way to go with this issue.

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

$ ansible --version
ansible [core 2.17.3]
  config file = /home/jpiatkowski/opera/gitlab/ansible/ansible.cfg
  configured module search path = ['/home/jpiatkowski/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jpiatkowski/opera/gitlab/ansible/.venv/lib/python3.12/site-packages/ansible
  ansible collection location = /home/jpiatkowski/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jpiatkowski/opera/gitlab/ansible/.venv/bin/ansible
  python version = 3.12.5 (main, Aug  9 2024, 08:20:41) [GCC 14.2.1 20240805] (/home/jpiatkowski/opera/gitlab/ansible/.venv/bin/python)
  jinja version = 3.1.4
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

# /home/jpiatkowski/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               8.2.1  
ansible.utils                            2.0.2  
community.crypto                         1.6.1  
community.general                        7.1.0  
kewlfft.aur                              0.10.0 

# /usr/lib/python3.12/site-packages/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               8.1.0  
ansible.netcommon                        6.1.3  
ansible.posix                            1.5.4  
ansible.utils                            4.1.0  
ansible.windows                          2.4.0  
arista.eos                               9.0.0  
awx.awx                                  24.6.1 
azure.azcollection                       2.6.0  
check_point.mgmt                         5.2.3  
chocolatey.chocolatey                    1.5.1  
cisco.aci                                2.10.1 
cisco.asa                                5.0.1  
cisco.dnac                               6.17.1 
cisco.intersight                         2.0.10 
cisco.ios                                8.0.0  
cisco.iosxr                              9.0.0  
cisco.ise                                2.9.3  
cisco.meraki                             2.18.1 
cisco.mso                                2.9.0  
cisco.nxos                               8.1.0  
cisco.ucs                                1.10.0 
cloud.common                             3.0.0  
cloudscale_ch.cloud                      2.4.0  
community.aws                            8.0.0  
community.ciscosmb                       1.0.9  
community.crypto                         2.21.1 
community.digitalocean                   1.26.0 
community.dns                            3.0.3  
community.docker                         3.12.1 
community.general                        9.3.0  
community.grafana                        1.9.1  
community.hashi_vault                    6.2.0  
community.hrobot                         2.0.1  
community.library_inventory_filtering_v1 1.0.1  
community.libvirt                        1.3.0  
community.mongodb                        1.7.6  
community.mysql                          3.9.0  
community.network                        5.0.3  
community.okd                            3.0.1  
community.postgresql                     3.4.1  
community.proxysql                       1.6.0  
community.rabbitmq                       1.3.0  
community.routeros                       2.18.0 
community.sap_libs                       1.4.2  
community.sops                           1.8.2  
community.vmware                         4.5.0  
community.windows                        2.2.0  
community.zabbix                         2.5.1  
containers.podman                        1.15.4 
cyberark.conjur                          1.3.0  
cyberark.pas                             1.0.27 
dellemc.enterprise_sonic                 2.4.0  
dellemc.openmanage                       9.5.0  
dellemc.powerflex                        2.5.0  
dellemc.unity                            2.0.0  
f5networks.f5_modules                    1.30.1 
fortinet.fortimanager                    2.6.0  
fortinet.fortios                         2.3.7  
frr.frr                                  2.0.2  
google.cloud                             1.3.0  
grafana.grafana                          5.4.0  
hetzner.hcloud                           3.1.1  
ibm.qradar                               3.0.0  
ibm.spectrum_virtualize                  2.0.0  
ibm.storage_virtualize                   2.4.1  
ieisystem.inmanage                       2.0.0  
infinidat.infinibox                      1.4.5  
infoblox.nios_modules                    1.6.1  
inspur.ispim                             2.2.3  
inspur.sm                                2.3.0  
junipernetworks.junos                    8.0.0  
kaytus.ksmanage                          1.2.2  
kubernetes.core                          3.2.0  
kubevirt.core                            1.5.0  
lowlydba.sqlserver                       2.3.3  
microsoft.ad                             1.6.0  
netapp.cloudmanager                      21.22.1
netapp.ontap                             22.12.0
netapp.storagegrid                       21.12.0
netapp_eseries.santricity                1.4.0  
netbox.netbox                            3.19.1 
ngine_io.cloudstack                      2.3.0  
ngine_io.exoscale                        1.1.0  
openstack.cloud                          2.2.0  
openvswitch.openvswitch                  2.1.1  
ovirt.ovirt                              3.2.0  
purestorage.flasharray                   1.30.2 
purestorage.flashblade                   1.18.0 
sensu.sensu_go                           1.14.0 
splunk.es                                3.0.0  
t_systems_mms.icinga_director            2.0.1  
telekom_mms.icinga_director              2.1.2  
theforeman.foreman                       4.1.0  
vmware.vmware                            1.4.0  
vmware.vmware_rest                       3.0.1  
vultr.cloud                              1.13.0 
vyos.vyos                                4.1.0  
wti.remote                               1.0.5  

AWS SDK versions

$ pip show boto boto3 botocore
Name: boto3
Version: 1.33.13
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /opt/python_ceph/lib/python3.7/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.33.13
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /opt/python_ceph/lib/python3.7/site-packages
Requires: python-dateutil, urllib3, jmespath
Required-by: s3transfer, boto3

Configuration

$ ansible-config dump --only-changed

OS / Environment

Target OS:

  • Debian GNU/Linux 10 (buster)
$ python3 --version
Python 3.7.3

Steps to Reproduce

I am working with Ceph backend locally, but this issue should be reproducible with any configuration. The failure seems to happen at the Python parsing stage

  - name: Create bucket
    s3_bucket:
      access_key: REDACTED
      ceph: true
      endpoint_url: REDACTED
      name: "dezeroku-test-bucket"
      public_access:
        block_public_acls: false
        block_public_policy: true
        ignore_public_acls: true
        restrict_public_buckets: true
      secret_key: REDACTED
      state: present
      versioning: true

Expected Results

Expected:

  • Bucket gets created

Actual Results

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: 'type' object is not subscriptable
fatal: [REDACTED]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 48, in invoke_module\n  File \"/usr/lib/python3.7/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.7/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_s3_bucket_payload_fapgl152/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py\", line 571, in <module>\nTypeError: 'type' object is not subscriptable\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

The "important" part:

ansible_collections/amazon/aws/plugins/modules/s3_bucket.py\", line 571, in <module>\nTypeError: 'type' object is not subscriptable\n"

Manually removing all tuple[...,...] typing from the collection and installing it locally allows Ansible to proceed

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
@tremble
Copy link
Contributor

tremble commented Sep 10, 2024

@dezeroku,

Thanks for taking the time to report this issue. Unfortunately we stopped running some of our tests against Python < 3.9 to try and bring the test matrix down to something sensible, so this slipped through.

FYI this collection dropped support for Python < 3.7 with release 7.0.0. While this isn't in line with the core Ansible requirements, the AWS SDK we use (botocore) has dropped support for older versions of Python, forcing our hand.

patchback bot pushed a commit that referenced this issue Sep 10, 2024
Fixes: #2287
SUMMARY
Support for type hinting generics in standard collections was first added in Python 3.9.
We're currently not running the sanity tests against Python 3.7 (including the compilation/import tests), so we missed this.  (We currently test 3.9-3.12)
Using quay.io/ansible/default-test-container:8.12.0 locally to run the tests against 3.7 I see:
(2.7 and 3.6 are skipped because this collection doesn't support them)
Running sanity test "import" on Python 2.7
Running sanity test "import" on Python 3.6
Running sanity test "import" on Python 3.7
ERROR: Found 1 import issue(s) on python 3.7 which need to be resolved:
ERROR: plugins/modules/s3_bucket.py:571:0: traceback: TypeError: 'type' object is not subscriptable
See documentation for help: https://docs.ansible.com/ansible-core/2.16/dev_guide/testing/sanity/import.html
Running sanity test "import" on Python 3.8
ERROR: Found 1 import issue(s) on python 3.8 which need to be resolved:
ERROR: plugins/modules/s3_bucket.py:571:0: traceback: TypeError: 'type' object is not subscriptable
See documentation for help: https://docs.ansible.com/ansible-core/2.16/dev_guide/testing/sanity/import.html
Running sanity test "import" on Python 3.9
Running sanity test "import" on Python 3.10
Running sanity test "import" on Python 3.11
Running sanity test "import" on Python 3.12

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
s3_bucket
ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis
(cherry picked from commit 55460c9)
softwarefactory-project-zuul bot pushed a commit that referenced this issue Sep 10, 2024
This is a backport of PR #2288 as merged into main (55460c9).
Fixes: #2287
SUMMARY
Support for type hinting generics in standard collections was first added in Python 3.9.
We're currently not running the sanity tests against Python 3.7 (including the compilation/import tests), so we missed this.  (We currently test 3.9-3.12)
Using quay.io/ansible/default-test-container:8.12.0 locally to run the tests against 3.7 I see:
(2.7 and 3.6 are skipped because this collection doesn't support them)
Running sanity test "import" on Python 2.7
Running sanity test "import" on Python 3.6
Running sanity test "import" on Python 3.7
ERROR: Found 1 import issue(s) on python 3.7 which need to be resolved:
ERROR: plugins/modules/s3_bucket.py:571:0: traceback: TypeError: 'type' object is not subscriptable
See documentation for help: https://docs.ansible.com/ansible-core/2.16/dev_guide/testing/sanity/import.html
Running sanity test "import" on Python 3.8
ERROR: Found 1 import issue(s) on python 3.8 which need to be resolved:
ERROR: plugins/modules/s3_bucket.py:571:0: traceback: TypeError: 'type' object is not subscriptable
See documentation for help: https://docs.ansible.com/ansible-core/2.16/dev_guide/testing/sanity/import.html
Running sanity test "import" on Python 3.9
Running sanity test "import" on Python 3.10
Running sanity test "import" on Python 3.11
Running sanity test "import" on Python 3.12

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
s3_bucket
ADDITIONAL INFORMATION

Reviewed-by: Mark Chappell
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants