Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

X | Y union syntax breaks GenericModel #4146

Closed
ntaylorwss opened this issue Jun 9, 2022 · 9 comments
Closed

X | Y union syntax breaks GenericModel #4146

ntaylorwss opened this issue Jun 9, 2022 · 9 comments
Labels
bug V1 Bug related to Pydantic V1.X

Comments

@ntaylorwss
Copy link

ntaylorwss commented Jun 9, 2022

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.9.1
            pydantic compiled: True
                 install path: /home/paystone/.local/lib/python3.10/site-packages/pydantic
               python version: 3.10.2 (main, Jan 29 2022, 02:55:36) [GCC 10.2.1 20210110]
                     platform: Linux-5.18.2-arch1-1-x86_64-with-glibc2.31
     optional deps. installed: ['typing-extensions']

Python 3.10 incorporated PEP 604, which added syntax for writing Unions as X | Y, instead of typing.Union[X, Y]. It seems that when this syntax is used in combination with TypeVars as part of a GenericModel, the binding of TypeVars to their types fails.

Below is first a working example using typing.Union, followed by a non-working example using | syntax, with the stack trace. The example seeks to construct a Relationship GenericModel, and a BidirectionalMultiRelationship model which contains a list of either Relationship[A, B] or Relationship[B, A] models:

from typing import Union, Generic, TypeVar
from pydantic.generics import GenericModel

SourceT = TypeVar("SourceT")
TargetT = TypeVar("TargetT")

class Relationship(GenericModel, Generic[SourceT, TargetT]):
    source: SourceT
    target: TargetT

class BidirectionalMultiRelationship(GenericModel, Generic[SourceT, TargetT]):
    relationships: list[Union[Relationship[SourceT, TargetT], Relationship[TargetT, SourceT]]]
    
BidirectionalMultiRelationship[str, int]
from typing import Generic, TypeVar
from pydantic.generics import GenericModel

SourceT = TypeVar("SourceT")
TargetT = TypeVar("TargetT")

class Relationship(GenericModel, Generic[SourceT, TargetT]):
    source: SourceT
    target: TargetT

class BidirectionalMultiRelationship(GenericModel, Generic[SourceT, TargetT]):
    relationships: list[Relationship[SourceT, TargetT] | Relationship[TargetT, SourceT]]
    
BidirectionalMultiRelationship[str, int]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [2], in <cell line: 14>()
     11 class BidirectionalMultiRelationship(GenericModel, Generic[SourceT, TargetT]):
     12     relationships: Relationship[SourceT, TargetT] | Relationship[TargetT, SourceT]
---> 14 BidirectionalMultiRelationship[str, int]

File ~/.local/lib/python3.10/site-packages/pydantic/generics.py:137, in GenericModel.__class_getitem__(cls, params)
    133     _generic_types_cache[(cls, params[0])] = created_model
    135 # Recursively walk class type hints and replace generic typevars
    136 # with concrete types that were passed.
--> 137 _prepare_model_fields(created_model, fields, instance_type_hints, typevars_map)
    139 return created_model

File ~/.local/lib/python3.10/site-packages/pydantic/generics.py:356, in _prepare_model_fields(created_model, fields, instance_type_hints, typevars_map)
    353 assert field.type_.__class__ is DeferredType, field.type_.__class__
    355 field_type_hint = instance_type_hints[key]
--> 356 concrete_type = replace_types(field_type_hint, typevars_map)
    357 field.type_ = concrete_type
    358 field.outer_type_ = concrete_type

File ~/.local/lib/python3.10/site-packages/pydantic/generics.py:263, in replace_types(type_, type_map)
    261         origin_type = getattr(typing, type_._name)
    262     assert origin_type is not None
--> 263     return origin_type[resolved_type_args]
    265 # We handle pydantic generic models separately as they don't have the same
    266 # semantics as "typing" classes or generic aliases
    267 if not origin_type and lenient_issubclass(type_, GenericModel) and not type_.__concrete__:

TypeError: 'type' object is not subscriptable

On inspection I found that at the point that the exception is thrown, origin_type is types.UnionType, where it should be Relationship.

Apologies if I missed an existing issue for this. I couldn't find anything in my search.

@ntaylorwss ntaylorwss added the bug V1 Bug related to Pydantic V1.X label Jun 9, 2022
@AMDmi3
Copy link

AMDmi3 commented Jun 15, 2022

Same thing with subscriptable types, with somehow simpler example:

# 1.py
from __future__ import annotations

from pydantic import BaseModel

class Model(BaseModel):
    foo: list[str]
% python3.8 1.py
Traceback (most recent call last):
  File "1.py", line 5, in <module>
    class Model(BaseModel):
  File "pydantic/main.py", line 188, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/typing.py", line 419, in pydantic.typing.resolve_annotations
    For example:
  File "/usr/local/lib/python3.8/typing.py", line 270, in _eval_type
    return t._evaluate(globalns, localns)
  File "/usr/local/lib/python3.8/typing.py", line 518, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
TypeError: 'type' object is not subscriptable

Obviously, python3.8 cannot evaluate list[str] here: while from __future__ import annotations (which is postponed evaluation of annotations) allows to use it in annotations, it's not a valid construct in python < 3.9 and cannot be evaluated directly. Thus I'm not sure if it's even fixable. The solution is to either use newer python, or use typing.List instead of list in annotations.

@thenx
Copy link

thenx commented Jan 22, 2023

While i believe #4146 (comment) is answered in #2314 (comment), I'd love to bump the original issue with a slightly reduced example.

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.10.4
            pydantic compiled: True
                 install path: {project_path}/.venv/lib/python3.10/site-packages/pydantic
               python version: 3.10.8 (main, Nov 18 2022, 12:43:38) [Clang 14.0.0 (clang-1400.0.29.102)]
                     platform: macOS-13.1-arm64-arm-64bit
     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']
from typing import Generic, TypeVar

import pydantic
from pydantic import generics

DataT = TypeVar("DataT")

class MyGenericModel(generics.GenericModel, Generic[DataT]):
    generic: list[DataT] | None


class A(pydantic.BaseModel):
    wrapper: MyGenericModel[int]

results in

Traceback (most recent call last):
  File "/<>/.pyenv/versions/3.10.8/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 12, in <module>
  File "<input>", line 13, in A
  File "<>/.venv/lib/python3.10/site-packages/pydantic/generics.py", line 145, in __class_getitem__
    _prepare_model_fields(created_model, fields, instance_type_hints, typevars_map)
  File "<>/.venv/lib/python3.10/site-packages/pydantic/generics.py", line 364, in _prepare_model_fields
    concrete_type = replace_types(field_type_hint, typevars_map)
  File "<>/.venv/lib/python3.10/site-packages/pydantic/generics.py", line 271, in replace_types
    return origin_type[resolved_type_args]
TypeError: 'type' object is not subscriptable

The same with Union

from typing import Generic, TypeVar, Union

import pydantic
from pydantic import generics

DataT = TypeVar("DataT")

class MyGenericModel(generics.GenericModel, Generic[DataT]):
    generic: Union[list[DataT], None]


class A(pydantic.BaseModel):
    wrapper: MyGenericModel[int]

seems to be working without the issue.

@samuelcolvin
Copy link
Member

Confirmed, should be fixed in V2, @dmontagu might be worth checking this is fixed in #4970.

Happy to review a PR to fix this in V1.10, but I suspect it's not trivial.

@thenx
Copy link

thenx commented Jan 22, 2023

A slightly more reduced example would be:

from pydantic import generics

generics.replace_types(list[str] | None, {str: int})

I guess the problem is that type(list | int) is types.UnionType and a dum-dum approach could be to translate types.UnionType back to typing.Union or just to do smth like functools.reduce(operator.or_, type_args) if origin_type is types.UnionType?

@dmontagu
Copy link
Contributor

dmontagu commented Jan 23, 2023

For what it's worth, on the #4970 PR, I checked that the following code runs without issue (on Python 3.11 anyway):

from typing import Generic, TypeVar

import pydantic

DataT = TypeVar("DataT")

class MyGenericModel(pydantic.BaseModel, Generic[DataT]):
    generic: list[DataT] | None

class A(pydantic.BaseModel):
    wrapper: MyGenericModel[int]

m = MyGenericModel[int](generic=None)
print(A(wrapper=m))
# wrapper=MyGenericModel[int](generic=None)

dmontagu pushed a commit that referenced this issue Feb 6, 2023
* Fix X | Y union syntax breaks GenericModel (#4146)

* Update changes/4146-thenx.md

Co-authored-by: ⬢ Samuel Colvin <s@muelcolvin.com>

* Improve tests

* Recreate newstyle union via typing._UnionGenericAlias

* Add basic pep-604 union args order caching detection test

---------

Co-authored-by: ⬢ Samuel Colvin <s@muelcolvin.com>
hexiro added a commit to hexiro/spotify-to-musi that referenced this issue Apr 19, 2023
@Kludex
Copy link
Member

Kludex commented Apr 27, 2023

Since this is working on V2, I'll be closing this issue. 🙏

@Kludex Kludex closed this as not planned Won't fix, can't repro, duplicate, stale Apr 27, 2023
@Cjkjvfnby
Copy link

I can reproduce it on Mac and windows with Python 3.9

pydantic.version.VERSION =2.2.1
sys.version = 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)]

from __future__ import annotations

from pydantic import BaseModel

class Model(BaseModel):
    foo: list[str]
    boo: int | str
TypeError: unsupported operand type(s) for |: 'type' and 'type'

Traceback (most recent call last):
  File "sample.py", line 12, in <module>
    class Model(BaseModel):
  File "...\lib\site-packages\pydantic\_internal\_model_construction.py", line 177, in __new__
    set_model_fields(cls, bases, config_wrapper, types_namespace)
  File "...\lib\site-packages\pydantic\_internal\_model_construction.py", line 405, in set_model_fields
    fields, class_vars = collect_model_fields(cls, bases, config_wrapper, types_namespace, typevars_map=typevars_map)
  File "...\lib\site-packages\pydantic\_internal\_fields.py", line 98, in collect_model_fields
    type_hints = get_cls_type_hints_lenient(cls, types_namespace)
  File "...\lib\site-packages\pydantic\_internal\_typing_extra.py", line 212, in get_cls_type_hints_lenient
    hints[name] = eval_type_lenient(value, globalns, localns)
  File "...\lib\site-packages\pydantic\_internal\_typing_extra.py", line 224, in eval_type_lenient
    return typing._eval_type(value, globalns, localns)  # type: ignore
  File "...\Python39\lib\typing.py", line 290, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "...\Python39\lib\typing.py", line 546, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'type' and 'type'

@thenx
Copy link

thenx commented Aug 19, 2023

I can reproduce it on Mac and windows with Python 3.9

PEP604-syntax requires python>=3.10

@Cjkjvfnby
Copy link

PEP604-syntax requires python>=3.10

With from __future__ import annotations you could use it in Python3.9.
It evaluates after parsing into the string. So when Pydantic tries to evaluate this string (with typing._eval_type) it fails.

Maybe catch this error and return a more meaningful message?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug V1 Bug related to Pydantic V1.X
Projects
None yet
Development

No branches or pull requests

7 participants