RecursionError in combination with Dataclasses and create_model

[mbillingr]

Initial Checks

• I have searched GitHub for a duplicate issue and I'm sure this is something new
• I have searched Google & StackOverflow for a solution and couldn't find anything
• I have read and followed the docs and still think this is a bug
• I am confident that the issue is with pydantic (not my code, or another library in the ecosystem like FastAPI or mypy)

Description

I'm getting a RecursionError when calling pydantic.create_model with a __base__ class that contains fields which are both pydantic and vanilla dataclasses.
Most of the recursion related problems I could find were related to recursive data types (such as #1370). There is no (obvious) recursion in the data structures I use.

Traceback (most recent call last):
  File "/home/martin/.config/JetBrains/PyCharm2022.3/scratches/scratch_26.py", line 15, in <module>
    pydantic.create_model("ArbitraryName", __base__=ResponseSchema)
  File "pydantic/main.py", line 1027, in pydantic.main.create_model
  File "pydantic/main.py", line 139, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/utils.py", line 695, in pydantic.utils.smart_deepcopy
  File "/home/martin/.pyenv/versions/3.11.0/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
[ whole bunch of deepcopy-related lines omitted ]
  File "/home/martin/.pyenv/versions/3.11.0/lib/python3.11/copy.py", line 272, in _reconstruct
    if hasattr(y, '__setstate__'):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "pydantic/dataclasses.py", line 255, in pydantic.dataclasses.DataclassProxy.__getattr__
    'type',

  [Previous line repeated 979 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

There is an easy workaround: Don't use both decorators.
Unfortunately, I need both because the class is defined in another module and I need to wrap it with pydantic for I/O validation.

Originally, I ran into this problem using FastAPI, but managed to reproduce it using pydantic alone. So I hope this is the correct place to ask.

Example Code

import dataclasses
import pydantic

@pydantic.dataclasses.dataclass
@dataclasses.dataclass
class ValueObj:
    name: str

class ResponseSchema(pydantic.BaseModel):
    value: ValueObj

pydantic.create_model("ArbitraryName", __base__=ResponseSchema)

Python, Pydantic & OS Version

pydantic version: 1.10.4
            pydantic compiled: True
                 install path: /home/martin/.cache/pypoetry/virtualenvs/private/lib/python3.11/site-packages/pydantic
               python version: 3.11.0 (main, Nov  7 2022, 07:51:46) [GCC 12.2.0]
                     platform: Linux-6.1.4-arch1-1-x86_64-with-glibc2.36
     optional deps. installed: ['typing-extensions']


Also reproducible with slightly older versions:

             pydantic version: 1.10.2
            pydantic compiled: True
                 install path: /home/martin/.cache/pypoetry/virtualenvs/private/lib/python3.9/site-packages/pydantic
               python version: 3.9.7 (default, Nov  7 2022, 07:48:33)  [GCC 12.2.0]
                     platform: Linux-6.1.4-arch1-1-x86_64-with-glibc2.36
     optional deps. installed: ['typing-extensions']

Affected Components

• Compatibility between releases
• Data validation/parsing
• Data serialization - .model_dump() and .model_dump_json()
• JSON Schema
• Dataclasses
• Model Config
• Field Types - adding or changing a particular data type
• Function validation decorator
• Generic Models
• Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
• Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

[samuelcolvin]

You can resolve this by removing the extra @dataclasses.dataclass.

Might have similar origin to #4907.

[mbillingr]

I applied both decorators to shorten the example. In my actual code I can't remove the extra dataclass because I have the following situation:

# can't change this definition
@dataclasses.dataclass
class ValueObj:
    name: str


# IO module
ValueObj = pydantic.dataclasses.dataclass(domain.ValueObj)

P.S. Your response time is impressive!

[samuelcolvin]

That was luck, often it's not so good. :-) As mentioned on the linked issue, the current dataclass support is very involved, most of our bug fixes have been related to trying to patch dataclass edge cases. I'd strongly prefer not to change dataclass support again in v1, but hopefully we can fix this properly in v2. Unless we can find a trivial solution.

…

On Fri, 13 Jan 2023, 16:34 mbillingr, ***@***.***> wrote: I applied both decorators to shorten the example. In my actual code I can't remove the extra dataclass because I have the following situation: # can't change this definition @dataclasses.dataclass class ValueObj: name: str # IO module ValueObj = pydantic.dataclasses.dataclass(domain.ValueObj) P.S. Your response time is impressive! — Reply to this email directly, view it on GitHub <#4949 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA62GGN3EVBLS5NK3MWUXSDWSF7XPANCNFSM6AAAAAAT2SPHYU> . You are receiving this because you commented.Message ID: ***@***.***>

[mbillingr]

I can understand that you don't feel the urge to touch the dataclass support in v1.

For me it's already a win to know that this is really a bug and not just me misusing your API. If it can't be fixed upstream I'll find a workaround. (Even though I'm not particularly fond of the idea, it should be easy to just duplicate the ValueObj definition.)

Starting Monday, I could free some time to look into this in more detail. Would you accept a pull request if I managed to find a simple fix? (I don't expect to, but you never know... :))

[samuelcolvin]

Possibly, it's really up to @PrettyWood who is the "master of dataclasses".

[mbillingr]

I dug deeper and found that the recursion occurs while deep-copying a DataclassProxy.
At some point Python's copy._reconstruct function creates a new uninitialized DataclassProxy, then checks if this has a __setstate__ attribute.
DataclassProxy.__getattr__ wants to delegate the attribute access to it's __dataclass__ field, but being uninitialized, this field does not exist and looking it up invokes DataclassProxy.__getattr__ again...

I think this problem could be solved by adding a __deepcopy__ method to DataclassProxy:

class DataclassProxy:

    # ...

    def __deepcopy__(self, memo: Any) -> "DataclassProxy":
        return DataclassProxy(deepcopy(self.__dataclass__, memo))

This seems to fix my example above (I did not check if it breaks anything else, though).

[PrettyWood]

Yup @mbillingr ! Seems like the right fix 👍

[mbillingr]

I guess it's up to you then to decide what to do next...
I have no idea if this issue will become obsolete with V2. So if you want me to make a PR against any branch/version let me know. If you prefer to fix it yourself because a PR is too much trouble I'm fine as well.
And if you decide not to do anything my client code monkey-patch will live a long happy life :)

[samuelcolvin]

I'm pretty sure this will become obsolete in V2, as the dataclass implementation will be radically simplified.

But if you'd like to create a PR again the 1.10.X-fixes branch, we'll happily review.

[PrettyWood]

fixed by #4963