fix: Fix broken parametrized bases with GenericModels

[MarkusSintonen]

This PR broke the usage of parametrized bases with GenericModels. Problem arises especially in applications having large number of generic data models (in our case a large FastAPI based application). The issue roots to the dictionary size limiting in pydantic/generics.py. When the limit is hit it starts to break usage of parametrized bases like here #4131 (comment) and here #4483

Original PR tried to fix a memory leak by limiting the size of the internal dictionaries. But it seems its not correct way to do it as it can break generics usage. So instead this PR relies on weak dictionaries to handle any leaks with dynamic model usage.

Fix #4131.

[MarkusSintonen]

FYI with this change we are not observing memory leak as the original issue did. We are running a similar patch in production like the change here.

[samuelcolvin]

Humm, in principle this all looks good, we should definitely use this in V2.

I'm a bit of a wimp about making changes like this in V1.10 - I've seen too many have unexpected consequences.

@pydantic/pydantic-maintainers what do you think?

[samuelcolvin]

@crazyscientist, would be amazing if you could review this and make sure it's working with your use case?

[MarkusSintonen]

@adriangb it seems pytest-memray is bugged with parametrized tests having long names 😞 It works when not using --memray-bin-path param. But then we dont get the full result artifacts.

Its using the test name as the file name here https://github.com/bloomberg/pytest-memray/blob/1.4.0/src/pytest_memray/plugin.py#L139 Which breaks with long generated test names. Maybe it would be worth reporting to them 🤔

[adriangb]

Thanks for tracking that down! 100% worth reporting. For now we can probably just put that test in it's own file and only run it on its own. What do you think?

[adriangb]

Oh wait you fixed the issue. Never mind, I think this should work!

[samuelcolvin]

We should definitely report this to mem ray.

I'll review this later.

[crazyscientist]

@samuelcolvin thank you for the heads up. When I run our test suite with this fix, I unfortunately keep getting the KeyError raised as described in #4131.

[samuelcolvin]

This is great, thanks all for your input.

In particular @MarkusSintonen thanks for your perseverance on this issue, I suspect you have the dubious honour of making the most complex use of pydantic generics.

We need this to be copied to main, probably worth waiting for #4970 to be merged. @MarkusSintonen would you be willing to contribute this?

Also, I'm yet again impressed with memray, I guess we should use it more.

[MarkusSintonen]

This is great, thanks all for your input. In particular @MarkusSintonen thanks for your perseverance on this issue, I suspect you have the dubious honour of making the most complex use of pydantic generics.

Haha no problem! 😊

We need this to be copied to main, probably worth waiting for #4970 to be merged. @MarkusSintonen would you be willing to contribute this?

Yeah I can open similar PR against main.

[MarkusSintonen]

@samuelcolvin I already opened the V2 PR so I wont forget it: #5056

[samuelcolvin]

awesome, thank you.

[pablogsal]

@adriangb it seems pytest-memray is bugged with parametrized tests having long names 😞 It works when not using --memray-bin-path param. But then we dont get the full result artifacts.

Its using the test name as the file name here https://github.com/bloomberg/pytest-memray/blob/1.4.0/src/pytest_memray/plugin.py#L139 Which breaks with long generated test names. Maybe it would be worth reporting to them 🤔

We are on it bloomberg/pytest-memray#68 . We are in the process of shipping some big stuff to memray core so this may take some more time than normal.