aptos-core is stuck on serde < 1.0.53 due to a change in 1.0.53 that breaks testsuite/generate-format

[banool]

Repro

git clone git@github.com:aptos-labs/aptos-core.git
# Change serde version in Cargo.toml to 1.0.153
cd testsuite/generate-format
cargo run -- --corpus aptos

You will see an error like this:

thread 'main' panicked at 'Incomplete tracing detected inside container: StructTag:
A final registry was requested but some formats are still unknown within the container StructTag. This can
happen if `tracer.trace_value` was called on a value `foo` which does not reveal some of the underlying
formats. E.g. if a field `x` of struct `Foo` has type `Option<String>` and `foo` is value of type
`Foo` such that `foo.x == None`, then tracing the value `foo` may result in a format `Option<Unknown>`
for the field `x`. The same applies to empty vectors and empty maps.

To fix this, avoid `trace_value` and prefer `trace_type` when possible, or make sure to trace at
least one value `foo` such that `foo.x` is not empty.
', testsuite/generate-format/src/lib.rs:50:17

Summary

This issue arises from this change: serde-rs/serde#2387.

I tried using the latest version of serde + serde_yaml and it doesn't fix the issue. I also tried to fork serde and fix it but then we get a bunch of other errors, see https://github.com/banool/serde/tree/banool/undo-2387 and https://github.com/aptos-labs/aptos-core/tree/banool/upgrade-serde.

I'm not sure right now how we could handle this issue in our codebase. I believe changing the format (I'm not 100% sure what format that is, btw) is out of the question so we'd have to do something on our side to undo whatever change serde-rs/serde#2387 introduces.

See more discussion here: https://aptos-org.slack.com/archives/C03N83P7QUC/p1691767293160939. I suspect this is relevant too: zefchain/serde-reflection#35 (meaning switching to this version of serde-reflection might not fix our issues).

Impact

This stops us from using crates that depend on a particular minimum version of serde higher than this version. To date this has happened twice:

• Once with me trying to use Bollard, in which case we can't because it depends on serde_with, which depends on serde 1.0.57+ (see more here: Question about minimum serde version jonasbb/serde_with#651).
• Once earlier on with @ibalajiarun in [devtools] (Re)Introducing target determinator cargo cli #9463.

As time goes on this is going to happen more and more frequently as our version of serde gets increasingly out of date.

[banool]

I opened an issue in serde: serde-rs/serde#2627.

[banool]

I tried updating to the latest zefchain and that didn't fix it, presumably because of this issue: zefchain/serde-reflection#35.