Flaky (Deadlock Detected) krate tests.

[adsnaider]

Current Behavior

While contributing to crates.io for #4690, I got failures from some krate tests during development. I reran the tests and the failures were gone. I figured I would post the issue so that we have a record of this behavior.

Here's the test output I got. However, it's hard to reproduce since there seems to be some flakiness.

failures:

---- krate::show::show_minimal stdout ----
thread 'krate::show::show_minimal' panicked at 'Unable to create crate foo_show_minimal: DatabaseError(__Unknown, "deadlock detected")', src/tests/builders/krate.rs:181:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- krate::summary::excluded_crate_id stdout ----
thread 'krate::summary::excluded_crate_id' panicked at 'Unable to create crate some_downloads: DatabaseError(__Unknown, "deadlock detected")', src/tests/builders/krate.rs:181:13

---- krate::search::index_queries stdout ----
thread 'krate::search::index_queries' panicked at 'called `Result::unwrap()` on an `Err` value: DatabaseError(__Unknown, "deadlock detected")', src/tests/krate/search.rs:85:14


failures:
    krate::search::index_queries
    krate::show::show_minimal
    krate::summary::excluded_crate_id

test result: FAILED. 253 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 14.26s

Expected Behavior

Tests are deterministic.

Steps To Reproduce

cargo test and hope for the worst.

Environment

• Browser: N/A (backend issue only)
• OS: PopOS 21.04

Anything else?

No response

[Turbo87]

I've seen similar things before when developing locally, but interestingly I've never observed this on CI. any clues what might be causing this? I'm wondering if it might be caused by a Postgres bug, rather than something wrong on our side.

[pietroalbini]

That depends on the amount of parallel tests you run: if you set a low concurrency (CI runs two tests at the time) you'll basically never see the error, while if you have a high concurrency (like 16 in parallel) you'll likely see one every time.

This is due to all tests operating on the same empty database inside their own transactions (rolled back when the test finishes): when two tests change the same table one of them will lock it, and in some cases there is some combinations of locks resulting in a deadlock.

[Turbo87]

Since the issue was opened we switched to a system where each test gets its own fresh database, so the problem should hopefully be resolved now :)