server: fix deadlock when closing concurrently with transport

[sukunrt]

When the listener and transport are closed concurrently. We can get a deadlock

Listener.Close is in Listener.CloseOnce.Do block and calls transport.closeServer for notifyOnClose. transport.closeServer requires Transport.mutex

Transport.Close holds the Transport.mutex and calls Listener.Close and gets stuck at the Listener.closeOnce.Do block

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.27%. Comparing base (c786a46) to head (121b82c).
Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4332      +/-   ##
==========================================
+ Coverage   84.80%   85.27%   +0.47%     
==========================================
  Files         150      150              
  Lines       14251    14466     +215     
==========================================
+ Hits        12085    12335     +250     
+ Misses       1668     1644      -24     
+ Partials      498      487      -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[marten-seemann]

Does this PR target #4266?

[sukunrt]

Oh sorry. Yes. I didnt realise there was an open PR. Happy to close this.

[marten-seemann]

#4266 is the issue, not a PR. Please keep this one open.

[marten-seemann]

@sukunrt Did you have a way to reliably trigger this deadlock? If so, would it make sense to add this as a test case here?

[sukunrt]

On my machine this test is consistently deadlocking on master.

[marten-seemann]

Thank you @sukunrt!