Flaky Test - org.eclipse.jetty.client.HttpClientTLSTest.testEncryptedOutputBufferRepoolingAfterNetworkFlushReturnsFalse

[joakime]

Jetty version(s)
Jetty 12

Description

org.eclipse.jetty.client.HttpClientTLSTest expected an empty collection but <[java.nio.DirectByteBuffer[pos=0 lim=0 cap=8192]]> within 5 seconds.

org.awaitility.core.ConditionTimeoutException: org.eclipse.jetty.client.HttpClientTLSTest expected an empty collection but <[java.nio.DirectByteBuffer[pos=0 lim=0 cap=8192]]> within 5 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:86)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:691)
	at org.eclipse.jetty.client@12.0.0-SNAPSHOT/org.eclipse.jetty.client.HttpClientTLSTest.testEncryptedOutputBufferRepoolingAfterNetworkFlushReturnsFalse(HttpClientTLSTest.java:964)

Running org.eclipse.jetty.client.HttpClientTLSTest.testEncryptedOutputBufferRepoolingAfterNetworkFlushReturnsFalse([1] true)
2022-12-14 17:51:59.035:INFO :oejs.Server:main: jetty-12.0.0-SNAPSHOT; built: 2022-12-14T17:42:41.341Z; git: 1eda33397755f2b90ebfe60e2e23ea8cc9a9e6a5; jvm 17.0.2+8
2022-12-14 17:51:59.188:INFO :oejus.SslContextFactory:main: x509=X509@31da0434(mykey,h=[localhost],a=[/127.0.0.1, /0:0:0:0:0:0:0:1],w=[]) for Server@6e24ce51[provider=null,keyStore=file:///home/jenkins/agent/workspace/jetty.project_PR-9053/jetty-core/jetty-client/src/test/resources/keystore.p12,trustStore=null]
2022-12-14 17:51:59.241:INFO :oejs.AbstractConnector:main: Started ServerConnector@101bdd1c{SSL, (ssl, http/1.1)}{0.0.0.0:43579}
2022-12-14 17:51:59.241:INFO :oejs.Server:main: Started oejs.Server@2cfe272f{STARTING}[12.0.0-SNAPSHOT,sto=0] @118965ms
2022-12-14 17:51:59.381:INFO :oejus.SslContextFactory:main: x509=X509@74960e9d(mykey,h=[localhost],a=[/127.0.0.1, /0:0:0:0:0:0:0:1],w=[]) for Client@4001d8c1[provider=null,keyStore=file:///home/jenkins/agent/workspace/jetty.project_PR-9053/jetty-core/jetty-client/src/test/resources/keystore.p12,trustStore=null]
2022-12-14 17:51:59.433:WARN :oejusS.config:main: No Client EndPointIdentificationAlgorithm configured for Client@4001d8c1[provider=null,keyStore=file:///home/jenkins/agent/workspace/jetty.project_PR-9053/jetty-core/jetty-client/src/test/resources/keystore.p12,trustStore=null]
2022-12-14 17:52:04.469:INFO :oejs.Server:main: Stopped oejs.Server@2cfe272f{STOPPING}[12.0.0-SNAPSHOT,sto=0]
2022-12-14 17:52:04.469:INFO :oejs.AbstractConnector:main: Stopped ServerConnector@101bdd1c{SSL, (ssl, http/1.1)}{0.0.0.0:0}

[sbordet]

This test fails with parameter close=true, because the EndPoint is closed (rather than just oshut).

Closing the EndPoint causes the execution of HttpConnection.onClose() (called by ManagedSelector.DestroyEndPoint.run()) in a different thread.

HttpConnection.onClose() acts on HttpConnection._sendCallback and triggers #9059, so that the custom state of SendCallback is "failed" in SendCallback.onCompleteFailure() which calls SendCallback.release().

The call to SendCallback.release() happens concurrently with another thread currently inside process() trying to generate a response, so eventually the scheduling is such that the test fails flakily.

In particular, the ByteBuffer that is leaked is the one requested by HttpGenerator via NEED_HEADER.

One failure path is when Thread1 tries to generate, NEED_HEADER is returned, a ByteBuffer is acquired, then Thread2 calls SendCallback.release(), SendCallback._info is nulled out, which now confuses HttpGenerator that, when called again by Thread1, produces NEED_INFO that throws without releasing the header ByteBuffer.

[sbordet]

Closing as part of fixing #9059.