Fixes #9237 - Decouple QTP idleTimeout from pool shrink rate. (#9498)

Introduced `QueuedThreadPool.maxEvictCount` to be the number of idle threads that are evicted in one idle timeout.

When set to 1 (the default), the old behavior is reproduced: expiring 1 thread every idle timeout.
When set to larger values, allows to keep around the threads for the idle timeout (in case of further load spikes), but allows to quickly recover OS memory when they are truly idle.

For example, with 2000 threads, 30 seconds idle timeout and idleTimeoutMaxShrinkCount=1, it will take 995 minutes (about 16.5 hrs) to shrink the pool back to 10 threads.
By setting idleTimeoutMaxShrinkCount=100, the thread pool can be shrunk to 10 threads in about 10 minutes.

Note also that the new algorithm is more aggressive at shrinking the thread pool.
Previously, a small load might have been sufficient to never evict any thread, because all threads could take turns at executing jobs so that threads were mostly idle but would never really idle time out.
The new algorithm is more aggressive even in presence of a small load, so that if `minThreads` are sufficient to cope with the small load, then the other threads are evicted.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Signed-off-by: gregw <gregw@webtide.com>
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
Co-authored-by: gregw <gregw@webtide.com>
Co-authored-by: Ludovic Orban <lorban@bitronix.be>

documentation/jetty-documentation/src/main/asciidoc/programming-guide/arch-threads.adoc

@@ -188,6 +188,19 @@ This value represents the maximum number of threads that can be reserved and use
 A negative value for `QueuedThreadPool.reservedThreads` means that the actual value will be heuristically derived from the number of CPU cores and `QueuedThreadPool.maxThreads`.
 A value of zero for `QueuedThreadPool.reservedThreads` means that reserved threads are disabled, and therefore the xref:pg-arch-threads-execution-strategy-epc[`Execute-Produce-Consume` mode] is never used -- the xref:pg-arch-threads-execution-strategy-pec[`Produce-Execute-Consume` mode] is always used instead.
 
+`QueuedThreadPool` always maintains the number of threads between `QueuedThreadPool.minThreads` and `QueuedThreadPool.maxThreads`; during load spikes the number of thread grows to meet the load demand, and when the load on the system diminishes or the system goes idle, the number of threads shrinks.
+
+Shrinking `QueuedThreadPool` is important in particular in containerized environments, where typically you want to return the memory occupied by the threads to the operative system.
+The shrinking of the `QueuedThreadPool` is controlled by two parameters: `QueuedThreadPool.idleTimeout` and `QueuedThreadPool.maxEvictCount`.
+
+`QueuedThreadPool.idleTimeout` indicates how long a thread should stay around when it is idle, waiting for tasks to execute.
+The longer the threads stay around, the more ready they are in case of new load spikes on the system; however, they consume resources: a Java platform thread typically allocates 1 MiB of native memory.
+
+`QueuedThreadPool.maxEvictCount` controls how many idle threads are evicted for one `QueuedThreadPool.idleTimeout` period.
+The larger this value is, the quicker the threads are evicted when the `QueuedThreadPool` is idle or has less load, and their resources returned to the operative system; however, large values may result in too much thread thrashing: the `QueuedThreadPool` shrinks too fast and must re-create a lot of threads in case of a new load spike on the system.
+
+A good balance between `QueuedThreadPool.idleTimeout` and `QueuedThreadPool.maxEvictCount` depends on the load profile of your system, and it is often tuned via trial and error.
+
 [[pg-arch-threads-thread-pool-virtual-threads]]
 ===== Virtual Threads
 Virtual threads have been introduced in Java 19 as a preview feature.

jetty-server/src/main/config/etc/jetty-threadpool-virtual-preview.xml

@@ -8,6 +8,7 @@
     <Set name="maxThreads" type="int"><Property name="jetty.threadPool.maxThreads" deprecated="threads.max" default="200"/></Set>
     <Set name="reservedThreads" type="int"><Property name="jetty.threadPool.reservedThreads" default="-1"/></Set>
     <Set name="idleTimeout" type="int"><Property name="jetty.threadPool.idleTimeout" deprecated="threads.timeout" default="60000"/></Set>
+    <Set name="maxEvictCount" type="int"><Property name="jetty.threadPool.maxEvictCount" default="1"/></Set>
     <Set name="detailedDump" type="boolean"><Property name="jetty.threadPool.detailedDump" default="false"/></Set>
     <Get id="namePrefix" name="name" />
     <Call class="java.lang.Thread" name="ofVirtual">

jetty-server/src/main/config/etc/jetty-threadpool.xml

@@ -26,6 +26,7 @@
     <Set name="reservedThreads" type="int"><Property name="jetty.threadPool.reservedThreads" default="-1"/></Set>
     <Set name="useVirtualThreads" property="jetty.threadPool.useVirtualThreads" />
     <Set name="idleTimeout" type="int"><Property name="jetty.threadPool.idleTimeout" deprecated="threads.timeout" default="60000"/></Set>
+    <Set name="maxEvictCount" type="int"><Property name="jetty.threadPool.maxEvictCount" default="1"/></Set>
     <Set name="detailedDump" type="boolean"><Property name="jetty.threadPool.detailedDump" default="false"/></Set>
   </New>
 </Configure>

jetty-server/src/main/config/modules/threadpool-virtual-preview.mod

@@ -30,6 +30,9 @@ etc/jetty-threadpool-virtual-preview.xml
 ## Thread idle timeout (in milliseconds).
 #jetty.threadPool.idleTimeout=60000
 
+## The max number of idle threads that can be evicted in one idleTimeout period.
+#jetty.threadPool.maxEvictCount=1
+
 ## Whether to output a detailed dump.
 #jetty.threadPool.detailedDump=false
 

jetty-server/src/main/config/modules/threadpool.mod

@@ -31,6 +31,9 @@ etc/jetty-threadpool.xml
 ## Thread idle timeout (in milliseconds).
 #jetty.threadPool.idleTimeout=60000
 
+## The max number of idle threads that are evicted in one idleTimeout period.
+#jetty.threadPool.maxEvictCount=1
+
 ## Whether to output a detailed dump.
 #jetty.threadPool.detailedDump=false
 # end::documentation[]

jetty-util/src/main/java/org/eclipse/jetty/util/AtomicBiInteger.java

@@ -251,6 +251,13 @@ public void add(int deltaHi, int deltaLo)
         }
     }
 
+    @Override
+    public String toString()
+    {
+        long encoded = get();
+        return getHi(encoded) + "|" + getLo(encoded);
+    }
+
     /**
      * Encodes hi and lo values into a long.
      *