Proactive connection management

[jeet1995]

Features

Minimum connection pool size per endpoint

Using a system config, the application developer can specify the minimum no. of connections to be set for all endpoints for all physical partitions across all containers whose connections are opened using the openConnectionsAndInitCaches flow either through CosmosClient / CosmosAsyncClient or CosmosContainer / CosmosAsyncContainer. This can help alleviate the cold start latency associated with establishing connections on-demand when requests are made by the SDK.

Setting the minimum connection pool size per endpoint through a system config

// initialize the min. no. of connections required for all endpoints
int minConnectionPoolSizePerEndpoint = 1;
System.setProperty("COSMOS.MIN_CONNECTION_POOL_SIZE_PER_ENDPOINT", String.valueOf(minConnectionPoolSizePerEndpoint));

Aggressive connection establishment time duration

In the openConnectionsAndInitCaches flow for CosmosClient / CosmosAsyncClient, a duration can be specified within which connections are established aggressively in a blocking manner. Once this duration elapses, connections are established defensively but in a non-blocking manner. This improves on the previous flow which only opened connections in a blocking manner therefore preventing the SDK from making requests until all connections were established.

Setting the aggressive proactive connection establishment duration through the public API

// containers to which connections are to be proactively opened
CosmosContainerIdentity containerIdentity1 = new CosmosContainerIdentity("sample_db_id", "sample_container_id_1");
CosmosContainerIdentity containerIdentity2 = new CosmosContainerIdentity("sample_db_id", "sample_container_id_2");

// no. of regions to which connections are to be proactively opened
int proactiveConnectionRegionsCount = 1;

// duration for which connections are to be aggressively opened in a blocking manner
// beyond this duration connections will be defensively opened in a non-blocking manner
Duration aggressiveWarmupDuration = Duration.ofSeconds(1);

// building the client along with opening connections
CosmosAsyncClient clientWithOpenConnections = new CosmosClientBuilder()
          .endpoint("<account URL goes here")
          .key("<account key goes here>")
          .endpointDiscoveryEnabled(true)
          .preferredRegions(Arrays.asList("sample_region_1", "sample_region_2"))
          .openConnectionsAndInitCaches(new CosmosContainerProactiveInitConfigBuilder(Arrays.asList(containerIdentity1, containerIdentity2))
                .setProactiveConnectionRegionsCount(proactiveConnectionRegionsCount)
                .setAggressiveWarmupDuration(aggressiveWarmupDuration)
                .build())
          .directMode()
          .buildAsyncClient();

Proactive connection management during connection close and reset

This PR helps proactively re-establish connections which are reset by the server or if they have been gracefully closed or in cases where the channel was closed due to it being unhealthy or idle.

Design considerations

• Fairness: This done to ensure open connection tasks for a given endpoint are spaced out as much as possible.
• Retries: A connection attempt could fail and such failures could be transient. Therefore, a backoff retry is used to reattempt unsuccessful connection attempts.
• Concurrency control: Since the SDK could open connections in the background, it is important to set the concurrency here as low as possible (set as 1 by default). This could be too low for an application and hence a system config - COSMOS.DEFENSIVE_WARMUP_CONCURRENCY can be used to modify the defensive concurrency.

Benchmarking results

Insights

• Cold start latency has reduced for workloads with proactive connection management.
• Cold start latency differences is lesser between workloads with and without proactive connection management at lower concurrencies.
• Cold start latencies are higher when the aggressive connection establishment duration is lower.

Reads (Provisioned throughput : 1 million RUs, Concurrency : 512, Documents to read: 10 thousand, Operations count: 1 million)

Reads (Provisioned throughput : 1 million RUs, Concurrency : 128, Documents to read: 10 thousand, Operations count: 1 million)

Reads (Provisioned throughput : 1 million RUs, Concurrency : 512, Documents to read: 10 thousand, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Reads (Provisioned throughput : 1 million RUs, Concurrency : 128, Documents to read: 10 thousand, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Reads (Provisioned throughput : 20 thousand RUs, Concurrency : 10, Documents to read: 1 thousand, Operations count: 100 thousand)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 512, Operations count: 1 million)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 128, Operations count: 1 million)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 512, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 128, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Writes (Provisioned throughput : 20 thousand RUs, Concurrency : 10, Operations count: 100 thousand)

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

LGTM, thanks for the great work :)

[FabianMeiswinkel]

LGTM - thanks

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

Re-ran the following tests locally.

• ReadMyWritesConsistencyTest - all tests pass locally
• CosmosItemTest - all tests pass locally
• FaultInjectionServerErrorRuleTests - investigate cases where the write region server is gone and yet fault injection rule is applied for query operation.

[jeet1995]

/check-enforcer override

[jeet1995]

fixes #33082
fixes #33079
fixes #33080
fixes #33087