Is it possible to follow cluster mode redirects with rueidis?

[Ghilteras]

Hey guys I have an Elasticache instance in cluster mode and we are using rueidis to connect via HAProxy (because the client is in another VPC) and we get redirects to a shard from the configuration endpoint so the connection goes back to the client and then the client tries to connect directly to the shard, but it fails because HAProxy only forwards to the configuration endpoint.

Example with redis-cli:

redis-cli -h elasticache-cluster.us-east-2.aws.k8s.sendgrid.net --tls -p 6379 -a 'password' -c

> get "some key"

-> Redirected to slot [15500] located at elasticache-cluster-0001-001.6wgagi.use2.cache.amazonaws.com:6379
Could not connect to Redis at elasticahe-cluster-0001-001.6wgagi.use2.cache.amazonaws.com:6379: SSL_connect failed: Connection reset by peer

Is there a way to tell rueidis to follow the redirect? Since this is Redis TCP protocol I don't think we can instrument HAProxy to do so because it only handles raw TCP forwards. Would it make sense to use a Redis proxy instead of HAProxy here?

[rueian]

Hi @Ghilteras, rueidis does follow the redirect. The error you saw is the proof. And you are right. You can't proxy an Elasticache Cluster with just one HAProxy in total. You need one HAProxy for each Elasticache node and a private DNS resolver that solves Elasticache addresses to corresponding HAProxies or you may consider VPC peering so that you will not need to setup HAProxies.

I don't recommend Redis proxy since it is not actively maintained.

[Ghilteras]

Thanks for the answer. How about Envoy Redis plugin? https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/redis#arch-overview-redis

[Ghilteras]

Specifically the enable redirect option https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/redis_proxy/v3/redis_proxy.proto#envoy-v3-api-field-extensions-filters-network-redis-proxy-v3-redisproxy-connpoolsettings-enable-redirection

[rueian]

Worth trying. Remember to set ForceSingleClient to true on rueidis.

rueidis/rueidis.go

Line 167 in 1ae39a7

ForceSingleClient bool

[Ghilteras]

@rueian may I ask how would that flag be useful with Envoy?

[rueian]

Hi @Ghilteras,

rueidis sends CLUSTER SLOTS to redis in order to determine which client implementation (cluster or single) to use. In your case, you can't use the cluster client implementation because you use Envoy to hide cluster details.

Setting the ForceSingleClient to true allows rueidis to skip the CLUSTER SLOTS process and just use the single client implementation to connect to your Envoy.

[rueian]

Hi @Ghilteras, does envoy work for you?

[Ghilteras]

yes it seems it works. We just need to wrap Envoy into an ingress controller (we're currently using nginx which does not have envoy, so we have envoy as sidecar of nginx which is not ideal) to expose elasticache through the ingress (we can't use the Gateway API due to our k8s version being 1.21)