RedisCluster in multi process applications

[mrAndersen]

I have some strange problems creating RedisCluster() instance in different process environments.

I am using swoole addProcess() mechanics to create new processes inside my app. And I want to have different objects of RedisCluster in each process to maintain different connects. However sometimes I am getting the same spl_object_hash for created RedisCluster instance in different PIDs, and because of that on that I am getting some random errors "Error processing response from Redis node!"

Any advises how I can force every time new manager object, or maybe some hints of using cluster in multi process enviornments?
Basically what swoole addProcess does - is fork

[michael-grunder]

PhpRedis doesn't understand anything about forking so I would expect this sort of thing (instability, random failures, etc).

It may be possible to work around this in the short term by manually "cloning" the RedisCluster object post fork if there is a hook you can get at with Swoole when that happens.

A more robust solution would require changes to PhpRedis to keep track of the owning PID so we can reconnect when that changes.

[mrAndersen]

@michael-grunder thx for reply. This is exactly what I am doing. Post fork in freshly created process I am creating RedisCluster by simply calling new RedisCluster(...).

The problem is that in different PIDs I am getting the same spl_object_hash for that fresh instance of RedisCluster. And therefore as far as I understand that is basically the same heap allocation and thus the same object. And because what you mentioned previously now 2 processes accessing the same object.

This happens not all the time, like out of 8 processes created I am getting only 2-3 same RedisCluster objects

[michael-grunder]

therefore as far as I understand that is basically the same heap allocation and thus the same object

If you are creating a new object in the forked child it can't be the same heap allocation. Process memory is duplicated on fork using a copy-on-write sheme.

Perhaps there is an edge case where the RedisCluster is not recreated?

If you are using persistent connections with RedisCluster that certainly could cause problems. On reinitialization of the object phpredis finds the persistent connection from the. parent (which is duplicated on fork) and tries to use the same underlying kernel file descriptor.

I've never really used Swoole. If you could provide a very simple example program that uses the mechanisms you're talking about I can certainly take a look.

[mrAndersen]

@michael-grunder Here you go! https://github.com/mrAndersen/swoole-phpredis-mutliprocess

To run this, execute

• docker-compose up -d --build
• docker exec -it t_swoole composer i
• ./cluster.sh (This will create actual redis cluster locally)
• And then you can test and run HTTP server + 6 subprocesses with docker exec -it bin/console t:run

Basically you will see the following:

Created Redis, pid=1316, spl_hash=00000000000000560000000000000000
Hello from 1316, i = 0
Created Redis, pid=1315, spl_hash=00000000000000560000000000000000
Created Redis, pid=1314, spl_hash=00000000000000560000000000000000
Hello from 1315, i = 0
Hello from 1314, i = 0
Created Redis, pid=1318, spl_hash=00000000000000560000000000000000
Hello from 1318, i = 0
Created Redis, pid=1317, spl_hash=00000000000000560000000000000000
Hello from 1317, i = 0
Created Redis, pid=1319, spl_hash=00000000000000560000000000000000
Hello from 1319, i = 0
Hello from 1314, i = 1
Hello from 1316, i = 1
Hello from 1318, i = 1
Hello from 1315, i = 1
Hello from 1317, i = 1
Hello from 1319, i = 1
Hello from 1318, i = 2
Hello from 1315, i = 2
Hello from 1317, i = 2
Hello from 1314, i = 2
Hello from 1316, i = 2
Hello from 1319, i = 2

And as you see we are getting the same object hash for each RedisCluster instance, therefore as long as you will increase number of commands per process you will likely encounter some random errors and some other bugs. As far as I know i should get different spl_object_hash for different objects.

To be more precise I included symfony, which is indeed used in production, however I doubt that it has any impact on that problem.

[mrAndersen]

Seems like spl_object_hash is working per process. Seems like even if it shows same hashes, in reality those objects are different objects. You can take a look at last commit - where I am doing the following:

 if ($pid % 2 == 0) {
                    $a = new \stdClass();
                }

Before each new RedisCluster()

Thus incrementing internal php allocator I guess. And therefore I am getting different spl_object_hash in odd\even PIDs ;)
But anyway you should take a look closer how correctly phpredis will work in this kind of environment.

[mrAndersen]

So in continuation - the exact errors I am getting are

RedisException: read error on connection to redis1:6380

OR

RedisException: socket error on read socket

[michael-grunder]

Thanks. I'll run your container myself this evening.

Those errors are what I would expect if the forked child was attempting to grab a connection actually cached by the parent but I'll need to replicate locally to say for sure.

[mrAndersen]

@michael-grunder Meanwhile, maybe I can somehow disable that caching mechanic? Maybe I can pass some specific $context into RedisCluster instance? Because really everything works fine, except for that random errors, I really can't exactly replicate when they occur, but surely this is due to some process-process problems.

Maybe I can disable persistent connection? Because as far as I understood I really don't need that feature, because my processes simply live forever, and as far as I understood the connection will stay connected as long as process lives even if it isn't set persisted during object creation, because this is not php-fpm with request-response dying pattern. (But I do need that keepalive up to "timeout" from redis-server itself obviously)

Anyway thx for help

[michael-grunder]

Maybe I can disable persistent connection? Because as far as I understood I really don't need that feature

You can specify whether or not to use persistent connections when constructing a RedisCluster object. You're currently requesting a persistent connection, so this is potentially the issue.

// change `true` to false in the 5th argument
$this->pool[$pid] = new RedisCluster(
    null, ["redis:6379", "redis1:6380", "redis2:6381"], 10, 10, true, "");

There are INI settings involved as well.

redis.pconnect.pooling_enabled
redis.clusters.persistent

There is also the slot cache, although I'm not sure why that would cause an issue.

redis.clusters.cache_slots

[mrAndersen]

Thx for advise. For the record. I managed to make it working setting persistent = false during object creation and setting redis.pconnect.pooling_enabled = 0, however the actual TCP connections established from each process are indeed not getting closed and working fine, so as I mentioned I really don't need persistent to be set to true, because processes are not dying anyway.