How to bring Redis cluster back up if it is in a failed state?

Hi,

I have set-up a Redis cluster with 5 master nodes. I noticed there are times when the entire cluster enters a failed state e.g. when a node goes down. When I bring the affected node back up again, only sometimes the cluster returns to an OK state depending on how quickly I bring back the node once it is down. How do I ensure I can reliably and consistently bring back an entire cluster to an OK state if I detect it is in a failed state?

E.g. Redis node in a cluster that was down for 5 minutes and brought back up again:
127.0.0.1:6379> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:3
cluster_stats_messages_pong_sent:4
cluster_stats_messages_sent:7
cluster_stats_messages_ping_received:4
cluster_stats_messages_pong_received:3
cluster_stats_messages_received:7
127.0.0.1:6379> cluster nodes
212fcd0b1e80b9d22ee7c0efd5f1d8ccdaaa5ae1 10.44.163.102:6379@16379 myself,master - 0 1653291601879 2 connected

And another node in the same cluster:
127.0.0.1:6379> cluster info
cluster_state:fail
cluster_slots_assigned:16384
cluster_slots_ok:13107
cluster_slots_pfail:0
cluster_slots_fail:3277
cluster_known_nodes:5
cluster_size:5
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:57607
cluster_stats_messages_pong_sent:51714
cluster_stats_messages_fail_sent:6
cluster_stats_messages_sent:109327
cluster_stats_messages_ping_received:51710
cluster_stats_messages_pong_received:57605
cluster_stats_messages_meet_received:4
cluster_stats_messages_fail_received:1
cluster_stats_messages_received:109320
127.0.0.1:6379> cluster nodes
ee2106f0115a5d70664702dd6998283ef9a15c74 10.44.163.107:6379@16379 master - 0 1653348703379 3 connected 6554-9829
df007adf6ecc0c52ddbc8afde9b7af48c8ce562b 10.44.163.110:6379@16379 master - 0 1653348704384 5 connected 13107-16383
cd7c11c57a844c6e6cc591c15bf545e9e6b9445c 10.44.162.108:6379@16379 myself,master - 0 1653348702000 1 connected 0-3276
d7dd0a990d15048c92920a513304b87276e7e2c5 :0@0 master,fail,noaddr - 1653291291862 1653291288000 2 disconnected 3277-6553
faf3f0aae6f34a193f37bca4ec117c976a18f791 10.44.163.108:6379@16379 master - 0 1653348703000 4 connected 9830-13106

Never mind, I found the answer. The Redis server must be started from the same directory every time because Redis uses the current directory to store cluster configuration settings. If a Redis node is restarted from the same directory then the entire cluster returns back to an OK state.