How do you handle failed redis connections

Question

I am using the Redis Windows implementation together with StackExchange.Redis client. My question is this, how do you handle reconnection attempts if the initial connection fails. I am thinking of the worst case scenario when all Redis master and slave services are down. The problem is that every time my app needs something from cache it will try and reconnect to Redis (if the intial connection failed) and this is very time consuming. My factory class looks like this:

 private ConnectionMultiplexer GetConnection()
        {

            if (connection != null && connection.IsConnected) return connection;

            lock (_lock)
            {
                if (connection != null && connection.IsConnected) return connection;

                if (connection != null)
                {
                    logger.Log("Redis connection disconnected. Disposing connection...");
                    connection.Dispose();
                }

                logger.Log("Creating new instance of Redis Connection");


                connection = ConnectionMultiplexer.Connect(connectionString.Value);
            }

            return connection;

        }

        public IDatabase Database(int? db = null)
        {
            try
            {
                return !redisConnectionValid ? null : GetConnection().GetDatabase(db ?? settings.DefaultDb);
            }
            catch (Exception ex)
            {
                redisConnectionValid = false;
                logger.Log(Level.Error, String.Format("Unable to create Redis connection: {0}", ex.Message));
                return null;
            }
        }

You can see that I am using a singleton pattern for creating the connection. If the initial connection fails I am setting a flag (redisConnectionValid) so that subsequent calls are prevented from attempting to recreate the connection (which takes approx 5-10 seconds). Is there a better approach than this? Our design goal is for our application to work as normal even if Redis caching is unavailable. We do not want the app performance to suffer because of continous Redis connection attempts that will ultimately fail/time out in worst case scenarios.

Answer 1

You should let StackExchange.Redis handle the reconnect, rather than checking IsConnected yourself. Here is the pattern we recommend:

private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => {
    return ConnectionMultiplexer.Connect("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");
});

public static ConnectionMultiplexer Connection {
    get {
        return lazyConnection.Value;
    }
}

Note that "abortConnect" is set to "false". This means if the first connection attempt fails, ConnectionMultiplexer will retry in the background rather than throwing an exception.