ServiceStack Redis (AWS ElastiCache implementation) using .Net core causing error No master found in: redis-cluster-xxxxxxxx:637

Question

I have implemented the following version of ServiceStack .net Core Redis library:

ServiceStack.Redis.Core 5.9.2

I am using the library to access a Redis cache I have created to persist values for my AWS Serverless Application using .NET Core 3.1. I have paid for a commercial license for ServiceStack Redis.

Periodically and without warning, my application captures the following error when trying to create a Redis client:

Exception: System.Exception: No master found in: redis-cluster-api-prd-lcs.in-xxxxxxxx:6379
   at ServiceStack.Redis.RedisResolver.CreateRedisClient(RedisEndpoint config, Boolean master) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisResolver.cs:line 116
   at ServiceStack.Redis.RedisResolver.CreateMasterClient(Int32 desiredIndex) in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisResolver.cs:line 142
   at ServiceStack.Redis.RedisManagerPool.GetClient() in C:\BuildAgent\work\b2a0bfe2b1c9a118\src\ServiceStack.Redis\RedisManagerPool.cs:line 174
   at LCSApi.UtilityCommand.Cache.IsCacheValueExists(String cacheKey, RedisManagerPool pool) in D:\a\1\s\testapi\Utility.cs:line 167
   at LCSApi.Functions.LcsConfigurationSweeper(ILambdaContext context) in D:\a\1\s\testapi\Function.cs:line 2028
Exception: System.Exception: No master found in: redis-cluster-api-prd-lcs.in

Other times, the same code works fine. My implementation is quite simple:

private readonly RedisManagerPool _redisClient;

 _redisClient = new RedisManagerPool(Environment.GetEnvironmentVariable("CACHE_URL") + ":" +
Environment.GetEnvironmentVariable("CACHE_PORT"));

        public static T GetCacheValue<T>(string cacheKey, RedisManagerPool pool)
        {
            T cacheValue;

            try
            {
                //StackExchange.Redis.IDatabase cache = Functions._redisConnect.GetDatabase();
                //string value = cache.StringGet(cacheKey);
                //cacheValue = (T)Convert.ChangeType(value, typeof(T));
                using (var client = pool.GetClient())
                {
                    client.RetryCount = Convert.ToInt32(Environment.GetEnvironmentVariable("CACHE_RETRY_COUNT"));
                    client.RetryTimeout = Convert.ToInt32(Environment.GetEnvironmentVariable("CACHE_RETRY_TIMEOUT"));
                    cacheValue = client.Get<T>(cacheKey);
                }
            }
            catch (Exception ex)
            {
                //Console.WriteLine($"[CACHE_EXCEPTION] {ex.ToString()}");
                cacheValue = GetParameterSSMFallback<T>(cacheKey);
                //Console.WriteLine($"[CACHE_EXCEPTION] Fallback SSM parameter --> {cacheValue}");
            }

            return cacheValue;
        }

It happens enough I've had to write a 'fallback' routine to fetch the value from the AWS Parameter Store where it originates from. Not ideal. Here is the Redis configuration:

I can find next to nothing about this error online anywhere. I've tried to sign up to the ServiceStack forums without success, it won't let me sign up for some reason, even though I have a commercial license. Can anyone assist?

Answer 1

The error is due to the network instance not connecting to a master instance as identified by the redis ROLE command. This could be happening during an ElastiCache failover where it eventually appears that the master instance will return under the original DNS name:

Amazon ElastiCache for Redis will repair the node by acquiring new service resources, and will then redirect the node's existing DNS name to point to the new service resources. Thus, the DNS name for a Redis node remains constant, but the IP address of a Redis node can change over time.

ServiceStack.Redis will try to connect to a master instance using all specified master connections (typically only 1). If it fails to connect to a master instance it has to give up as the client expects to perform operations on the read/write master instance.

If it's expected the master instance will return under the same DNS name we can use a custom IRedisResolver to continually retry connecting on the same connection for a master instance for a specified period of time, e.g:

public class ElasticCacheRedisResolver : RedisResolver 
{
    public override RedisClient CreateRedisClient(RedisEndpoint config, bool master)
    {
        if (master)
        {
            //ElastiCache Redis will failover & retain same DNS for master
            var firstAttempt = DateTime.UtcNow;
            Exception firstEx = null;
            var retryTimeSpan = TimeSpan.FromMilliseconds(config.RetryTimeout);
            var i = 0;
            while (DateTime.UtcNow - firstAttempt < retryTimeSpan) {
                try
                {
                    var client = base.CreateRedisClient(config, master:true);
                    return client;
                }
                catch (Exception ex)
                {
                    firstEx ??= ex;
                    ExecUtils.SleepBackOffMultiplier(++i);
                }
            }
            throw new TimeoutException(
              $"Could not resolve master within {config.RetryTimeout}ms RetryTimeout", firstEx);
        }
        return base.CreateRedisClient(config, master:false);
    }
}

Which you configure with your Redis Client Manager, e.g:

private static readonly RedisManagerPool redisManager;

redisManager = new RedisManagerPool(...) {
    RedisResolver = new ElasticCacheRedisResolver()
};

Note: there you should use only 1 shared instance of Redis Client Managers like RedisManagerPool so the share the same connection pool. If the class containing the redisManager is not a singleton it should be assigned to a static field ensuring the same singleton instance is used to retrieve clients.