Multiple 101 responses when querying CosmosDB using Gremlin

Question

Trying to diagnose some odd behaviour from CosmosDB when querying using Gremlin:

I've got a Blazor site that is querying Cosmos. It's regularly timing out, or taking a long time to pull back results from Cosmos. I've reduced the code down to:

var gremlinServer = new GremlinServer(
    Hostname, 
    Port, 
    true, 
    $"/dbs/{Database}/colls/{Collection}", 
    AuthKey
);

using var gremlinClient = new GremlinClient(
    gremlinServer, 
    new GraphSON2Reader(), 
    new GraphSON2Writer(), 
    GremlinClient.GraphSON2MimeType
);

var query = $"g.V().has('itemid', '{PartitionKey}')";

StringBuilder sb = new StringBuilder();

try
{
    var results = await gremlinClient.SubmitAsync<dynamic>(query);
    foreach (var result in results)
    {
        sb.AppendLine(result);
    }

    Logger.LogInformation(sb.ToString());
}
catch (Exception ex)
{
    Logger.LogError($"Error executing Gremlin query: {ex.Message}");
}

When checking App Insights to see what is going on behind the scenes, I noticed a number of long running requests/redirects which seem to be causing the issues:

When it's working correctly those 101 responses return in < 100ms and the query executes correctly. However, fairly regularly they can take > 30 secs to run and the query times out.

In terms of the database, there is < 1000 records in it. Running the query directly on the DB returns in < 1 sec. I've disabled any other processes that interact with the DB, so it doesn't seem to be a rate-limit issue.

Any ideas?

Answer 1

Managed to resolve this - somewhere else in the project the WebSockets.KeepAliveInterval and ConnectionPoolSettings had been set to a non-default value which was causing this behaviour. Removing that stopped the long running 101 requests.