HttpClient using content encoding gzip and transfer encoding chunked leads to incomplete response on connection loss

Question

I have a .NET Core 3.1 WEB API with gzip compression enabled

            services.AddResponseCompression(options =>
            {
                options.EnableForHttps = true;
                options.Providers.Add<GzipCompressionProvider>();
            });

            services.Configure<GzipCompressionProviderOptions>(options =>
            {
                options.Level = CompressionLevel.Fastest;
            });

And an endpoint that offers some JSON formatted entries

        public async Task<ActionResult> Get(int category)
        {
            var entries = await _service.getEntries(category);

            return Content(entries, MediaTypeNames.Application.Json);
        }

Response Headers

HTTP/1.1 200 OK
Date: Wed, 04 Jan 2023 08:15:35 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
Transfer-Encoding: chunked
Content-Encoding: gzip
Vary: Accept-Encoding

The client - Xamarin Forms App using .NET Standard 2.1 libs - fetches data like this

            var handler = new HttpClientHandler();
            if (handler.SupportsAutomaticDecompression)
                handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
            
            var httpClient = new HttpClient(handler );
            httpClient.Timeout = TimeSpan.FromMinutes(7);
            httpClient.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
            httpClient.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));

            var response = await httpClient.GetAsync(url,cancellationToken);
            if (response.IsSuccessStatusCode)
                return response;

// ...deserialize JSON and do stuff with it

The code works as expected if the connection to the api is not interrupted during the download of the data. But if the connection gets lost (e.g. out of Wi-Fi range), the response still contains a HTTP 200 status code. This behavior might be expected, as that is how chunked transmission works (Header first, then content by chunks).

Still, I didn't expect the client not to throw an error and let the code continue until it ultimately runs into a parsing error.

Shouldn't the request fail because the last byte and the end message are missing? What am I doing wrong or how should this problem be handled?

I could disable chunking by setting a content length in the response header, or by some configuration?

But that doesn't seem to be the best approach.

Update:

Maybe I have a wrong understanding of how AutomaticDecompression works. If I omit this setting and use the default HttpClient without automatic decompression and the connection is interrupted, the following exception occurs:

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: Network subsystem is down.
      at System.Net.Http.HttpConnection.SendAsyncCore (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x012bb] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:745 
  at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync (System.Net.Http.HttpConnection connection, System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x000e6] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:330 
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x00101] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:296 
  at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00070] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:32 
  at System.Net.Http.HttpClient.FinishSendAsyncBuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x0017e] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:506

When using AutomaticDecompression, no exception is thrown, but the response is incomplete.

What is the reason for this? I would expect the decompression handler to check whether the last (zero-length) chunk is present or not and behave appropriately.

Answer 1

At first, you can try to compare the content length between the begin and the end to check if the transfer is completed or not.

Actually, the connection lost must cause a error or a problem for the user unless you get the full response content before the connection interred. So you can just use the try catch to deal with the exception and try to get the data from the web api again.

No matter the connection lost or the content wrong. You can just add a try catch when you deserialize JSON. Or you can just use the HttpClient.GetAsync() to get the web api data.

Generally speaking, the try catch is the best solution I can think about. Maybe there are some better solution.