Libcurl Write_data is being called multiple-times leading to truncated data in buffer

Question

I have a python server running on my desktop that parsers a large CSV file and sends each row to an embedded device in response to a HTTP request as cjson file setup like so which can be received just fine when I test it on the samedevice using a C program but fails when I try to run the same code on an embedded Linux ubuntu device. The code compiles and all data is send from the server appropriately but issues arise with the recived data being randomly missing values

self.send_response(200)
            self.send_header('Content-type', 'application/json')
            self.send_header('Content-Encoding', 'chunked')  # Specify gzip encoding
            self.end_headers()
            response_data = self.read_csv_lines(param_value)
            response_json = json.dumps(response_data)

        # Compress the JSON data
        buf = io.BytesIO()
        with gzip.GzipFile(fileobj=buf, mode='w') as f:
            f.write(response_json.encode('utf-8'))
        compressed_data = buf.getvalue()

        # Send the compressed JSON response
        self.wfile.write(compressed_data)

Here is my C code for sending responses as well as handling the incoming data.

'''

size_t write_callback(void *contents, size_t size, size_t nmemb, void *userp) {
    size_t realsize = size * nmemb;
    char *data = (char *)userp;

    // Append the received data to the user buffer
    memcpy(data, curl_buffer, realsize);

    return realsize;
}

void readDatasetLineByLine(CURL* curl ,CURLcode res ,int line_number){

// Read float values from file

float value;

char url_query[50];
sprintf(url_query,"http://192.168.137.1:8080?param=%d",line_number);
curl_easy_setopt(curl, CURLOPT_URL, url_query);

// Set the callback function to handle received data
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);

// Create a buffer to store the received data
// Perform the HTTP request
res = curl_easy_perform(curl);

test_input_memory[0] = value;
 if (res != CURLE_OK) {
        fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
    } else {
        // Print the received text
        cJSON *root = cJSON_Parse(curl_buffer);
        if (root != NULL) {
            // Check if the root is an array
            if (cJSON_IsArray(root)) {
                // Iterate through the array and print the float values
                int count = 0;
                cJSON *value;
                cJSON_ArrayForEach(value, root) {
                    if (cJSON_IsNumber(value)) {
                        test_input_memory[count] = (float)value->valuedouble;
                        count += 1;
                    }
                }
            }
            // Free cJSON objects
            cJSON_Delete(root);                
        } 
        else 
        {
            fprintf(stderr, "Error parsing JSON\n");
        }
}

What i'm finding most curious is that the write_callback function runs multiple times randomly splitting up some responses but leaving others whole in a way that shows no obvious pattern. How do I get the entire response copied without any errors or missing values ?

Answer 1

I managed to solve it, the issue was that my write_callback function did not have support for being called multiple time by the same response as I was unaware of this behaviour being possible. By modifying it to keep track of an offset when the res get's broken up into multiple interactions I was able to keep the code working regardless of how lib-curl split up each request.