How to https `GET` a fresh copy of a file from azure storage CDN?

Question

Basically, I'd like to GET https a fresh copy during the development process out of the azure storage CDN. But I don't want to wait for 20h for the CDN to be rehydrated.

I tried to add the header Cache-control: no-cache, no-store, must-revalidate without success.

My file has Cache-control: no-cache, no-store, must-revalidate property as follows:

What's strange is that when I GET from Chrome or Safari from my Macbook Pro, I get the latest (fresh) version while when I do the GET from my ESP32-based custom PCB thru the same SSID WiFi I get the old version.

My code is as follows:

/// @brief Writes the GET on [path] and write mandatory constant headers
/// @param host The host name
/// @param path The path of he document to get
static void httpsGetWriteMandatoryHeaders(char const *hostname, char const * path) {
    client.printf("GET %s HTTP/1.1", path);             client.println();
    client.printf("Host: %s", hostname);                client.println();

    client.printf("Accept-encoding: gzip, deflate");    client.println();
    client.printf("Cache-control: no-cache, no-store, must-revalidate"); client.println();
}

bool NanoWifi :: httpsGet(
    char const *hostname, char const *path, 
    byte *buffer, size_t bufferCapacity, 
    size_t &bodyLength, bool &bufferOverflows,
    void (*downloadChunkCallback)(byte * buffer, size_t bufferReadCount, size_t contentLength),
    bool verbose
    ) {

    log("Time is: %d %d %d %02d:%02d:%02d", day(), month(), year(), hour(), minute(), second());

    char const * subModule = "httpsGet: ";
    
    if (verbose) log("%sbegin %s%s…",subModule, hostname, path);
    
    // === if you get a connection, report back via serial:
    if (!client.connect(hostname, 443)) {
        // if you didn't get a connection to the server:
        error("%sconnection failed", subModule);
        return false;
    }

    unsigned chunkIndex = 0;

    success("%sConnected!",subModule);

    // Make a HTTP request:
    httpsGetWriteMandatoryHeaders(hostname, path);

    client.printf("Range: bytes=%d-%d", chunkIndex*bufferCapacity, (chunkIndex+1)*bufferCapacity);              client.println();
    client.println();


// etc.

Note that I use Range: header to fetch small chunks as I haven't enough memory to fetch bigger chunks of the firmware file. But I don't think it makes a difference regarding the cache issue I have.

Any idea on how I should do the GET?

Answer 1

I have a solution, which I don't really get so would be very happy someone gives a rationale for this.

By setting the Accept-Encoding header with br that works. My code becomes:

/// @brief Writes the GET on [path] and write mandatory constant headers
/// @param host The host name
/// @param path The path of he document to get
static void httpsGetWriteMandatoryHeaders(char const *hostname, char const * path) {
    client.printf("GET %s HTTP/1.1", path);             client.println();
    client.printf("Host: %s", hostname);                client.println();

    client.print ("Cache-Control: no-cache, no-store, must-revalidate"); client.println();
    client.print ("Accept-Encoding: gzip, deflate, br"); client.println();
}

Also note that my file property is now set back to (which does not seem to have any effect though):

CacheControl: max-age=0

I also conformed with curl: without br, doesn't work; So this works:

curl -H "Accept-Encoding: gzip, deflate, br" <url> --output test.bin

But this does not:

curl -H "Accept-Encoding: gzip, deflate" <url> --output test.bin