Implement flexible rate limits with X-Ratelimit headers

Question

I have limits for API accessing: 100 requests per second. But, I also want to use power of X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Reset headers that API returns. So, if the rate limits will be changed I can use new limit and stay with these limits without global constant modifications.

If I got X-Ratelimit-Remaining=0 I have to wait (lock goroutine) until X-Ratelimit-Reset and only then I can perform new requests to API.

package main

import (
    "context"
    "fmt"
    "time"

    "golang.org/x/time/rate"
)

func main() {
    l := rate.NewLimiter(rate.Every(time.Second*5), 100)

    for i := 0; i < 200; i++ {
        when := time.Now()
        if err := l.Wait(context.TODO()); err != nil {
            panic(err)
        }
        if i == 100 {
            // Too Many Requests
            l.SetLimitAt(time.Now().Add(time.Second*5), rate.Every(time.Second*5)) // X-Ratelimit-Reset
            l.SetBurstAt(time.Now().Add(time.Second*5), 100)                       // X-Ratelimit-Remaining
            fmt.Println("Got new limits")

        }
        fmt.Println(i, "Done", time.Now().Sub(when).Milliseconds(), "ms")
    }
}

The first 100 requests were burst, it's OK. Then 101 request is locked, it's OK! But the next 99 requests are waiting for 5 seconds every time. But it has to be performed without locking! What am I doing wrong? Or maybe I have misunderstanding about rate limits idiom?