.net core MemoryCache strange behavior

Question

I have a singleton ActionFilter and I want to store data in cache. What I did:

Program.cs

builder.Services.AddSingleton<MyActionFilter>();
builder.Services.AddMemoryCache();

ActionFilter

public class MyActionFilter : ActionFilterAttribute
{
    private readonly IServiceScopeFactory _serviceScopeFactory;
    private readonly IMemoryCache _cache;
    public MyActionFilter(IServiceScopeFactory serviceScopeFactory, IMemoryCache cache)
    {
        _serviceScopeFactory = serviceScopeFactory;
        _cache = cache;
    }

Getting

if (_cache.TryGetValue(token, out int temp))
{
    await next.Invoke();
    return;
}

Setting

if (response.Success)
{
    if (response.Score >= _minimumAllowedScore)
    {
        _cache.Set(token, response.Score, new TimeSpan(0, _expirationMinutes, 0));
        await next.Invoke();
        return;
    }

It sets the value, but can't find it by the same key on the next request, count shows that there is one entry (as it should). What am I doing wrong? Also, is using singleton action filter bad for performance if I have like 5 rps max?