Nuget restore throwing errors in build pipelines with compatible libraries

Question

I have a .NET Core project created in Visual Studio 2019 with target framework .NET 5.0. I am also using Azure Devops 2019 on-premise server as a repository and pipeline manager.

Everything works fine if I build and publish manually using Visual Studio and deploying it to IIS on the same build server, but running into issues when using the build pipeline on the nuget command task.

I get this error:

The nuget command failed with exit code(1) and error(NU1202: Package IdentityServer4.AspNetIdentity 4.1.2 is not compatible with net50

Which should not be the case since that version is compatible with .NET Core 3.1 or higher right?

My pipeline task looks like this:

steps:
- task: NuGetToolInstaller@0

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

I also saw similar issues in the internet and tried using Install nuget task to install 5.4 nuget version but still get the same error