Install-PackageProvider NuGet failing on docker even with TLS12 present

Question

I am trying to install Nuget package manager in a docker image

WARNING: MSG:UnableToDownload
«https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409» «»
WARNING: Unable to download from URI
'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409' to ''.
WARNING: Unable to download the list of available providers. Check your
internet connection.
Install-PackageProvider : No match was found for the specified search criteria
for the provider 'NuGet'. The package provider requires 'PackageManagement'
and 'Provider' tags. Please check if the specified package has the tags.
At line:1 char:128
+ ... :SecurityProtocol; Install-PackageProvider -Name NuGet -Force ; [Net. ...
+                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvi
   der:InstallPackageProvider) [Install-PackageProvider], Exception
    + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.Pac
   kageManagement.Cmdlets.InstallPackageProvider

Tls 12 is present on the machine. I have set the user to Container admin

USER ContainerAdministrator

Output of

RUN powershell [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; [Net.ServicePointManager]::SecurityProtocol; Install-PackageProvider -Name NuGet -Force ; [Net.ServicePointManager]::SecurityProtocol

is

Tls12
WARNING: MSG:UnableToDownload
«https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409» «»
WARNING: Unable to download from URI
'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409' to ''.
WARNING: Unable to download the list of available providers. Check your
internet connection.
Install-PackageProvider : No match was found for the specified search criteria
for the provider 'NuGet'. The package provider requires 'PackageManagement'
and 'Provider' tags. Please check if the specified package has the tags.
At line:1 char:128
+ ... :SecurityProtocol; Install-PackageProvider -Name NuGet -Force ; [Net. ...
+                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvi
   der:InstallPackageProvider) [Install-PackageProvider], Exception
    + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.Pac
   kageManagement.Cmdlets.InstallPackageProvider

Tls, Tls11, Tls12

Details about the image:

Base image: mcr.microsoft.com/windows/servercore:ltsc2019

PS Version: 5.1.17763.4644