Powertools for AWS Lambda (.NET) throws IAM security credentials error in one VS project, but works in other project defined similarly

Question

I have a Visual Studio 2022 net6 AWS Serverless Application Model (SAM) project that that throws an IAM credentials exception when I try to test a function in my unit testing project. I built it using Powertools for AWS Lambda (.NET), which creates all of the necessary scaffolding (including the Test project).

The maddening thing is that I have a previous VS 2022 AWS project which was built the exact same way, using the same version of AWS Power Tools, using the exact same \.aws\credentials file and the same profile within that file, and it doesn't give these errors!

Here is the exception I receive (I have xxxed out company-identifying information):

Amazon.Runtime.AmazonServiceException
  HResult=0x80131500
  Message=Unable to get IAM security credentials from EC2 Instance Metadata Service.
  Source=AWSSDK.Core
  StackTrace:
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
   at Integrations.Documentation.Auth.POC.Lib.Secrets.SecretsHelper.<GetSecret>d__0.MoveNext() in C:\Users\xxx\source\repos\xxx\xxx\Secrets\SecretsHelper.cs:line 23

  This exception was originally thrown at this call stack:
    [External Code]
    xxx.xxx.xxx.xxx.xxx.Secrets.SecretsHelper.GetSecret(string) in SecretsHelper.cs

My FunctionTest class is defined as follows:

public class FunctionTest
{
    readonly TestLambdaContext _context;
    readonly APIGatewayProxyRequest _request;
    APIGatewayProxyResponse _response;
    readonly Functions _functions;
    readonly string dhKeySecretId = "DeveloperHub_API_Key";

    public FunctionTest()
    {
        Environment.SetEnvironmentVariable("POWERTOOLS_METRICS_NAMESPACE", "AWSLambdaPowertools");
        Environment.SetEnvironmentVariable("POWERTOOLS_TRACE_DISABLED", "1");
        _functions = new Functions();
        _request = new APIGatewayProxyRequest();
        _context = new TestLambdaContext();
        _response = new APIGatewayProxyResponse();
    }

    [Fact]
    public async Task TestGetSecret()
    {
        _request.PathParameters = new Dictionary<string, string> { { "secretId", dhKeySecretId } };
        _response = await _functions.GetSecret(_request, _context);

        Assert.Equal(200, _response.StatusCode);
        var secret = JsonSerializer.Deserialize<GetSecretValueResponse>(_response.Body);
        Assert.NotNull(secret);
    }

My Functions.cs file is defined as follows:

using System.Net;
using Amazon.Lambda.Core;
using Amazon.Lambda.APIGatewayEvents;
using AWS.Lambda.Powertools.Logging;
using AWS.Lambda.Powertools.Metrics;
using AWS.Lambda.Powertools.Tracing;
using Integrations.Documentation.Auth.POC.AWS.Util;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Integrations.Documentation.Auth.POC.AWS;

/// <summary>
/// Learn more about Powertools for AWS Lambda (.NET) at https://awslabs.github.io/aws-lambda-powertools-dotnet/
/// Powertools for AWS Lambda (.NET) Logging: https://awslabs.github.io/aws-lambda-powertools-dotnet/core/logging/
/// Powertools for AWS Lambda (.NET) Tracing: https://awslabs.github.io/aws-lambda-powertools-dotnet/core/tracing/
/// Powertools for AWS Lambda (.NET) Metrics: https://awslabs.github.io/aws-lambda-powertools-dotnet/core/metrics/
/// </summary>
public class Functions
{
    /// <summary>
    /// Retrieves a single certificate from Secrets Manager matching the unique identifier/name.
    /// </summary>
    /// <param name="request"></param>
    /// <returns>The API Gateway response.</returns>
    [Logging(LogEvent = true, CorrelationIdPath = CorrelationIdPaths.ApiGatewayRest)]
    [Metrics(CaptureColdStart = true)]
    [Tracing(CaptureMode = TracingCaptureMode.ResponseAndError)]
    public async Task<APIGatewayProxyResponse> GetSecret(APIGatewayProxyRequest request, ILambdaContext context)
    {
        Logger.LogInformation(nameof(GetSecret));

        return await FunctionHelper.GetSecret(request, context);
    }
}

FunctionHelper.GetSecret is defined thusly:

        public static async Task<APIGatewayProxyResponse> GetSecret(APIGatewayProxyRequest request, ILambdaContext context)
        {
            if (!request.PathParameters.TryGetValue("secretId", out var secretId)
                || string.IsNullOrWhiteSpace(secretId))
            {
                return await Task.FromResult(new APIGatewayProxyResponse
                {
                    StatusCode = (int)HttpStatusCode.BadRequest,
                    Body = "secretId cannot be empty."
                });
            }

            return CreateAPIGatewayProxyResponse<GetSecretValueResponse>(await SecretsHelper.GetSecret(secretId));
        }

        private static APIGatewayProxyResponse CreateAPIGatewayProxyResponse<T>(T response)
            where T : AmazonWebServiceResponse
        {
            return new APIGatewayProxyResponse
            {
                StatusCode = response == null ? (int)HttpStatusCode.PartialContent : (int)response.HttpStatusCode,
                Body = response == null ? null : JsonSerializer.Serialize(response),
                Headers = new Dictionary<string, string> { { "Content-Type", "text/plain" } }
            };
        }
    }

And finally, SecretsHelper.GetSecret() is defined as follows:

    public static class SecretsHelper
    {
        public static async Task<GetSecretValueResponse> GetSecret(string secretId)
        {
            try
            {
                var client = new AmazonSecretsManagerClient();
                var secretResp = await client.GetSecretValueAsync(new GetSecretValueRequest { SecretId = secretId });
                if (secretResp == null || secretResp.HttpStatusCode != HttpStatusCode.OK)
                {
                    var secretJson = secretResp == null ? "" : JsonSerializer.Serialize(secretResp);
                    var status = secretResp == null ? HttpStatusCode.NoContent : secretResp.HttpStatusCode;
                    throw new Exception($"Request for secret '{secretId} returned status code '{status}'.  Response: '{secretJson}'");
                }

                return secretResp;
            }
            catch (Exception ex)
            {
                return await Task.FromResult(new GetSecretValueResponse
                {
                    HttpStatusCode = HttpStatusCode.InternalServerError,
                    SecretString = ex.Message
                });
            }
        }
    }

Again, the above is more or less how my working AWS app is designed (different business logic, of course).

AWS Explorer shows that I am connected/authenticated to the AWS instance (and this is exactly what I see in the other, working project as well):

Therefore, I know that my VS Project is able to connect via my profile.

The aws-lambda-tools-defaults.json file is almost exactly the same in both projects except that s3-prefix refers to the name of my new app instead of the other app (just for the heck of it, I tried setting it to the name of the working app, but that didn't work).

What could be going wrong? (I researched other similar errors on StackOverflow, but none of them applied to my situation. I'm hoping this is just a simple fix.)

Answer 1

It turns out that I simply needed to add the nuget packages AWSSDK.SSO and AWSSDK.SSOOIDC to my project. That simple changed fixed it! I didn't need to change any application code. I hope this helps anyone else who may be struggling with the same problem.