I've recently set up an ECS Fargate service in a private subnet having CannotPullContainerError: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open: unexpected status code https://xxx.dkr.ecr.ap-southeast-2.amazonaws.com/v2/xxx/blobs/sha256:xxx: 403 Forbidden error during startup. I've done some troubleshooting myself, checked private subnet routes, ACLs, service task execution role and security group, which all seem correct (comparing to a running env). But I'm still getting this error, I'm hoping to get some help, many thanks!
Checked VPC private subnet and ACL - routes to local within VPC, to 0.0.0.0 via NAT GW. ACL now allows all traffic for troubleshooting purpose.
ECS task exec role - allow ecr:* for troubleshooting purpose.
ECS security group - all traffic on service port (8080)
I also found something confusing, the error containing an URL:
https://xxxx.dkr.ecr.ap-southeast-2.amazonaws.com/v2/xxx/blobs/sha256:xxxx, and the sha is different from the ECR image sha, not sure if this is expected.