Unable to do curl on a service in Kubernetes

Question

I am using EKS and created a deployment. I am trying to do curl on the pod but it is not working for me if I am doing it from outside or from some other pod. But if I am doing it from within the pod it is woking fine.

Here is the Pod details.

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test get po -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP             NODE                           NOMINATED NODE   READINESS GATES
dataserver-57f54b4969-m4tss   1/1     Running   0          17h   10.39.192.4    ip-10-211-29-70.ec2.internal   <none>           <none>
utils                         1/1     Running   0          16h   10.36.128.12   ip-10-211-28-99.ec2.internal   <none>           <none>

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test get po --show-labels
NAME                          READY   STATUS    RESTARTS   AGE   LABELS
dataserver-57f54b4969-m4tss   1/1     Running   0          18h   app=dataserver,pod-template-hash=57f54b4969
utils                         1/1     Running   0          16h   <none>

Here is the service details

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dataserver   ClusterIP   172.20.126.85   <none>        8282/TCP   18h

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test describe svc dataserver
Name:              dataserver
Namespace:         ankit-test
Labels:            <none>
Annotations:       service.beta.kubernetes.io/aws-load-balancer-internal: true
Selector:          app=dataserver
Type:              ClusterIP
IP Families:       <none>
IP:                172.20.126.85
IPs:               172.20.126.85
Port:              https  8282/TCP
TargetPort:        8282/TCP
Endpoints:         10.39.192.4:8282
Session Affinity:  None
Events:            <none>

I tried doing curl on an API from within the pod and it is working fine for me.

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test exec -it dataserver-57f54b4969-m4tss -- bash
ankit@dataserver-57f54b4969-m4tss:/workdir$ curl -kv https://localhost:8282/services/healthcheck
*   Trying 127.0.0.1:8282...
* Connected to localhost (127.0.0.1) port 8282 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=CA; L=Fremont; O=ankit; OU=Data Security; CN=dataserver.ankit.com
*  start date: Jan 20 10:33:54 2023 GMT
*  expire date: Jan 19 10:33:54 2025 GMT
*  issuer: C=US; ST=CA; L=Fremont; O=ankit; OU=Data Security; CN=dataserver.ankit.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /services/healthcheck HTTP/1.1
> Host: localhost:8282
> User-Agent: curl/7.74.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: Ankit Data Server
< date: Tue, 24 Jan 2023 05:03:20 GMT
< content-length: 217
< content-type: text/plain
<
* Connection #0 to host localhost left intact
{"name":"Ankit DataServer","version":"V2","uptime":"18:14:36","buildVersion":"10.11.0.1-SNAPSHOT.1","buildDate":"2023-01-20 03:26:18 UTC","buildVersionTag":"10.11.0.1-SNAPSHOT.1","buildBranch":"origin

But When I am trying to access it from other pod, it is not working.

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test exec -it utils -- bash
root@utils:/# curl -kv https://dataserver:8282/services/healthcheck
*   Trying 172.20.126.85...
* TCP_NODELAY set

Even tested with fully qualified domain name, still not working.

root@utils:/# curl -kv https://dataserver.ankit-test.svc.cluster.local:8282/services/healthcheck
*   Trying 172.20.126.85...
* TCP_NODELAY set

Initially I thought that by default may be some network-policy will be there to deny the traffic. So I created a network policy as well and attached to the pod to allow traffic from all sources at any pod. Still no luck.

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test get networkpolicy
NAME        POD-SELECTOR     AGE
allow-all   app=dataserver   39m

[ankit@ip-10-211-20-76 ~]$ kubectl -n ankit-test describe networkpolicy allow-all
Name:         allow-all
Namespace:    ankit-test
Created on:   2023-01-24 04:30:18 +0000 UTC
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     app=dataserver
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From: <any> (traffic not restricted by source)
  Not affecting egress traffic
  Policy Types: Ingress

I don't understand what is missing here. Any help will be appreciated. Thank you in advance.