Curl from App Container failing with Istio

Question

Have istio integrated to the k8s cluster. I am trying to curl to a etcd pod behind headless service (one of three pods) from the ubuntu pod (this pod also has envoy) in the same namespace.

I am seeing 404 page not found when I curl to the headless service etcd pods.

I tried creating a virtual service and destination rule

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: etcd-route-rule
  namespace: aks-istio-system
spec:
  hosts:
  - "*"
  http:
   - route:
      - destination:
          host: <etcd-pod>.<headless-svc-name>.ns.svc.cluster.local
          port:
            number: 2380

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: etcd-mtls-dstrule
  namespace: aks-istio-system
spec:
  host: <etcd-pod>.<headless-svc-name>.ns.svc.cluster.local
  trafficPolicy:
    tls:
     mode: ISTIO_MUTUAL

I started getting 503 error. What am I doing wrong here?

Answer 1

To set up your environment correctly and avoid the errors you're encountering with Istio and your etcd service, follow these steps:

Deploy Your etcd Cluster with a Headless Service

1. Deploy etcd Pods: Ensure your etcd pods are deployed in Kubernetes. You might be using a StatefulSet for this, which is typical for etcd clusters.

2. Create a Headless Service: Define a headless service in Kubernetes for your etcd cluster. Here's an example service definition:

apiVersion: v1
kind: Service
metadata:
  name: etcd-headless
  namespace: aks-istio-system
spec:
  ports:
  - port: 2380
    name: peer
  - port: 2379
    name: client
  clusterIP: None  # This makes the service headless
  selector:
    app: etcd  # Adjust this selector to match your etcd pods

Install and Configure Istio to enable automatic sidecar injection

kubectl label namespace aks-istio-system istio-injection=enabled

Ensure that your etcd pods have the Istio sidecar injected. If they were running before Istio was enabled, you might need to recreate the pods.

next configure Istio Networking by creating a VirtualService to manage the traffic to your etcd headless service and a DestinationRule to specify traffic policies, such as TLS settings.

virtualService.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: etcd-route-rule
  namespace: aks-istio-system
spec:
  hosts:
  - "etcd-headless.aks-istio-system.svc.cluster.local"
  http:
  - route:
    - destination:
        host: "etcd-headless.aks-istio-system.svc.cluster.local"
        port:
          number: 2379  # or 2380 depending on the port you want to access

DestinationRule.yaml

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: etcd-destination-rule
  namespace: aks-istio-system
spec:
  host: "etcd-headless.aks-istio-system.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE  # Use ISTIO_MUTUAL if you want mTLS

Now that the Istio networking objects are in place, you can test the connectivity. Since the etcd pods are likely to be part of the etcd cluster itself and might not have curl installed, you'll need a separate pod within the aks-istio-system namespace that has curl available. If you don't have a suitable pod already, you can create a temporary one for testing purposes. Here's how you can deploy a simple Ubuntu pod and then use it to test the connectivity. Create a file named ubuntu-pod.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-test
  namespace: aks-istio-system
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    command: ["/bin/sleep", "infinity"]

Deploy the pod

kubectl apply -f ubuntu-pod.yaml

Once pod is up, exec into it.

kubectl exec -it ubuntu-test -n aks-istio-system -- bash

and install curl

apt-get update && apt-get install curl -y

and finally curl http://etcd-headless.aks-istio-system.svc.cluster.local:2379/health