Include X-Forwarded-Port header on requests through an Istio proxy

Question

I'm looking at enabling Istio across a cluster. Inside the cluster are several services that generate links for clients to follow. Since enabling the istio sidecars the links are being generated incorrectly.

A request will be made to service a at something like http://service-a.foo.svc.cluster.local:7001/x/y/z, but the response from that will be missing the port on any generated links. i.e. http://service-a.foo.svc.cluster.local/x/y/z

It looks like this is because the envoy proxy sidecar is not including an X-Forwarded-Port header, which the service needs to construct a viable link.

Is it possible to configure the sidecars to include that header? Envoy docs suggest some config that is required to add the header, but I don't see how to do that in Istio.

We've had some success with VirtualService resources where the X-Forwarded-Port is manually specified, but we'd have to roll that out to every affected service, and it seems like a workaround rather than a proper fix.

Answer 1

I have made good progress with this. First I should signpost to some links that really helped.

An issue was raised on envoyproxy's github about this exact type of issue. One of the comments talks about fixing this in an Istio deployment using an EnvoyFilter.

I also used this link to debug the headers that were coming through the sidecar. Very useful to see what was going on.

In that comment, the user adds an X-Forwarded-For header specifically for the Spring Actuator endpoints that require it.

I've adapted that answer and applied a filter to the workload in question. I check for the presence of the X-Forwarded-For header, and only add it if one wasn't already specified. In debugging this, I saw many requests coming into the pod with the header set, and envoy happily passes that along - it just doesn't seem to add the header itself.

Here is my filter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: eo-cloud-envoy-fiter
  namespace: foo
spec:
  workloadSelector:
    labels:
      app: cre-services
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value: # lua filter specification
          name: envoy.filters.http.lua
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
            inlineCode: |
              local function isempty(s)
                return s == nil or s == ''
              end
              
              local function print_request_details(request_headers)
                local authority = request_headers:get(":authority")
                local path = request_headers:get(":path")
                print(string.format("Authority = %s", authority))
                print(string.format(Path = "%s", path))
              end
              
              local function add_forwarded_port(request_handle)
                local request_headers = request_handle:headers()
                local forwarded_port_header = request_headers:get("x-forwarded-port")
                if isempty(forwarded_port_header) then
                  local localAddressWithPort = request_handle:streamInfo():downstreamLocalAddress()
                  local t={}
                  
                  for str in string.gmatch(localAddressWithPort, "([^"..":".."]+)") do 
                    table.insert(t, str)
                  end
                  print(string.format("x-forwarded-port header is missing, adding it with value: %s", t[#t]))
                  request_handle:headers():add("X-Forwarded-Port", t[#t])
                else
                  print("Request already had an X-Forwarded-Port header. No change necessary") 
                end                
              end
              
              function envoy_on_request(request_handle)
                print_request_details(request_handle:headers())
                add_forwarded_port(request_handle)
                print("")
              end

Here is the log output from the Istio proxy, showing a request having the header added using the port that was provided:

2023-07-07T13:08:54+01:00 /agent-service/devtenant/agents/admin/permissions?profile=SystemAdmin: 2023-07-07T13:08:54+01:00 x-forwarded-port header is missing, adding it with value: 7001