I have set up a Kubernetes cluster with a single master node and three worker nodes running in VMs. Within this cluster, I deployed an nginx pod and created a LoadBalancer service using MetalLB to allocate an external IP address. The LoadBalancer service is exposing port 80, and when I SSH into the VM hosting the Kubernetes cluster and curl the external IP address, I receive the expected response from nginx.
However, when attempting to access the service from my local device using the same external IP address, I encounter a connection failure with the error message: "Failed to connect to port 80 after 348 ms: Couldn't connect to server."
I suspect the issue may lie in the IP address range specified in the IPAddressPool of MetalLB. Currently, I've assigned an external IP address within the range of my master node. How can I determine the appropriate value for the IP address range in MetalLB's IPAddressPool to enable access to the LoadBalancer service from devices outside the Kubernetes cluster?
Any guidance or suggestions would be greatly appreciated.
I created an nginx pod using below command
kubectl create deploy nginx --image=quay.io/redhattraining/hello-world-nginx
The result of kubectl get deploy -oyaml looks like
apiVersion: v1
items:
- apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2024-03-30T08:34:01Z"
generation: 1
labels:
app: nginx
name: nginx
namespace: default
resourceVersion: "5748"
uid: ff20a336-075e-45cd-9a2e-29110d3e9bce
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: quay.io/redhattraining/hello-world-nginx
imagePullPolicy: Always
name: hello-world-nginx
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2024-03-30T08:34:20Z"
lastUpdateTime: "2024-03-30T08:34:20Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2024-03-30T08:34:02Z"
lastUpdateTime: "2024-03-30T08:34:20Z"
message: ReplicaSet "nginx-774b9c499f" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
kind: List
metadata:
resourceVersion: ""
IPAddressPool
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- <IP Range>
autoAssign: true
avoidBuggyIPs: false
Service
apiVersion: v1
kind: Service
metadata:
name: nginx-lb-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: LoadBalancer
kubectl get svc gives below result
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-lb-service LoadBalancer <CLUSTER-IP> <First IP in the pool> 80:30732/TCP 13h
when I take ssh into the vm which host k8s cluster and then run curl I get the expected response
<html>
<body>
<h1>Hello, world from nginx!</h1>
</body>
</html>
but when I try the same from my local device it is giving me below response
curl: (7) Failed to connect to <IP> port 80 after 348 ms: Couldn't connect to server.
when nothing was not working I created L2Advertisement like
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: nginx-expose
namespace: metallb-system
spec:
ipAddressPools:
- first-pool
But that also didn't helped.
Note: I am not using minikube or kind cluster.
The error is probably caused by the way MetalLB's IP Adress pool is configured.
In your current configuration As of right now, MetalLB is configured to automatically assign an external IP address from the range that is provided in IPAdressPool.The service is accessible from within the cluster. The connection failed because the VM itself has a route for that IP range, but your local device does not have a route for the IP range that the MetalLB uses.
To resolve your issue :
This solution should be working for your on-premises or cloud-based Kubernetes cluster since you aren't using Minikube or any similar product.
Refer to this gitlink for more information
EDIT1
That's a positive step forward. You have verified that the external IP address that metalLB assigned to your network may be reached. The fact that curl fails and traceroute reaches the IP address yet ping succeeds shows that there may be a security restriction or an application layer port 80 problem.
Follow below Troubleshooting steps to resolve the issue
Verify your firewall settings on the VMS where Kubernetes is hosted .Make sure they permit inbound traffic for the particular interface that the Kubernetes pods utilize on port 80.To confirm this, you can use tools like IPtables or the firewall control console provided by your cloud provider.
In the K8s cluster, confirm that your Nginx service is set up to listen on port 80 and offer the proper application path. To inspect the service description and verify the port configuration, use kubectl get svc nginx-lb-service -o yaml all.
To rule out any issues with local settings on your Mac, try using curl to access the service from another device on your network, such as a separate PC or phone.
Check the NGINX container logs within the k8s cluster if the preceding methods are ineffective in identifying the issue. To find out whether NGINX has reported any faults or access issues, you can use the kubectl logs deployment/NGINX command.
Remember that there are security concerns associated with exposing your service to the internet. Make that the Internet access control settings for your NGINX container are set correctly. It is not advised to expose sensitive data without appropriate authentication.
To find out if requests are reaching the NGINX container, try capturing network traffic on port 80 using a tool like tcpdump on the VM hosting the K8S service.
When utilizing a cloud provider, go to their official doc for detailed instructions on how to configure metalLB and external load balancing.