Openshift 3 , 503 Error (No server is available to handle this request)

Question

I have created a web application using jsp/tiles/struts/mysql/tomcat. I created new project on Openshift 3 console (Openshift online) https://console.preview.openshift.com/console/ then added tomcat/mySql. I was getting 503 error sometimes and other times, same page was working as expected. 503 error came randomly for any page from my project. When I get 503 error, I refresh some no of times and it goes away, and my page is correctly displayed.

Error that I see is: "503 Service Unavailable No server is available to handle this request. "

I did some research: What I understand from this openshift 2 link: https://blog.openshift.com/how-to-host-your-java-ee-application-with-auto-scaling/

is that to correct 503 error:

SSH into your application gear using rhc ssh --app <app_name>
Change directory to haproxy/conf
change the following in haproxy.cfg option httpchk GET / to option httpchk GET /api/v1/ping
Restart the HAProxy cartridge from your local machine using RHC rhc cartridge-restart --cartridge haproxy

I dont know if it is also applicable to openshift 3. In openshift 3 where is haproxy.log, haproxy.cfg, haproxy/conf or its slightly different in openshift 3. (Nut thanks to Warrens comments, yes he saw 503 error in openshift related to HAProxy)

Now after 1 week after posting this question: I am getting Quota Reached Error. I am able to build my project but all deployments are failing. I wonder if 503 error that I was getting earlier(either completely or partially) was related to Quota reached. How should I proceed now.

curl -i localhost:8080/GEA

HTTP/1.1 302 Found Server: 
Apache-Coyote/1.1 
Location: http://localhost:8080/GEA/ 
Transfer-Encoding: chunked Date: Tue, 11 Apr 2017 18:03:25 GMT

Tomcat logs do not show any application error.

Will Readiness Probe and Liveness Probe help me? I have not set them yet. Nor do I know how to set them.

Will scaling help me (I dont know how to set it either)

Do I have to set memory/... all at maximum allowed to ensure project runs smooth?

Answer 1

For me I had a similar situation of getting 503's sometimes and sometimes getting my actual page. the reason was because you have haproxy on the frontend handling the requests. Depending on your setup you may even have a few haproxy pods and your request could be funneled between one of the pods. So as in my case one pod was working and the other not.

So basically

oc get pods -n default
NAME                            READY     STATUS    RESTARTS   AGE
docker-registry-7-i02rh         1/1       Running   0          75d
registry-console-12-wciib       1/1       Running   0          67d
router-1-533cg                  1/1       Running   3          76d
router-1-9utld                  1/1       Running   1          76d
router-1-uwf64                  1/1       Running   1          76d

As you can see in my output default namespace is where my router(haproxy) pods live. If I change to that namespace

oc project default

Then run

oc logs -f router-1-533cg

on each of the pods you will most likely find a sepcific pod that is behaving bad. You can simply delete, and the replication controller will create a new one