Nginx + Php Fpm scaling issues on AWS EC2 instance

Question

We are performing a load test using locust(1000 Users) on a webpage of our application.

Instance type: t3a.medium The instance is running behind a load balancer. And we are using RDS Aurora Database which peaks at around 70% CPU utilization. EC2 instance metrics are healthy. EDIT: Instance memory consumption is within 800 MB out of available 4 GB

There are multiple 502 Server error: Bad Gateway and sometimes 500 and 520 errors as well.

Error 1:

2020/10/08 16:58:21 [error] 4344#4344: *41841 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: <PublicIP>, server: <Domain name>, request: "GET <webpage> HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "<Domain name>"

Error 2(Alert):

2020/10/08 19:15:11 [alert] 9109#9109: *105735 socket() failed (24: Too many open files) while connecting to upstream, client: <PublicIP>, server: <Domain name>, request: "GET <webpage> HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "<Domain name>"

Listing down configuration files:

Nginx Configuration

server {
        listen 80;
        listen [::]:80;

        root /var/www/####;
        index index.php;

    access_log /var/log/nginx/###access.log;
    error_log  /var/log/nginx/####error.log ;   

    server_name  #####;

        client_max_body_size 100M;

        autoindex off;

     location / {
        try_files $uri $uri/ /index.php?$query_string;
      }

    location ~ \.php$ {
        include fastcgi_params;
        fastcgi_intercept_errors on;
        fastcgi_index index.php;
        fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
    }
}

/etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  8096;
    multi_accept        on;
    use                 epoll;
    epoll_events        512;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay on;

    keepalive_timeout  65;

    gzip  on;
    gzip_comp_level  2;
    gzip_min_length  1000;
    gzip_types  text/xml text/css;
    gzip_http_version 1.1;
    gzip_vary  on;
    gzip_disable "MSIE [4-6] \.";
    
include /etc/nginx/conf.d/*.conf;
}

/etc/php/7.2/fpm/php-fpm.conf

  emergency_restart_threshold 10
  emergency_restart_interval 1m
  process_control_timeout 10s

Php-fpm Important Parameters:

user = www-data
group = www-data
listen = /run/php/php7.2-fpm.sock
listen.owner = www-data
listen.group = www-data
;listen.mode = 0660
pm = static
pm.max_children = 300

/etc/security/limits.conf

nginx       soft    nofile  30000
nginx       hard    nofile  50000

/etc/sysctl.conf

net.nf_conntrack_max = 131072
net.core.somaxconn = 131072
net.core.netdev_max_backlog = 65535
kernel.msgmnb = 131072
kernel.msgmax = 131072
fs.file-max = 131072

What are we missing? Can anyone please point to the right direction?

Answer 1

So we were able to resolve this issue. The problem was php-fpm did not have access to access system resources. You may need to change values according to hardware specifications. So, our final configuration looks like this:

1. In /etc/security/limits.conf, add following lines:

   nginx soft nofile 10000

   nginx hard nofile 30000

   root soft nofile 10000

   root hard nofile 30000

   www-data soft nofile 10000

   www-data hard nofile 30000

2. In /etc/sysctl.conf, add following values

   net.nf_conntrack_max = 231072

   net.core.somaxconn = 231072

   net.core.netdev_max_backlog = 65535

   kernel.msgmnb = 231072

   kernel.msgmax = 231072

   fs.file-max = 70000

3. In /etc/nginx/nginx.conf, change or add so finally it should have these values(kindly change them according to your use case and server capacity):

   worker_processes auto;

   worker_rlimit_nofile 30000;

   events { worker_connections 8096; multi_accept on; use epoll; epoll_events 512; }

   sendfile on;

   tcp_nopush on;

   tcp_nodelay on;

   keepalive_timeout 65;

   gzip on;

   gzip_comp_level 2;

   gzip_min_length 1000;

   gzip_types text/xml text/css;

   gzip_http_version 1.1;

   gzip_vary on;

   gzip_disable "MSIE [4-6] .";

4. In /etc/php/7.2/fpm/php-fpm.conf , change values to look like this:

   emergency_restart_threshold = 10

   emergency_restart_interval = 1m

   process_control_timeout = 10s

   rlimit_files = 10000

5. In /etc/php/7.2/fpm/pool.d/www.conf , change values to look like this:

   user = www-data

   group = www-data

   listen.backlog = 4096

   listen.owner = www-data

   listen.group = www-data

   ;listen.mode = 0660

   pm = static

   pm.max_children = 1000