During tsung load testing tcp connections are getting closed after 1024 connections

Question

I am working on distributed load testing with tsung to test my mqtt message broker. My message broker can ideally handle 10k connections now. When iam testing it with tsung for 10k parallel connections i realized that my tcp connections in server are getting closed.

I set the port range and increased ulimit but still i could generate users with tsung but couldn't get 10k concurrent connections at server.

I even tested the broker with other tool called mqtt-bench in this i could generate parallel connections and here the tcp connections are not getting closed. Is there any configuration that iam missing on tsung?

and tsung version is 1.7.0, erlang version is 10.1

Answer 1

This is most likely you hitting the default ulimits for the server process.

Normally processes are limited to 1024 open file handles at a time (a open socket is backed by a file handle).

You can use the ulimit command to see what the current soft/hard limits are and change the limit temporarily up to the hard limit. You can make permanent changes by editing the /etc/security/limits.conf file.

Answer 2

I had similar issues when handling multiple clients. One of the reason is that you are using same client id to connect to the mqtt server.

For example if client A is connected to MQTT server with name(client1) and then client B is connected to MQTT server with the same name(client2) then it will disconnect the old connection. It can be one of the simplest reason

Answer 3

Without proof, we'll take your word for it that with mqtt-bench you get more than 1024 connections. (A netstat -a -n | grep 1883 | wc -l would be sufficient to prove it to me.)

If so, that would indicate that the problem is not with any broker configuration, but with your TSUNG configuration. Ie. you don't prove that your tsung process has higher limits, eg. cat /proc/PROCNUMBER/limits where PROCNUMBER is the process number of your tsung.

Once you have proven this, then next step would be to start the max number of connections that works with TSUNG, then start a wireshark, and start one more connection with some other client. The PCAP will show exactly what's happening. If the new connection fails, then the reasons would be different than if the connection succeeds, and some other client gets disconnected.

We do this all the time, as detailed in this blog post https://gambitcomm.blogspot.com/2016/06/how-is-realistic-mqtt-testing-different.html and others on our blog.

Answer 4

Have you increased the file descriptor limit as fs.file-max = 100000 in the following location of the client instance /etc/sysctl.conf

As you increased the open limits the above file descriptor limit also need to be updated.

Kindly check whether you have sufficient memory in the client instance (say 8 Core, 16GB RAM). This configuration is compatible for you to create a load of 10k concurrent users with 3 transactions (connect, authenticate and send message) from the tsung script.

Note: Monitor the CPU utilization while running the test!