Tsung distributed load testing throwing internal server error

Question

I am running a distributed load test for mqtt message broker using tsung. I have configured the nodes with shh keys and checked that shh is happening properly between the nodes and with single client the test is running succesfully but when i start the tsung.xml file with 2 nodes i encounter the internal server error.

I have checked on versions and paths of erlang and tsung installed in both nodes. They have same verions and paths even.

<clients>
    <client host="1.2.3.4" cpu="1" maxusers="300"/>
    <client host="2.3.4.6" cpu="1" maxusers="100"/>
</clients>
<servers>
    <server host="1.2.3.4" port="1883" type="tcp" />
</servers>

when i run tsung for reports at 1.2.3.4:8091

Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator unknown@unknown, and inform them of the time the error occurred and anything you might have done that may have caused the error.

And [email protected] says

ts_config_server:(0:<0.158.0>) Can't start newbeam on host '2.3.4.6' (reason: timeout) ! Aborting!

Is their any reason for this issue or a solution of how to solve the error?

Answer 1

You have not written what operating system you are using. I assume you are using Linux. I easily reproduced the bug (I just deleted the ssh-key line from another node's ~/.ssh/authorized_keys file).

I describe my experiences(Linux Debian):

• You need to install Tsung properly on every node. I mean, you have to able to start Tsung from your home directory.
• There must be a working ssh-key connection between the nodes (without a password of course). Important! When you use Tsung with root user, then your node will use root user to connect to another node.
• My nodes work only with a registered hostname
• Linux kernel needs some tweak

I have a Debian install script for Tsung 1.6:

wget http://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
sudo dpkg -i erlang-solutions_1.0_all.deb
yes | sudo apt-get update
yes | sudo apt-get install elixir esl-erlang build-essential git gnuplot libtemplate-perl
wget http://tsung.erlang-projects.org/dist/tsung-1.6.0.tar.gz
tar -xvf tsung-1.6.0.tar.gz
cd tsung-1.6.0/
./configure
make
sudo make install
cd ..

ssh key authentication: https://debian-administration.org/article/530/SSH_with_authentication_key_instead_of_password

Copy these lines to the end of your /etc/hosts file on every node.

1.2.3.4    n1
2.3.4.6    n2

Your config should be like this:

<clients>
    <client host="**n1**" cpu="1" maxusers="300"/>
    <client host="**n2**" cpu="1" maxusers="100"/>
</clients>
<servers>
    <server host="1.2.3.4" port="1883" type="tcp" />
</servers>

this is how I start:

tsung -k -f tsung.xml start

I use this tweak script on Linux:

# Increase system file descriptor limit
sudo sysctl -w fs.file-max=300000

# Discourage Linux from swapping idle processes to disk (default = 60)
# vm.swappiness = 10

# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.core.rmem_default=16777216
sudo sysctl -w net.core.wmem_default=16777216
sudo sysctl -w net.core.optmem_max=40960
sudo sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sudo sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
sudo sysctl net.core.netdev_max_backlog=50000
sudo sysctl net.ipv4.tcp_max_syn_backlog=30000
sudo sysctl net.ipv4.tcp_max_tw_buckets=2000000
sudo sysctl net.ipv4.tcp_tw_reuse=1
sudo sysctl net.ipv4.tcp_fin_timeout=10

# Disable TCP slow start on idle connections
sudo sysctl net.ipv4.tcp_slow_start_after_idle=0

# Disable source routing and redirects
sudo sysctl net.ipv4.conf.all.send_redirects=0
sudo sysctl net.ipv4.conf.all.accept_redirects=0
sudo sysctl net.ipv4.conf.all.accept_source_route=0

# Log packets with impossible addresses for security
sudo sysctl net.ipv4.conf.all.log_martians=1