TrueNAS slow performance with NFS

Question

hardware and system

CPU:Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz * 2
MEMORY: 128G
POOL: 2T * 10 (raidz3) + 512G * 2 L2ARC
TRUENAS: TrueNAS-13.0-U3.1

dd test

# on truenas
root@freenas[/mnt/dev1/docker]# dd if=/dev/zero of=test.dat bs=1M count=400 conv=fsync oflag=direct
400+0 records in
400+0 records out
419430400 bytes transferred in 0.126497 secs (3315728885 bytes/sec)
root@freenas[/mnt/dev1/docker]# dd if=/dev/zero of=test.dat bs=512K count=800 conv=fsync oflag=direct
800+0 records in
800+0 records out
419430400 bytes transferred in 0.128923 secs (3253346675 bytes/sec)
root@freenas[/mnt/dev1/docker]# dd if=/dev/zero of=test.dat bs=4K count=20000 conv=fsync oflag=direct
20000+0 records in
20000+0 records out
81920000 bytes transferred in 0.135577 secs (604232041 bytes/sec)

# on nfs client (1GB network)
[root@node1 docker]# dd if=/dev/zero of=test.dat bs=1M count=400 conv=fsync oflag=direct
400+0 records in
400+0 records out
419430400 bytes (419 MB) copied, 4.01762 s, 104 MB/s
[root@node1 docker]# dd if=/dev/zero of=test.dat bs=512K count=800 conv=fsync oflag=direct
800+0 records in
800+0 records out
419430400 bytes (419 MB) copied, 4.39617 s, 95.4 MB/s
[root@node1 docker]# dd if=/dev/zero of=test.dat bs=4K count=20000 conv=fsync oflag=direct
20000+0 records in
20000+0 records out
81920000 bytes (82 MB) copied, 6.95326 s, 11.8 MB/s

nfs

# client mount 
192.168.10.16:/mnt/dev1/docker on /docker type nfs4 (rw,noatime,nodiratime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.120,local_lock=none,addr=192.168.10.16,_netdev)

other

Truenas pool already set sync disable

HELP

We see 4k test is very slow on nfs client. Can someone help me troubleshoot?

Answer 1

The problem with TrueNas is that it ships preconfigured with periodic security checks that hog the system. You should do the following:

1. Edit using vim the file /usr/local/lib/python3.9/site-packages/middlewared/etc_files/local/collectd.conf and disable python plugin by commenting the line "LoadPlugin python"

2. Go to Truenas UI, in System, reporting, and change points from 1200 to 100, so it purges and regenerate RRD data.

3. Then execute: service collectd onestatus service collectd onerestart

4. Then edit with vi in a shell session the file /etc/defaults/periodic.conf and set security_status_neggrpperm_enable="NO" See other recommended settings in https://gist.github.com/adam12/b2a645adc55c4c076d9887838ab4bae7

Edit also the file /conf/base/etc/defaults/periodic.conf because it will not persist a boot otherwise.

There is also some settings in the /data directory, in sqlite databases, this is written by the TrueNas UI, be careful.

Also if you have like me, a ntopng in your network, and see the message kex_exchange_identification pop every 30 minutes in your freenas terminal screen, then you have two options:

1. disable network device discovery in ntopng settings, or
2. Go to TrueNAS, Services menu, SSH, and change the port from 22 to other, for example 8022.

Best regards