Anti-virus in docker container - does fanotify works between host and container?

Question

I need to implement anti-virus on-access scanning solution for files inside docker containers using open-source software. Clamav On-Access works fine but have some requirements and limitations:

• require CAP_SYS_ADMIN capability for working inside a container
• needs to be run per-container, not per-host
• require 850Mb resident memory for signatures in each running container, even small one

Does this limitation - "fanotify not working for container events when watching from host", really exists or I just misconfigured ClamAV? I have no deep knowledge how fanotify works with namespaces, but it looks like kernel limitation to me.

UPDATE: Are there any workarounds for this limitation? Adding /var/lib/docker/overlay2/container_id/merged is one option, because of dynamic container nature clamd.conf needs to be updated on every container event. But even with added path ClamAV doesn't detect malicious files in the containers.

Running ClamAV per-container creates huge memory overhead, especially for small containers.

Links collection:

• fanotify events do not work between containers
• Using fanotify within Docker Containers
• Fanotify kernel interface does not support scanning inside containers
• Could not watch path /var/lib/docker/overlay2 error
• Patch for OnAccessIncludePath traversal across file systems

Answer 1

I have a solution with a patched ClamAV.

1. Must use ClamAV < 0.102.0 because of the splitting of scanning and detection: detected files can't be scanned because the path is observed from the container point of view
2. OnAccessMount doesn't work because you have to list each mount in ClamAV config then restart and docker creates mounts on the fly
3. Must use overlayfs not LVM so ClamAV can access the mount
4. OnAccessIncludePath doesn't work because the file and folder enumeration method doesn't traverse file systems (doesn't scan beyond mount for path specified)

I was able to get OnAccessIncludePath working with a patch I posted to clamav-devel mailing list: https://lists.gt.net/clamav/devel/77347#77347.

I ended up with one process using fanotify for static mounts and one using inotify to monitor /var/lib/docker ephemeral mounts. Having 2 instances is still much better than 1 per container. I did a fair bit of load testing and have had the patch in production since about the time I mailed the list.

Sophos didn't work for me but I gave up pretty quickly.

Answer 2

Yes, fanotify only monitors events in the mount namespace that it is running in.

Answer 3

Now fanotify can monitor events across the filesystem, regardless of the mount namespaces. You need to use the flag FAN_MARK_FILESYSTEM alongwith FAN_MARK_ADD (FAN_MARK_ADD | FAN_MARK_FILESYSTEM). From the fanotify_mark man page here is the snippet:

FAN_MARK_FILESYSTEM (since Linux 4.20)
       Mark the filesystem specified by pathname.  The filesystem
       containing pathname will be marked.  All the contained
       files and directories of the filesystem from any mount
       point will be monitored.