LTTNG grab kernel events to monitor processes and process actions

Question

I'm researching LTTNG with the purpose to gather kernel events from other processes (not by a specific pid, but any process). I have managed to get kernel syscalls for read and write operations, however, the data retrieved there is not quite what i need.

I need to be able to see a live feed for when a process is created (i need the PID, process path and the parent PID), and most important, i need to see whenever a process is performing read/write operations on files (and exactly what those files are).

On Windows i can receive this using ETW tracing. I need the same thing on linux and LTTNG seems to be the closest thing to achieving that from what i've researched.

Did anybody tried to do this before?

Thank you in advance!

Answer 1

Getting the write and read syscall is a good first step!

The clone and execve syscalls will provide you information regarding the process creation.

... syscall_entry_clone: { cpu_id = 2 }, { clone_flags = 0x1200011, newsp = 0x0, parent_tid = 0x0, child_tid = 0x7F4440D7EA10 }
... syscall_exit_clone: { cpu_id = 2 }, { ret = 1606323 }
... syscall_exit_clone: { cpu_id = 1 }, { ret = 0 }  
... syscall_entry_execve: { cpu_id = 1 }, { filename = "/bin/bash", argv = 0x55FA993F7EB0, envp = 0x55FA993BAF20 }

The execve syscall provides the process path.

Indeed, it would be nice to have the pid, ppid for each event to ease analysis. This can be done using the lttng add-context command. For this case:

lttng add-context -k -t vpid
lttng add-context -k -t vppid

We then get:

... syscall_entry_clone: { cpu_id = 0 }, { vpid = 1602589, vppid = 2996 }, { clone_flags = 0x1200011, newsp = 0x0, parent_tid = 0x0, child_tid = 0x7F4440D7EA10 }
... syscall_exit_clone: { cpu_id = 0 }, { vpid = 1602589, vppid = 2996 }, { ret = 1607998 }                                              
... syscall_exit_clone: { cpu_id = 1 }, { vpid = 1607998, vppid = 1602589 }, { ret = 0 }
... syscall_entry_execve: { cpu_id = 1 }, { vpid = 1607998, vppid = 1602589 }, { filename = "/bin/bash", argv = 0x55FA993B8C00, envp = 0x55FA993BAF20 }
... syscall_exit_execve: { cpu_id = 1 }, { vpid = 1607998, vppid = 1602589 }, { ret = 0 }

I encourage you to take a look at the available context using the lttng add-context --list command.

Now onto the read/write/open/close for files, a base event setup would be:

 lttng enable-event -k --syscall write,pwrite64,writev,pwritev
 lttng enable-event -k --syscall read,pread64,readv,preadv
 lttng enable-event -k --syscall open,openat,name_to_handle_at,open_by_handle_at
 lttng enable-event -k --syscall close

The open family is important since it give the starting point of the relation between the fd number and the file path. The close syscall is important to give you the end of the relation.

As for the "live feed", the live mode would fit the bill. But if you can spare some delay, I would encourage you to look into the rotation feature.

From there you should have all the information required. Unfortunately, Trace Compass does not seem to offer a baked-in analysis for this. If you are up to the task, it should not be too hard to implement in a Babeltrace2 python plugin or simply using the python TraceCollection API.

In recap:

lttng create my_session
lttng enable-event -k --syscall clone,exec
lttng enable-event -k --syscall write,pwrite64,writev,pwritev
lttng enable-event -k --syscall read,pread64,readv,preadv
lttng enable-event -k --syscall open,openat,name_to_handle_at,open_by_handle_at
lttng enable-event -k --syscall close
lttng add-context -k -t vpid
lttng add-context -k -t vppid