Is there a way to see what happens during program execution in detail on Linux?

Question

I am trying to debug a performance of my program. What would be ideal is to have a way to see in detail when was the thread doing useful work, when was it blocked by page faults, when was it executing some memory writes and reads, etc...

I would simply like to have a detailed understanding of whats going on. Is it possible?

Answer 1

The linux kernel sources come with the perf tool that can measure a large number of performance counter, all of those you listed included, and can print statistics about it, annotate symbols, instructions and source lines with them (if debug symbols are available), and can track any process or also logical cpu cores.

Your Linux distribution will have the tool probably in a standalone package. Some hardening options of the kernel may limit what information root or non-root users can collect with it.

Answer 2

You can use perf and visualizing a perf output file graphically with hotspot