How to trace individual point-to-point events of MPI collective routines?

Question

MPI Collective routines are implemented using p2p routines. I am trying to find out what p2p events (in terms of sender,receiver,message size) make up the collective routines. In other words, I want to find out which ranks are communicating with which other ranks during a collective.
Is there is a tool that can trace such events? If not, is it possible to do so somehow?

Answer 1

MPI allows you to define your own reduction operator. You could write one that prints out its inputs and output. Given a sufficiently different set of elements on the processors that would allow you to reconstruct the reduction.

Caveat #1: good MPI implementations use multiple algorithms, and switch dynamically between algorithms based on the message size. You may be better off reading the documentation. The choice of algorithms is often controlled by environment variables.

Caveat #2: if you define your own reduction it may very well be that it uses different routing from the default. For instance, your hardware may have support for short collective operations. As an example, the IBM BlueGene had a separate network for collectives, which 1. makes it untraceable and 2. is not a sequence of p2p operations.