I have an MPI program and the assignment is to create a barrier function that halts execution of all processes until all processes enter the barrier function. There is an additional timing constraint as well and the barrier function must operate within roughly 2log_2(n) units of time. When I print the timings of when the processes enter my barrier function and when they send/receive from other processes, the timings are not making sense for what I expected. Am I not understanding how MPI send works? I thought that it would block until the other process reaches the receive corresponding to that send.
This is the program code:
#include<iostream>
#include<mpi.h>
#include<time.h>
#include<windows.h>
#define MESSAGE_TAG 999
void barrier();
using namespace std;
int main(int argc, char* argv[])
{
MPI_Init(NULL, NULL);
int my_rank;
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
barrier();
MPI_Finalize();
return 0;
}
void barrier() {
int my_rank;
int world_size;
MPI_Status status;
char message[6] = "";
clock_t startTime, endTime;
double duration;
startTime = clock();
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
if (my_rank == 2) {
Sleep(5000);
}
endTime = clock();
duration = ((double)endTime - startTime) / CLOCKS_PER_SEC;
printf("Beginning of barrier for process %d at time %f\n", my_rank, duration);
if (my_rank == 0) {
message[0] = 'r';
message[1] = 'e';
message[2] = 'a';
message[3] = 'd';
message[4] = 'y';
message[5] = '\0';
}
int stepNumber = 0;
int totalStepsNeeded = (int) (2*(floor(log2(world_size))-1));
int closestPower2 = (int) pow(2, ((totalStepsNeeded/2)+1));
int excessProcs = world_size - closestPower2;
if (my_rank < closestPower2) {
while (stepNumber < totalStepsNeeded) {
if (my_rank % 2 == 0) {
int reciever = (stepNumber * 2) + 1 + my_rank;
if (reciever > closestPower2) reciever = reciever - closestPower2;
MPI_Send(message, 10, MPI_CHAR, reciever, MESSAGE_TAG, MPI_COMM_WORLD);
endTime = clock();
duration = ((double)endTime - startTime) / CLOCKS_PER_SEC;
printf("Process %d sent to process %d at time %f\n", my_rank, reciever, duration);
}
else {
int sender = my_rank - (stepNumber * 2) - 1;
if (sender < 0) sender = closestPower2 + sender;
MPI_Recv(message, 10, MPI_CHAR, sender, MESSAGE_TAG, MPI_COMM_WORLD, &status);
sender = sender;
endTime = clock();
duration = ((double)endTime - startTime) / CLOCKS_PER_SEC;
printf("Process %d recieved from process %d at time %f\n", my_rank, sender, duration);
}
stepNumber++;
}
}
if (my_rank < excessProcs) {
MPI_Send(message, 10, MPI_CHAR, my_rank + closestPower2, MESSAGE_TAG, MPI_COMM_WORLD);
}
else if (my_rank >= closestPower2) {
MPI_Recv(message, 10, MPI_CHAR, my_rank - closestPower2, MESSAGE_TAG, MPI_COMM_WORLD, &status);
}
endTime = clock();
duration = ((double)endTime - startTime) / CLOCKS_PER_SEC;
printf("End of barrier for process %d at time %f", my_rank, duration);
}
When I made process 2 sleep, processes 0,4, and 6 finish their execution before process 2 wakes up. When I printed the execution times I noticed that process 4 is sending to process 3 and is finishing its send before 3 ever reaches its receive function.
The output is:
Beginning of barrier for process 4 at time 0.000000
Process 4 sent to process 5 at time 0.000000
Process 4 sent to process 7 at time 0.000000
Process 4 sent to process 1 at time 0.000000
Process 4 sent to process 3 at time 0.000000
End of barrier for process 4 at time 0.000000
Beginning of barrier for process 5 at time 0.000000
Process 5 recieved from process 4 at time 0.001000
Process 5 recieved from process 2 at time 5.005000
Process 5 recieved from process 0 at time 5.005000
Process 5 recieved from process 6 at time 5.005000
End of barrier for process 5 at time 5.005000
Beginning of barrier for process 1 at time 0.000000
Process 1 recieved from process 0 at time 0.001000
Process 1 recieved from process 6 at time 0.001000
Process 1 recieved from process 4 at time 0.001000
Process 1 recieved from process 2 at time 5.006000
End of barrier for process 1 at time 5.006000
Beginning of barrier for process 6 at time 0.000000
Process 6 sent to process 7 at time 0.000000
Process 6 sent to process 1 at time 0.000000
Process 6 sent to process 3 at time 0.000000
Process 6 sent to process 5 at time 0.000000
End of barrier for process 6 at time 0.000000
Beginning of barrier for process 2 at time 5.003000
Process 2 sent to process 3 at time 5.004000
Process 2 sent to process 5 at time 5.004000
Process 2 sent to process 7 at time 5.005000
Process 2 sent to process 1 at time 5.005000
End of barrier for process 2 at time 5.005000
Beginning of barrier for process 0 at time 0.000000
Process 0 sent to process 1 at time 0.001000
Process 0 sent to process 3 at time 0.001000
Process 0 sent to process 5 at time 0.001000
Process 0 sent to process 7 at time 0.001000
End of barrier for process 0 at time 0.001000
Beginning of barrier for process 7 at time 0.000000
Process 7 recieved from process 6 at time 0.001000
Process 7 recieved from process 4 at time 0.001000
Process 7 recieved from process 2 at time 5.005000
Process 7 recieved from process 0 at time 5.005000
End of barrier for process 7 at time 5.005000
Beginning of barrier for process 3 at time 0.000000
Process 3 recieved from process 2 at time 5.004000
Process 3 recieved from process 0 at time 5.004000
Process 3 recieved from process 6 at time 5.004000
Process 3 recieved from process 4 at time 5.004000
End of barrier for process 3 at time 5.004000