non blocking send receive: code executes arbitrarily openmpi c++

45 Views Asked by At

I am using openMPI for building a code to solve Eulers equations. For this I need to send an array of properties (cols= conservative properties, rows= node nos.) across different processes. A snippet of the code where I send and receive is as below

// SEND
for (int proc = 0; proc < numtasks; proc++) {
    if (proc != taskid) {
        size = send * 11; // typical size may be of the order of 10^3
        source = proc;
        p = &array2D[0][0];
        MPI_Send(&size, 1, MPI_INT, source, taskid, MPI_COMM_WORLD);
        MPI_Send(p, size, MPI_DOUBLE, source, taskid, MPI_COMM_WORLD);
  }
}

// RECEIVE
offset = 0;
for (int proc = 0; proc < numtasks; proc++) {
    if (proc != taskid) {
        source = proc;
        p = &array2D[offset][0];
        MPI_Recv(&recv_size[proc][0], 1, MPI_INT, source, source, MPI_COMM_WORLD,
                 &status); // recv_size is vector with rows equal to no of
                       // processors since a given process has to receive from
                       // multiple processes
        MPI_Recv(p, recv_size[proc][0], MPI_DOUBLE, source, source, MPI_COMM_WORLD,
             &status);
        offset = offset + recv_size[proc][0] / (11);
    }
}

When this code is executed in openmpiV3, it runs fine. However, when I execute the same in openmpiV1, it just stops before this block (no errors thrown) and doesn't execute further. (In V1 it runs fine for smaller size of data)

I read some posts on blocking and non blocking send and receive. So I modified the code to

// SEND
MPI_Request sendrequest, recvrequest;
for (int proc = 0; proc < numtasks; proc++) {
    if (proc != taskid) {
        size = send * 11; // typical size may be of the order of 10^3
        source = proc;
        p = &array2D[0][0];
        MPI_Send(&size, 1, MPI_INT, source, taskid, MPI_COMM_WORLD);
        MPI_Isend(p, size, MPI_DOUBLE, source, taskid, MPI_COMM_WORLD,
                  &sendrequest);
    }
}
// RECIEVE
MPI_Status status1, offset = 0;
for (int proc = 0; proc < numtasks; proc++) {
    if (proc != taskid) {
        source = proc;
        p = &array2D[offset][0];
        MPI_Recv(&recv_size[proc][0], 1, MPI_INT, source, source, MPI_COMM_WORLD,
                 &status);
        MPI_Irecv(p, recv_size[proc][0], MPI_DOUBLE, source, source, MPI_COMM_WORLD,
                  &recvrequest);
        offset = offset + recv_size[proc][0] / (11);
    }
}
MPI_Wait(&recvrequest, &status1);
    

The modified code executes successfully in openmpiV1 arbitrarily. Sometimes, it doesn't throw an error but the send and receive doesn't get completed and code gives erroneous results.

I am wondering if its the version change that is causing the issue or is my code badly written?

Is there any better way to do it?

0

There are 0 best solutions below