I am using openMPI for building a code to solve Eulers equations. For this I need to send an array of properties (cols= conservative properties, rows= node nos.) across different processes. A snippet of the code where I send and receive is as below
// SEND
for (int proc = 0; proc < numtasks; proc++) {
if (proc != taskid) {
size = send * 11; // typical size may be of the order of 10^3
source = proc;
p = &array2D[0][0];
MPI_Send(&size, 1, MPI_INT, source, taskid, MPI_COMM_WORLD);
MPI_Send(p, size, MPI_DOUBLE, source, taskid, MPI_COMM_WORLD);
}
}
// RECEIVE
offset = 0;
for (int proc = 0; proc < numtasks; proc++) {
if (proc != taskid) {
source = proc;
p = &array2D[offset][0];
MPI_Recv(&recv_size[proc][0], 1, MPI_INT, source, source, MPI_COMM_WORLD,
&status); // recv_size is vector with rows equal to no of
// processors since a given process has to receive from
// multiple processes
MPI_Recv(p, recv_size[proc][0], MPI_DOUBLE, source, source, MPI_COMM_WORLD,
&status);
offset = offset + recv_size[proc][0] / (11);
}
}
When this code is executed in openmpiV3, it runs fine. However, when I execute the same in openmpiV1, it just stops before this block (no errors thrown) and doesn't execute further. (In V1 it runs fine for smaller size of data)
I read some posts on blocking and non blocking send and receive. So I modified the code to
// SEND
MPI_Request sendrequest, recvrequest;
for (int proc = 0; proc < numtasks; proc++) {
if (proc != taskid) {
size = send * 11; // typical size may be of the order of 10^3
source = proc;
p = &array2D[0][0];
MPI_Send(&size, 1, MPI_INT, source, taskid, MPI_COMM_WORLD);
MPI_Isend(p, size, MPI_DOUBLE, source, taskid, MPI_COMM_WORLD,
&sendrequest);
}
}
// RECIEVE
MPI_Status status1, offset = 0;
for (int proc = 0; proc < numtasks; proc++) {
if (proc != taskid) {
source = proc;
p = &array2D[offset][0];
MPI_Recv(&recv_size[proc][0], 1, MPI_INT, source, source, MPI_COMM_WORLD,
&status);
MPI_Irecv(p, recv_size[proc][0], MPI_DOUBLE, source, source, MPI_COMM_WORLD,
&recvrequest);
offset = offset + recv_size[proc][0] / (11);
}
}
MPI_Wait(&recvrequest, &status1);
The modified code executes successfully in openmpiV1 arbitrarily. Sometimes, it doesn't throw an error but the send and receive doesn't get completed and code gives erroneous results.
I am wondering if its the version change that is causing the issue or is my code badly written?
Is there any better way to do it?