MPI: How to use MPI_Win_allocate_shared properly

Question

I would like to use a shared memory between processes. I tried MPI_Win_allocate_shared but it gives me a strange error when I execute the program:

Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0 internal ABORT

Here's my source:

    # include <stdlib.h>
    # include <stdio.h>
    # include <time.h>
    
    # include "mpi.h"
    
    int main ( int argc, char *argv[] );
    void pt(int t[], int s);
    
    int main ( int argc, char *argv[] )
    {
        int rank, size, shared_elem = 0, i;
        MPI_Init ( &argc, &argv );
        MPI_Comm_rank ( MPI_COMM_WORLD, &rank );
        MPI_Comm_size ( MPI_COMM_WORLD, &size );
        MPI_Win win;
        int *shared;
        
        if (rank == 0) shared_elem = size;
        MPI_Win_allocate_shared(shared_elem*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &shared, &win);
        if(rank==0)
        {
            MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, MPI_MODE_NOCHECK, win);
            for(i = 0; i < size; i++)
            {
                shared[i] = -1;
            }
            MPI_Win_unlock(0,win);
        }
        MPI_Barrier(MPI_COMM_WORLD);
        int *local = (int *)malloc( size * sizeof(int) );
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (avant): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
        
        MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, 0, win);
        
        MPI_Put(&rank, 1, MPI_INT, 0, rank, 1, MPI_INT, win);
        
        MPI_Win_unlock(0,win);
        
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (apres): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
        
        
        MPI_Win_free(&win);
        MPI_Free_mem(shared);
        MPI_Free_mem(local);
        MPI_Finalize ( );
        
        return 0;
    }
    
    void pt(int t[],int s)
    {
        int i = 0;
        while(i < s)
        {
            printf("%d ",t[i]);
            i++;
        }
        printf("\n");
    }

I get the following result:

processus 0 (avant): -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 
processus 0 (apres): 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 
processus 4 (avant): 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 
processus 4 (apres): 0 -1 -1 -1 4 -1 -1 -1 -1 -1 
Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0
internal ABORT - process 5
Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0
internal ABORT - process 6
Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0
internal ABORT - process 9

Can someone please help me figure out what's going wrong & what that error means ? Thanks a lot.

Answer 1

MPI_Win_allocate_shared is a departure from the very abstract nature of MPI. It exposes the underlying memory organisation and allows the programs to bypass the expensive (and often confusing) MPI RMA operations and utilise the shared memory directly on systems that have such. While MPI typically deals with distributed-memory environments where ranks do not share the physical memory address space, a typical HPC system nowadays consists of many interconnected shared-memory nodes. Thus, it is possible for ranks that execute on the same node to attach to shared memory segments and communicate by sharing data instead of message passing.

MPI provides a communicator split operation that allows one to create subgroups of ranks such that the ranks in each subgroup are able to share memory:

MPI_Comm_split_type(comm, MPI_COMM_TYPE_SHARED, key, info, &newcomm);

On a typical cluster, this essentially groups the ranks by the nodes they execute on. Once the split is done, a shared-memory window allocation can be executed over the ranks in each newcomm. Note that for a multi-node cluster job this will result in several independent newcomm communicators and thus several shared memory windows. Ranks on one node won't (and shouldn't) be able to see the shared memory windows on other nodes.

In that regard, MPI_Win_allocate_shared is a platform-independent wrapper around the OS-specific mechanisms for shared memory allocation.

Answer 2

There are several problems with this code and the usage. Some of these are mentioned in @Hristolliev's answer.

1. you have to run all the processes in the same node to have a intranode communicator or use "communicator split shared".
2. you need to run this code with at least 10 processes.
3. Third, local should be deallocated with free().
4. you should get the shared pointer from a query.
5. you should deallocate shared (I think this is taken care by Win_free)

This is the resulting code:

    # include <stdlib.h>
    # include <stdio.h>
    # include <time.h>
    
    # include "mpi.h"
    
    int main ( int argc, char *argv[] );
    void pt(int t[], int s);
    
    int main ( int argc, char *argv[] )
    {
        int rank, size, shared_elem = 0, i;
        MPI_Init ( &argc, &argv );
        MPI_Comm_rank ( MPI_COMM_WORLD, &rank );
        MPI_Comm_size ( MPI_COMM_WORLD, &size );
        MPI_Win win;
        int *shared;
    
    //    if (rank == 0) shared_elem = size;
    //    MPI_Win_allocate_shared(shared_elem*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &shared, &win);
    
       if (rank == 0)
       {
          MPI_Win_allocate_shared(size, sizeof(int), MPI_INFO_NULL,
                                  MPI_COMM_WORLD, &shared, &win);
       }
       else
       {
          int disp_unit;
        MPI_Aint ssize; 
          MPI_Win_allocate_shared(0, sizeof(int), MPI_INFO_NULL,
                                  MPI_COMM_WORLD, &shared, &win);
          MPI_Win_shared_query(win, 0, &ssize, &disp_unit, &shared);
       }
    
        
        if(rank==0)
        {
            MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, MPI_MODE_NOCHECK, win);
            for(i = 0; i < size; i++)
            {
                shared[i] = -1;
            }
            MPI_Win_unlock(0,win);
        }
    
        MPI_Barrier(MPI_COMM_WORLD);
    
        int *local = (int *)malloc( size * sizeof(int) );
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (avant): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
    
        MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, 0, win);
    
        MPI_Put(&rank, 1, MPI_INT, 0, rank, 1, MPI_INT, win);
    
        MPI_Win_unlock(0,win);
    
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (apres): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
    
    
        MPI_Win_free(&win);
     //   MPI_Free_mem(shared);
        free(local);
    //    MPI_Free_mem(local);
        MPI_Finalize ( );
        return 0;
    
    
    }
    
    void pt(int t[],int s)
    {
        int i = 0;
        while(i < s)
        {
            printf("%d ",t[i]);
            i++;
        }
        printf("\n");
    }