Sending large std::vector using MPI_Send and MPI_Recv doesn't complete

Question

I'm trying to send a std::vector using MPI. This works fine when the the vector is small, but just doesn't work when the vector is large (more than ~15k doubles in the vector). When trying to send a vector with 20k doubles, the program just sits there with the CPU at 100%.

Here is a minimal example

#include <vector>
#include <mpi.h>

using namespace std;

vector<double> send_and_receive(vector<double> &local_data, int n, int numprocs, int my_rank) {
    MPI_Send(&local_data[0], n, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD);

    if (my_rank == 0) {
        vector<double> global_data(numprocs*n);
        vector<double> temp(n);
        for (int rank = 0; rank < numprocs; rank++) {
            MPI_Recv(&temp[0], n, MPI_DOUBLE, rank, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
            for (int i = 0; i < n; i++) {
                global_data[rank*n + i] = temp[i];
            }
        }
        return global_data;
    }
    return vector<double>();
}

int main(int args, char *argv[]) {
    int my_rank, numprocs;
    // MPI initialization
    MPI_Init (&args, &argv);
    MPI_Comm_rank (MPI_COMM_WORLD, &my_rank);
    MPI_Comm_size (MPI_COMM_WORLD, &numprocs);

    int n = 15000;
    vector<double> local_data(n);

    for (int i = 0; i < n; i++) {
        local_data[i] = n*my_rank + i;
    }

    vector<double> global_data = send_and_receive(local_data, n, numprocs, my_rank);

    MPI_Finalize();

    return 0;
}

I compile using

mpic++ main.cpp

and run using

mpirun -n 2 a.out

When I run with n = 15000 the program completes successfully, but with n = 17000 or n = 20000 it never finishes, and the two CPU's sit at 100% until I force close the program.

Does anyone know what the problem could be?

Anycorn · Accepted Answer · 2013-09-11 16:35:44Z

2

MPI_Send is a funny call. If there is enough internal buffers to store the input, it may return - the only guarantee it makes is that input buffer is not going to be needed further by MPI. However, if there isn't enough internal buffer space, the call will block until the opposite MPI_Recv call begins to receive data. See where this is going? Both processes post MPI_Send that block due to insufficient buffer space. When debugging issues like that, it helps to replace MPI_Send with MPI_Ssend.

Your possible solutions are:

Use buffered send, MPI_Bsend.
Use MPI_Sendrecv
Alternate send/recv pair so that each send has a matching recv (e.g. odd proc sends, even recvs, then vice versa).
Use non-blocking send, MPI_Isend

See http://www.netlib.org/utk/papers/mpi-book/node39.html

answered Sep 11, 2013 at 16:35

Anycorn

51.8k45 gold badges173 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Filip S. Over a year ago

Thanks a lot, I was just looking into Isend/Irecv. If I just use Isend and Irecv instead in the example, should I add MPI_Barrier(MPI_COMM_WORLD) before receiving?

Anycorn Over a year ago

No need for Irecv, you only need to have Send return. No need for barrier either.

Collectives™ on Stack Overflow

Sending large std::vector using MPI_Send and MPI_Recv doesn't complete

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related