High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI [Electronic resources] نسخه متنی

14.3 Managing Communicators

Collective
communication simplifies the communication process but has the
limitation that you must communicate with every process in the
communicator or communication group. There are times when you may
want to communicate with only a subset of available processes. For
example, you may want to divide your processes so that different
groups of processes work on different tasks. Fortunately, the
designers of MPI foresaw that possibility and included functions that
allow you to define and manipulate new communicators. By creating new
communicators that are subsets of your original communicator,
you'll still be able to use collective
communication. This ability to create and manipulate communicators
has been described as MPI's key distinguishing
feature, i.e., what distinguishes MPI from other message passing
systems.

Communicators are composed of two parts: a group of processes and a
context. New communicators can be built by manipulating an existing
communicator or by taking the group from an existing communicator
and, after modifying that group, building a new communicator based on
that group. The default communicator
MPI_COMM_WORLD is usually the starting point, but
once you have other communicators, you can use them as
well.^[1]

^[1] Although it sounds like there is only one
default communicator, there are actually two. The other default
communicator is MPI_COMM_SELF. Since this is
defined for each process and contains only that process, it
isn't all that useful when defining new
communicators.

14.3.1 Communicator Commands

MPI provides a number of functions
for manipulating groups. The simplest way to create a new group is to
select processes from an existing group, either by explicitly
including or excluding processes. In the following example, process 0
is excluded from the group associated with
MPI_COMM_WORLD to create a new group. You might
want to do this if you are organizing your program using one process
as a master, typically process 0, and all remaining processes as
workers. This is often called a master/slave algorithm. At times, the
slave processes may need to communicate with each other without
including process 0. By creating a new communicator
(newComm in the following example), you can then
carry out the communication using collective functions.

#include "mpi.h"
#include <stdio.h>
int main( int argc, char * argv[  ] )
{
int processId, i, flag = 0;
int processes[1] = {0};
   MPI_Group  worldGroup, newGroup;
   MPI_Comm  newComm;
  
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &processId);
   MPI_Comm_group(MPI_COMM_WORLD, &worldGroup);
   MPI_Group_excl(worldGroup, 1, processes, &newGroup);
   MPI_Comm_create(MPI_COMM_WORLD, newGroup, &newComm);
  
fprintf(stderr, "Before: process: %d   Flag: %d\n", processId, flag); 
if (processId = = 1) flag = 1;  
if (processId != 0)
MPI_Bcast(&flag, 1, MPI_INT, 0, newComm);
fprintf(stderr, "After  process: %d   Flag: %d\n", processId, flag); 
   if (processId !=0)
   { MPI_Comm_free(&newComm);
   MPI_Group_free(&newGroup);
   }
MPI_Finalize( );
return 0;
}

Relevant portions of this program appear in boldface.

The first things to notice about the program are the new type
declarations using MPI_Group and
MPI_Comm. MPI_Group allows us
to define handles for manipulating process groups. In this example,
we need two group handles, one for the existing group from
MPI_COMM_WORLD and one for the new group, we are
defining. The MPI_Comm type is used to define a
variable for the new communicator being created.

14.3.1.1 MPI_Comm_group

Next, we need to extract the group
from MPI_COMM_WORLD, our starting point for the
new group. We use the function MPI_Comm_group to
do this. It takes two arguments: the first is the communicator; the
second argument is a handle used to return the group for the
specified communicator, in this case,
MPI_COMM_WORLD's group.

14.3.1.2 MPI_Group_incl and MPI_Group_excl

Once we have an existing group, we
can use it to create a new group. In this example, we exclude process
from the original group using the MPI_Group_excl
command. Exclusion is the easiest way to handle this particular case
since only one process needs to be specified.
MPI_Group_incl should be used when it is simpler
to list processes to include rather than exclude. The four arguments
to MPI_Group_incl and
MPI_Group_excl are the same: the first argument is
the original group you are using as a starting point; the second
argument is the number of processes that will be included or
excluded; the third argument is an integer array giving the ranks of
the processes to be included or excluded; and the last parameter is
the address of the group's handle.

In this example, since process 0 is excluded, we have used the array
process to list the single process rank that we
want excluded. We could have accomplished the same thing with the
array

int processes[3] = {1, 2, 3};

and the call

MPI_Group_incl(worldGroup, 3, processes, &newGroup);

Either way works fine.

14.3.1.3 MPI_Comm_create

Finally, we need to turn the new group
into a communicator. This is done with the
MPI_Comm_create command, which takes three
arguments: the original communicator, the new group, and the address
for the new communicator's handle. Once this call is
made, we have our communicator.

In the code sample given above, the next block of code shows how the
new communicator could be used. In the example, there is a variable
flag initially set to 0. It is changed in process
1 to 1 and then broadcast to the remaining processes within the new
communicator. Here is what the output for four processes looks like.

[sloanjd@amy COMM]$ mpirun -np 4 comm
Process: 0   Flag: 0
Process: 0   Flag: 0
Process: 1   Flag: 0
Process: 2   Flag: 0
Process: 3   Flag: 0
Process: 1   Flag: 1
Process: 2   Flag: 1
Process: 3   Flag: 1

Note that the value changes for every process except process 0.

There are a couple of things worth noting about how the new
communicator is used. First, notice that only the relevant processes
are calling MPI_Bcast. Process 0 has been
excluded. Had this not been done, the call in process 0 would have
returned a null communicator error since it is not part of the
communicator. The other thing to note is that the process with rank 1
in MPI_COMM_WORLD has a rank of 0 in the new
communicator. Thus, the fourth argument to
MPI_Bcast is 0, not 1.

14.3.1.4 MPI_Comm_free and MPI_Group_free

It is good housekeeping to release
any communicators or groups you are no longer using. For these two
functions, the handles will be set to
MPI_COMM_NULL and
MPI_GROUP_NULL, respectively. While releasing
these isn't absolutely necessary, it can be helpful
at times. For example, doing so may alert you to the inadvertent use
of what should be defunct groups or communicators. Each of these two
functions takes the address of the communicator or of the group as an
argument, respectively. It doesn't matter which
function you call first.

Since process 0 is not part of the new communicator in the last
example, we need to guard againt using the new communicator within
process 0. This isn't too difficult when a single
process is involved but can be a bit of a problem when more processes
are involved. So in some instances, splitting communicators is a
better approach. Here is a simple example.

#include "mpi.h"
#include <stdio.h>
int main( int argc, char * argv[  ] )
{
int processId, i, flag = 0, color = 0;
MPI_Comm    newComm;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &processId);
if (processId = = 0 || processId = = 1) color = 1;
MPI_Comm_split(MPI_COMM_WORLD, color, processId, &newComm);
fprintf(stderr, "Process: %d   Flag: %d\n", processId, flag); 
if (processId = = 0) flag = 1;  
MPI_Bcast(&flag, 1, MPI_INT, 0, newComm);
fprintf(stderr, "Process: %d   Flag: %d\n", processId, flag); 
MPI_Comm_free(&newComm);
MPI_Finalize( );
return 0;
}

Notice that, in this example, the communicator is manipulated
directly without resorting to dealing with groups.

14.3.1.5 MPI_Comm_split

The function
MPI_Comm_split is at the heart of this example. It
is used to break a communicator into any number of pieces. The first
argument is the original communicator. The second argument, often
referred to as the

color ,
is used to determine which communicator a process will belong to. All
processes that make the call to MPI_Comm_split
with the same color will be in the same communicator. Processes with
different values (or colors) will be in different communicators. In
this example, processes 0 and 1 have a color of 1 so they are in one
communicator while processes 2 and above have a color of 0 and are in
a separate communicator. (If the color is
MPI_UNDEFINED, the process is excluded from any of
the new communicators.) The third argument, often called the

key , is used to determine the rank ordering for
processes within a communicator. When keys are the same, the original
rank is used to break the tie. The last argument is the address of
the new communicator.

Table 14-1 gives a slightly more complicated
example of how this might work. Using the data in this table, three
new communicators are created. The first communicator consists of
processes A, C, and D with ranks in the new communicator of 1, 0, and
2, respectively. The second communicator consists of processes B and
E with ranks 0 and 1, respectively. The last communicator consists of
the single process F with a rank of 0. Process G is not included in
any of the new communicators.

Returning to the code given above, with four processes, two
communicators will be created. Both will be called
newComm. The first will have the original
processes 0 and 1 with the same ranks in the new communicator. The
second will have the original processes 2 and 3 with new ranks 0 and
1, respectively. Notice that a communicator is defined for every
process, all with the same name.

These two examples should give you an idea of why communicators are
useful and how they are used. Group management functions include
functions to access groups (e.g., MPI_Group_size,
MPI_Group_rank, and
MPI_Group_compare) and functions to construct
groups (e.g., MPI_Group_difference,
MPI_Group_union,
MPI_Group_incl, and
MPI_Group_range_incl). There are also a number of
different communicator management functions
(e.g., MPI_Comm_size,
MPI_Comm_dup, MPI_Comm_compare,
and MPI_Comm_create).