High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI [Electronic resources] نسخه متنی

9.4 MPICH

Message Passing Interface Chameleon
(MPICH) was developed
by William
Gropp and Ewing
Lusk and is freely available from Argonne National Laboratory
(http://www-unix.mcs.anl.gov/mpi/mpich/). Like
LAM, it is both a library and an execution environment. It runs on a
wide variety of Unix platforms and is even available for Windows NT.

Documentation can be downloaded from the web site. There are separate
manuals for each of the communication models. This chapter provides
an overview of the installation process and a description of how to
use MPICH. For more up-to-date and detailed information, you should
consult the appropriate manual for the communications model you are
using.

9.4.1 Installing

There
are five different "flavors" of
MPICH
reflecting the type of machine it will run on and how interprocess
communication has been implemented:

ch_p4

This is probably the most common version. The
"ch" is for channel and the
"p4" for portable programs for
parallel processors.

ch_p4mpd

This extends ch_p4 mode by including a set of daemons built to
support parallel processing. The MPD is for multipurpose daemon. MPD
is a new high-performance job launcher designed as a replacement for
mpirun.

ch_shmem

This is a version for shared memory or SMP systems.

globus2

This is a version for computational grids. (See http://www.globus.org for more on the

Globus project.)

ch_nt

This is a version of MPI for Windows NT machines.

The best choice for most clusters is either the ch_p4 model or
ch_p4mpd model. The ch_p4mpd model assumes a homogenous architecture
while ch_p4 works with mixed architectures. If you have a homogenous
architecture, ch_p4mpd should provide somewhat better performance.
This section will describe the ch_p4 since it is more versatile.

The first step in installing MPICH is to download the source code for
your system. MPICH is not available in binary (except for Windows
NT). Although the available code is usually updated with the latest
patches, new patches are occasionally made available, so
you'll probably want to check the patch list at the
site. If necessary, apply the patches to your download file following
the directions supplied with the patch file.

Decide where you want to install the software. This example uses
/usr/local/src/mpich. Then download the source
to the appropriate directory, uncompress it, and unpack it.

[root@fanny src]# gunzip mpich.tar.gz
[root@fanny src]# tar -xvf mpich.tar
...

Expect lots of output! Change to the directory where the code was
unpacked, make a directory for the installation, and run
configure.

[root@fanny src]# cd mpich-1.2.5.2
[root@fanny mpich-1.2.5.2]# mkdir /usr/local/mpich-1.2.5.2
[root@fanny mpich-1.2.5.2]# ./configure --prefix=/usr/local/mpich-1.2.5.2 \
> -rsh=ssh
...

As with LAM, this installation configures MPICH to use SSH.^[4] Other configuration
options are described in the installation and user's
guides.

^[4] Alternatively, you could use the environmental variable
$RSHCOMMAND to specify SSH.

Next, you'll make, install, and clean up.

[root@fanny mpich-1.2.5.2]# make
...
[root@fanny mpich-1.2.5.2]# make install
...
[root@fanny mpich-1.2.5.2]# make clean
...

Again, you'll see lots of output after each of these
steps. The first make builds the software while
the make install, which is optional, puts it in
a public directory. It is also a good idea to make the tests on the
head node.

Before you can use MPICH, you'll need to tell it
which machines to use by editing the file
machine.architecture.
For Linux clusters, this is the file
machine.LINUX and is located in the directory
../share under installation directory. If you
use the same file layout used here, the file is
/usr/local/mpich-1.2.5.2/share/machines.LINUX.
This file is just a simple list of machines with one hostname per
line. For SMP systems, you can append a :n where
n is the number of processors in the host. This
file plays the same role as the schema with LAM. (You can specify a
file with a different set of machines as a command-line argument when
you run a program if desired.)

9.4.2 User Configuration

Since individual users
don't set up schemas for MPICH, there is slightly
less you need to do compared to LAM. Besides this difference, the
user setup is basically the same. You'll need to set
the $PATH variable appropriately (and $MANPATH, if you wish). The
same concerns apply with MPICH as with LAMyou need to
distinguish between LAM and MPICH executables if you install both,
and you need to ensure the path is set for both interactive and
noninteractive logins. You'll also need to ensure
that you can log onto each machine in the cluster using SSH without a
password. (For more information on these issues, see the subsection
on user configuration under LAM/MPI.)

9.4.3 Using MPICH

Unlike
LAM, you don't need to boot or shut down the runtime
environment when running an MPICH program. With MPICH
you'll just need to write, compile, and run your
code. The downside is, if your program crashes, you may need to
manually kill errant processes on compute nodes. But this
shouldn't be a common problem. Also,
you'll be able to run programs as root provided you
distribute the binaries to all the nodes. (File access can be an
issue if you don't export root's
home directory via NFS.)

The first step is to write and
enter your program using your favorite text editor. Like LAM, MPICH
supplies a set of wrapper programs to simplify
compilationmpicc,
mpiCC, and mpif77, and
mpif90 for C, C++, FORTRAN 77, and FORTRAN 90,
respectively. Here is an example of compiling a C program:

[sloanjd@fanny sloanjd]$ mpicc -o cpi cpi.c

cpi.c is one of the sample programs included
with MPICH. It can be found in the directory
../examples/basic under the source directory.

You can see the options supplied by the wrapper program without
executing the code by using the -show option. For
example,

[sloanjd@fanny sloanjd]$ mpicc -show -o cpi cpi.c
gcc -DUSE_STDARG -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_UNISTD_H=1 -DHAVE_
STDARG_H=1 -DUSE_STDARG=1 -DMALLOC_RET_VOID=1 -L/opt/mpich-1.2.5.10-ch_p4-gcc/
lib -o cpi cpi.c -lmpich

Obviously, you'll want to use the wrapper programs
rather than type in arguments manually.

To run a program, you use the
mpirun command. Again, before the code will
run, you must have copies of the binaries on each machine and you
must be able to log into each machine with SSH without a password.
Here is an example of running the code we just compiled.

[sloanjd@fanny sloanjd]$ mpirun -np 4 cpi
Process 0 of 4 on fanny.wofford.int
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.008783
Process 2 of 4 on hector.wofford.int
Process 1 of 4 on george.wofford.int
Process 3 of 4 on ida.wofford.int

The argument -np 4 specified running the program
with four processes. If you want to specify a particular set of
machines, use the -machinefile argument.

[sloanjd@fanny sloanjd]$ mpirun -np 4 -machinefile machines cpi
Process 0 of 4 on fanny.wofford.int
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.007159
Process 1 of 4 on george.wofford.int
Process 2 of 4 on fanny.wofford.int
Process 3 of 4 on george.wofford.int

In this example, four processes were run on the two machines listed
in the file machines. Notice that each machine
was used twice. You can view the mpicc(1) and
mpirun(1) manpage for more details.

9.4.4 Testing the Installation

You can
test connectivity issues and the like with the MPICH-supplied script
tstmachines, which is located in the
../sbin directory under the MPICH installation.
This script takes the architecture as an argument. For example,

[sloanjd@fanny sloanjd]$ /usr/local/mpich-1.2.5.2/sbin/tstmachines LINUX

If all is well, the script runs and terminates silently. If there is
a problem, it makes suggestions on how to fix the problem. If you
want more reassurance that it is actually doing something, you can
run it with the -v argument.

For more thorough testing, MPICH provides a set of tests with the
distribution. You'll find a thorough collection of
tests supplied with the source files. These are in the directory
../examples/test. You run these tests by
executing the command:

[sloanjd@fanny test]$ make testing | tee make.log
...

You'll need to do this in the
test directory. This directory must be shared
among all the nodes on the cluster, so you will have to either mount
this directory on all the machines or copy its contents over to a
mounted directory. When this runs, you'll see a lot
of output as your cluster is put through its paces. The output will
be copied to the file make.log, so
you'll be able to peruse it at your leisure.

9.4.5 MPE

The
Multi-Processing Environment (MPE) library
extends MPI. MPE provides such additional facilities as libraries for
creating log files, an X graphics library, graphical visualization
tools, routines for serializing sections of parallel code, and
debugger setup routines. While developed for use with MPICH, MPE can
be used with any MPI implementation. MPE is included with MPICH and
will be built and installed. MPE includes both a library for
collecting information and a viewer for displaying the collected
information. A user's guide is available that
provides greater detail. Use of MPE is described in greater detail in
Chapter 17.

MPE
includes four
viewersupshot,
nupshot,
jumpshot-2,
and
jumpshot-3.
These are not built automatically since the software required for the
build may not be present on every machine. Both
upshot and nupshot require
Tcl/Tk and Wish. jumpshot-2 and
jumpshot-3 require Java.

There are three different output
formats for MPE log
filesalog,
an ASCII format provided for backwards compatibility;
clog,
alog's binary equivalent; and
slog,
a scalable format capable of handling very large files.
upshot reads alog files,
nupshot and jumpshot-2 read
clog files, and jumpshot-3
reads slog files. MPE includes two utilities,
clog2slog and clog2alog, to
convert between formats. The basic functionality of the viewers is
similar, so installing any one of them will probably meet your basic
needs.

Although the requirements are different, the compilation process is
similar for each tool. You can build the viewers collectively or
individually. For example, to compile
jumpshot-3, you'll need to
install Java if you don't already have it. JDK-1.1,
JDK-1.2, or JDK-1.3 can be used. (jumpshot-2
compiles only with JDK-1.1.) If you don't have the
appropriate Java, you can download it from http://www.blackdown.org or http://java.sun.com and follow the
installation directions given at the respective site. Once Java has
been installed, make sure that you add its directory to your path.
Next, change to the ../mpe/viewer/jumpshot-3
subdirectory under the MPICH directory, for example,
/usr/local/src/mpich-1.2.5.2/mpe/viewers/jumpshot-3.
Now you can configure and build jumpshot-3.

[root@fanny jumpshot-3]# ./configure
...
[root@fanny jumpshot-3]# make
...
[root@fanny jumpshot-3]# make install
...

jumpshot-3 will be installed in the
/usr/local/bin directory as
jumpshot. (You will only need to install it on
the head node.) For details on the installation of the other viewer,
see the MPE installation and user's guide.

To test your installation, you'll need to compile a
program using the -mpilog option and run the code
to create a log file.

[sloanjd@fanny sloanjd]$ mpicc -mpilog -o cpi cpi.c
[sloanjd@fanny sloanjd]$ mpirun cpi
...

When you run the code, the log file cpi.clog
will be created. You'll need to convert this to a
format that jumpshot-3 can read.

[sloanjd@fanny sloanjd]$ clog2slog cpi.clog

The conversion routines are in the directory
../mpich-1.2.5.2/bin. Now you can view the
output. Of course, you must have a graphical login for this to work.
With this command, several windows should open on your display.

[sloanjd@fanny sloanjd]$ jumpshot cpi.slog

As noted, the use of MPE will be described in greater detail in Chapter 17.