MPI

• MPI working group
• MPI for Grids (1): MPICH-G2 http://www3.niu.edu/mpi/
• MPI for Grids (2): PACX-MPI http://www.hlrs.de/organization/amt/projects/pacx-mpi/
• MPI intro by Apple: http://developer.apple.com/hardwaredrivers/hpc/mpionmacosx.html
• MPI Tutorial (part 1): http://www.lam-mpi.org/tutorials/nd/part1/
• MPI Tutorial (part 2): http://www.lam-mpi.org/tutorials/nd/part2/
• MPI Tutorial (part 3): http://www.lam-mpi.org/tutorials/nd/part3/

• OpenMPI Workshop 06: http://www.open-mpi.org/papers/workshop-2006/
• OpenMPI FAQ: http://www.open-mpi.org/faq/
• OpenMPI FAQ (What does MPI mean to sysops?): http://www.open-mpi.org/faq/?category=sysadmin
• "8. Does Open MPI support execution in heterogeneous environments? Open MPI currently requires that the representation of C, C++, and Fortran datatypes be the same on all platforms within a single parallel application. Endian and size difference between nodes in a parallel job will cause undefined behavior in Open MPI. Support for heterogeneous run-time environments is on the to-do list, but has not yet been implemented."

OpenMPI versus MPICH

• OpenMPI is 'the new kid on the block', should become the de facto implementation of MPI (?)
• OpenMPI merges (is the successor of) LAM-MPI, PACX-MPI (MPI for grids) and FT-MPI

• "OpenMPI - recommended for macs, compiles smoothly but takes quite a while to compile. Further information is available at http://en.wikipedia.org/wiki/Open_MPI"
• "MPICH - a portable, open source MPICH. Can be emerged on Gentoo (ssh) where it will install in /usr. Further information on MPICH is available at: http://www-unix.mcs.anl.gov/mpi/mpich/, including documentation and manual pages."

• "The various MPI distributions (OpenMPI, LamMPI, MPICH) all attempt

conform to a set of MPI standards. Thus, sometimes they differ slightly in implementation, but normally a code will run on all of them (though you will need to compile it usuing the correct libraries)."

• At Darthmouth, they state:
  • Examples of Different Implementations
    • MPICH - developed by Argonne Nationa Labs (freeware)
    • MPI/LAM - developed by Indiana, OSC, Notre Dame (freeware)
    • MPI/Pro - commerical product
    • Apple's X Grid
    • OpenMPI -recent project, MPI-2 compliant, thread safe
  • Similiarities in Various Implementations
    • source code compatibility (except parallel I/O)
    • programs should compile and run as is
    • support for heterogeneous parallel architectures
      • clusters, groups of workstations, SMP computers, grids
  • Difference in Various Implementations
    • commands for compiling and linking
    • how to launch an MPI program
    • parallel I/O (from MPI-2)
    • debugging
  • Programming Approaches
    • SPMD - Single Program Multiple Data (same program on all processors)
    • MPMP - Multiple Program Multiple Data ( different programs on different processors)

MPICH on LCG howto

• Notes on gLite-3.0+MPI
• Example MPI job (jdl and shellscript)
• Another example script (MPItest.c, MPItest.jdl and MPItest.sh)
• ImprovingMpiSupport
• https://csd.vpac.org/twiki/bin/view/Tech/MpichHowTo
• Pre-gLite (i.e., LCG-2.x): "ssh hostbased authentication MUST BE well configured between all the WNs." (OBSOLETE)
• MPI_Support_with_Torque
• NA4 working group on parallel jobs
• Cal Loomis' RPMS for MPI
• leading RPMS for Torque for LCG?
• leading RPMS for Maui for LCG?

• "Unlike the LCG middleware, gLite WMS is able to support both configurations (shared and not shared) automatically for both LSF and Torque. With gLite-1.4 job wrapper will take care to mirror the working directory in all nodes dedicated to the mpi job if the home are not shared."

JMS-MPI Interoperability

• OpenMPI/Condor are OK from Condor-6.7: https://lists.cs.wisc.edu/archive/condor-users/2006-March/msg00468.shtml
• From OpenMPI FAQ:

17. How do I run with the SLURM and PBS/Torque launchers?

If support for these systems are included in your Open MPI installation (which you can check with the ompi_info command -- look for components named "slurm" and/or "tm"), Open MPI will automatically detect when it is running inside such jobs and will just "do the Right Thing."

Specifically, if you execute an mpirun command in a SLURM or a PBS/Torque job, it will automatically use the SLURM- or PBS/Torque-native mechanisms to launch and kill processes. There is no need to specify what nodes to run on -- Open MPI will obtain this information directly from SLURM or PBS/Torque. For example:

# Allocate a SLURM job with 4 nodes
shell$ srun -N 4 -A
# Now run a 4-process Open MPI job
shell$ mpirun -np 4 a.out

This will run the 4 MPI processes on the nodes that were allocated by SLURM. Similar results occur with PBS/Torque:

# Allocate a PBS job with 4 nodes
shell$ qsub -I -lnodes=4
# Now run a 4-process Open MPI job
shell$ mpirun -np 4 a.out

Other stuff

• Tool for checking correctness of MPI-code for use on Grids: http://savannah.fzk.de/projects/cg-wp2-2/
• CampusGrid LRM w/MPI support?: http://savannah.fzk.de/projects/cg-broker/
• CampusGrid Job Scheduler w/MPI support?: http://savannah.fzk.de/projects/cg-wp3-2/
• GridIe has implemented MPI in the LCG-RB: http://www.grid.ie/wiki/RbMpiSupport

-- RichardDeJong - 14 Jun 2006