Filter
Top Products
9 Programming and User Environment
This chapter contains information that applies to the programming and user environment.
9.1 MPI and OFED InfiniBand Stack Fork Restrictions
With the introduction of the OFED InfiniBand stack in this release, MPI applications cannot call
fork(), popen(), and system() between MPI_Init and MPI_Finalize. This is known to
affect some applications like NWChem.
9.2 InfiniBand Multiple Rail Support
HP-MPI provides multiple rail support on OpenFabric through the MPI_IB_MULTIRAIL
environment variable. This environment variable is ignored by all other interconnects. In multi-rail
mode, a rank can use up to all cards on its node, but it is limited to the number of cards on the
node to which it is connecting.
For example, if rank A has three cards, rank B has two cards, and rank C has three cards, then
connection A--B uses two cards, connection B--C uses two cards, and connection A--C uses three
cards. Long messages are striped among all the cards on that connection to improve bandwidth.
By default, multi-card message striping is off. To turn it on, specify -e MPI_IB_MULTIRAIL=N
where N is the number of cards used by a rank:
• If N <= 1, message striping is not used.
• If N is greater than the maximum number of cards M on that node, all M cards are used.
• If 1 < N <= M, message striping is used on N cards or less.
If you specify -e MPI_IB_MULTIRAIL , the maximum possible cards are used.
On a host, all the ranks select all the cards in a series. For example: given 4 cards and 4 ranks per
host:
• rank 0 will use cards 0, 1, 2, 3
• rank 1 will use cards 1, 2, 3, 0
• rank 2 will use cards 2, 3, 0, 1
• rank 4 will use cards 3, 0, 1, 2
The order is important in SRQ mode because only the first card is used for short messages. The
selection approach allows short RDMA messages to use all the cards in a balanced way.
For HP-MPI 2.2.5.1 and older, all cards must be on the same fabric.
9.3 Benign Messages From HP-MPI Version 2.2.5.1
When running jobs with XC Version 3.2, OFED InfiniBand, and HP-MPI Version 2.2.5.1 the
following messages are printed once for each rank:
libibverbs: Warning: fork()-safety requested but init failed
HP-MPI Version 2.2.5.1 has support for fork() using OFED 1.2, but only for kernels more recent
than version 2.6.12. HP XC Version 3.2 is currently based on kernel version 2.6.9. This message
is a reminder that fork() is not supported in this release.
You can suppress this message by defining the MPI_IBV_NO_FORK_SAFE environment variable,
as follows:
% /opt/hpmpi/bin/mpirun -np 4 -prot -e MPI_IBV_NO_FORK_SAFE=1 -hostlist nodea,nodeb,nodec,noded /my/dir/hello_world
9.1 MPI and OFED InfiniBand Stack Fork Restrictions 35