2. Getting help

If you have a problem or question, it is highly recommended that you execute the following steps in order. Many people have similar problems with configuration and initial setup of Open MPI — it is possible that your question has already been answered.

  1. Validate your Open MPI installation. This ensures that you have a nominally-correct Open MPI installation.

  2. Check prior GitHub issues and see if others have asked the same question and had it answered.

  3. Check the mailing list archives “search” features (or use Google) to check old posts and see if others have asked the same question and had it answered.

  4. If you do not find a solution to your problem in the above resources, proceed to the Where to send? section.

2.1. Where to send?

Different types of questions and problems should be sent to different places. If you have:

  1. A general end-user question or problem: you should probably subscribe to the Open MPI user’s mailing list and post it there.

    Note

    Because of spam, only subscribers to the mailing list are allowed to post to the mailing list. Specifically: you must subscribe to the mailing list before posting.

    Important

    The more information you include in your report, the better. E-mails/bug reports simply stating, “It doesn’t work!” are not helpful; we need to know as much information about your environment as possible in order to provide meaningful assistance.

    The best way to get help is to provide a “recipe” for reproducing the problem. This will allow the Open MPI developers to see the error for themselves, and therefore be able to fix it.

    Important

    Please use a descriptive “subject” line in your email! Some Open MPI question-answering people decide whether to read a mail based on its subject line (e.g., to see if it’s a question that they can answer). So please please please use a good subject line that succinctly describes your problem.

  2. A bug report: you should probably post it to Open MPI’s Github issue tracker. Follow the template to submit all the requested information.

  3. A patch, bug fix, or other code submission: please post a Github Pull Request to the Open MPI Github repository.

  4. A developer-level / internal question about Open MPI itself: you should probably subscribe to the Open MPI developer’s mailing list and post it there.

If you’re unsure where to send your question, subscribe and send an email to the user’s mailing list (i.e., option #1, above).

2.2. For problems building or installing Open MPI

If you cannot successfully configure, build, or install Open MPI, please provide all of the following information:

  1. The version of Open MPI that you’re using.

  2. The stdout and stderr from running configure.

  3. All config.log files from the Open MPI build tree.

  4. Output from when you ran make V=1 all to build Open MPI.

  5. Output from when you ran make install to install Open MPI.

The script below may be helpful to gather much of the above information (adjust as necessary for your specific environment):

#!/usr/bin/env bash

set -euxo pipefail

# Make a directory for the output files
dir="`pwd`/ompi-output"
mkdir $dir

# Fill in the options you want to pass to configure here
options=""
./configure $options 2>&1  | tee $dir/config.out
tar -cf - `find . -name config.log` | tar -x -C $dir -

# Build and install Open MPI
make V=1 all 2>&1          | tee $dir/make.out
make install 2>&1          | tee $dir/make-install.out

# Bundle up all of these files into a tarball
filename="ompi-output.tar.bz2"
tar -jcf $filename `basename $dir`
echo "Tarball $filename created"

Then attach the resulting ompi-output.tar.bz2 file to your report.

Caution

The mailing lists have a 150 KB size limit on messages (this is a limitation of the mailing list web archives). If attaching the tarball makes your message larger than 150 KB, you may need to post the tarball elsewhere and include a link to that tarball in your mail to the list.

2.3. For problems launching MPI or OpenSHMEM applications

If you cannot successfully launch simple applications across multiple nodes (e.g., the non-MPI hostname command, or the MPI “hello world” or “ring” sample applications in the examples/ directory), please provide all of the information from the For problems building or installing Open MPI section, and all of the following additional information:

  1. The output of the ompi_info --all command from the node where you are invoking mpirun(1).

  2. If you have questions or problems about process mapping or binding, send the output from running the lstopo -v and lstopo --of xml commands from a recent version of Hwloc.

  3. If running on more than one node, also include the output of the ompi_info --version command from each node on which you are trying to run.

  4. The output of running mpirun --map-by ppr:1:node --prtemca plm_base_verbose 100 --prtemca rmaps_base_verbose 100 --display alloc hostname. Add in a --hostfile argument if needed for your environment.

The script below may be helpful to gather much of the above information (adjust as necessary for your specific environment).

Note

It is safe to run this script after running the script from the building and installing section.

#!/usr/bin/env bash

set -euxo pipefail

# Make a directory for the output files
dir="`pwd`/ompi-output"
mkdir -p $dir

# Get installation and system information
ompi_info --all 2>&1       | tee $dir/ompi-info-all.out
lstopo -v                  | tee $dir/lstopo-v.txt
lstopo --of xml            | tee $dir/lstopo.xml

# Have a text file "my_hostfile.txt" containing the hostnames on
# which you are trying to launch
for host in `cat my_hostfile.txt`; do
    ssh $host ompi_info --version 2>&1 | tee $dir/ompi_info-version-$host.out
    ssh $host lstopo -v                | tee $dir/lstopo-v-$host.txt
    ssh $host lstopo --of xml          | tee $dir/lstopo-$host.xml
done

# Have a my_hostfile.txt file if needed for your environment, or
# remove the --hostfile argument altogether if not needed.
set +e
mpirun \
     --hostfile my_hostfile.txt \
     --map-by ppr:1:node \
     --prtemca plm_base_verbose 100 \
     --prtemca rmaps_base_verbose 100 \
     --display alloc \
     hostname 2>&1                     | tee $dir/mpirun-hostname.out

# Bundle up all of these files into a tarball
filename="ompi-output.tar.bz2"
tar -jcf $filename `basename $dir`
echo "Tarball $filename created"

2.4. For problems running MPI or OpenSHMEM applications

If you can successfully launch parallel MPI or OpenSHMEM applications, but the jobs fail during the run, please provide all of the information from the For problems building or installing Open MPI section, all of the information from the For problems launching MPI or OpenSHMEM applications section, and then all of the following additional information:

  1. A detailed description of what is failing. The more details that you provide, the better. Please include at least the following information:

    • The exact command used to run your application.

    • Any relevant MCA parameters that were set (or unset) when you ran (from either the command line, environment, parameter file, etc.).

    • The value of the PATH and LD_LIBRARY_PATH environment variables (did you set them correctly to point to all relevant executables, the Open MPI libraries, and any required support libraries, such as libraries required for high-speed networks such as InfiniBand).

  2. The source code of a short sample program (preferably in C or Fortran) that exhibits the problem.

  3. If you are experiencing networking problems, include detailed information about your network.

    Error

    TODO Update link to IB FAQ entry.

    1. For RoCE- or InfiniBand-based networks, include the information in this FAQ entry.

    2. For Ethernet-based networks (including RoCE-based networks), include the output of the ip addr command (or the legacy ifconfig command) on all relevant nodes.

      Note

      Some Linux distributions do not put ip or ifconfig in the default PATH of normal users. Try looking for it in /sbin or /usr/sbin.