13.7. Source code

13.7.1. Code style

We intentionally do not have too many code conventions in the Open MPI code base.

13.7.1.1. All languages

  • 4 space tabs. No more, no less.

  • NEVER use actual tab characters; always use spaces. Both emacs and vim have secret mojo that can automatically use spaces when you hit the <TAB> key. This makes the code look the same in every browser, regardless of individual tab display settings.

13.7.1.2. C / C++

  • When comparing constants for equality or inequality, always put the constant on the left. This is defensive programming: if you have a typo in the test and miss a ! or =, you’ll get a compiler error. For example:

    /* Do this */
if (NULL == foo) { ... }

/* Because if you have a typo (i.e., = instead of ==), this will
   be a compile error rather than a subtle bug */
if (NULL = foo) { ... }

  • More defensive programming: always include blocks in curly braces { }, even if they are only one line long. For example:

    /* Do this */
if (whatever) {
    return OMPI_SUCCESS;
}

/* Not this */
if (whatever)
   return OMPI_SUCCESS;

  • Starting with Open MPI 1.7, Open MPI requires a C99-compliant compiler.

    • C++-style comments are now allowed (and preferred).

    • C99-style mixing declarations are allow allowable (and preferred).

  • ALWAYS include <level>_config.h as your first #include file, where <level> is one of ompi, oshmem, or opal — the level that you’re writing in. There are very, very few cases where this is not true (E.g., some bizarre Windows scenarios). But in 99.9999% of cases, this file should be included first so that it can affect system-level #include files if necessary.

  • Filenames and symbols must follow the prefix rule (see [e-mail thread](http://www.open-mpi.org/community/lists/devel/2009/07/6389.php)):

    • Filenames must be prefixed with <framework>_<component>.

    • Public symbols must be prefixed in components with <project>_<framework>_<component>, where <project> is one of mca, ompi, oshmem, or opal. Note that mca used to be the most common, but it has fallen out of favor compared to the other <project> prefixes. When in doubt about whether a symbol is public, be safe and add the prefix.

    • Non-public symbols must be declared static or otherwise made to not appear in the global scope.

  • ALWAYS #define macros, even for logical values.

    • The GNU Way is to #define a macro when it is “true” and to #undef it when it is “false”.

    • In Open MPI, we always #define a logical macro to be either 0 or 1 — we never #undef it.

    • The reason for this is defensive programming: if you are only checking if a preprocessor macro is defined (via #ifdef FOO or #if defined(FOO)), you will get no warning when compiling if you accidentally misspell the macro name. However, if you use the logic test #if FOO with an undefined macro (e.g., because you misspelled it), you’ll get a compiler warning or error.

      Rationale

      Misspelled macro names can be tremendously difficult to find when they are buried in thousands of lines of code; we will take all the help from the preprocessor/compiler that we can get!

    /* GNU Way - you will get no warning from the compiler if you
   misspell "FOO"; the test will simply be false */
#ifdef FOO
...
#else
...
#endif

/* Open MPI Way - you will get a warning from the compiler if you
   misspell "FOO"; the result of the test is a different value
   than whether you spelled the macro name right or not */
#if FOO
...
#else
...
#endif

13.7.1.3. Fortran

We do not have specific coding style guidelines for Fortran. Please read some of the existing Fortran code in the source code tree and try to use a similar style.

13.7.1.4. Shell scripting

Please read some of the existing shell code in the source code tree and try to use a similar style.

  • Always enclose evaluated shell variables in quotes to ensure that multi-token values are handled properly.

    # This is bad
if test $foo = bar; then

# This is good
if test "$foo" = "bar"; then

    • The one exception to this is that when doing an assignment to a shell variable from another shell variable, it is not necessary to use quotes on the right hand side:

      # This is harmless, but unnecessary
foo="$bar"

# This is actually sufficient, even for multi-token values of $bar
foo=$bar

  • Do not use the == operator for test — this is a GNU extension and can cause portability problems on BSD systems. Instead, use the single = operator.

    # This is bad
if test "$foo" == "bar"; then

# This is good
if test "$foo" = "bar"; then

13.7.1.5. m4

We do not have specific coding style guidelines for m4 (the language used to create the configure script). Please read some of the existing m4 code in the source code tree and try to use a similar style.

13.7.2. Tree layout

There are a few notable top-level directories in the source tree:

  • The main sub-projects:

    • oshmem: Top-level OpenSHMEM code base

    • ompi: The Open MPI code base

    • opal: The OPAL code base

  • config: M4 scripts supporting the top-level configure script mpi.h

  • etc: Some miscellaneous text files

  • docs: Source code for Open MPI documentation

  • examples: Trivial MPI / OpenSHMEM example programs

  • 3rd-party: Included copies of required core libraries (either via Git submodules in Git clones or via binary tarballs).

    Note

    While it may be considered unusual, we include binary tarballs (instead of Git submodules) for 3rd party projects that are:

    1. Needed by Open MPI for correct operation, and

    2. Not universally included in OS distributions, and

    3. Rarely updated.

Each of the three main source directories (oshmem, ompi, and opal) generate at least a top-level library named liboshmem, libmpi, and libopen-pal, respectively. They can be built as either static or shared libraries. Executables are also produced in subdirectories of some of the trees.

The libopen-pal top-level library is built internally in two parts:

  • libopen-pal_core Internal “core” portion of OPAL containing the essential source and MCA needed for tools like mpicc/mpirun to link against. The “core” library is not installed.

    • Includes the following MCA frameworks: backtrace, dl, installdirs, threads, timer

    • Includes all of the source under opal/class and most of opal/util

    • Includes the files suffixed with _core in opal/runtime

  • libopen-pal Includes “core” plus all of the other OPAL project sources. This is installed.

Each of the sub-project source directories have similar (but not identical) directory structures under them:

  • class: C++-like “classes” (using the OPAL class system) specific to this project

  • include: Top-level include files specific to this project

  • mca: MCA frameworks and components specific to this project

  • runtime: Startup and shutdown of this project at runtime

  • tools: Executables specific to this project

  • util: Random utility code

There are other top-level directories in each of the sub-projects, each having to do with specific logic and code for that project. For example, the MPI API implementations can be found under ompi/mpi/LANGUAGE, where LANGUAGE is c, fortran, or java.

The layout of the mca trees are strictly defined. They are of the form:

PROJECT/mca/FRAMEWORK/COMPONENT

To be explicit: it is forbidden to have a directory under the mca trees that does not meet this template (with the exception of base directories, explained below). Hence, only framework and component code can be in the mca trees.

That is, framework and component names must be valid directory names (and C variables; more on that later). For example, the TCP BTL component is located in opal/mca/btl/tcp/.

The name base is reserved; there cannot be a framework or component named base. Directories named base are reserved for the implementation of the MCA and frameworks. Here are a few examples (as of the v5.1.x source tree):

# Main implementation of the MCA
opal/mca/base

# Implementation of the btl framework
opal/mca/btl/base

# Implementation of the sysv framework
oshmem/mcs/sshmem/sysv

# Implementation of the pml framework
ompi/mca/pml/base

Under these mandated directories, frameworks and/or components may have arbitrary directory structures, however.

13.7.3. Symbol Visibility

The *_DECLSPEC macros provide a method to annotate symbols to indicate their intended visibility when compiling dynamically shared object files (e.g., libmpi.so). The macros are defined on a per project basis:

  • Open MPI: OMPI_DECLSPEC

  • Open PAL: OPAL_DECLSPEC

  • OpenSHMEM: OSHMEM_DECLSPEC

The macros expand to the appropriate compiler and platform flags for marking whether a symbol should be explicitly made public in the target project’s library namespace. The *_DECLSPEC attributes are used to declare that a symbol is to be visible outside of that library/DSO’s scope. For example, OMPI_DECLSPEC is used to control what symbols are visible in the libmpi.so scope.

Note

This is entirely related to dynamic library compilation and does not apply to static compilation.

Note

The macros were originally introduced when Open MPI supported Windows (circa Open MPI v1.0.0) and are motivated by the Windows __declspec. While support for Windows has been dropped from Open MPI, the symbol visibility macros remain.