14.8. Internal frameworks

The Modular Component Architecture (MCA) is the backbone of Open MPI – most services and functionality are implemented through MCA components.

14.8.1. MPI layer frameworks

Here is a list of all the component frameworks in the MPI layer of Open MPI:

  • bml: BTL management layer

  • coll: MPI collective algorithms

  • fbtl: point to point file byte transfer layer: abstraction for individual read: collective read and write operations for MPI I/O

  • fcoll: collective file system functions for MPI I/O

  • fs: file system functions for MPI I/O

  • hook: Generic hooks into Open MPI

  • io: MPI I/O

  • mtl: Matching transport layer, used for MPI point-to-point messages on some types of networks

  • op: Back end computations for intrinsic MPI_Op operators

  • osc: MPI one-sided communications

  • pml: MPI point-to-point management layer

  • sharedfp: shared file pointer operations for MPI I/O

  • topo: MPI topology routines

  • vprotocol: Protocols for the “v” PML

14.8.2. OpenSHMEM component frameworks

  • atomic: OpenSHMEM atomic operations

  • memheap: OpenSHMEM memory allocators that support the PGAS memory model

  • scoll: OpenSHMEM collective operations

  • spml: OpenSHMEM “pml-like” layer: supports one-sided, point-to-point operations

  • sshmem: OpenSHMEM shared memory backing facility

14.8.3. Miscellaneous frameworks

  • allocator: Memory allocator

  • backtrace: Debugging call stack backtrace support

  • btl: Point-to-point Byte Transfer Layer

  • dl: Dynamic loading library interface

  • hwloc: Hardware locality (hwloc) versioning support

  • if: OS IP interface support

  • installdirs: Installation directory relocation services

  • memchecker: Run-time memory checking

  • memcpy: Memory copy support

  • memory: Memory management hooks

  • mpool: Memory pooling

  • patcher: Symbol patcher hooks

  • pmix: Process management interface (exascale)

  • rcache: Memory registration cache

  • reachable: Network reachability determination

  • shmem: Shared memory support (NOT related to OpenSHMEM)

  • smsc: Shared memory single-copy support

  • threads: OS and userspace thread support

  • timer: High-resolution timers

14.8.4. Framework notes

Each framework typically has one or more components that are used at run-time. For example, the btl framework is used by the MPI layer to send bytes across different types underlying networks. The tcp btl, for example, sends messages across TCP-based networks; the ucx pml sends messages across InfiniBand-based networks.

14.8.5. MCA parameter notes

Each component typically has some tunable parameters that can be changed at run-time. Use the ompi_info(1) command to check a component to see what its tunable parameters are. For example:

shell$ ompi_info --param btl tcp

shows some of the parameters (and default values) for the tcp btl component (use --all or --level 9 to show all the parameters).

Note that ompi_info (without --all or a specified level) only shows a small number a component’s MCA parameters by default. Each MCA parameter has a “level” value from 1 to 9, corresponding to the MPI-3 MPI_T tool interface levels. See the LEVELS section in the ompi_info(1) man page for an explanation of the levels and how they correspond to Open MPI’s code.

Here’s rules of thumb to keep in mind when using Open MPI’s levels:

  • Levels 1-3:

    • These levels should contain only a few MCA parameters.

    • Generally, only put MCA parameters in these levels that matter to users who just need to run Open MPI applications (and don’t know/care anything about MPI). Examples (these are not comprehensive):

      • Selection of which network interfaces to use.

      • Selection of which MCA components to use.

      • Selective disabling of warning messages (e.g., show warning message XYZ unless a specific MCA parameter is set, which disables showing that warning message).

      • Enabling additional stderr logging verbosity. This allows a user to run with this logging enabled, and then use that output to get technical assistance.

  • Levels 4-6:

    • These levels should contain any other MCA parameters that are useful to expose to end users.

    • There is an expectation that “power users” will utilize these MCA parameters — e.g., those who are trying to tune the system and extract more performance.

    • Here’s some examples of MCA parameters suitable for these levels (these are not comprehensive):

      • When you could have hard-coded a constant size of a resource (e.g., a resource pool size or buffer length), make it an MCA parameter instead.

      • When there are multiple different algorithms available for a particular operation, code them all up and provide an MCA parameter to let the user select between them.

  • Levels 7-9:

    • Put any other MCA parameters here.

    • It’s ok for these MCA parameters to be esoteric and only relevant to deep magic / the internals of Open MPI.

    • There is little expectation of users using these MCA parameters.

See this section for details on how to set MCA parameters at run time.