13.8. Internal frameworks
The Modular Component Architecture (MCA) is the backbone of Open MPI – most services and functionality are implemented through MCA components.
13.8.1. MPI layer frameworks
Here is a list of all the component frameworks in the MPI layer of Open MPI:
bml: BTL management layer
coll: MPI collective algorithms
fbtl: point to point file byte transfer layer: abstraction for individual read: collective read and write operations for MPI I/O
fcoll: collective file system functions for MPI I/O
fs: file system functions for MPI I/O
hook: Generic hooks into Open MPI
io: MPI I/O
mtl: Matching transport layer, used for MPI point-to-point messages on some types of networks
op: Back end computations for intrinsic MPI_Op operators
osc: MPI one-sided communications
pml: MPI point-to-point management layer
sharedfp: shared file pointer operations for MPI I/O
topo: MPI topology routines
vprotocol: Protocols for the “v” PML
13.8.2. OpenSHMEM component frameworks
atomic: OpenSHMEM atomic operations
memheap: OpenSHMEM memory allocators that support the PGAS memory model
scoll: OpenSHMEM collective operations
spml: OpenSHMEM “pml-like” layer: supports one-sided, point-to-point operations
sshmem: OpenSHMEM shared memory backing facility
13.8.3. Miscellaneous frameworks
allocator: Memory allocator
backtrace: Debugging call stack backtrace support
btl: Point-to-point Byte Transfer Layer
dl: Dynamic loading library interface
hwloc: Hardware locality (hwloc) versioning support
if: OS IP interface support
installdirs: Installation directory relocation services
memchecker: Run-time memory checking
memcpy: Memory copy support
memory: Memory management hooks
mpool: Memory pooling
patcher: Symbol patcher hooks
pmix: Process management interface (exascale)
rcache: Memory registration cache
reachable: Network reachability determination
shmem: Shared memory support (NOT related to OpenSHMEM)
smsc: Shared memory single-copy support
threads: OS and userspace thread support
timer: High-resolution timers
13.8.4. Framework notes
Each framework typically has one or more components that are used at
run-time. For example, the
btl framework is used by the MPI layer
to send bytes across different types underlying networks. The
btl, for example, sends messages across TCP-based networks; the
pml sends messages across InfiniBand-based networks.
13.8.5. MCA parameter notes
Each component typically has some tunable parameters that can be changed at run-time. Use the ompi_info(1) command to check a component to see what its tunable parameters are. For example:
shell$ ompi_info --param btl tcp
shows some of the parameters (and default values) for the
--level 9 to show all the parameters).
--all or a specified level) only
shows a small number a component’s MCA parameters by default. Each
MCA parameter has a “level” value from 1 to 9, corresponding to the
MPI-3 MPI_T tool interface levels. See the LEVELS section in
the ompi_info(1) man page for an explanation
of the levels and how they correspond to Open MPI’s code.
Here’s rules of thumb to keep in mind when using Open MPI’s levels:
These levels should contain only a few MCA parameters.
Generally, only put MCA parameters in these levels that matter to users who just need to run Open MPI applications (and don’t know/care anything about MPI). Examples (these are not comprehensive):
Selection of which network interfaces to use.
Selection of which MCA components to use.
Selective disabling of warning messages (e.g., show warning message XYZ unless a specific MCA parameter is set, which disables showing that warning message).
Enabling additional stderr logging verbosity. This allows a user to run with this logging enabled, and then use that output to get technical assistance.
These levels should contain any other MCA parameters that are useful to expose to end users.
There is an expectation that “power users” will utilize these MCA parameters — e.g., those who are trying to tune the system and extract more performance.
Here’s some examples of MCA parameters suitable for these levels (these are not comprehensive):
When you could have hard-coded a constant size of a resource (e.g., a resource pool size or buffer length), make it an MCA parameter instead.
When there are multiple different algorithms available for a particular operation, code them all up and provide an MCA parameter to let the user select between them.
Put any other MCA parameters here.
It’s ok for these MCA parameters to be esoteric and only relevant to deep magic / the internals of Open MPI.
There is little expectation of users using these MCA parameters.
See this section for details on how to set MCA parameters at run time.