Message passing is an inherent element of all computer clusters. All computer clusters, ranging from homemade Beowulfs to some of the fastest supercomputers in the world, rely on message passing to coordinate the activities of the many nodes they encompass. Message passing in computer clusters built with commodity servers and switches is used by virtually every internet service. Recently, the use of computer clusters with more than one thousand nodes has been spreading. As the number of nodes in a cluster increases, the rapid growth in the complexity of the communication subsystem makes message passing delays over the interconnect a serious performance issue in the execution of parallel programs. Specific tools may be used to simulate, visualize and understand the performance of message passing on computer clusters. Before a large computer cluster is assembled, a trace-based simulator can use a small number of nodes to help predict the performance of message passing on larger configurations. Following test runs on a small number of nodes, the simulator reads the execution and message transfer log files and simulates the performance of the messaging subsystem when many more messages are exchanged between a much larger number of nodes.
Messages and computations
Approaches to message passing Historically, the two typical approaches to communication between cluster nodes have been PVM, the Parallel Virtual Machine and MPI, the Message Passing Interface. However, MPI has now emerged as the de facto standard for message passing on computer clusters. PVM predates MPI and was developed at the Oak Ridge National Laboratory around 1989. It provides a set of software libraries that allow a computing node to act as a "parallel virtual machine". It provides run-time environment for message-passing, task and resource management, and fault notification and must be directly installed on every cluster node. PVM can be used by user programs written in C, C++, or Fortran, etc. Unlike PVM, which has a concrete implementation, MPI is a specification rather than a specific set of libraries. The specification emerged in the early 1990 out of discussions between 40 organizations, the initial effort having been supported by ARPA and National Science Foundation. The design of MPI drew on various features available in commercial systems of the time. The MPI specifications then gave rise to specific implementations. MPI implementations typically use TCP/IP and socket connections. MPI is now a widely available communications model that enables parallel programs to be written in languages such as C, Fortran, Python, etc. The MPI specification has been implemented in systems such as MPICH and Open MPI.
Testing, evaluation and optimization
Computer clusters use a number of strategies for dealing with the distribution of processing over multiple nodes and the resulting communication overhead. Some computer clusters such as Tianhe-I use different processors for message passing than those used for performing computations. Tiahnhe-I uses over two thousand FeiTeng-1000 processors to enhance the operation of its proprietary message passing system, while computations are performed by Xeon and Nvidia Tesla processors. One approach to reducing communication overhead is the use of local neighborhoods (also called locales) for specific tasks. Here computational tasks are assigned to specific "neighborhoods" in the cluster, to increase efficiency by using processors which are closer to each other. However, given that in many cases the actual topology of the computer cluster nodes and their interconnections may not be known to application developers, attempting to fine tune performance at the application program level is quite difficult. Given that MPI has now emerged as the de facto standard on computer clusters, the increase in the number of cluster nodes has resulted in continued research to improve the efficiency and scalability of MPI libraries. These efforts have included research to reduce the memory footprint of MPI libraries. From the earliest days MPI provided facilities for performance profiling via the PMPI "profiling system". The use of the PMIPI- prefix allows for the observation of the entry and exit points for messages. However, given the high level nature of this profile, this type of information only provides a glimpse at the real behavior of the communication system. The need for more information resulted in the development of the MPI-Peruse system. Peruse provides a more detailed profile by enabling applications to gain access to state-changes within the MPI-library. This is achieved by registering callbacks with Peruse, and then invoking them as triggers as message events take place. Peruse can work with the PARAVER visualization system. PARAVER has two components, a trace component and a visual component for analyze the traces, the statistics related to specific events, etc. PARAVER may use trace formats from other systems, or perform its own tracing. It operates at the task level, thread level, and in a hybrid format. Traces often include so much information that they are often overwhelming. Thus PARAVER summarizes them to allow users to visualize and analyze them.
Performance analysis When a large scale, often supercomputer level, parallel system is being developed, it is essential to be able to experiment with multiple configurations and simulate performance. There are a number of approaches to modeling message passing efficiency in this scenario, ranging from analytical models to trace-based simulation and some approaches rely on the use of test environments based on "artificial communications" to perform synthetic tests of message passing performance. Systems such as BIGSIM provide these facilities by allowing the simulation of performance on various node topologies, message passing and scheduling strategies.
… excerpt ends here. Continue reading the full article.
