Programming for heterogeneous networks of computers is a diffi cult task. Among others, performance, fault tolerance, and arithmetic heterogeneity are perhaps the most important and challenging issues of heterogeneous parallel and distributed programming.
Performance is one of the primary issues of parallel programming for any parallel architecture, but it becomes particularly challenging for programming for parallel heterogeneous networks. Performance is also one of the primary issues of high -performance distributed computing.
Fault tolerance has always been one of the primary issues of distributed computing. Interestingly, this has not been the case for parallel applications running on traditional homogeneous parallel architectures. The probability of unexpected resource failures in a centralized dedicated parallel computer system was quite small because the system had a relatively small number of processors. This only becomes an issue for modern large -scale parallel systems counting tens of thousands of processors with different interconnection schemes. At the same time, this probability reaches quite high fi gures for common networks of computers of even a relatively small size. First, any individual computer in such a network may be switched off or rebooted unexpectedly for other users in the network. The same may happen with any other resource in the network. Second, not all elements of the common network of computers are equally reliable. These factors make fault tolerance a desirable feature for parallel applications intended to run on common networks of computers; and the longer the execution time of the application is, the more critical the feature becomes.
Arithmetic heterogeneity has never been an issue of parallel programming for traditional homogeneous parallel architectures. All arithmetic data types are uniformly represented in all processors of such a system, and their transfer between the processors does not change their value. In heterogeneous platforms, the same arithmetic data type may have different representations in different processors. In addition, arithmetic values may change in the heterogeneous communication network during transfer even between processors