The solution of a large class of problems requires the repeated evaluation of matrix vector products: y = Ax. An appropriate data decomposition and Communications system to exchange x components among processors is necessary for efficient evaluation of these vector products on an MIMD concurrent computer. A communications system is presented for the case of a sparse matrix A that arises from a finite element or finite difference discretization of a partial differential equation on an irregular region, or from some kind of finite range interaction between particles. The method presented here uses a domain decomposition of the physical space to distribute A and x among processors. A packed form of the matrix is used which turns out to be very convenient to set up the data structures necessary to send and receive the extra x components. The resulting communications scheme has been used in a multigrid solver for finite element static elasticity problems and in a program which solves an eigenvalue problem. Speed up factors were determined on a 32 processor Caltech/JPL Mark II hypercube with good results. The communications system is not hypercube specific and can easily be implemented on other types of MIMD parallel computers.