(\mathrm{SPH}) (smoothed particle hydrodynamics) is a gridless Lagrangian technique which is appealing as a possible alternative to numerical techniques currently used to analyze large deformation events. Recent tests of the standard SPH method using the cubic B-spline kernel indicated the possibility of an instability in the tensile regime, even though no such difficulties were observed in compression. A von Neumann stability analysis of the SPH algorithm has been carried out which identifies the criterion for stability or instability in terms of the stress state and the second derivative of the kernel function. The analysis explains the observation that the method is unstable in tension while apparently stable in compression but shows that it is possible to construct kernel functions which are stable in tension and unstable in compression. The instability is shown to result from an effective stress with a negative modulus (imaginary sound speed) being produced by the interaction between the constitutive relation and the kernel function and is not caused by the numerical time integration algorithm. That is, changes in the effective stress act to amplify, rather than reduce, perturbations in the strain. The analysis and the stability criterion provide insight into possible methods for removing the instability. (1995 Academic Press, Inc.