The behavior of neural networks may be influenced by transmission delays and many studies have derived constraints on parameters such as connection weights and output functions which ensure that the asymptotic dynamics of a network with delay remains similar to that of the corresponding system without delay. However, even when the delay does not affect the asymptotic behavior of the system, it may influence other important features in the system's dynamics such as the boundary of the basin of attraction of the stable equilibria. In order to better understand such effects, we study the dynamics of a system constituted by two neurons interconnected through delayed excitatory connections. We show that the system with delay has exactly the same stable equilibrium points as the associated system without delay, and that, in both the network with delay and the corresponding one without delay, most trajectories converge to these stable equilibria. Thus, the asymptotic behavior of the network with delay and that of the corresponding system without delay are similar. We obtain a theoretical characterization of the boundary separating the basins of attraction of two stable equilibria, which enables us to estimate the boundary. Our numerical investigations show that, even in this simple system, the boundary separting the basins of attraction of two stable equilibrium points depends on the value of the delays. The extension of these results to networks with an arbritrary number of units is discussed.