In this paper we consider simple neural network models consisting, o/two to three continuous nonlinear neurons, with no intrinsic svnaptic plasticity and with delay in neural signal transmission. I1~,, investL¢ate the diJferent dynamic regimes which may ~:xist ./or these "minimal" neural network structures. Examples of stable. oscillatory (periodic or quasi-periodic), and chaotic re¢imes are reported arid ana/.l'zed. For chaotic re~imes, classical characteristicw sttch as b~[itrcation diagrams, sensitive dependellce oil initial conditions. L)'apttnov ~:-pollenls. pseudo phase space attractors, are presented. It is shown that the d)'namic regime o/a network call be changed through mod~/k'ations o.f either internal or evterna/ parametetw, such as a s)'naplic weight or an c~vternal neuron input. The resulting d)'namic regimes o[]'er fi'anleworks to represent various nettra/./imctions. For instance, oscil/atoo' regimes provide a mechanism to implement controllable neltra/ oscillators. The sensitive dependence oil initial conditions. which is shon'n to ¢:'ist even for veo' small networks, sets a limit to an)' hm.~,, term predictiot? eoncertting the evolution of the nettral O,stem. tmless the network adjust its parameters through plasticiO' ill order tO avoid chaotic re, ffimes.