While significant non-linear behavior has been observed in many vibration mounting applications, most design studies are typically based on the concept of linear system theory in terms of force or motion transmissibility. In this paper, an improved analytical strategy is presented for the design optimization of complex, active or passive, non-linear mounting systems. This strategy is built upon the computational Galerkin method of weighted residuals, and incorporates order reduction and numerical continuation in an iterative optimization scheme. The overall dynamic characteristics of the mounting system are considered and vibratory power transmission is minimized via adjustment of mount parameters by using both passive and active means. The method is first applied through a computational example case to the optimization of basic passive and active, non-linear isolation configurations. It is found that either active control or intentionally introduced non-linearity can improve the mount's performance; but a combination of both produces the greatest benefit. Next, a novel experimental, active, non-linear isolation system is studied. The effects of non-linearity on vibratory power transmission and active control are assessed via experimental measurements and the enhanced Galerkin method. Results show how harmonic excitation can result in multiharmonic vibratory power transmission. The proposed optimization strategy offers designers some flexibility in utilizing both passive and active means in combination with linear and non-linear components for improved vibration mounts.