Light impurities in covalent materials, such as H or O in Si, are characterized by high-frequency local vibrational modes (LVMs) far above the phonon frequencies of the host crystal. Upon excitation, such LVMs are expected to have long lifetimes since their decay involve multi-phonon processes. Yet, time-resolved infrared absorption spectroscopy reveals that the vibrational lifetimes of almost identical LVMs sometimes differ by up to two orders of magnitude. Indeed, the low-temperature lifetimes of the 2062 cm Γ1 mode of the H Γ 2 complex, the 1998 cm Γ1 mode of bond-centered hydrogen H ΓΎ BC and the 2072 cm Γ1 mode of the di-vacancy di-hydrogen complex VH AE HV are 4, 8 and 295 ps, respectively. More surprising still, the measured lifetime of the asymmetric stretch of interstitial O in Si changes by almost an order of magnitude with the isotope of O or one of its Si neighbors. In this paper, ab-initio molecular dynamic simulations in periodic supercells are used to calculate vibrational lifetimes. The calculations predict the correct lifetimes of the various defects in the range 50 < T < 200 K and provide critical insight into the decay processes.