Many first-generation interspecific hybrids between the five species of the Drosophila virilis subgroup are fertile. Fertile hybrids can be utilized in the analysis of sterile hybrids, by creating new combinations of genetic elements to identify cross-incompatible chromosomes or chromosome segments. Here, crosses starting from D. lummei, D. virilis, D. novamexicana and D. americana were produced to analyse the 96.4% male sterility in the progeny of D. virilis females mated with D. novamexicana males (sterility based on sperm motility). In autosomally identical heterozygous combination, the substitution of a large segment of the D. virilis X chromosome with a double or a triple inversion from the sister taxa decreased male sterility from 96.4% (Xvirilis) to 11% (Xvirilis/novamexicana), 19% (Xvirilis/americana) and 34% (Xvirilis/lummei). This result shows that the Xvirilis interaction with D. novamexicana elements (Y and/or autosomes) is a major component in the F1 sterility observed. When males with a substituted Ylummei in otherwise pure D. novamexicana were crossed with D. virilis females, all the F1 males were fertile. We constructed combinations of intact Xvirilis, recombinant Xvirilis/novamexicana, Ynovamexicana and Ylummei chromosomes in homozygous D. virilis and heterozygous D. virilis/D. novamexicana autosomal backgrounds. In such combinations, the Y chromosome interacted strongly not only with the nonconspecific X chromosome, but with several autosomes. Autosomal hybrid sterility contributions tend to be recessive and not present in the F1. In this species group, the role of the Y chromosome is central in contributing to the phenomenon called Haldane's rule of sterility, but the Y chromosomal incompatibility factor of D. novamexicana is the most divergent one.