In recognition memory tests, feelings of familiarity for stimuli vary in strength. Increasing levels of felt familiarity should modulate activity in brain structures that mediate familiarity memory. We used this expectation to identify the neural system that underlies scene familiarity memory. Normal subjects studied pictures of scenes and 2 days later while undergoing event-related functional magnetic resonance imaging (fMRI) rated old and new pictures as novel, slightly familiar, moderately familiar, very familiar, or recollected, although they were specifically instructed not to try and recollect. Familiarity strength was, therefore, judged as absent (misses) or present at three levels of increasing strength. A parametric analysis showed that, as perceived strength of familiarity increased activity in the perirhinal cortex, insula and left superior temporal cortex declined linearly whereas activity in the left dorsomedial thalamus, left ventrolateral and anteromedial frontal cortex, posterior cingulate cortex, and left parietal neocortex increased linearly. Hippocampal activity was not modulated linearly or quadratically by changes in familiarity strength. Recollection activated the hippocampus, and left anterior and inferolateral frontal and parietal cortices more than strong familiarity. In contrast, no brain region that was unaffected by recollection (relative to misses and correct rejections) was modulated by variations in familiarity strength. The implications of these findings for the functional and neural bases of familiarity and recollection are considered.