Recent mechanistic models argue for a key role of rhythm processing in both speech production and speech perception. Patients with the non-fluent variant (NFV) of primary progressive aphasia (PPA) with apraxia of speech (AOS) represent a specific study population in which this link can be examined. Previously, we observed impaired rhythm processing in NFV with AOS. We hypothesized that a shared neurocomputational mechanism structures auditory input (sound and speech) and output (speech production) in time, a “temporal scaffolding” mechanism. Since considerable white matter damage is observed in NFV, we test here whether white matter changes are related to impaired rhythm processing. Forty-seven participants performed a psychoacoustic test battery: 12 patients with NFV and AOS, 11 patients with the semantic variant of PPA, and 24 cognitively intact age- and education-matched controls. Deformation-based morphometry was used to test whether white matter volume correlated to rhythmic abilities. In 34 participants, we also obtained tract-based metrics of the left Aslant tract, which is typically damaged in patients with NFV. Nine out of 12 patients with NFV displayed impaired rhythmic processing. Left frontal white matter atrophy adjacent to the supplementary motor area (SMA) correlated with poorer rhythmic abilities. The structural integrity of the left Aslant tract also correlated with rhythmic abilities. A colocalized and perhaps shared white matter substrate adjacent to the SMA is associated with impaired rhythmic processing and motor speech impairment. Our results support the existence of a temporal scaffolding mechanism structuring perceptual input and speech output.