Background: Research has shown that engaging in leisure-time physical activity (LTPA) and consuming dairy foods can lead to better sleep. Combining these two non-invasive prescriptions may be more effective for helping people fall asleep. This study investigates whether participating in LTPA in conjunction with consuming milk and milk products has a beneficial association with difficulty initiating sleep (DIS) among older adults. Methods: The present study looked at 421 community-dwelling older people aged 65 years and older living in Ibaraki prefecture, Japan (mean age 74.9 ± 5.5 years, male 43.7%). We measured LTPA and sleep latency with the Physical Activity Scale for the Elderly and the Pittsburgh Sleep Quality Index, respectively. Participants who needed 30 minutes or more to fall asleep were defined as having DIS. We assessed dairy consumption as participants’ habitual intake of milk, yogurt and cheese. Results: After adjusting for covariates, participants who engaged in sufficient levels of LTPA as well as consumed milk (OR = 0.27, 95% CI = 0.10-0.73) or cheese (OR = 0.34, 95% CI = 0.14-0.85) were less likely to complain of DIS compared with people who neither engaged in LTPA nor ingested milk or cheese. Conclusions: Our findings suggest that the combination of engaging in LTPA and consuming milk or cheese is necessary as a prescription to improve falling asleep for older adults suffering from DIS. Additionally, engaging in LTPA along with dairy consumption may effectively improve a problem with falling asleep.