Background: Little is known about the association between adrenarche and cognition in general populations of children. We therefore studied the associations of dehydroepiandrosterone sulfate (DHEAS), androstenedione (A4), testosterone, insulin-like growth factor-1 (IGF-1), and adrenarche with cognition among prepubertal children. Methods: These cross-sectional analyses are based on baseline data of the Physical Activity and Nutrition in Children Study. A total of 387 children (183 girls, 204 boys) were included in the analyses. Raven’s Coloured Progressive Matrices (CPM) score was used to assess nonverbal reasoning. Serum adrenal androgens and IGF-1 concentrations were measured and clinical signs of androgen action were evaluated. Results: Higher IGF-1 among boys (β = 0.149, p =0.033) was related to a better Raven’s CPM score after adjustment for age and parental education. Adrenal androgens in girls or boys or IGF-1 in girls were not associated with the score. There were no differences in Raven’s CPM score between children with biochemical adrenarche (DHEAS ≥1.08 µmol/L; ≥40 µg/dL) or with clinical signs of androgen action and children without them. Conclusion: The results suggest that higher serum IGF-1 among boys is related to better cognition in prepubertal children. We could not provide evidence for the associations of adrenal maturation with cognition in prepubertal children.