Polygenic risk scores (PRS) for breast cancer have potential to improve risk prediction, but there is limited information on their utility in various clinical situations. Here we show that among 122,978 women in the FinnGen study with 8401 breast cancer cases, the PRS modifies the breast cancer risk of two high-impact frameshift risk variants. Similarly, we show that after the breast cancer diagnosis, individuals with elevated PRS have an elevated risk of developing contralateral breast cancer, and that the PRS can considerably improve risk assessment among their female first-degree relatives. In more detail, women with the c.1592delT variant in PALB2 (242-fold enrichment in Finland, 336 carriers) and an average PRS (10–90th percentile) have a lifetime risk of breast cancer at 55% (95% CI 49–61%), which increases to 84% (71–97%) with a high PRS (> 90th percentile), and decreases to 49% (30–68%) with a low PRS (< 10th percentile). Similarly, for c.1100delC in CHEK2 (3.7–fold enrichment; 1648 carriers), the respective lifetime risks are 29% (27–32%), 59% (52–66%), and 9% (5–14%). The PRS also refines the risk assessment of women with first-degree relatives diagnosed with breast cancer, particularly among women with positive family history of early-onset breast cancer. Here we demonstrate the opportunities for a comprehensive way of assessing genetic risk in the general population, in breast cancer patients, and in unaffected family members.