Abstract We investigate the problem of multi-hop scheduling in self-backhauled millimeter wave (mmWave) networks. Owing to the high path loss and blockage of mmWave links, multi-hop paths/routes between the macro base station and the intended users via full-duplex small cells need to be carefully selected. This paper addresses the fundamental question: “how to select the best paths and how to allocate rates over these paths subject to latency constraints?” To answer this question, we propose a new system design, which factors in mmWave-specific channel variations and network dynamics. The problem is cast as a network utility maximization subject to a bounded delay constraint and network stability. The studied problem is decoupled into: (i) a path/route selection and (ii) rate allocation, whereby learning the best paths is done by means of a reinforcement learning algorithm, and the rate allocation is solved by applying the successive convex approximation method. Via numerical results, our approach ensures reliable communication with a guaranteed probability of 99.9999%, and reduces latency by 50.64% and 92.9% as compared to baselines.