Abstract. In this talk, we consider learning rates and momentum in the context of the randomized Kaczmarz algorithm, which is an instance of stochastic gradient descent for a linear least squares loss function. First, we consider the problem of determining an optimal learning rate schedule for the Kaczmarz algorithm for noisy linear systems. Second, we consider how momentum affects how the randomized Kaczmarz algorithm converges in the direction of singular vectors of the matrix defining the linear loss function. Finally, we explore open questions and potential directions for future research.