This work addresses the problem of adaptive modulation and power in wireless systems with a strict delay constraint. Modulation and transmit power are dynamically adapted to minimize the outage probability for a fixed data rate. A discrete-time stationary Markov chain is used to model the time-varying channel. The problem is formulated as a finite-horizon MDP. The solution is a set of power/modulation allocation policies to be used during the transmission, as a function of the channel and system state. Numerical results show the benefits of such adaptation policies.