We incorporate modeling features from Markov decision processes (MDPs) in a class of multi-stage stochastic programs. This class includes structured MDPs with continuous states and actions. We extend Markov-chain based policy graphs to include a form of statistical learning and to include decision-dependent one-step transition probabilities. We illustrate the expressiveness of our modeling approach. To computationally handle these new features, we extend the growing class of stochastic dual dynamic programming algorithms.