We consider a $k$-player sequential bargaining model in which the size of the cake and the order in which players move follow a general Markov process. For games in which one agent makes an offer in each period and agreement must be unanimous, we characterize the sets of subgame perfect and stationary subgame perfect payoffs. With these characterizations, we investigate the uniqueness and efficiency of the equilibrium outcomes, the conditions under which agreement is delayed, and the advantage to proposing. Our analysis generalizes many existing results for games of sequential bargaining which build on the work of Stahl (1972), Rubinstein (1982), and Binmore (1987).