This paper considers a problem faced by players who are involved in a sequence of games: not necessarily the same games, not necessarily with the same opponents, and not necessarily under conditions of complete information. The players are assumed to act in response to stationary Markovian hypotheses which they form about the actions of their opponents. Conditions are explored which require that these hypotheses be correct on average and that the players actions be optimal in response to their hypotheses.