Markov Chain - Markov Chains

Markov Chains

The probability of going from state i to state j in n time steps is

and the single-step transition is

For a time-homogeneous Markov chain:

and

The n-step transition probabilities satisfy the Chapman–Kolmogorov equation, that for any k such that 0 < k < n,

where S is the state space of the Markov chain.

The marginal distribution Pr(X_n = x) is the distribution over states at time n. The initial distribution is Pr(X₀ = x). The evolution of the process through one time step is described by

Note: The superscript (n) is an index and not an exponent.