A discrete-time Markov chain is a sequence of random variables ${\displaystyle X_{0},X_{1},X_{2},...}$  with the Markov property, namely that the probability of moving to the next state depends only on the present state and not on the previous states:

${\displaystyle \Pr(X_{n+1}=x\mid X_{1}=x_{1},X_{2}=x_{2},\ldots ,X_{n}=x_{n})=\Pr(X_{n+1}=x\mid X_{n}=x_{n}),}$  if both conditional probabilities are well defined, that is, if ${\displaystyle \Pr(X_{1}=x_{1},\ldots ,X_{n}=x_{n})>0.}$ 

The possible values of X_i form a countable set S called the state space of the chain.^[1]

Markov chains are often described by a sequence of directed graphs, where the edges of graph n are labeled by the probabilities of going from one state at time n to the other states at time n + 1, ${\displaystyle \Pr(X_{n+1}=x\mid X_{n}=x_{n}).}$  The same information is represented by the transition matrix from time n to time n + 1. However, Markov chains are frequently assumed to be time-homogeneous (see variations below), in which case the graph and matrix are independent of n and are thus not presented as sequences.

These descriptions highlight the structure of the Markov chain that is independent of the initial distribution ${\displaystyle \Pr(X_{1}=x_{1}).}$  When time-homogeneous, the chain can be interpreted as a state machine assigning a probability of hopping from each vertex or state to an adjacent one. The probability ${\displaystyle \Pr(X_{n}=x\mid X_{1}=x_{1})}$  of the machine's state can be analyzed as the statistical behavior of the machine with an element ${\displaystyle x_{1}}$  of the state space as input, or as the behavior of the machine with the initial distribution ${\displaystyle \Pr(X_{1}=y)=[x_{1}=y]}$  of states as input, where ${\displaystyle [P]}$  is the Iverson bracket.^{[citation needed]}

Variations edit

• Time-homogeneous Markov chains (or stationary Markov chains) are processes where

${\displaystyle \Pr(X_{n+1}=x\mid X_{n}=y)=\Pr(X_{n}=x\mid X_{n-1}=y)}$ 

for all n. The probability of the transition is independent of n.^[1]

• A Markov chain with memory (or a Markov chain of order m)

where m is finite, is a process satisfying

${\displaystyle {\begin{aligned}&\Pr(X_{n}=x_{n}\mid X_{n-1}=x_{n-1},X_{n-2}=x_{n-2},\dots ,X_{1}=x_{1})\\={}&\Pr(X_{n}=x_{n}\mid X_{n-1}=x_{n-1},X_{n-2}=x_{n-2},\dots ,X_{n-m}=x_{n-m}){\text{ for }}n>m\end{aligned}}}$ 

In other words, the future state depends on the past m states. It is possible to construct a chain ${\displaystyle (Y_{n})}$  from ${\displaystyle (X_{n})}$  which has the 'classical' Markov property by taking as state space the ordered m-tuples of X values, ie. ${\displaystyle Y_{n}=\left(X_{n},X_{n-1},\ldots ,X_{n-m+1}\right)}$ .^{[citation needed]}

The probability of going from state i to state j in n time steps is

${\displaystyle p_{ij}^{(n)}=\Pr(X_{n}=j\mid X_{0}=i)}$ 

and the single-step transition is

${\displaystyle p_{ij}=\Pr(X_{1}=j\mid X_{0}=i).}$ 

For a time-homogeneous Markov chain:

${\displaystyle p_{ij}^{(n)}=\Pr(X_{k+n}=j\mid X_{k}=i)}$ 

and

${\displaystyle p_{ij}=\Pr(X_{k+1}=j\mid X_{k}=i).}$ 

The n-step transition probabilities satisfy the Chapman–Kolmogorov equation, that for any k such that 0 < k < n,

${\displaystyle p_{ij}^{(n)}=\sum _{r\in S}p_{ir}^{(k)}p_{rj}^{(n-k)}}$ 

where S is the state space of the Markov chain.^[1]

The marginal distribution Pr(X_n = x) is the distribution over states at time n. The initial distribution is Pr(X₀ = x). The evolution of the process through one time step is described by

${\displaystyle \Pr(X_{n}=j)=\sum _{r\in S}p_{rj}\Pr(X_{n-1}=r)=\sum _{r\in S}p_{rj}^{(n)}\Pr(X_{0}=r).}$ 

(The superscript (n) is an index, and not an exponent).

A state j is said to be accessible from a state i (written i → j) if a system started in state i has a non-zero probability of transitioning into state j at some point. Formally, state j is accessible from state i if there exists an integer n_ij ≥ 0 such that

${\displaystyle \Pr(X_{n_{ij}}=j\mid X_{0}=i)=p_{ij}^{(n_{ij})}>0.}$ 

A state i is said to communicate with state j (written i ↔ j) if both i → j and j → i. A communicating class is a maximal set of states C such that every pair of states in C communicates with each other. Communication is an equivalence relation, and communicating classes are the equivalence classes of this relation.^[1]

A communicating class is closed if the probability of leaving the class is zero, namely if i is in C but j is not, then j is not accessible from i.^[1] The set of communicating classes forms a directed, acyclic graph by inheriting the arrows from the original state space. A communicating class is closed if and only if it has no outgoing arrows in this graph.

A state i is said to be essential or final if for all j such that i → j it is also true that j → i. A state i is inessential if it is not essential.^[2] A state is final if and only if its communicating class is closed.

A Markov chain is said to be irreducible if its state space is a single communicating class; in other words, if it is possible to get to any state from any state.^[1][3]

Periodicity edit

A state ${\displaystyle i}$  has period ${\displaystyle k}$  if any return to state ${\displaystyle i}$  must occur in multiples of ${\displaystyle k}$  time steps. Formally, the period of state ${\displaystyle i}$  is defined as

${\displaystyle k=\gcd\{n>0:\Pr(X_{n}=i\mid X_{0}=i)>0\}}$ 

(where ${\displaystyle \gcd }$  is the greatest common divisor) provided that this set is not empty. Otherwise the period is not defined.^[1] Note that even though a state has period ${\displaystyle k}$ , it may not be possible to reach the state in ${\displaystyle k}$  steps. For example, suppose it is possible to return to the state in ${\displaystyle \{6,~8,~10,~12,\dots \}}$  time steps; ${\displaystyle k}$  would be ${\displaystyle 2}$ , even though ${\displaystyle 2}$  does not appear in this list.

If ${\displaystyle k=1}$ , then the state is said to be aperiodic. Otherwise (${\displaystyle k>1}$ ), the state is said to be periodic with period ${\displaystyle k}$ . Periodicity is a class property—that is, if a state has period ${\displaystyle k}$  then every state in its communicating class has period ${\displaystyle k}$ .^[1]

Transience and recurrence edit

A state i is said to be transient if, given that we start in state i, there is a non-zero probability that we will never return to i. Formally, let the random variable T_i be the first return time to state i (the "hitting time"):

${\displaystyle T_{i}=\inf\{n\geq 1:X_{n}=i\}.}$ 

The number

${\displaystyle f_{ii}^{(n)}=\Pr(T_{i}=n\mid X_{0}=i)}$ 

is the probability that we return to state i for the first time after n steps. Therefore, state i is transient if

${\displaystyle \Pr(T_{i}<\infty \mid X_{0}=i)=\sum _{n=1}^{\infty }f_{ii}^{(n)}<1.}$ 

State i is recurrent (or persistent) if it is not transient. Recurrence and transience are class properties, that is, they either hold or do not hold equally for all members of a communicating class.^[1]

A state i is recurrent if and only if the expected number of visits to i is infinite:^[1]

${\displaystyle \sum _{n=0}^{\infty }p_{ii}^{(n)}=\infty .}$ 

Positive recurrence edit

Even if the hitting time is finite with probability 1, it need not have a finite expectation. The mean recurrence time at state i is the expected return time M_i:

${\displaystyle M_{i}=E[T_{i}]=\sum _{n=1}^{\infty }n\cdot f_{ii}^{(n)}.}$ 

State i is positive recurrent (or non-null persistent) if M_i is finite; otherwise, state i is null recurrent (or null persistent). Positive and null recurrence are classes properties.^[1]

Absorbing states edit

A state i is called absorbing if it is impossible to leave this state. Therefore, the state i is absorbing if and only if

${\displaystyle p_{ii}=1{\text{ and }}p_{ij}=0{\text{ for }}i\not =j.}$ 

If every state can reach an absorbing state, then the Markov chain is an absorbing Markov chain.^[3]

Reversible Markov chain edit

A Markov chain is said to be reversible if there is a probability distribution π over its states such that

${\displaystyle \pi _{i}\Pr(X_{n+1}=j\mid X_{n}=i)=\pi _{j}\Pr(X_{n+1}=i\mid X_{n}=j)}$ 

for all times n and all states i and j. This condition is known as the detailed balance condition (or local balance equation).

Considering a fixed arbitrary time n and using the shorthand

${\displaystyle p_{ij}=\Pr(X_{n+1}=j\mid X_{n}=i)\,,}$ 

the detailed balance equation can be written more compactly as

${\displaystyle \pi _{i}p_{ij}=\pi _{j}p_{ji}\,.}$ ^[1]

The single time-step from n to n + 1 can be thought of as each person i having π_i dollars initially and paying each person j a fraction p_ij of it. The detailed balance condition states that upon each payment, the other person pays exactly the same amount of money back.^[4] Clearly the total amount of money π each person has remains the same after the time-step, since every dollar spent is balanced by a corresponding dollar received. This can be shown more formally by the equality

${\displaystyle \sum _{i}\pi _{i}p_{ij}=\sum _{i}\pi _{j}p_{ji}=\pi _{j}\sum _{i}p_{ji}=\pi _{j}\,,}$ 

which essentially states that the total amount of money person j receives (including from himself) during the time-step equals the amount of money he pays others, which equals all the money he initially had because it was assumed that all money is spent (that is, p_ji sums to 1 over i). The assumption is a technical one, because the money not really used is simply thought of as being paid from person j to himself (that is, p_jj is not necessarily zero).

As n was arbitrary, this reasoning holds for any n, and therefore for reversible Markov chains π is always a steady-state distribution of Pr(X_n+1 = j | X_n = i) for every n.

If the Markov chain begins in the steady-state distribution, that is, if ${\displaystyle \Pr(X_{0}=i)=\pi _{i}}$ , then ${\displaystyle \Pr(X_{n}=i)=\pi _{i}}$  for all ${\displaystyle n}$  and the detailed balance equation can be written as

${\displaystyle \Pr(X_{n}=i,X_{n+1}=j)=\Pr(X_{n+1}=i,X_{n}=j)\,.}$ 

The left- and right-hand sides of this last equation are identical except for a reversing of the time indices n and n + 1.

Kolmogorov's criterion gives a necessary and sufficient condition for a Markov chain to be reversible directly from the transition matrix probabilities. The criterion requires that the products of probabilities around every closed loop are the same in both directions around the loop.

Reversible Markov chains are common in Markov chain Monte Carlo (MCMC) approaches because the detailed balance equation for a desired distribution π necessarily implies that the Markov chain has been constructed so that π is a steady-state distribution. Even with time-inhomogeneous Markov chains, where multiple transition matrices are used, if each such transition matrix exhibits detailed balance with the desired π distribution, this necessarily implies that π is a steady-state distribution of the Markov chain.

Closest reversible Markov chain edit

For any time-homogeneous Markov chain given by a transition matrix ${\displaystyle P\in \mathbb {R} ^{n\times n}}$ , any norm ${\displaystyle ||\cdot ||}$  on ${\displaystyle \mathbb {R} ^{n\times n}}$  which is induced by a scalar product, and any probability vector ${\displaystyle \pi }$ , there exists a unique transition matrix ${\displaystyle P^{*}}$  which is reversible according to ${\displaystyle \pi }$  and which is closest to ${\displaystyle P}$  according to the norm ${\displaystyle ||\cdot ||.}$  The matrix ${\displaystyle P^{*}}$  can be computed by solving a quadratic-convex optimization problem.^[5]

For example, consider the following Markov chain:

This Markov chain is not reversible. According to the Frobenius Norm the closest reversible Markov chain according to ${\displaystyle \pi =\left({\frac {1}{3}},{\frac {1}{3}},{\frac {1}{3}}\right)}$  can be computed as

If we choose the probability vector randomly as ${\displaystyle \pi =\left({\frac {1}{4}},{\frac {1}{4}},{\frac {1}{2}}\right)}$ , then the closest reversible Markov chain according to the Frobenius norm is approximately given by

A distribution ${\displaystyle \pi }$  is a stationary distribution of the Markov chain with stochastic matrix ${\displaystyle P}$  if and only if ${\displaystyle \pi P=\pi }$ . This can be written as:^[1]

${\displaystyle \forall j\in \mathbb {S} :\sum _{i\in \mathbb {S} }\pi _{i}p_{ij}=\pi _{j}}$ .

This condition implies that ${\displaystyle \pi P^{n}=\pi }$  and hence if the Markov chain ${\displaystyle (X_{n},n\in \mathbb {N} )}$  has initial distribution ${\displaystyle X_{0}=\pi }$  then ${\displaystyle X_{n}=\pi }$  (in distribution) for any ${\displaystyle n\in \mathbb {N} }$ .^[1]

If a Markov chain is irreducible then it has a stationary distribution if and only if it is positive recurrent,^[6] in which case the unique such distribution is given by ${\displaystyle \pi _{i}={\frac {1}{M_{i}}}}$  where ${\displaystyle M_{i}=\mathbb {E} (T_{i})}$  is the mean recurrence time of i.^[1]

If a chain has more than one closed communicating class, its stationary distributions will not be unique (consider any closed communicating class ${\displaystyle C_{i}}$  in the chain; each one will have its own unique stationary distribution ${\displaystyle \pi _{i}}$ . Extending these distributions to the overall chain, setting all values to zero outside the communication class, yields that the set of invariant measures of the original chain is the set of all convex combinations of the ${\displaystyle \pi _{i}}$ 's). However, if a state j is aperiodic, then ${\displaystyle \lim \nolimits _{n\rightarrow \infty }p_{jj}^{(n)}={\tfrac {C}{M_{j}}}}$  and for any other state i, let f_ij be the probability that the chain ever visits state j if it starts at i, ${\displaystyle \lim \nolimits _{n\rightarrow \infty }p_{ij}^{(n)}=C{\tfrac {f_{ij}}{M_{j}}}.}$ 

If a state i is periodic with period k > 1 then the limit ${\displaystyle \lim \nolimits _{n\rightarrow \infty }p_{ii}^{(n)}}$  does not exist, although the limit ${\displaystyle \lim \nolimits _{n\rightarrow \infty }p_{ii}^{(kn+r)}}$  does exist for every integer r.

Steady-state analysis and the time-inhomogeneous Markov chain edit

A Markov chain need not necessarily be time-homogeneous to have an equilibrium distribution. If there is a probability distribution over states ${\displaystyle {\boldsymbol {\pi }}}$  such that

${\displaystyle \pi _{j}=\sum _{i\in S}\pi _{i}\,\Pr(X_{n+1}=j\mid X_{n}=i)}$ 

for every state j and every time n then ${\displaystyle {\boldsymbol {\pi }}}$  is an equilibrium distribution of the Markov chain. Such can occur in Markov chain Monte Carlo (MCMC) methods in situations where a number of different transition matrices are used, because each is efficient for a particular kind of mixing, but each matrix respects a shared equilibrium distribution.

The hitting time is the time, starting in a given set of states until the chain arrives in a given state or set of states. The distribution of such a time period has a phase type distribution. The simplest such distribution is that of a single exponentially distributed transition.^{[citation needed]}

Expected hitting times edit

For a subset of states A ⊆ S, the vector k^A of hitting times (where element ${\displaystyle k_{i}^{A}}$  represents the expected value, starting in state i that the chain enters one of the states in the set A) is the minimal non-negative solution to^[7]

${\displaystyle {\begin{aligned}k_{i}^{A}=0&{\text{ for }}i\in A\\-\sum _{j\in S}q_{ij}k_{j}^{A}=1&{\text{ for }}i\notin A.\end{aligned}}}$ 

An instance of ergodic theory, the ergodic theorem for states that for an irreducible aperiodic Markov chain, for any two states i and j,^[1]

${\displaystyle p_{i,j}^{(n)}\rightarrow {\frac {1}{M_{j}}}}$  as ${\displaystyle n\rightarrow \infty }$ .