Connecting Local Models

Graphical models

Conditional independence

The variable \(\rvx_1\) is said to be conditionally independent of \(\rvx_3\) given \(\rvx_2\) if \(\rvx_1\) and \(\rvx_3\) are independent given \(\rvx_2\):

\[ \begin{aligned} Pr(\rvx_1|\rvx_2,\rvx_3) &= Pr(\rvx_1|\rvx_2)\\ Pr(\rvx_3|\rvx_2,\rvx_1) &= Pr(\rvx_3|\rvx_2) \end{aligned} \]

Then, the joint distribution can be factorized as:

\[ Pr(\rvx_1,\rvx_2,\rvx_3) = Pr(\rvx_3|\rvx_2)Pr(\rvx_2 |\rvx_1)Pr(\rvx_1) \]

Directed graphical models

Directed graphical model represents probability distribution that factorizes as a product of conditional probabilities distributions:

\[ Pr(\rvx_{1\dots N}) = \prod_{i=1}^N Pr(\rvx_n|\rvx_{\mathrm{pa}_{[n]}}) \]

where pa[n] denotes the parents of node n.

A variable is conditionally independent of all other variables given its Markov blanket: its parents, its children, and parents of its children.

Undirected graphical models

Undirected graphical model represents probability distribution that factorizes as a product of potential functions \(\phi_c\):

\[ Pr(\rvx_{1\dots N}) = \frac{1}{Z} \prod_{c=1}^C \phi_c(\rvx_{1\dots N}) \]

where \(Z\) is the normalization constant, \(C\) is the set of cliques in the graph. It is better to write undirected model with clique \(\mathcal{S}_c \subset \{x_n\}_{n=1}^N\):

\[ Pr(\rvx_{1\dots N}) = \frac{1}{Z} \prod_{c=1}^C \phi_c(\mathcal{S}_c) \]

One set of nodes is conditionally independe of another given a third if the third set separates the first two (i.e., blocks all paths between them).

Models for chains and trees

Models for grids