A Brief Intruction of Mutual Information and Demonstration With R
$\newcommand{\entropfrac}[2]{\frac{#1}{#2} \log \left( \frac{#1}{#2} \right)}$
Mututal Information (MI)
Introduction
Mutual Information (MI) distance is used to measure the distance between two genes vectors, for example $x_1 = {1, 0, 1, 1, 1, 1, 0}$ and $y_1 = {0, 1, 1, 1, 1, 1, 0}$. It is easily to transfer the two vectors into a binary table:
X/Y  1(Presence)  0(Absence)  Sum 

1(Presence)  a  b  a+b 
0(Absence)  c  d  c+d 
Sum  a+c  b+d  n=a+b+c+d 
Typically, here we give the example of two discrete variables, the mutual information between $x_1$ and $y_1$ is
The $\eqref{eq:1}$ is equal to
$p(x)$ is the probability that a symbol (here is 0 or 1) appears in the gene vector X regardless that what the symbol is in gene vector Y. $p(y)$ has a similar definition of $p(x). $$p(x, y)$ is probability of a symbol combination appears in gene vector X and Y. In this example, there are four kinds of symbol combination $(1, 1)$, $(1, 0)$, $(0, 1)$ and $(0, 0)$.
If we use the binary table to illustrate this equation, the $\eqref{eq:1}$ is:
The $\eqref{eq:3}$ is mathmatically equal to:
Example
We can use R to directly calculate the MI between two gene vectors mentioned above.
 Use basic R function
1 2 3 4 5 6 7 8 9 10 

 Use R package bioDist
1 2 3 4 

Reference

SeungSeok Choi, SungHyuk Cha, Charles C. Tappert: A Survey of Binary Similarity and Distance Measures.

Huynen M, Snel B, Lathe W 3rd, Bork P: Predicting protein function by genomic context: quantitative evaluation and qualitative inferences.Genome Res. 2000;10(8):120410.

Korber BT, Farber RM, Wolpert DH, Lapedes AS: Covariation of mutations in the V3 loop of human immunodeficiency virus type 1 envelope protein: an information theoretic analysis. Proc Natl Acad Sci U S A. 1993;90(15):717680.

Kensche PR, van Noort V, Dutilh BE, Huynen MA: Practical and theoretical advances in predicting the function of a protein by its phylogenetic distribution. J R Soc Interface. 2008;5(19):15170.
Update record
02/11/2016