0. Basics

0.1 3 Variable Probabilities

Es gilt, dass:

• $p(a,b\mid c)=p((a,b)\mid c)=\frac{p(a,c,b)}{p(c)}$
• $p(a|b,c)=p(a\mid (b,c))=\frac{p(a,b,c)}{p(b,c)}$

0.2 Entropy

Entropy basically measures how uniform the PMF is. The more uniformity, the more less we are sure of what comes next, the more entropy.

Inverse as in physics, lowest entropy is achieved for uniform gas.

0.3.1 Equations

We can reconstruct basically all identities from the two rules:

• log turns products into sums
  • $p(x,y)=p(x)\cdot p(y|x)$ turns into $H(X,Y)=H(X)+H(Y|X)$
• expectation is linear

Conditioning is just something you wrap around any identity:

• if it holds for any distribution, it holds for $P_{\cdot \mid Z=z}$
  • We then average with $p(z)$ and sum, and we get the identity with "$\cdot \mid Z$" appended to ever term

0.3 Mutual Information

Three forms:

• $I(X;Y)$ is symmetric! → it’s the reduction in uncertainty
• Overlap of entropies (Venn diagram)
  • if X,Y indep. describing would cost $H(X)+H(Y)$
  • mutual information measures how much is saved by describing them together
• Distance from independence (real definition)
  • $I(X;Y)=D(p_{XY}||p_X{p}_Y)$