Let $X, Y \in \text{Random}(\mathcal{X})$ each be a
D5723: Simple random variable such that
| (i) |
\begin{equation}
\forall \, x \in \mathcal{X} :
p(x) : = \mathbb{P}(X = x)
\end{equation}
|
| (ii) |
\begin{equation}
\forall \, x \in \mathcal{X} :
q(x) : = \mathbb{P}(Y = x)
\end{equation}
|
Let $\log_a$ be the
D866: Standard real logarithm function in base $a \in (0, \infty) \setminus \{ 1 \}$.
Then
| (1) |
\begin{equation}
- \sum_{x \in \mathcal{X}} p(x) \log_a p(x)
\leq - \sum_{x \in \mathcal{X}} p(x) \log_a q(x)
\end{equation}
|
| (2) |
\begin{equation}
- \sum_{x \in \mathcal{X}} p(x) \log_a p(x)
= - \sum_{x \in \mathcal{X}} p(x) \log_a q(x)
\quad \iff \quad
\forall \, x \in \mathcal{X} : p(x) = q(x)
\end{equation}
|