Information Theory is awesome so here is a TL;DR about Shannon's entropy.
Information Theory is awesome so here is a TL;DR about Shannon's entropy.
-This field is about quantifying the amount ``of information'' contained
+This field is about quantifying the amount of ``information'' contained
in a signal and how much can be transmitted under certain conditions.
What makes it awesome IMO is that it is very intuitive, and like thermodynamics in Physics it give exact bounds about what is possible or not.
in a signal and how much can be transmitted under certain conditions.
What makes it awesome IMO is that it is very intuitive, and like thermodynamics in Physics it give exact bounds about what is possible or not.
@@ -62,17+62,23 @@ What makes it awesome IMO is that it is very intuitive, and like thermodynamics
This is the key concept from which everything is defined.
This is the key concept from which everything is defined.
-Imagine that you have a distribution of probabilities p on a finite set of symbols and that you generate a stream of symbols by sampling them one after another independently with that distribution.
+Imagine that you have a distribution of probabilities p on a finite
+set of symbols and that you generate a stream of symbols by sampling
+them one after another independently with that distribution.
-To transmit that stream, for instance with bits over a communication line, you can design a coding that takes into account that the symbols are not all as probable, and decode on the other side.
+To transmit that stream, for instance with bits over a communication
+line, you can design a coding that takes into account that the symbols
+are not all as probable, and decode on the other side.
For instance if $P('\!\!A')=1/2$, $P('\!\!B')=1/4$, and
$P('\!\!C')=1/4$ you would transmit ``0'' for a ``A'' and ``10'' for a
``B'' and ``11'' for a ``C'', 1.5 bits on average.
For instance if $P('\!\!A')=1/2$, $P('\!\!B')=1/4$, and
$P('\!\!C')=1/4$ you would transmit ``0'' for a ``A'' and ``10'' for a
``B'' and ``11'' for a ``C'', 1.5 bits on average.
-If the symbol is always the same, you transmit nothing, if they are equiprobable you need $\log_2$(nb symbols) etc.
+If the symbol is always the same, you transmit nothing, if they are
+equiprobable you need $\log_2$(nb symbols) etc.
-Shannon's Entropy (in base 2) is the minimum number of bits you have to emit on average to transmit that stream.
+Shannon's Entropy (in base 2) is the minimum number of bits you have