From: Osher Doctorow, [log in to unmask], Fri. Sept. 29, 2000, 7:27PM
In response to Nantana Naranong, I will say a few words about Markov chains
and mutual information/entropy. Let me first tell you that the best
technical source of information is the original papers on these topics,
which go back rather far in mathematics (probability and statistics in
particular for Markov chains) and engineering (for mutual information).
Look under C. Shannon for the original work on information theory including
mutual information/entropy around the 1950s. Likewise for Markov, whose
first initial I do not recall offhand (I think it is Y.). For an intuitive
understanding of what these things are, you can try internet mathematics
encyclopedias or glossaries or dictionaries which are usually free. An
especially good presentation is by Reza in the 1960s on information and
mutual information. Then there are intuitive presentations, which you
might be able to squeeze out of a Lecturer, Instructor, Professor, etc., if
he/she respects you very much or is unusually communicative. I will
volunteer intuitive definitions: a Markov chain is a sequence of random
events or variables which are "memoryless" in the sense that once you know
any fixed event in the chain, the probability of the next event/variable in
the sequence is known and does not depend on any events other than the one
previous (fixed) event. Mutual information is the information contained in
two random variables/events, and analogously for mutual entropy (which is
often not distinguished from mutual information). It turns out that the
equation for Markov chains is extremely similar to the equation for
independent events/random variables, i.e., events or random variables which
do not influence each other. For this reason, Markov chains can be regarded
as "almost independent" events or random variables, or as "one step removed
from independence". Put another way, they are slightly dependent but almost
independent. It turns out that (Bayesian) conditional probability (my
abbreviation BCP), a mainstream probability/statistics tool, is very
applicable to Markov chains and in general to events or random variables
that are of low or even negligible dependence. Logic-based probability
(LBP) has been found to be very useful for events of fair to high dependence
(see some of my earlier papers for LBP). A textbook on university
probability theory, preferably second or third year, should be adequate for
(Bayesian) conditional probability. Similar comments apply to mutual
information, to which both BCP and LBP apply and in fact under circumstances
remarkably similar to the above - LBP is especially applicable to highly
dependent and also highly rare events (for the latter, BCP
information/entropy tend to "blow up").
----- Original Message -----
From: "nantana naranong" <[log in to unmask]>
To: <[log in to unmask]>
Cc: <[log in to unmask]>
Sent: Friday, September 29, 2000 5:33 PM
Subject: markov chains
> Hello,
> Does anyone know the meaning of these words as below:
> Texture analysis, Kullback-Leiblen Distance, Gibbs Inequality,
> Markov Chains and Mutual information.
>
> I would also appreciate some comments about these.
>
> Thanks in advance.
>
> Nantana Naranong
>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|