what does relative entropy mean?

[W_Heisenberg]

anyone helps?

 

[Shug]

It is a measure of the distance between two discrete probability distributions. The same probability distribution would have a relative entropy of 0 with itself, while a large relative entropy indicates a large difference between two distributions. It is also known as the Kullback-Leibler distance between two distributions. I believe it is often used to show that one probability distribution is converging to another via a shrinking relative entropy.

 

[W_Heisenberg]

what does " converging to another via a shrinking relative entropy" mean?

shrinking ? any physical meanning of the relative entropy here?

It is a measure of the distance between two discrete probability distributions. The same probability distribution would have a relative entropy of 0 with itself, while a large relative entropy indicates a large difference between two distributions. It is also known as the Kullback-Leibler distance between two distributions. I believe it is often used to show that one probability distribution is converging to another via a shrinking relative entropy.

 

[Shug]

"converging to another via a shrinking relative entropy" means that as you increase n, the relative entropy is decreasing (going to 0), so the distribution is converging. I do not know of a physical meaning to relative entropy; as I stated above, it is a measure of distance between two probability distributions.

 

[W_Heisenberg]

I do not understand this "distance".

But thanks a lot though.

"converging to another via a shrinking relative entropy" means that as you increase n, the relative entropy is decreasing (going to 0), so the distribution is converging. I do not know of a physical meaning to relative entropy; as I stated above, it is a measure of distance between two probability distributions.

 

[Shug]

Think about it this way: you define a distance metric for two numbers, say the absolute value of their difference. This one is straightforward to define. Another measure of distance is to take their difference squared; once again, this is pretty straightforward.

However, how do you define the distance between two vectors? One way to do it is to measure the 2-norm of the difference vector; once again, pretty straightforward.

Now, how do you define the distance between two distributions? One answer is to use the relative entropy. That is likely where the motivation for relative entropy comes from; people were trying to judge the distance between two distributions, and in order to do so they came up with relative entropy.

 

[FvM]

Shug mentioned the meaning of relative entropy in statistics. You can review the topic at Wikipedia.

The original post doesn't mention a context, but apparently are you assuming relative entropy could have a meaning in thermodynamics, too. Any indications?

Curiously, you posted the question in the Digital Design and Programming.

 

[W_Heisenberg]

I think this is digital communication question.
Maybe in wrong place?

Shug mentioned the meaning of relative entropy in statistics. You can review the topic at Wikipedia.

The original post doesn't mention a context, but apparently are you assuming relative entropy could have a meaning in thermodynamics, too. Any indications?

Curiously, you posted the question in the Digital Design and Programming.

 

[zorro]

Dear W_H,

you should be more explicit. Where does the question came from?
I guess that the question is in the context of information theory, where the term has a meaning.
In that case, search for "relative entropy" "information theory".
Regards

Z

 

[W_Heisenberg]

Think about it this way: you define a distance metric for two numbers, say the absolute value of their difference. This one is straightforward to define. Another measure of distance is to take their difference squared; once again, this is pretty straightforward.

However, how do you define the distance between two vectors? One way to do it is to measure the 2-norm of the difference vector; once again, pretty straightforward.

Now, how do you define the distance between two distributions? One answer is to use the relative entropy. That is likely where the motivation for relative entropy comes from; people were trying to judge the distance between two distributions, and in order to do so they came up with relative entropy.

Pro Answer!