In most of cases, a circuit is stable when it does not have any positive feedback. This means that the phase shift of the feedback path is not 360 degrees. Then why in the phase margin definition 180 degree is used to check the stability. I believe that phase margin should be measured with 360 degree reference because at 360 degree phase shift, positive feedback occurs, which causes stability problems.
So do you mean that 180 degree phase shift is provided by the negative phase shift and rest of 180 degrees are provided by system closed loop response?
Yes. That is what happens at the frequency for which |GH|=1 when the system is in the limit of stability.
(There is an additional condition related with the slope of the phase, but let's leave that in this moment.)
Suppose that at zero-frequency G and H are both real and positive . Then we have negative feedback. If increasing frequency we find that at the frequency for which the loop gain is unity the phase is 180° (i.e. zero phase margin) the total phase shift is the 360 deg you mention in the first post.