You are right, the reference is the stable element in the system. So if it is stable, everything else should be stable.
That is almost true. The problem is that the error amplifier (the one that compares the output to the reference) has a limited gain, it is not infinity. The actual error between the P/S output and the reference is in fact
ε=ΔV/A
Where ε is the final error, ΔV is the difference that would exist if the error amp was not present and A is the gain of the error amp. As you can see, if the gain of the error amp was infinity, then the final error would be zero, in other words the output would faithfully follow the reference and it would not change, except for changes in reference. (We are talking about the DC gain here, or gain at very low frequency).
However, the error amp has a finite gain, which in the end results in a non-zero error. Typical DC gains for the entire chain can be anywhere from 60 to 120dB. 60dB is not that good, it means a gain of 1000, so the error will only be 1/1000 the variation that would occur in open-loop. That is not too great, since the output can change a lot without regulation. So dividing that "a lot" by only 1000 still results in some significant change in the output when the input changes.