Well, you pretty much got it. The noise will get the oscillator started. However, you should remember that there is a filter element in this oscillator, that is, something depends on the frequency, either an LC tank, or and RC network, etc. (Usually β is made to be frequency dependent).
That means that out of the many components of the noise (it's pretty much white noise), only the correct frequency will be returned to the input with enough amplitude and phase to make the oscillator run. So the oscillator will run at that frequency.
Mathematical explanations exist, of course, but they may not be so easy to grasp.
However, here is a simplified one, which you probably know already:
the oscillator can be viewed as an inverting amplifier (gain A), with a feedback network ("gain" β, in fact it's in most cases a loss). Now calculate the output, as a function of the input, for the amplifier:
Vout=A*Vin
But Vin is really Vout multiplied by β: Vin=Vout*β.
For the oscillator to work, the input signal Vin must be sufficient to produce Vout when amplified.
So now we can write:
Vout=A*Vout*β, or 1=A*β, which is the condition you already knew.
If the amplifier is inverting, then its gain is -A and the final condition is -1=A*β, meaning the phase of the signal at the amplifier input has to be -180°, instead of 0°.
Since β is really a loss, you can see that the condition requires that the amplifier make up for it, or else the oscillator will not work.
If you have A*β<1, you can see that that A*Vout*β<Vout, so the oscillation cannot be sustained, since all the time the output amplitude will decrease, instead of building up.