No the noise isn't a problem - more annoying than anything else. Think of the coil as a sort of loudspeaker.
Even if your circuit works outside the audio frequency, say 100kHz, you can still get a audible noise under certain loads. Depending on the circuit design and the feedback loop the circuit can "pulse" - doing a burst of oscillations then stopping for a short while. This could be a burst for 100us followed by a 200us gap, for example. That will make a sound with 3.3kHz components in it, so it will be audible.
Keith.