This is my reasoning, I may have gotten sound dB power and energy mixed up (10*log vs 20*log).
Take average speaker sensitivity 92dB SPL 1W/1m and headphones 92dB SPL 1mW, and using an 8R speaker at 1m away, verses a 32R headphone with no air loss.
This means headphones need 1/500th the power of a loudspeaker to make the same SPL at your ears, in this example.
But the audio amp LM383 hum and noise ends up present at the output regardless of the volume control setting. The way the circuit is right now, a loudspeaker or headphones will see the same voltage, so the headphones make the noise appear much much louder. 1/500 is almost 27dB which is a huge difference.
It is traditional to attenuate headphone drive with 100-220R in series with each headphone channel. This is done on receivers and other audio gear.
I assumed OP's noise problem would certainly show up when using headphones. Otherwise, it's upstream of the volume control or oscillations.
OP's circuit has some component values out to lunch, for example 100uF coupling cap then 470pF feed to the op-amp. So it might be losing all the signal there and trying to get it back with lots of gain afterwards, which makes for poor S/N ratio. I would not have a 0.22uF cap directly across the LM383 output as a capacitive load, there should be a 1-10R resistor in a Zobel network lest the IC go unstable. I had TDA2030/LM1875 oscillate at 13MHz which seemed like high noise and distortion until the scope revealed the output network needed optimization.
http://www.sengpielaudio.com/calculator-soundlevel.htm
edit: The original circuit is from 73 Magazine by W6QIF
http://n5dux.com/ham/files/pdf/The Magical Audio Filter.pdf