I suggest that you buffer the oscillator part from the amplifier part. That way you can adjust the gain without affecting the frequency.
Also what is on the other side of the crystal/resonator?
Without spending a lot of time analysing this in detail, what I suspect is happening is that you are altering the impedance as seen by the crystal. The equivalent circuit of a crystal is a capacitor, resistor and inductor in series, all in parallel with a capacitor. It has a very high Q which typically can made it very sensitive to value changes in the surrounding circuit and load. By altering the source resistance you are altering the characteristics/load and so pulling the crystal frequency.
By the way, if the values on your schematic are correct, then the trimmer resistor in the source line of the FET at 100K will swamp the 100R that it is in series with. I know that you can say that the 100R provides a minimum resistance in the source line, but unless the trimmer is right at the end of the range, then it will have very little effect.
Susan