A common source stage will only be a unity gain amp
either under very specific bias, load and process
conditions, or when driven in a feedback loop. It will
be hard to stabilize across all. The common {source,
emitter} stage is a voltage amplifier and to reduce
its gain to 1, needs a (compensating) fractional
gain. And because gain variability is high, feedback.
A common drain stage will never be truly unity gain
unless also feedback controlled, but may be "close
enough" without feedback, and can be easier to
stabilize - sometimes a very simple diff amp can
do the job without compensation, provided you
don't need rail-rail input range. Any plain common-
{drain, collector} stage is sub-unity voltage gain
on its own, and to get unity gain you need to have
a preceding gain stage (not much, but some).