A source follower is a buffer. It can provide current gain
but not voltage gain (always less than 1.00000000...).
It -could- be used to boost "op amp" output current.
But for RF you would not likely be able to use an op amp,
for frequency reasons especially but also because it's
likely to have very poor gain @ frequency and higher
than desirable noise.
Why, or whether you need a designed PA, depends on
the goals for the assembly. If you can do it as described,
in fact, then have at it. But expect to find reasons why
you need to do better, as you proceed with your first
stab at lineup design and analysis.