Would this perhaps be an argument in favour of the source follower:
Charging up a large capacitive load would require a large peak current. With the common source amp, its peak current will be its bias current (what makes it class A). However, with the source follower, its peak current is only limited by the size of the output transistor.
Therefore, in theory you could supply a larger peak current for a given bias current with the source follower than with the common source amp. This would mean you would charge your output capacitive load quicker and with less power using the source follower over the common source amp.
Does this make sense?
If so, is it correct to view a source follower as a class A amplifier? By definition a class A amp is one whose peak current is equal to its bias current which (from above I think) need not necessarily be the case for a source follower.