The LM380 had sufficient stability issues with their vertical-PNP output stage, to complicate this further with external transistors.
You would lose voltage swing, which is not great to start with, causedt by the transistor's Vbe.
The transistors would be outside the feedback loop. No distortion correction available.
The DC biasing would have to be investigated to ensure that it would not cause a DC current flow from the 380.
I'm sure AudioGuru will rspond to this thread and add further insights of his own.
It actually it would be better if you used a modern audio amplifier IC to get the power level you require.