It's all about the airflow. You might consider putting the fins
outside, or through, the enclosure (or with proper insulators,
use a metal enclosure as the sink).
You can also find itty bitty 5V fans that could help things
(look inside junk laptops, etc.) if the box ends up being
tucked inside some dead space and natural convection
can't be counted on.
I have also seen 12V-adaptors that have the regulator
up in the plug. Might modify your plug to put the regulator
up in it, and export 12V raw and 5V regulated to the electronics
package; that would put the regulator dissipation and the
payload dissipation in separate "sinks" to the benefit of
both.