you have to start with the maximum ambient temperature. This is where your heat is going to, say 40 degrees C. At the other end you have to consider the maximum permissable temperature for all components, especially electrolytic capacitors (they have a short life at elevated temperatures) and semi conductors ( which generate heat). Once you have found the most heat sensitive semiconductors, mount them on heat sinks on/as part of the case. If the layout is particularly compact them a heat pipe will be required to get the heat to the outside and then into the atmosphere. This is why fans are used, much more compact then a big array of heatsinks!
Frank