Hi,
1)
You ask about mounting on a transformer, but we don't see the transformer.
Usually a transformer is no heatsink. If you can use it this way depends on the thermal situation of the transformer. It's own power dissipation, it's temperature, the ambient temperature and the thermal resistance of the transformer to the ambient.
If you consider to use the iron core..then you should know that the thermal resistance of iron is way worse than aluminum.
Avoid conductive material directely touching the iron core to avoid eddy currents.
2)
You need to take care about waveform. Thus we can't verify your calculations.
For power dissipation calculation use the
* RMS current for the Mosfet
* average current for the diode
We don't see a schematic...so don't know if the schottky diode is necessaray at all....what's the use of it?
3)
We don't have enough information. Can you be sure the torque is always within specified limits? Can you be sure the heat spreading is within specified limits? If you can answer "yes" on both, then you could use a current measurement technique. A motor protection fuse with thermal trigger. A current measurement...I2t trigger...
If you are unsure: use a thermofuse directely at the motor.
Klaus