Hi,
First step would be to calculate PD of 7805: (Vin - Vout) * I. If it's 12V in, then 7V * (Modbus + ATMega currents).
Good idea to review datasheet graphs for 7805 about current draw at different Vin - Vout. I think about 100mA is a nice current for that size voltage drop without a heatsink or forced cooling.
Possibly - depending on current draw - need to heatsink the 7805.
If 7805 is the regulator you must use you could bypass it with a BJT, figure 13 and 14 of STMicro 78xx datasheet has the circuit diagrams and formula.
Use a dropping resistor on input to 7805, but that just wastes the same amount of energy, at least it would be outside the regulator. You can do the same thing with a mosfet (using it as a dropping resistor), I believe.