Your calculation and concept of power are wrong.
The Zener is in shunt configuration - it 'soaks up' the excess current that isn't flowing into the load and in doing so it dissipates heat. The less load you place on it, the more it has to dissipate and the hotter it gets.
1) Max current through zener= bener wattage/zener volt=1.3/5.1 = 0.255amp,
Gives you the amount of current that would make the diode dissipate 1.3W, that isn't what you want. The intention is only to ensure the total load current maintains 5.1V after dropping through the input capacitors.
Calculate it like this:
1. Find your total
maximum load current, that is the LED, relays and anything else added together and activated.
2. Ignoring the voltage drop in the bridge rectifier, work out the RESISTANCE you would need to drop the incoming AC to 5.1V at the current from step 1. (230 - 5.1)/I
3. Work out the capacitor value with equivalent reactance, this will be your input capacitance. Xc = 1/(2 * pi* f * C)
If you remove the Zener altogether that would then give you 5.1V at the output
under full load.
The problem is the voltage will rise if the load current drops, that is where the Zener comes into play.
4. Find the total
minimum load current, that is with the circuit working but in a state where it's current is as low as it can be. Relays not energized etc.
5. Subtract result 4 from 1 to get the difference between maximum and minimum.
6. Choose a 5.1V Zener rated to carry the current from step 5.
For example, if your circuit draws 100mA maximum and 40mA minimum, the Zener would need to pass 60mA which means a rating of at least (5.1 * 0.06) = 0.3W
As your components are getting too hot, it suggests the input capacitors are too high in value and letting more current than necessary flow through them. Note that this kind of simple circuit has NO safety isolation and must, including whatever it supplies, be completely insulated. Do not connect the output to any other equipment. The input capacitors should be rated for AC use and the 1M resistor across them should be rated for high voltage as well.
Where is Rs in your schematic? If it is the resistor in series with the LED, the value should be (5.1V - Vf)/Iled. For example for a standard red LED with Vf = 1.6V and I=10mA it would be (5.1 - 1.6)/0.01 = 350 Ohms.
Brian.