Ceramic cap can surely handle 5V. Most have a 25 or 50 volt rating. Put one at the V+ pin of each IC with shortest lead possible. The picture of the power supply shows longer leads than you would like. Also keep caps from touching any hot devices, like regulators.
what you have connected is 100V-400V range capacitor.
What voltage transformer you are using
before bridge rectifier no capacitor required
after bridge rectifier atleast connect 100-1000uf capacitor
and if 7805 is kept near to bridge rectifier output capacitor then 0.1uf ceramic is ok but 10uf/25V electrolytic will be still better.
Ceramic capacitor (like the one on the pic) changes its value with temperature (you can test it if you have a simple capacitance meter, by heating the capacitor body with your fingers). But the exact value is not important if used as bypass. And in your case it can be used at the regulator output for sure.