Provide heat sink.
Probably it is not able to provide sufficient current when all the displays on,with around 7V drop across itself.That could be a problem.
Its better if you use heatsink other wise any time the regulator will burn,, why dont you use 7 segment display from direct supply without 7805?? I mean, use 7805 for the controller circuit and supply 7 segment direct from rectifier. use rectifier upto 9v - 10v.
regards