What did you expect?
Most of the port pins are connected internally via diodes to GND and VDD for protection purposes. When the potential of the port pin is under GND, or above VDD (with approximately 0.6V), one of the 2 diodes begins conducting in trying to equalize the 2 potentials.
In your case is like supplying the uC through the pull-up and port pin and this is a faulty design, because VDD & GND are the last signals in a circuit to be "disconnected" from the power supply.
This is not only annoying, but it could also destroy the internal diode (at least) if the current through it is big enough.
Correct this by tying the pins via pull-ups (pull-down's) to uC's VDD (GND) signal(s) and not separately to the power supply. Then you connect this "power-plane" to the power supply, so that everything gets "disconnected" at the same time (VDD and ports alike).
Arthur