Do not think of a single plate of a capacitor being charged to a potential. It does not work like that.
A capacitor is energized by a potential difference across its plates, to a charge given by Q=CV. Each plate is energized to the same magnitude of charge, Q, but with opposite polarity.
So, if you connect a capacitor between 5V and 3V, all that matters is the potential difference of 2V. The capacitor's plates will energize to a charge given by Q=CV. Assuming you give the system enough time to settle. Conversely, if you now remove the power supply, the capacitor is left with a charge, Q, on its plates. This gives rise to a potential difference across the plates again given by Q=CV. If nothing has changed (assuming a perfect/ideal capacitor with no leakage, etc) then the potential difference across it will be the same, 2V, as you energized it with in the first place.
The time taken to fully energize depends on how quickly enough energy can be transferred; that depends on the resistance in the circuit between the plates (and even of the plates themselves) and the power supply.
There is nothing to say it must be 'slow'. 'Slowly is an ambiguous and relative term; it has no meaning here without giving it context.
The charge on the capacitor with time is given by:
Q = Qmax ( 1-e-t/RC)
Since Q is proportional to V, you can also say:
V = Vmax ( 1-e-t/RC)
So, you can see that as R gets smaller, the -e-t/RC term becomes less significant, so the capacitor reaches full energy quicker. In theory, it never quite gets to full energy in a real system (with non-zero resistance), but in practice it gets close enough that we don't care.