I would like to add here a special warning regarding tantalum capacitors:
They may burn violently, and cause fire hazard, and also emit toxic fumes, if they are failing either due to over-voltage or reversal. That happens very easily, if the power supply has sufficient current capacity!
Therefore: If you "misuse" aluminum electrolytic capacitors, they usually "just" leak or vent -and sometimes burst, causing a sticky mess. Very seldom they explore or behave otherwise violently (except maybe very big capacitors, and I mean physically so big, that they have screw terminals). I don't have much experience with those huge can sizes, but have seen many smaller capacitors "giving up".
However, a tantalum capacitor is often much more dangerous, if used beyond their specifications - were it then reversal of polarity, too high voltage, or too large AC current. I have seen many times burned circuit boards and other secondary damage as a result!
-ted
p.s. Also a quick summary of already earlier explanations, why polarized capacitors short-circuit when reversed: When the isolating oxide is damaged due to conditions it is not intended for, such as reversed polarity, it breaks down. The result is either high leakage current, or so low resistance, that we call it short-circuit. Depending on materials used and energy available, the consequences can vary from zero Vcc to destroyed circuit board and even to a fire. Thus it is not a good idea to reverse or overload those components.