That's asking for a variety of problems:
- 5V DC used by switches is coming from same source as IC's 5V?
- If separate 5V supplies: what happens if switch inputs 5V, but XC95144 isn't powered up? Are there series resistors in place to account for this situation?
Better use pull-ups with switch to GND.
Also you might want to check for short circuits between I/O pins, and have a good look at how the outputs are loaded. Is this burning up a regular event, or 1st time this happened? (maybe lightning struck or something...)