It's all about energy. High energy electromagnetic wave can transfer the high energy to the matter and potebtially cause an upset on the silicon circuit, but cross-section of electromagnetic wave to the matter is usually not high enough for the high energy photon. Most of the interaction between the photon and the matter is with photoelectric effect that happens at high probability below 1MeV, and around 1MeV, the cross-section hits the bottom, and then increases slowly from there, but not by much. It means, overall, there is less chance for an high energy photon to interact with the matter.
if you look at the cross-section plot of gamma ray or X ray, you can see the probability.