Higher frequency is less efficient to produce, lower frequency can't carry as much data per second.
It is a compromise between the two but it has become a standard over time so that interchangeable detector modules can be used. The standardization also defined a band of frequencies in the 38KHz region that other emitting devices, such a CFL should avoid to minimize interference. Please remember that the frequency was first used in the 1970s when LED and IC technology was in it's infancy. You can of course choose any frequency you like if you want to design your own but you will need an unfiltered IR detector to recover the data.
Brian.