Are you using an anti-aliasing filter? I think you are not and this is my reasoning. If your ADC is clocked at fs=100MHz, and you put in a tone at 41.4MHz, below Nyquist a good thing, your harmonics I think are aliasing down. So your HD2 is 82.8MHz, the alias would be 100MHz-82.8MHz=17.2MHz. If you have a HD3, this would be at 41.4MHz*3=124.2MHz, therefore, 200MHz-124.2MHz=75.8MHz, alias again, =>100MHz-75.8MHz=24.2MHz. Since it is a double aliasing of the HD3 thats why it is smaller. Does this make sense?
JGK