I'm not getting the point of this - IIP3 is supposed
to be a measure of channel linearity, for "pretty
good" signals. If you're going to clip it, that seems
like you'd be measuring garbage.
IIP3 is a frequency-point and so to get it you
need measurements at multiple frequencies
so you can extrapolate. Using a RC filter on a
varying frequency square wave will make you
a varying-amplitude varying-frequency sorta-
sine (with embedded harmonics, varying also)
and how are you going to take that effect out
of the data?