You can't get rid of it, if you need inversion. Now you
might flip comparator inputs and lose one inverter stage
(presuming no other circuit downside, like loss of drive
and actually increasing net delay as a result).
More likely your comparator analog section, not some
inverter, is the bulk of the lag. That wants more current,
lower signal swings and more front end gain especially
in low-overdrive applications - and prevention of any
large-underdrive initial conditions from winding up the
gain lineup. Comparator speed being very dependent
on overdrive, especially in simple topologies.