To my opinion, the CIC filter has low latency (group delay) compared to other filters, of course depending on the order. For minimum delay, try an IIR or a low order FIR filter at the input sampling rate and resample without additional averaging.
But I don't know your application requirements, so I can't make specific suggestions. In the general downsampling case, you need a low pass filter, that cuts off frequency components above the Nyquist rate of the downsampled signal, to prevent from aliasing. Additionally, you're possibly trying to achieve a particular frequency or time domain characteristics. Typically, increasing the filter order and/or stop band attenuation also increases the group delay. Thus a minimum delay decimation filter always requires a trade-off.