Averaging Image
You are complaining that your calculations are takes too long but you don't say how you have implemeted them.
As far as I know, you need to do 2-D convolution of your input image with a certain mask. So you should do a lot of matrix operation, look at the assembly code to check if your compiled used Multiply and Accumulate operations. Since the mask values are the same for the whole image, it is faster to store this values on fast acces registers, you shouldn't load this values for each pixels.
If you can do a 2D FFT and inverseFFT faster than image convolution then you could go for frequency domain.