Well, first you just need to define the array factor for circular array. LMS and RLS basically give weights that wil help in removing interference, noise or whatever.
The array factor for a Uniform circular array is
AF(theta, phi) = exp(j*(2*pi/lambda)*radius*sin(theta)*cos(phi-phi_n))
theta is elevation, phi is azimuth (it depends on the coordinates u're taking as references)
phi_n=(2*pi/M)*m
M=no. of elements, and m=1,2...M
to apply LMS or RLS,
Array Factor will become:
Wm*AF
Wm are the wts.