The smaller OA, the more A influences O. (Similarly with B, C.)
I don't know how to do root mean square error, but this is how I would calculate the motion of point O. The method is to determine how much percentage weight each point holds, in its influence on O.
Add 1/OA + 1/OB + 1/OC. Take the reciprocal. Call the result Z. (Identical to the formula used to find the equivalent R of resistors in parallel.)
Calculate a weighting factor for each point's motion. For point A, this is Z /AO. Etc.
Move O in proportion to the weighting factor for that point.
- - - Updated - - -
Example, if the weighting factor for A is 33%, then move O parallel to AA', for a distance of 33% of AA'.
Etc.