Have two sets of sensors, one sensitive to blue light, the other sensitive to red light (e.g. using filters or illuminated with blue and red light
separately. Possibly better to use a red and blue LED rather than a white one).
A reflection off a white surface will trigger both sensors. A reflection off a blue, green or yellow background will only trigger one or zero
sensors.
A lot of experimentation will be needed. Sometimes surfaces are glossy, etc. Also, what you think is (say) a green surface may not be
pure green, and so on.