Can you test the color sensor to see what kind of signal it produces, when exposed to various colors?
Does it output a particular volt level?
Or does it output 3 signals representing red/ yellow/ blue levels (red/ green/ blue)?
Is the signal analog or is it digital?
The object should be illuminated with white light. If room lighting is too dim then your robot needs to carry its own headlamp. Dim light may cause your color sensor to 'see' a wrong color and thus it produces wrong readings.