Well, differentiation would solve the problem if you had a function to differentiate. Don't forget that the expression for transistor current is only an approximate one, that doesn't describe all effects.
What actually happens is following:
1. For smaller Vds, as you increase Vds, rout increases due to the CHANNEL LENGTH MODULATION
2. In short-channel devices, as you increase Vds further, there is an additional effect called DRAIN-INDUCED BARRIER LOWERING which reduces threshold voltage and increases the current, thus effectively canceling the effect 1. In this area of Vds, rout is relatively constant and independent from Vds.
3. As you further increase Vds, IMPACT IONIZATION near the drain produces large current, lowering the output impedance.
Hope I could help
Djordje