In all probability you apply the higher voltage through one device, and the lesser voltage through a different device. This occurs internally in an op amp which is a common level shifting device. Is that what you are using?
The output stage has two different transistors which turn on to greater or lesser extent. To apply a higher voltage, one of its output transistors sources current.
To apply a lower voltage, the other transistor sinks current.
If one transistor has more conductance than the other, then it creates dissimilar rise/fall times.