concept of hole in effective mass
To understand why mobilities of electrons and holes are different you need to understand the concept of effective mass of electrons and holes.
Its true the movement is always of electrons. But in the conduction band of the semiconductor the scenario is that electrons are very few and there are a lot of empty energy states and so these electrons move quite freely. We calculate this Energy movement through Scrodinger Equations and then equate it to the classical F=ma equation and thus get expression for the "effective" mass of the elctron moving in the conduction band. So note the mass we use for electrons in the semiconductor calculation is not even the actual mass of the electron but some "effective" mass, which changes from semiconductor to semiconductor and even worse it also changes from the direction in which the electrons flow.
Now for holes, their movement is always considered in the Valence band, where there are lots of electrons and very few empty states of energy. We call these empty states as holes. So what we do is we consider holes as particles and then for these holes all electron occupied energy states now become empty energy states! But still since the movement of holes is in the valence band which has a different stucture of the available energy states therefore when you do the actual calculation through schrodinger wave equation and equate the result to the equation F=ma, you get a different "effective" mass. And that is the effective mass used for the hole.
So finally back to your question. The mobility of a particle is inversely proportianal to the effective mass as it turns out when you derive the expression for mobility. So since the effective mass of holes is greater than the effective mass of electrons the mobility of holes is less than that of electrons.
Hope this gave you a general idea, whats going on.