A controller is the hardware. For instance a micro controller.
A driver (in this context) is a piece of software to control (with a micro controller or PC ) a part of the hardware.
Example: a piece of software to control a motor or display.
Later you call in your main software only the function to the motor driver.
The controller is the logic (hardware and/or software) that decides upon and specifies the driver's output. It's decision may be based on feedback from the device it is managing or it may run open loop.
The driver is responsible for supplying the actual output power to the device. This may be something like a solid state relay, although there are many other types of drivers.