It may be possible to do the job using a buck-boost, but you need to experiment to discover how to get the most from it.
Try all possible configurations of the buck-boost...
By using an Nmos, then a Pmos.
By putting the converter stage adjacent to the supply rail, then adjacent to the ground rail. (Paying attention to diode direction as you change things around.)
By taking your output from the supply rail and center node, then from the ground and center node.
This gives 8 combinations you need to test.
The mosfet needs to turn on with 3V at the gate if an Nmos (whereas a Pmos will turn on with 0V but you don't say you can apply 3V to the more positive terminal so that it has a differential of 3V).
There may be such a type of mosfet that will work in this low range. While you are experimenting, however, apply 4 or 5 V to the gate. Once you see whether one of the 8 configurations will work, you can start reducing volt level at the gate.