Or the 'dirty' fix is to add a Schottky diode in series with the battery, it should drop a fairly constant 0.5V or so. You should follow the ESP32 data sheet and add a large value capacitor across the supply pins as Klaus suggested, without it and a regulated power source you risk it crashing as the Wifi transmitter starts running.
My experience with ESP32, and I use quite a few, is that if the supply voltage drops too low (about 2.5V) they crash and the transmitter tends to turn on by itself and current consumption shoots upward. It doesn't do any damage but if the voltage is then raised back up to 3.3V and if you don't reset it, the transmitter will continue to run and current consumption will remain high.
Brian.