Hi Brendon,
A couple of things to try...
The voltage across the battery, while not connected to anything (the open circuit voltage) might not be the same when its under load.
Once you have connected the DC/DC convertors, try testing the battery voltage with a multimeter while its actually under load. You may find its less than 4.2v.
The stepdown convertor will only be able to "reduce" the voltage (hence "step down") so if the battery falls to less that 3.3v the stepdown convertor wont be able to maintain 3.3v.
There is always a drop out voltage across any step down convertor. You will need to check the data sheet for yours, but it could be 0.5v for example. In which case, the battery voltage will need to be at least 3.8v for the DC/DC convertor to produce 3.3v, as 0.5v is "lost" in the convertor.
If you haven't already, check the ESP01 module works using a separate power supply. if you don't have a bench power supply, just charge your 18650 to 3.3v and connect it directly. But be careful.
DOUBLE CHECK the polarity. every time you connect the power. the output voltage dropping to 2.8v and a buzzing sound could be you have it connected back to front, or the module is already toast.
I would suggest you buy a simple USB to serial dongle to program the ESP01. They normally have a voltage regulator on them too, so its a good what to test everything works, before plugging the module in to something else.
And to answer your other questions .. yes, the Wemos D1 mini battery shield has under voltage protection. So it cuts out when the battery drains. It also cuts out when its fully charged too.
Regards,
Jon.