The fast flashing of the blue LED is (supposed to be) when it is running, but does not have a GNSS fix. Once it has a fix, the blue LED stays on, if it is transmitting. Do you have an OLED display on it? In the recent versions, if you turn it off, and then plug into USB power (and there is a battery in it) then instead of booting up normally, it shows "CHARGE MODE" on the OLED for a second and then turns itself off but the blue LED stays on while it charges. In any case, if USB power is not connected (or even if it is) and you press and hold the button (the one closest to the USB jack) until it says "OFF" on the OLED display (and then stop holding the button) then it should turn off completely. Maybe you keep holding the button too long, so it starts up again?
If power_ext is set to 1 in Advanced Settings then the "charge mode" does not happen. And it turns itself off if all the following conditions are met: the device was running for at least an hour, the aircraft is not airborne, the external (USB) power was on and has been turned off, and the battery voltage has decreased to under 3.9 volts (which may take a long time). This setting is intended for installation in an aircraft with both an internal battery and USB power from the aircraft. But even with that setting, you should be able to turn it off via the button.
By the way, which exact model of the T-Beam do you have? In the early
models (version 0.7) there was a real slide switch to disconnect power.
In versions 1.0 and 1.1 of the hardware, there was an AXP192 Power
Management Unit and that first pushbutton actually talks to the PMU not
the ESP32. In version 1.2 they switched to an AXP2101 PMU. SoftRF
works OK with either PMU type.