Hi Mathias,
The yellow light turning on suggests that you're getting appropriate power to the charging circuit, so I think your chargers are good. If the batteries are not charging at all I would suspect it's the button being off or the battery itself going bad. You mention the buttons are all on, can you check that to make sure that's true? If you have a bad battery you should be able to confirm that by putting it into another FED? I'd start there and make sure you get any problematic batteries out of the mix.
Four other ideas :)
1) The battery is 4400mAh and the Feather charges at 100mA, so it theoretically will take 44 hours to completely charge one from dead. Have you tried leaving it plugged in for 2 nights?
4) (Longer term solution) I am talking with the Open Ephys Production Site (
https://www.oeps.tech/) about them building and selling FEDs. They are making some changes to the PCB to simplify manufacture that include removing the button and putting a power switch right on the PCB. This will also eliminate this issue of FED not charging unless the switch is on. We hope to have this arrangement completed by the fall, and they expect to sell complete circuitry for not much more than building it from scratch. I'll keep you posted!
-Lex