I'll have to ask our team to make sure you're looking at the right security.yml file, but the general process you've described should work, even for custom groups. One tip: when there are multiple groups in the security file, the format is a bit different. Here's an example from the accessions module that might be useful for reference:
Make sure to add the double square brackets and preserve the same spacing and commas between group names. Ideally, make your custom group one word to avoid spacing in the group name having unintended consequences. Clear the cache and restart PHP-FPM after making changes.
Let us know if it helps! If not, I'll see if any of the developers have further thoughts.