The group size is the number of steps in the population size function.
In the BSP you can't estimate more population size changes than there
are coalescent intervals in your tree. So the alignments that are not
working will be the ones with a small number of sequences. This can be
fixed by reducing the number of groups from the default of 10 to less
than or equal to n-1, where n is the number of sequences in the
alignment.
FYI: Estimating highly parametric demographic functions from single
gene alignments of just a handful of sequences is not likely to be
very illuminating. For the single-population coalescent, sequencing
more individuals will help, but there is very little return for
sequencing more than 20-40 individuals. Longer sequences are better as
long as they are completely linked (i.e. the mitochondrial genome). By
far the best thing to do is sequence multiple independent loci and use
a method that can combine the information across multiple loci (like
EBSP).