Hello,
The algorithm for the GO terms based on the hierarchical structure of Gene Ontology. We essentially start from the leaves of the hierarchy and move up the structure until we find a node that is between 20 and 500 genes. This is the non-redundant set. Since most databases are not in a hierarchical structure, we can't apply this technique to most gene sets.
WebGestalt does have redundancy removal in the Advanced Parameters section below where you enter in your gene lists. This is not dependent on a hierarchy and can be used for any organism or database. If you would like to remove redundancy from a GMT file before analysis, you could try using one of these methods yourself. k-Medoid and affinity propagation would be my recommendations. cluster and apcluster are R packages that implement these methods and could be a good starting point.
Let me know if you have any questions.