Our high schoolers run the SciOly club; the Executive Committee (ExCom) is a small group of elected students chosen the prior year. During the summer, they participate in the Captains' Summer Exchange to get access to tests written by other high schoolers across the country. At the start of the school year, before rules drop, they start the application process: every student has to fill out an application form asking what classes they've taken that might help with SciOly, what other extracurriculars are vying for their attention, how many hours they will devote to SciOly, what their priorities are, what events they want to do, and if there is anyone they'd like to partner with (or not partner with). Every student also is interviewed by ExCom and takes a 1-hour test that is a distillation of questions from the available Captains' Summer Exchange tests in all non-build events.
Students are chosen for their devotion and past performance, which might put a great newcomer on a lower team, but those great newcomers are encouraged to continue the next year with expectation of advancing to a better team. Team announcements are made the week after rules drop.
While I agree that students who want to do Scioly shouldn't be turned away, the reality is that some students are just there to bolster their college applications, so our process seeks to weed out those who are just there to say they were on the team but aren't willing to put the time into it. We have had 4 teams for the last several years and will probably not increase the number -- we've found that you can realistically have 2 really good teams and 2 other teams of reasonable quality, but beyond that it's hard to find really motivated students. Plus 4 teams is about as many as an event coach can realistically mentor. We have occasionally made adjustments to the roster, but that's rare.
Hope that helps!