the API design team probably shouldn’t be too big so you can collaborate effectively, but that doesn’t mean you should rush and have a one-sided team of only the API’s backend developers.
If you have tech writers for API documentation they should be able to give their feedback early on, so include them.
If your API is a backend for frontends, such as mobile or single-page-apps, then the frontend developers as API consumers should be the primary drivers of the process.
For external APIs you also need to consider non-technical requirements so include business people responsible for them, too.
Getting feedback from potential clients is also a good idea for external APIs so you won’t build something nobody wants.
Hope this helps!