Well, yes and no. If you look at the logs when the build failed, you might see that some files were not downloaded. They are not heavy files though. Ofen times, I start my project offline using android studio. Though I develop while connected to the internet hence I don't get issues as yours frequently but yes, I have encountered a similar issue.
Gradle issues with flutter are not as severe as that of native android(java and kotlin) {from my own experience, although I may be wrong}. For every failed build, one should check the error logs to have an idea of what went wrong.
Failed builds could be a result of many things - certain files need to be downloaded, androidX incompatibility in a certain plugin... One just have to check the logs to be sure what went wrong.