App Inventor is not a professional game design platform. It's an educational tool to help people understand the concepts of programming. It meets and exceeds its purpose.
There's a lot of things that can be done in App Inventor in a semi professional way, but advanced programming like a fast game, are not really what the developers have in mind when making App Inventor.
I also felt frustrated when I reached the limits of what I can do with graphics, but then somebody in this forum explained this same thing to me.
If you are really into game developing, you should try learning more advanced platforms, like Unity, Gideros Mobile or Android Studio, for example.