I think the answer is ultimately 'it depends', but I think there are a couple of main points:
1) Making everything a widget is probably easier from an api point of view. You suggest that it might make it harder to do stuff like zooming, but this really wouldn't be the case, it's easy to bind positions etc. in a relative way. Plus as widgets you get all the normal kivy ways to position and interact with them, rather than needing to manually handle the extra layer of lower level stuff.
2) A bigger issue is that everything being a widget gets *slow*. That's not necessarily a problem, depending on how many you have and what you do with them, but widgets do carry some overhead that adds up. I think this is the single really important factor. For instance, I think all the blocks are widgets in tito's Flat Jewels game (
https://play.google.com/store/apps/details?id=com.meltingrocks.flatjewels), without being a performance problem, but if there were many more blocks maybe it would become slow.
On this basis, kovak has been creating a nice game engine for kivy, which you can see at
https://github.com/Kovak/KivEnt . It's a work in progress and I don't know the current status, but it's definitely usable...I don't know if kovak reads this list, but you can find him on irc sometimes. The engine *doesn't* use widgets for everything, so that it can scale well beyond where widgets become a problem, plus it builds in lots of other useful stuff like physics and particle effects. If you're interested, I suggest contacting kovak on the kivy irc channel (#kivy on freenode).
If you're new to kivy, in the short term I'd suggest just trying something and seeing what happens. Widgets might work fine for you, and if they don't at least you get an appreciation for why, when and how - and also for what you lose when switching to managing everything more manually.