You don't see a lot moving on the screen do you :) ?
I think if you need raw speed, then you need to get as minimal as possible, and svg.js I would imagine faster at setting stuff, and no library even faster. If you really want to get speed and talking about gaming, you probably want canvas.
However, as mentioned, nothing moves on the screen. I don't know jsperf, so I'm quite likely talking cobblers, but I'm guessing this isn't timing svg draws, just setting attributes in reality so relatively insignificant. The thing that takes the time with SVG is the actual drawing, transformations, painting etc. Browsers don't draw SVG on a frame by frame basis, with a nice wait, they try and shortcut, and there is no way to wait until one part is finished.
To try and highlight what I mean, if you look at a loop on a random draw
http://jsfiddle.net/vy2vycn4/ You won't see it move apart from once (feel free to increase loop to 1000).
The thing that really takes the time in a different way, is coding. My main advice would be not to get bogged down in testing stuff before you've written anything, you will never finish it. Get writing with sensible logic and functions, and you will find that if you need you can drop in svg.js, snap, canvas, whatever.