OK I see the problem.
Comparing xvel or yvel to 0 in this case will not work.
You need to have a minimum value other than zero to compare it to to indicate the the user did in fact intend to swipe in that direction.
That would be called a threshold value.
If the swipe exceeds the threshold value then the user was swiping in that direction.
If not then ignore it.
This threshold value looks like it could be around +1 or -1 for flinging in any direction.
if xvel > 1 then flinging right
if xvel < -1 then flinging left
if vvel > 1 then flinging down
if yvel < -1 then flinging up
You may need to adjust the threshold value for the anticipated swipe speed.
---
Scott