An extra idea, I haven't tried it though.
Maybe the slowdown is a backlog of sprite drag events that are waiting to fire because
you passed over them with your drag of another sprite, even though they will
decide to take no action because of your test in the drag block.
This might happen if you have lots of closely spaced sprites (china shop?)
(conjecture).
If all you have are sprite touchDown and touchUp events, you might
be able to capture the dragged sprite component at touchDown time, and
drag it under control of a Canvas.Dragged event.
(Eliminate all those Sprite.Dragged event blocks)
That might reduce the number of drag events by an order of magnitude.
(Again, I haven't tried this.)
ABG