Personally, I wouldn't bother including the jQuery library at all.
The dom flavour of jsPlumb doesn't have a depedency on jQuery UI and, if you're new to Angular, having jQuery to fall back on will make it very tricky to grasp the Angular way of doing things.
Our approach is to add connection objects to an array on the $scope. Each connection object contains info on the connection, such as the source id, target id, type of connection (straight or flowchart) etc.
In the template, we use ng-repeat to render a list of <connection> elements which we pass in the connection data to. (<connection data="model.data"></connection>).
The connection directive uses this data to initialize jsPlumb and create the connection. It also handles events (click, dblclick etc).
There is really no reason to use jQuery.