I think what Anthony might be suggesting, which I would agree with, is that there should be layers to the implementation: an external API for laying out and providing data to a plot, the actual generation of plots in terms of vector elements, and then the conversion of those vector elements into rasterized results. The top two layers would be defined in platform-independent Swift code. It's what to do with the bottom layer that's the question.
On iOS and Mac, you have Core Graphics (and Core Animation for layering). I was one of the founding contributors to the Core Plot project years ago (an Objective-C plotting library for Mac and iOS):
and because we were solely targeting platforms where you had Core Graphics for rasterization, we didn't worry about anything else. We used Core Animation layers for the composited elements in the plot, and Core Graphics to draw lines, arcs, text, etc.
From working on that, I can see how you could define the interface to the lower rasterization layer in terms of a protocol that the upper two layers could interact with. The protocol would define things like telling the rasterizer to draw a line from one coordinate to another, fill an area with a color, draw text at a location, etc. With a generic interface like that, you could have the possibility of using Core Graphics on Mac and iOS for easy UI integration, maybe OpenGL for accelerated rendering on some platforms, or another rasterization implementation elsewhere.
Ideally, it would be nice to have the minimal number of external dependencies be pulled in to use this graphing library, even to the point where we could look at rolling our own Swift rasterization engine as a fallback for platforms that don't have a native one. Even there, I imagine there will be platform-specific elements, like image outputs, display interaction, etc.
I'd be glad to help with this, especially on the developer-facing external API. We made a few mistakes on the early interface for Core Plot that made it a lot harder to use (eventually corrected), and I think some of those lessons could be applied here, especially with what Swift provides in language capabilities.