UI / Overlay Graphics cross-platform library

87 views
Skip to first unread message

PierreJasmin_REVisionEffects

unread,
Mar 21, 2020, 1:12:43 PM3/21/20
to ofx-discussion
Prepping agenda for OFX meeting sometime week of april 20-24 (TBA)
One of the item is GPU but actually there is a subset we never discuss that is also an issue which is related to Interact Suite and drawing in overlays...

Question is:  With OpenGL, DirectX, Metal, Vulkan perhaps being native UI for that for a particular host I am wondering if there is a simple open source library for drawing that handles the particular of drawing in overlays and supports all the graphics libraries we might need for that. For those who are familiar with Adobe APIs, that would be somehow similar to the adobesdk/DrawbotSuite.h abstraction for OFX plugin developers...

Below might be hard to understand for someone not familiar with Adobe API (hint: their UI canvas is -32768 to 32768 I think if the 65535 multiplication is not clear below), but here is a simple example and trying to see if there is a way out of having need for an OFX abstraction just for that... (I think we are fine for the non draw events).


/////

void RVAE_DrawLineBot (float x0,float y0, float x1, float y1,
            float r, float g, float b, // PF_Pixel            some_color
            PF_InData            *in_data,
            PF_EventExtra        *event_extra) {

    PF_Err     err = PF_Err_NONE;
    RVAE_SuiteHandler    suites(in_data->pica_basicP);   

    DRAWBOT_DrawRef     drawbot_ref = NULL;
    suites.CustomUISuite()->PF_GetDrawingReference(event_extra->contextH, &drawbot_ref);

    DRAWBOT_SurfaceRef     surface_ref;
    suites.DrawbotSuite()->GetSurface(drawbot_ref, &surface_ref);   

    DRAWBOT_SupplierRef     supplier_ref = NULL;
    suites.DrawbotSuite()->GetSupplier(drawbot_ref, &supplier_ref);

//make new pen
 
    DRAWBOT_PenRef    pen_ref = NULL;
 
    DRAWBOT_ColorRGBA    pixel_color;
    pixel_color.red= r;
    pixel_color.green=g;
    pixel_color.blue=b;
    pixel_color.alpha = 1.0;
 
    suites.SupplierSuite()->NewPen(supplier_ref, &pixel_color, 1.0, &pen_ref);

// make new path
 
    DRAWBOT_PathP    pathP(suites.SupplierSuite(), supplier_ref);
    DRAWBOT_PathRef    path_ref = NULL;
    suites.SupplierSuite()->NewPath(supplier_ref, &path_ref);
 
    PF_FixedPoint point;
    point.x = x0*65536.0;point.y= y0*65536.0;
    if (PF_Window_COMP        ==    (*event_extra->contextH)->w_type) {
        event_extra->cbs.layer_to_comp(event_extra->cbs.refcon,event_extra->contextH,in_data->current_time,in_data->time_scale,&point);
    }
    event_extra->cbs.source_to_frame(event_extra->cbs.refcon,event_extra->contextH,&point);

     PF_Point pnt;
    pnt.h = (float)(point.x/65536.0F + 0.0F);
    pnt.v = (float)(point.y/65536.0F +  0.0F);
 
    suites.PathSuite()->MoveTo(path_ref,pnt.h,pnt.v);
 
    point.x = x1*65536.0;point.y= y1*65536.0;
    if (PF_Window_COMP        ==    (*event_extra->contextH)->w_type) {
        event_extra->cbs.layer_to_comp(event_extra->cbs.refcon,event_extra->contextH,in_data->current_time,in_data->time_scale,&point);
    }
    event_extra->cbs.source_to_frame(event_extra->cbs.refcon,event_extra->contextH,&point);
    pnt.h = (float)(point.x/65536.0F + 0.0F);
    pnt.v = (float)(point.y/65536.0F +  0.0F);
 
    suites.PathSuite()->LineTo(path_ref,pnt.h,pnt.v);
 

// draw
    PF_Boolean draw_shadowB = 0;
     suites.CustomUIOverlayThemeSuite()->PF_StrokePath(drawbot_ref, pathP.Get(), 0);
 

    if (path_ref) {
        suites.SupplierSuite()->ReleaseObject((DRAWBOT_ObjectRef)path_ref);
    }
 
    if (pen_ref) {
        suites.SupplierSuite()->ReleaseObject((DRAWBOT_ObjectRef)pen_ref);
    }
 
 
//    return err;

}

Paul Miller

unread,
Mar 21, 2020, 6:00:08 PM3/21/20
to ofx-dis...@googlegroups.com

That came up recently here too.

Can we start with a survey of what plugin developers actually use in their overlays?

I'll start - in DFX/DFT it's just line strips/loops and some rectangles, in different colors.

--
You received this message because you are subscribed to the Google Groups "ofx-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ofx-discussion/9297d266-0e79-4ca4-b615-8d85e16816f5%40googlegroups.com.

PierreJasmin_REVisionEffects

unread,
Mar 21, 2020, 6:17:08 PM3/21/20
to ofx-discussion

Similar here (RE:Vision): rectangles, lines, circles, curve paths opened or closed e.g. ellipses, and of course with choice of color

useful too if linewidth can have a fraction (e.g. line width 1.5) and semi-transparency if an option (not a deal killer) would be useful sometimes. I can draw dotted lines myself and fancy stuff like that if need be :)

Would be useful to have a way to write text too - no need for fancy fonts, just start location, font size and /n strings would work for me
It's then more like overlay tidbits in viewer to avoid popping up message box or adding more UI controls, happy to match how the host does it.

Pierre

To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussion+unsubscribe@googlegroups.com.

PierreJasmin_REVisionEffects

unread,
Mar 23, 2020, 2:33:50 PM3/23/20
to ofx-discussion
Actually let me translate what Pierre and Paul are discussing using less talk and more code (have to assume some younger folks might start life not using OpenGL).

1. Loop example: 

const float DEG2RAD = 3.14159/180;
 
void drawCircle(float radius)
{
   glBegin(GL_LINE_LOOP);
 
   for (int i=0; i < 360; i++)
   {
      float degInRad = i*DEG2RAD;
      glVertex2f(cos(degInRad)*radius,sin(degInRad)*radius);
   }
 
   glEnd();
}

2. From glew.h

#define GL_LINES 0x0001
#define GL_LINE_LOOP 0x0002
#define GL_POINT_BIT 0x00000002
#define GL_CLIENT_VERTEX_ARRAY_BIT 0x00000002
#define GL_LINE_STRIP 0x0003
#define GL_LINE_BIT 0x00000004

From Gl.h,  glLineStipple  -- i.e. we can live without that, point here is lowest common denomimator between OpenGL, DirectX, Metal, Vulkan etc

For simplification we might also consider a 2 level suite, mandatory and extra options
We are trying to avoid like some suites having large amount of functions where it's not clear to start with what is supported by a given host.
So to at least have a way to draw lines would be mandatory and for example support for passing a string to an on-screen tidbit would be optional. 



Paul Miller

unread,
Apr 16, 2020, 10:20:48 AM4/16/20
to ofx-dis...@googlegroups.com

I'm working on a design proposal for this and could use some more input from host and plugin implementers.

I think there are two ways to go here:

drawRect()
fillRect()
drawLine()
drawLines()
drawPolygon()

etc...

Or something more directly mappable to OpenGL/Vulkan style:

draw(point_array, point_count, type)

where type is an enum: point, lines, rectangles, line strip, line loop, polygon, etc. This would more closely emulate what GL does with glBegin/end or glDrawBuffer while keeping the API simple. I'm leaning toward the latter.

Notes:

  • I'm not intending to include a matrix stack- plugins will need to do their own transforms a la modern GPU APIs
  • no initial texture/image support
  • text would be optional

Anyone have any comments?

--
You received this message because you are subscribed to the Google Groups "ofx-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ofx-discussion/89f3aaff-2f2a-47da-b372-e4e2e8b0f2d7%40googlegroups.com.

Gary Oberbrunner

unread,
Apr 16, 2020, 12:55:20 PM4/16/20
to ofx-dis...@googlegroups.com
I like your second idea. What about color, width, dash style etc.? Those would be separate calls I presume, always set to defaults by the host before it calls the plugin? (So just to be clear, no single draw() call with multiple colors.)

Paul Miller

unread,
Apr 16, 2020, 1:00:34 PM4/16/20
to ofx-dis...@googlegroups.com
On 4/16/20 11:55 AM, Gary Oberbrunner wrote:
I like your second idea. What about color, width, dash style etc.? Those would be separate calls I presume, always set to defaults by the host before it calls the plugin? (So just to be clear, no single draw() call with multiple colors.)

Yes, additional calls for color, dash pattern (maybe from a predefined set?), line width, etc.

Although I personally dislike capability flags, but maybe at least a query to see if text is supported. I don't want to force hosts to implement text-on-OGL (though maybe not too bad with OGLFT?), but text on top of Vulkan or Metal? No clue.

What about a query to find out what the underlying implementation is? It might be useful to know it's still OpenGL, in case you want to do some extra OpenGLy things. But could add complexity for getting at the details of different display contexts.


Adams, Dennis

unread,
Apr 16, 2020, 2:51:34 PM4/16/20
to ofx-dis...@googlegroups.com

I also like the “pass the data structure” (like Vulkan) over the “make many calls” approach.

 

If those other items (color, width, etc.) are attributes of the vertices (or segments between them), it seems like they should be passed with the vertices. The call should indicate which items are being passed (make it future extensible). That way the API can remain stateless.

 

///d@

PierreJasmin_REVisionEffects

unread,
Apr 16, 2020, 3:47:02 PM4/16/20
to ofx-discussion
I also like as stateless as possible even if it becomes a call with a lot of arguments

By text I only would like a text string with /n maybe with a screen coord - not to do fancy custom on-screen UI, that would be another issue.
Pix below as semi-example, most hosts already have some sort of Overlay UI for text internally, would be useful to users particularly with multiple items what is selected for example with a label - e.g. left-eye instead of screen coord in pix attached. This way it's drawn like any other items in host UI. This is also useful if there is parameter UI associated then that helps user can match overlay to UI parameters easily... Like all good things if an host does not support that, they return unsupported as state back...

Overlay.jpg

Pix

To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussion+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ofx-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussion+unsubscribe@googlegroups.com.


 

--

Gary

--
You received this message because you are subscribed to the Google Groups "ofx-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussion+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "ofx-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussion+unsubscribe@googlegroups.com.

PierreJasmin_REVisionEffects

unread,
May 19, 2020, 7:27:07 PM5/19/20
to ofx-discussion

Paul two short orthogonal notes to your first draft:

1. There is a 3 options OFX prop that describe where 0,0 in UI is -  as opposed to image pixels that are 0,0 lower-left except if rowbytes is negative -
For example in Hitfilm UI 0,0 is center and in Silhouette UI is top-left is 0,0??? (or that is just pixel data)

2.  We are disccussing only overlay render, but I noticed some apps handle differently how often Instance Changed is called for the interaction suite part, this is related to moving the interactive overlay, updating the parameter value in UI, triggering a render frame... There is sort of 3 things we want here: no render until I release I am just placing overlays, background image does not change  2)  plugin caches an intermediate result to speed up interaction - so some ephemeral buffer we release when interaction is done (think of a magnifying window on a part of image under cursor  for example and only that showing interactive change)  and 3) regular update of render while interacting (while mouse down)...

Pierre

Peter Loveday

unread,
Mar 16, 2021, 9:11:58 AM3/16/21
to ofx-dis...@googlegroups.com
Hey guys…


Did we ever get anywhere with this proposal?


Is there a draft kicking around at all?


Love, Light and Peace,
- Peter Loveday
Blackmagic Design



--
You received this message because you are subscribed to the Google Groups "ofx-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ofx-discussio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ofx-discussion/dfb8e4e3-a016-4764-9f71-2f10672485cf%40googlegroups.com.

Paul Miller

unread,
Mar 16, 2021, 9:54:25 AM3/16/21
to ofx-dis...@googlegroups.com

I somehow missed Pierre's notes from last May. Nothing new on my proposal.

I believe OFX by default sets the view transform for overlay interacts so that coordinates are mapped to the image plane, with 0,0 in the bottom-left corner, and the proposed overlay suite follows that norm. Maybe it would be useful to have some simpler mechanism of converting coordinates into Window coordinates?

As for interact edit relationship with the parameters, I don't believe OFX currently defines the exact behavior, and is left up to the host. If changing parameters during pen move, Silhouette (for example) bundles them all under one undo event, and triggers a render with each (potential set) of changes. If the plugin wants to wait until pen-up to set the new values, that's always an option. But this doesn't have anything to do with the DrawSuite.

Peter Loveday

unread,
Mar 16, 2021, 10:06:06 AM3/16/21
to ofx-dis...@googlegroups.com
I think the mapping makes total sense as is.  No reason to change that?  Hosts can presumably convert as needed (as we would).  I think forcing the plugin to be aware of the hosts’ mapping conventions is a bit too onerous?  *Much* easier for the host to handle that consistently than expect every plugin to get it right.

And also agree that there’s no particular need to make the edit relationship explicit… we do the same, being an undo event on pen down, close it on pen up (if any changes).  Again, leave it host side… the host knows best what users expect from its behaviour.

In any case, specifics aside, I guess I’m raising this again because this is becoming increasingly important as we transition to alternate APIs… Supporting fixed-function GL on Vulkan/Metal is painful.

Love, Light and Peace,
- Peter Loveday
Blackmagic Design

Paul Miller

unread,
Mar 16, 2021, 11:04:52 AM3/16/21
to ofx-dis...@googlegroups.com
On 3/16/21 9:05 AM, 'Peter Loveday' via ofx-discussion wrote:
I think the mapping makes total sense as is.  No reason to change that?  Hosts can presumably convert as needed (as we would).  I think forcing the plugin to be aware of the hosts’ mapping conventions is a bit too onerous?  *Much* easier for the host to handle that consistently than expect every plugin to get it right.

I agree completely.

But with the current OpenGL usage, the plugin is free to change the transform into a different space. The current proposal doesn't provide for that. I can think of a few cases (as a plugin writer) where I might want to work in window coordinates. Should we offer that as an option, or a rudimentary matrix stack?

And also agree that there’s no particular need to make the edit relationship explicit… we do the same, being an undo event on pen down, close it on pen up (if any changes).  Again, leave it host side… the host knows best what users expect from its behaviour.

In any case, specifics aside, I guess I’m raising this again because this is becoming increasingly important as we transition to alternate APIs… Supporting fixed-function GL on Vulkan/Metal is painful.
I feel like we're going to lose OpenGL "any day now" on Macs, and deciding this now should save some headaches later.

Peter Loveday

unread,
Mar 16, 2021, 11:54:00 AM3/16/21
to ofx-dis...@googlegroups.com

On 17 Mar 2021, at 01:34, Paul Miller <pa...@fxtech.com> wrote:

On 3/16/21 9:05 AM, 'Peter Loveday' via ofx-discussion wrote:
I think the mapping makes total sense as is.  No reason to change that?  Hosts can presumably convert as needed (as we would).  I think forcing the plugin to be aware of the hosts’ mapping conventions is a bit too onerous?  *Much* easier for the host to handle that consistently than expect every plugin to get it right.

I agree completely.

But with the current OpenGL usage, the plugin is free to change the transform into a different space. The current proposal doesn't provide for that. I can think of a few cases (as a plugin writer) where I might want to work in window coordinates. Should we offer that as an option, or a rudimentary matrix stack?

Perhaps we could replace ’matrix stack’ with ‘a matrix’.  I dunno, this isn't hard for us (or any) host to do.  I’d say if it helps plugins be consistent, then let’s do so.

OTOH it’s a bit tricky because OFX treats overlays as a per-instance thing, rather than a parameter level thing… which arguably could change transform requirements.

I’m open to plugin vendors’ needs here.

The other thing to consider is efficiency.  With a modern API, using vertex buffers etc on-card is the go.  Not fixed function arguments.  This was alluded to by the suggestion a call could take many attributes for a draw(). BUT… for efficiency we could consider a way to avoid re-uploading that vertex/attrib buffer if it hadn’t changed.  This is an optimisation, bot a requirement, but…


And also agree that there’s no particular need to make the edit relationship explicit… we do the same, being an undo event on pen down, close it on pen up (if any changes).  Again, leave it host side… the host knows best what users expect from its behaviour.

In any case, specifics aside, I guess I’m raising this again because this is becoming increasingly important as we transition to alternate APIs… Supporting fixed-function GL on Vulkan/Metal is painful.
I feel like we're going to lose OpenGL "any day now" on Macs, and deciding this now should save some headaches later.

Well Apple will do what they do do.

But we will move to new APIs as needed, and the lack of abstraction on this is not consistent with OFX in general, I think.

The last thing we want is plugins having to do three+ versions of overlays.

PierreJasmin_REVisionEffects

unread,
Mar 16, 2021, 4:20:16 PM3/16/21
to ofx-discussion
Paul, can you just push it as a candidate if you have not done it?  So it can go in official review clock, an host at least has to implement it so we can test it works.

Notes:

I know aside Vegas another host going on Windows to DirectX instead of VK 

The host UI orientation is orthogonal, we just need this here when grabbing something and updating parameters from that interaction so our values have same orientation as host.  Out-of-scope likely here.

The pen down etc interaction callbacks are also orthogonal, was just being complete and noting we have 3 kinds of hosts, to summarize:  one that returns continuously instance changed (which is fine but perhaps we need a sort of sub-context as we have to trap this);  one will trigger a render only on pen up which is not very interactive and one that will degenerate resolution to speed up interaction but somehow some render refresh happens. This is out of scope in this discussion.

Pierre

PierreJasmin_REVisionEffects

unread,
Mar 22, 2021, 3:26:33 PM3/22/21
to ofx-discussion


I have a good use case generic OpenFX example for that, that would be something real (something an host can use as a test reference and some users might actually use).

Check this link, (select camera, then aspect ratio, then scaling (safe margin), and aspect ratio...)

https://www.arri.com/en/learn-help/learn-help-camera-system/tools/frame-line-lens-illumination-tool

See the UI elements used (nothing really special for us but a real example)

I happen to be on an ASC committee for framing - nothing too exciting for FX :) - I am there because we have been a consultant for a cine-camera space company....
But one application is in-camera framelines (and then maintaining that info across production - i.e. the director/dp framing intent versus the recorded in pixel images)
-- e.g. shoot with 10% margin as this will require stabilization.... or there are two output ratios for this show...
There will be a small format reference later this year, currently named FDL (maybe this could also later become meta-data in OpenFX for hosts that can't easily handle in their architecture different image width and height - in and out...?)

The Arri current implementation now uses xml to load the info ...  I think Red uses a different UI units for this (% from center).
Because BMD and Sony on this list also have cameras, more context :
(Peter, PeterC has joined some meetings, unclear about any eventual support of this in BMD Camera and Dennis, michaelF and SimonM from your company are on that list - Phil, PeterP is on that group for FilmLight)

As per previous discussion, the additional item was display text/numbers - fonts, if you press FrameLeader (which would sort of be a Generator for us) -  note two extra UI items:  some sort of text display model and some image icon...these could be optional if too hard for an host - not for me to decide - I would be happy with small and big font, and string start location.

Pierre

Reply all
Reply to author
Forward
0 new messages