noVNC jquery dialog

[Scott Haynes]

I've been using noVNC for a few days now, and I've got a fairly impressive demo in 2d working already.  I'm having some issues using the 'Display' API to get the VNC application displayed properly into a dialog window( this dialog will be re-sizable so I need to set up the dialogs resize event to modify the scale of the canvas as well ).  I'm able to get the size and position that I want( by calling resize, but this is not a full view of the canvas though ), but I need to scale the display and not have the canvas move with in the dialog.  I'm more of a 3D applications engineer, so html isn't my strong point.  When I scale the Display, I noticed that a webkit-transformation => translation is begin applied.  This moves the canvas outside of the dialog.  I found a viewportChange function, but that didn't seem like what I needed to call either.  

Another issue is that I plan to map the VNC canvas to a texture in 3D.  I won't have as many issues there with the scale as I will be mapping the canvas to the entire surface, The texture dimensions will need to be a power of 2.  Is there an event that says a change to the canvas has occurred?

Any help or suggestions would be great appreciated,

Scott  

[Joel Martin]

On Wed, Aug 15, 2012 at 3:01 PM, Scott Haynes <scottn...@gmail.com> wrote:

I've been using noVNC for a few days now, and I've got a fairly impressive demo in 2d working already.  I'm having some issues using the 'Display' API to get the VNC application displayed properly into a dialog window( this dialog will be re-sizable so I need to set up the dialogs resize event to modify the scale of the canvas as well ).  I'm able to get the size and position that I want( by calling resize, but this is not a full view of the canvas though ), but I need to scale the display and not have the canvas move with in the dialog.  I'm more of a 3D applications engineer, so html isn't my strong point.  When I scale the Display, I noticed that a webkit-transformation => translation is begin applied.  This moves the canvas outside of the dialog.  I found a viewportChange function, but that didn't seem like what I needed to call either.  

Scaling is a feature that has only been partially implemented. This issue tracking this:

https://github.com/kanaka/noVNC/issues/12

There is also a recent pull request that I need to review soon to see if it fully addresses the problem:

https://github.com/kanaka/noVNC/pull/117

One of the challenges here is that scaling needs to work correct with viewport size/clipping. Having a smaller viewport than the remote desktop works well and allows noVNC to be used on phone/tablets. The viewport clipping works quite well, but the logic around this and the half-working scaling code is pretty messy. :-(

The display.rescale(factor) method is what is used to change the scaling value. This is certainly buggy although I think the mouse event scaling works correctly.

The display.viewportChange(deltaX, deltaY, width, height) is used to setup the viewport clip area. This works great by itself although I'm certain it doesn't work together with scaling.

 

Another issue is that I plan to map the VNC canvas to a texture in 3D.  I won't have as many issues there with the scale as I will be mapping the canvas to the entire surface, The texture dimensions will need to be a power of 2.  Is there an event that says a change to the canvas has occurred?

There are two callbacks related to that: onFBUReceive and onFBUComplete. The first is called right before an update is made and the second right after. The second argument to both is an object with the frameBufferUpdate metadata: x, y, width, height, encoding, and encodingName.

Sounds like really interesting work and hope you are able to figure it out. Will it be available somewhere to easily check out?

Regards,

Joel Martin (kanaka)

[Scott Haynes]

Thanks for the response.  The core project, Virtual World Framework, is currently open source and can be found on github(  http://www.virtualworldframework.com/web/about.html  ).  The experiment that I'm working on however is being funded by internal research and development funds and is not available on github yet.  

A coworker and I checked out the scaling on Thursday of last week, and she was pretty convinced that she could get the scaling the dialog to work.  I can understand the situation and complexities of getting clipping and scaling working.  Good luck with that, I'll check on the issue you sent a link to.

I'm not too worried about the 3D texture implementation.  I've already got drawing on a 3D surface working via a canvas, so VNC shouldn't be much different.  In 3D I'll do a pick for mouse events, and I'll get back a surface and the texture coordinates for the VNC surface, and then I'll have to call the mouse events inside of VNC.  I'm imagining those handle with events from the canvas normally.  Do you see any issues with me calling those same events?  I'm assuming that I'll need to duplicate the event parameter that is passed into the canvas event handlers.

Thanks for all of the efforts to get this project up and running,

Scott 

[Joel Martin]

On Mon, Aug 20, 2012 at 11:16 AM, Scott Haynes <scottn...@gmail.com> wrote:

In 3D I'll do a pick for mouse events, and I'll get back a surface and the texture coordinates for the VNC surface, and then I'll have to call the mouse events inside of VNC.  I'm imagining those handle with events from the canvas normally.  Do you see any issues with me calling those same events?  I'm assuming that I'll need to duplicate the event parameter that is passed into the canvas event handlers.

Yeah that should work. The functions for sending direct mouse or keyboard events aren't exposed in the API (although there is no reason they shouldn't be really). In the RFB object you will need to add mouseButton and mouseMove to the public API functions (i.e. "that"). Those don't need constructed events objects, they just need x, y, down, bmask and x, y respectively. You could also do this at a lower level (in the Mouse object in input.js) but it's lower level and there is a bit more involved there.

Good luck!

Joel Martin (kanaka)