Mousover detection for any collage Form (proof of concept)

[Max Goldstein]

Have you ever wanted to know which Form in a collage the mouse was on top of? Well, now you can. (Sort of.)

The basic idea is to render to a backbuffer which is never shown to the user. Everything is rendered a single unique color with no transparency. Since only the computer sees this buffer, those colors can all be shades of black. Then we read the color value at the pixel of interest, and that tells us the form responsible for that pixel. This approach is completely non-geometric, so you can use it for crazy splines and stuff. It reuses the existing render pipeline, only with copies of the shape that are of the right color. 

Unfortunately this requires access to some opaque types, so I had to put the code inside the Collage class itself. The commit sits directly on 2.1.0 so you can run these commands in your project root to get the forked source:

curl https://raw.githubusercontent.com/mgold/core/backbuffer/src/Graphics/Collage.elm -o elm-stuff/packages/elm-lang/core/2.1.0/src/Graphics/Collage.elm

curl https://raw.githubusercontent.com/mgold/core/backbuffer/src/Native/Graphics/Collage.js -o elm-stuff/packages/elm-lang/core/2.1.0/src/Native/Graphics/Collage.js 

You can then run the example at the end of this post. Some caveats: I haven't tested it with every type of Form yet and it's possible it has some bugs for lines, text, elements, or groups. (Even in theory, all members of a group will be indistinguishable). Also, on the border between two shapes, it may infer another shape, due to interpolation.

This is a proof of concept and should not be relied on. But if there's interest, I can work on making this more stable. It would have to be merged into core eventually; it can't be a third party lib.

Incidentally, you can also use this trick for arbitrary collision detection, i.e. you get everything under the mouse, not just the topmost one. You assign each form a bit in the color space, and set it to 1 wherever the form renders. This means you can only use 24 forms (1 byte for r,g,b; I don't think you can use alpha) and you can't reuse the render pipeline without looking at each pixel you draw and doing some bit manipulation. But hey, in theory...

Example:

import Graphics.Element as E

import Graphics.Collage as C

import Color exposing (blue, red, green)

import Mouse

forms = [ C.rect 200 100 |> C.filled blue

        , C.oval 40 250 |> C.filled red

        , C.polygon [(20, 20), (20, 70), (80, 40), (30, 50)] |> C.filled green |> C.move (20, -50)

        ]

collage = C.collage 500 500 forms

bbuf = C.backbuffer 500 500 forms

hovering = Signal.map (\p -> C.getAt p bbuf |> E.show) Mouse.position

render hoverState = collage `E.above` hoverState

main = Signal.map render hovering

[Pete Vilter]

Wow, that's quite a hack! I've heard of this being used for 3d as well.

Shameless plug: I needed to be able to do this (and a couple other things), so I ended up writing the entire elm-diagrams library (here's the over-detection function, called `pick`). It renders to Graphics.Collage. It doesn't have support for crazy splines though, because in my approach that would require some fancy geometric algorithms. This is probably also really fast.

[Pete Vilter]

(I meant the backbuffer approach is probably really fast). I wonder what browsers' SVG implementations do for over detection on splines — fancy geometric algorithms or back buffer?

[Jeff Smits]

The back buffer approach sounds like the kind of hack you may need for speed, but has a scale limitation and then there's the anti-alias gotcha. I like it, but I also like the 'picking' in diagrams (that Pete plugged) as a "pure" solution. Crazy geometric functions? No problem, let's just figure it out, put it in the library and be done :)

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Smits]

Hmm, that may come across more negatively than I meant. I like this idea and the effort! But I'm not sure it should go into the core library, given the limitations. In general, I think the Graphics.* modules could use a do-over. Maybe there will be an opportunity then, to expose more to allow this as a separate library? Do you think that's possible Max?

[Max Goldstein]

Jeff: No worries, I appreciate the feedback. It would be interesting to see a geometric solution. But point location is tricky, and it gets trickier if you ever want to add true curves like splines. I see Pete has not implemented point detection, so let's try all viable solutions? Although I think Diagram has me beat for text - the bounding box is better behavior than the actual letters.

This is a proof of concept and I think issues like aliasing can be worked out. If you sacrifice some of the 2^24 possibilities (1024 ought to be enough, right?), you can better the odds of detecting a collision. A sophistical approach might use something like cubehelix: anti-aliasing will use linear interpolation, but cubehelix is nonlinear, so you can detect bad encodings. (Although each value is triple-encoded, so you have a ceiling of 255 and more reasonably 32 or 64.) Or use some other kind of error-correcting code - any EEs in the house?

Or, geometrically, give every filled shape a white stroke. There are ways.

I agree with the sentiment that Element and Collage are starting to show their age, but I also think that there are other things we can focus on. But I've been thinking about this for a while and I thought it was cool that I was able to do it.