additional graphics synchronization?

[Dave Dyer]

In android builds

Is some additional synchronization needed to be sure readers and writers of
offscreen images see the correct  things?

I have a writer process that prepares a full screen bitmap in a game process,
which is painted into the real display by paint() in the EDT process.  The two
processes are synchronized, so they ought to be non-interfering.

The behavior I'm seeing is partial paints which would be consistent with the
writer process not really being finished when the java method exits, or with
the reader process not being finished when the EDT function finishes. This
could plausible arise if the GPU was still busy transferring bits after the call
that initialted it finished.

[Shai Almog]

Double buffering is really bad in modern mobile OS's. Our original implementation relied heavily on this practice and we got awful performance as double buffering can't be truly optimized by the GPU when drawing offscreen and you end up with something that is seriously sub par.

On iOS drawing to a mutable image is only supported on the EDT because opening an offscreen buffer is a global operation in the C level API's (this is old stuff they got from Next) so its impossible to mimic this on a separate thread without an even heavier performance penalty. To make things worse synchronization is generally slow on iOS since there is no JIT hence no lock elimination...

The behavior you are seeing specifically relates to the fact that the EDT itself is an abstraction. There is a native EDT which we completely hide from the Codename One API as its behavior is radically different in all mobile OS's. We do the synchronization between that and the EDT so you don't have to do it.

[Dave Dyer]

The fact that the reader and writer are in different threads is probably not relevant, given
that they are synchronized to run serially.  The problem is that there are two graphics
instances, one of which is writing to the mutable image, while the other is reading from
it.  Under the hood, unless you are really clever, there's no way to tell that incomplete
operations on one will affect the other.     I presume there must be some underlying
primitive in the real edt thread to force a wait-until-complete for the graphic operations
associated with a Graphics instance.   I haven't seen any sign of it, or of any operations
I could perform to force a stall in the graphics pipeline.    For example, if there were
a primitive to read a pixel value from the screen, that would have to be implemented
by waiting for all pending operations to complete first.

[Shai Almog]

No the issue is that there is an abstraction to an underlying graphics platform that varies a lot and we only enable a relatively simple high level abstraction.
The C level code in iOS is REALLY different from what we have on Android/JavaSE. Worse in iOS we have 2 completely separate graphics pipelines...

[Dave Dyer]

I can't tell if you're claiming that there is no problem or that the problem is insoluble.

I find that forcing the both sides of the display into the edt thread makes the problem
better, but doesn't completely solve it.   Is there a reliable way to tell that graphics
activity as a result of a paint operation has completed?  I think there needs to be.

[Shai Almog]

Every problem is solvable. Synchronization is not solvable in a way that is performant in GUI programming.

If you are modifying an image don't draw on it. A simple trick would be to have two images and only when you finished drawing on one of them swap to the other. To keep the background identical (up to date) draw the previous image onto the one you are flipping.

[Dave Dyer]

That is already the case, except that after my visible process finishes with the image and starts reusing it, some invisible process is still reading from it.

If I actually abandon the bitmap and allocate a new one, I suppose it is mandatory that the OS is smart enough to wait until it is safe to GC it,
but surely churning images this way would be stressful for the OS.

Explicitly double buffering would probably work, but only probably, aince there is no guarantee how far the graphics engine is allowed to lag.

On Saturday, January 23, 2016 at 8:41:24 PM UTC-8, Shai Almog wrote:

Every problem is solvable. Synchronization is not solvable in a way that is performant in GUI programming.

If you are modifying an image don't draw on it. A simple trick would be to have two images and only when you finished drawing on one of them swap to the other. To keep the background identical (up to datecase) draw the previous imtheage onto the one you are flipping.

[Shai Almog]

Not abandon, just switch between two explicit images.
Notice that you should still do all the changes on the EDT as that is expected.

[Dave Dyer]

Using explicit double buffering seems to be good enough, but I still wish for an explicit synchronization
protocol. For example, if the size of the image is small enough, merely double buffering would not be
enough.

[Shai Almog]

I don't understand that use case.

[Dave Dyer]

there's no guarantee that for some combination of image size and other graphic activity, the writer won't fill two buffers and start on a third while the emptier is still working on the first, which is the same as the third.

[Shai Almog]

There is a guarantee on the EDT that all operations done on the EDT will render in sequence both to mutable images and buffers.

[Dave Dyer]

At face value, wouldn't that mean that double buffering would be superfluous?

[Shai Almog]

It isn't superfluous. Its HARMFUL. It effectively disables GPU accelerated rendering.

Double buffering was a solid approach 5-10 years ago but on modern mobile hardware its considered harmful.

[Dave Dyer]

Something isn't adding up.  The upshot of the previous portions of this thread
was that double buffering was necessary to deglitch rendering. 

My working assumption based on the observed behavior
was that rendering through a single graphics object was
synchronized, but if more than one was involved, some
additional synchronization was needed.

My render loop (all int he EDT thread) is to get a graphics for an image,
render into it, then draw the rendered image into the screen using the
graphics supplied by paint.   This loop repeats, but it appeared that
occasionally the rewriting of the image had started before the image
got painted onto the screen.

[Dave Dyer]

I've set up my app so I can turn double buffing on or off while the UI is running.
I can arrange for you to try it for yourself, or I can send you a screen capture
movie showing the effect.

[Shai Almog]

We need small test cases in the issue tracker not large apps. If you have a test case that produces flickering check whether it happens on the device/simulator etc.
Also in the simulator make sure you are working with scrolling mode checked since you might see artifacts in scaling mode.

[Dave Dyer]

The simulator works correctly with or without double buffering.

Both my real android tablet and Duos on windows exhibit the flicker problem
without double buffering.

I'm willing to put some effort into helping you identify the problem, if you're willing
to engage at a similar level.   First step would be for you to try my app and acknowlege
that there probably is a problem.

[Dave Dyer]

Here's a trivial test program that illustrates the problem.  it "works" (ie; produces glitchy graphics)
on both real android devices and DuOS under windows.

package com.mycompany.myapp;

import com.codename1.io.Log;
import com.codename1.ui.Button;
import com.codename1.ui.Component;
import com.codename1.ui.Container;
import com.codename1.ui.Display;
import com.codename1.ui.FontImage;
import com.codename1.ui.Form;
import com.codename1.ui.Graphics;
import com.codename1.ui.Image;
import com.codename1.ui.Label;
import com.codename1.ui.animations.CommonTransitions;
import com.codename1.ui.events.ActionEvent;
import com.codename1.ui.events.ActionListener;
import com.codename1.ui.geom.Dimension;
import com.codename1.ui.layouts.BorderLayout;
import com.codename1.ui.layouts.BoxLayout;
import com.codename1.ui.layouts.FlowLayout;
import com.codename1.ui.layouts.LayeredLayout;
import com.codename1.ui.plaf.UIManager;
import com.codename1.ui.util.Resources;
import com.codename1.ui.util.UITimer;

import java.io.IOException;

class DisplayComponent extends Component
{    public Image fixed = null;
    public Image front = null;
    public void fillFixed()
    {    int w = getWidth();
        int h = getHeight();
        Graphics g = fixed.getGraphics();
        g.setColor(0xd0d0d0);
        g.fillRect(0,0,w,h);
    }
    int frame = 0;
    public Dimension getPreferredSize()
    { Container parent = getParent();
      return(new Dimension(parent.getWidth(),parent.getHeight()));
    }
    public void fillFront()
    {    int w = getWidth();
        int h = getHeight();
        frame++;
        Graphics g = front.getGraphics();
        g.drawImage(fixed,0,0);
        g.setColor(0);
        g.fillRect(w/3,0,w/4,h);
        g.setColor(0xff);
        g.drawLine(0,0,w,h);
        g.setColor(0xff00);
        g.drawString("Frame "+frame,w/2,h/2);
    }
    public void paint(Graphics g)
    {    int w = getWidth();
        int h = getHeight();
        if(fixed==null) { fixed = Image.createImage(w,h); fillFixed(); }
        if(front==null) { front = Image.createImage(w,h); fillFront(); }
        if(front!=null)
        {
        g.drawImage(front,0,0);
        // prepare the next frame
        fillFront();
        repaint();
        }
    }
   
}

public class MyApplication {

    private Form current;
    private Resources theme;
    private DisplayComponent display = null;
    public void init(Object context) {
        theme = UIManager.initFirstTheme("/theme");

        // Pro only feature, uncomment if you have a pro subscription
        // Log.bindCrashProtection(true);
    }

    public void start() {
        if (current != null) {
            current.show();
            return;
        }
        Form hi = new Form("Welcome", new BorderLayout(BorderLayout.CENTER_BEHAVIOR_CENTER_ABSOLUTE));
       
        display = new DisplayComponent();
        display.setVisible(true);
        hi.add("Center",display);

        hi.show();
    }

    public void stop() {
        current = Display.getInstance().getCurrent();
    }

    public void destroy() {
    }

}

[Shai Almog]

Calling repaint() from within paint is bad form and might trigger issues.
If this is happening when you use registerAnimated() on the form and return true from animate() (which is the official way to do animations like that) then please file an issue into the issue tracker with that code.

[Dave Dyer]

This isn't intended to be an animation.  The same behavior occurs when the program is ticking along at 1 frame per second, and the only thing changing is a digital clock.

[Shai Almog]

I never said animate() is an animation. I did say you shouldn't call repaint() from within paint().

[Dave Dyer]

It is only this test program that calls repaint this way.

[Shai Almog]

Does it happen if you switch to animate()?

[Dave Dyer]

adding registerAnimation and animate has no effect.  It still glitches.

[Shai Almog]

We'll look into it.

[Dave Dyer]

Any news of this?

I've seen lately that even double buffering doesn't quite do it.  Occasionally there is a breakthrough
of bad bits from the other bitmap.  Today I made a small change intended to reduce the latency
between preparing a frame and displaying it, and now I'm seeing  a lot more of these breakthroughs.

[Shai Almog]

Is there an issue associated with this thread?
There are too many messages here and it's impossible to follow by now.

[Dave Dyer]

Yes, it was filed. https://github.com/codenameone/CodenameOne/issues/1660
That was when you said "we'll look into it" on Feb 1.

[Shai Almog]

Where is this happening. The issue doesn't state the OS this is occurring on?

[Dave Dyer]

It's an Android only problem.  IOS works smoothly with single buffering.

Latest info: 
 As previously mentioned, A change intended to reduce the latency between preparing and displaying
frames made this problem much worse.  I've papered over the problem by instituting a mandatory
20 microsecond delay between the time drawImage completes, and the time the image in question
is recycled and redrawn for the next frame.   Its pretty clear that taking the bits from the image is still
in progress when drawImage completes, and that subsequent getGraphics do not take this into account.

[Dave Dyer]

I meant 20 millisecond delay.

[Shai Almog]

I scheduled the issue

[Dave Dyer]

I object to this issue being closed without action.   The minimal action would be to
document it as a known problem, and leave the issue unresolved.

On Sunday, March 13, 2016 at 9:33:14 PM UTC-7, Shai Almog wrote:

I scheduled the issue

[Shai Almog]

OK, how/where would you document it?

[Dave Dyer]

The current state is pretty embarrasing to document. Basically any kind of draw image using mutable images
Can produce graphic junk if the image is rewritten "too soon" afterward. And you can't promise how
Long to wait is long enough.

...and please keep the issue open so it doesn't disappear forever.

[Shai Almog]

Native OS's don't support double buffering today as the hardware from the past 10 years moved away from that architecture.
I repeatedly told you that using this approach is a bad idea. Our API specifically has a method called areMutableImagesSlow() which returns true for iOS/Android discouraging their use.
None of our samples/guides make reference to using them in the way you chose because of the obvious problems that will happen in any complex pipeline.

[Dave Dyer]

The problem being that my frames are sufficiently complex that I can't get a decent frame rate for animations
if the whole scene has to be drawn.  Using a backing image for the fixed elements is the only way to go, but
that backing image isn't fixed either.

Anyway, the point shouldn't be that mutable images aren't recommended, there should be that they
can theoretically work.

[Shai Almog]

You can't get them to theoretically work natively on iOS or modern Android. New GPU pipelines are so deep it's impractical (performance wise).

You can cache some elements in mutable images but I suggest rethinking these assumptions.

[Dave Dyer]

What about using images that are allocated in the frame buffer?  Possibly these wouldn't support getRGB,
and would be writable only using getGraphics

[Shai Almog]

In modern graphics pipelines there is no frame buffer.
On iOS we wrote the pipeline and have two completely separate and unrelated pipelines for mutable images/screen.
On Android we use the builtin pipeline which google implemented as two completely different pipelines...