ITextureEffect Sync Problem

[paulscode]

I have been playing around with ITextureEffect, and I keep coming across the same problem.  It seems that at random points in time, the texture effect is sometimes not fully drawn before a frame gets rendered.  The result is light-blue shapes randomly showing up for a split second on the texture.  It seems to happen in both hardware and software rendering modes, and it occurs more frequently during heavy processing.

I assume the problem is that I am somehow not synchronizing things correctly, but I can't figure out what I need to change.  I created a simple test case to demonstrate.  Because this applet is so simple (i.e. no heavy processing going on), the problem doesn't show up very frequently (especially if you have a fast computer), but you can simulate the problem by dragging the browser window around, which seems to create a strain on the rendering, resulting in the behavior described above.  I made it so you can switch between software and hardware rendering modes by right-clicking on the applet.

Test Case

Here is the source code for the above test case.  Hopefully you can see what I am doing wrong:

Code Select Expand

import java.awt.Canvas;
import java.awt.Color;
import java.awt.BorderLayout;
import java.awt.Graphics;
import java.awt.Image;
import java.awt.image.BufferedImage;
import java.awt.image.ImageObserver;
import java.awt.image.PixelGrabber;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
import java.awt.event.MouseMotionListener;

// From the jPCT API (http://www.jpct.net):
import com.threed.jpct.FrameBuffer;
import com.threed.jpct.IRenderer;
import com.threed.jpct.ITextureEffect;
import com.threed.jpct.Lights;
import com.threed.jpct.Matrix;
import com.threed.jpct.Object3D;
import com.threed.jpct.SimpleVector;
import com.threed.jpct.Texture;
import com.threed.jpct.TextureManager;
import com.threed.jpct.World;

/**  IMPORT THIS FOR AN APPLET: **/
/**/
import javax.swing.JApplet;
/**/
/*********************************/

/**  IMPORT THESE FOR AN APPLICATION: **/
/*
import java.awt.Insets;
import javax.swing.JFrame;
*/
/***************************************/

/**  EXTEND JApplet FOR AN APPLET **/
/**  EXTEND JFrame FOR AN APPLICATION **/
public class ITextureEffectSyncProblem extends /*JFrame*/ /**/JApplet/**/
                         implements MouseListener, MouseMotionListener, Runnable
{
    // JApplet/JFrame dimensions:
    private int width = 640;
    private int height = 480;
    // top and left insets (both zero for an applet)
    private int titleBarHeight = 0;
    private int leftBorderWidth = 0;

    private Canvas myCanvas = null;

    private final Object jPCTSync = new Object();
    private FrameBuffer buffer = null;
    private World world = null;
    private Object3D object = null;
    private Texture texture = null;
    private Image blankImage = null;

    private boolean alive = true;
    private boolean initialized = false;


/**  USE THIS FOR AN APPLICATION  **/
/*
    public static void main( String[] args )
    {
        new ITextureEffectSyncProblem();
    }

    public ITextureEffectSyncProblem()
    {
        setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE );
        setTitle( "ITextureEffect Sync Problem" );
        pack();
        Insets insets = getInsets();
        titleBarHeight = insets.top - 1;
        leftBorderWidth = insets.left - 1;
        setSize( width + leftBorderWidth + insets.right - 1,
                 height + titleBarHeight + insets.bottom - 1 );
        setResizable( false );
        setLocationRelativeTo( null );
        setVisible( true );

        init();
    }
*/
/***********************************/

/**  USE Override FOR AN APPLET  **/
/**/
    @Override
/**/
/***********************************/
/**
* Shuts down the main game-loop thread, and cleans up.
*/
    public void destroy()
    {
    alive = false;
    }


/**  USE Override FOR AN APPLET  **/
/**/
    @Override
/**/
/***********************************/
/**
* Initializes all components of the program.
*/
    public void init()
    {
        synchronized( jPCTSync )
        {
            // sign the applet up to receive mouse messages:
            world = new World();  // create a new world

            World.setDefaultThread( Thread.currentThread() );

            // create a new buffer to draw on:
            buffer = new FrameBuffer( width, height, FrameBuffer.SAMPLINGMODE_HARDWARE_ONLY );
            buffer.disableRenderer( IRenderer.RENDERER_SOFTWARE );

            myCanvas = buffer.enableGLCanvasRenderer();

            add( myCanvas, BorderLayout.CENTER);
            myCanvas.setVisible( true );

            texture = new Texture( 512, 512, Color.BLACK );
            TextureManager.getInstance().addTexture( "TextureEffect", texture );
            texture.setEffect( new BlankTextureEffect() );
            blankImage = new BufferedImage( 512, 512,
                                            BufferedImage.TYPE_4BYTE_ABGR );
            Graphics g = blankImage.getGraphics();
            g.setColor( new Color( 25, 25, 25 ) );
            g.fillRect( 0, 0, 512, 512 );
            object = createQuad( 50, 0, "TextureEffect" );
            object.setAdditionalColor( Color.WHITE );
            world.addObject( object );
            object.translate( new SimpleVector( 0, 0, 100 ) );

            world.buildAllObjects();

            letThereBeLight();  // create light sources for the scene

            // receive mouse input from the main applet:
            addMouseListener( this );
            addMouseMotionListener( this );

            // also get mouse input picked up by the canvas:
            myCanvas.addMouseListener( this );
            myCanvas.addMouseMotionListener( this );
        }

        initialized = true;

        new Thread(this).start();
    }

    @Override
    public void run()
    {
        while( alive )
        {
            synchronized( jPCTSync )
            {
                texture.applyEffect();
            }
            this.repaint();
            try
            {
                Thread.sleep(10);
            }
            catch(Exception e)
            {}
        }
    }

    // Draw the scene
    @Override
    public void paint( Graphics g )
    {
        if( !initialized )
            return;

        synchronized( jPCTSync )
        {
            buffer.clear();   // erase the previous frame

            // render the world onto the buffer:
            world.renderScene( buffer );
            world.draw( buffer );
            buffer.update();

            if( buffer.usesRenderer( IRenderer.RENDERER_SOFTWARE ) )
            {
                // Paint this frame onto the applet/frame (software mode)
                buffer.display( g, leftBorderWidth, titleBarHeight );
            }
            else
            {
                // Repaint the canvas (hardware mode)
                buffer.displayGLOnly();
                myCanvas.repaint();
            }
        }
    }

    // create light sources for the scene
    private void letThereBeLight()
    {
        world.getLights().setOverbrightLighting (
            Lights.OVERBRIGHT_LIGHTING_DISABLED );
        world.getLights().setRGBScale( Lights.RGB_SCALE_2X );

        // Set the overall brightness of the world:
        world.setAmbientLight( 50, 50, 50 );

        // Create a main light-source:
        world.addLight( new SimpleVector( 50, -50, -300 ), 20, 20, 20 );
    }

/**
* Creates a billboarded quad with a z-offset (handles like a cube rather than
* a plane).
* @param width Quad width.
* @param zoffset Z-offset (toward the camera).
* @param texture Name of the texture to use on the quad.
* @return Handle to the billboarded quad.
*/
    public static Object3D createQuad( float width, float zoffset,
                                       String texture )
    {
        float offset = width / 2.0f;
        Object3D obj = new Object3D( 2 );

        obj.addTriangle( new SimpleVector( -offset, -offset, 0 ),
                         0, 0,
                         new SimpleVector( -offset, offset, 0 ), 0, 1,
                         new SimpleVector( offset, offset, 0 ), 1, 1,
                         TextureManager.getInstance().getTextureID( texture ) );
        obj.addTriangle( new SimpleVector( offset, offset, 0 ),
                         1, 1,
                         new SimpleVector( offset, -offset, 0 ), 1, 0,
                         new SimpleVector( -offset, -offset, 0 ), 0, 0,
                         TextureManager.getInstance().getTextureID( texture ) );
        // Make it billboard:
        obj.setBillboarding( Object3D.BILLBOARDING_ENABLED );
        // Set up the transparency:
        obj.setTransparency( 0 );
        obj.setTransparencyMode( Object3D.TRANSPARENCY_MODE_ADD );
        obj.setLighting( Object3D.LIGHTING_NO_LIGHTS );
        obj.build();

        SimpleVector o = new SimpleVector( 0, 0, -zoffset );
        Matrix m = obj.getTranslationMatrix();
        obj.setTranslationMatrix( new Matrix() );
        obj.translate( o );
        obj.translateMesh();
        obj.setTranslationMatrix( m );

        return obj;
    }

    @Override
    public void mouseClicked( MouseEvent e )
    {
        if( (e.getModifiers() & MouseEvent.BUTTON3_MASK) != 0 )
        {
            synchronized( jPCTSync )
            {
                if( buffer.usesRenderer( IRenderer.RENDERER_SOFTWARE ) )
                {
                    // Switch to hardware mode
                    buffer.disableRenderer( IRenderer.RENDERER_SOFTWARE );
                    myCanvas = buffer.enableGLCanvasRenderer();

                    this.add( myCanvas, BorderLayout.CENTER);
                    this.validate();
                    myCanvas.setVisible( true );
                    myCanvas.validate();
                    // get mouse input picked up by the canvas:
                    myCanvas.addMouseListener( this );
                    myCanvas.addMouseMotionListener( this );
                }
                else
                {
                    // Switch to software mode
                    buffer.disableRenderer( IRenderer.RENDERER_OPENGL );
                    buffer.enableRenderer(IRenderer.RENDERER_SOFTWARE );
                    this.remove( myCanvas );
                    this.validate();
                }
            }
        }
    }

    @Override
    public void mouseDragged( MouseEvent e )
    {}

    @Override
    public void mouseEntered( MouseEvent e )
    {}

    @Override
    public void mouseExited( MouseEvent e )
    {}

    @Override
    public void mouseMoved( MouseEvent e )
    {}

    @Override
    public void mousePressed( MouseEvent e )
    {}

    @Override
    public void mouseReleased( MouseEvent e )
    {}

/**
* The BlankTextureEffect class copies a solid color image onto the texture.
*/
    private class BlankTextureEffect implements ITextureEffect
    {
        /**
         * Ignored by BlankTextureEffect.
         * @param tex The video texture.
        */
        @Override
        public void init( Texture tex)
        {}

        /**
         * Copies the solid color image onto the texture.
         * @param dest Pixels to change.
         * @param dest Current pixels.
        */
        @Override
        public void apply(int[] dest, int[] source)
        {
            synchronized( jPCTSync )
            {
                if( blankImage != null )
                {
                    PixelGrabber pg = new PixelGrabber( blankImage, 0, 0, 512,
                                                        512, dest, 0, 512 );
                    try
                    {
                        pg.grabPixels();
                    }
                    catch( InterruptedException e )
                    {
                        System.err.println( "interrupted waiting for pixels!" );
                        return;
                    }
                    if( ( pg.getStatus() & ImageObserver.ABORT ) != 0 )
                    {
                        System.err.println( "image fetch aborted or errored" );
                        return;
                    }
                }
            }
        }

        /**
         * Whether the texture contains alpha information (true for
         * BlankTextureEffect).
         * @return True.
        */
        @Override
        public boolean containsAlpha()
        {
            return true;
        }
    }
}

[EgonOlsen]

I'm not sure...in case of the AWTGLRenderer this could be caused by the fact that it renders in the AWT event dispatch thread in its own paint method, to which you can't synchronize by synchronizing to your own jPCTSync. But that's not the case with the software renderer... ...anyway, you can get a lock object from the FrameBuffer (getLock()). Try to synchronize your applyEffect to that object too and see if that helps. However, i suggest to put such code in one of the IPaintListeners methods in possible, because it ensures that it happens in the same thread as the rendering then.

[paulscode]

Ok, will do.  Thanks for the great tips.  I'll let you know what I find out.

[EgonOlsen]

...but it still doesn't make any sense that it happens in the software renderer.  Especially as your example applet doesn't change the image at all. I don't see how copying uniform colored pixels can result in blue shapes even if there were a synchronization problem. Have you checked that it isn't your system ram? It could very well be a weak memory chip that starts to go crazy under load. Maybe you should run something like Prime95 for an hour in stress testing mode to make sure that this isn't the problem.
I haven't tried the example but i've tried your video applet and i wasn't able to spot any artifacts on my machine.

[paulscode]

That's a good idea, checking if it is specific to the one machine.  I'll run it on some of my other test machines to see if I get the same behavior.

[paulscode]

I ran the test applet on my laptop, and it didn't experience the problem there either.  However, the blue shapes do still appear on the Britney Spears music video ITextureEffect demo applet I posted in an earlier thread.  I was wondering if you see the same behavior when you run it?  It would at least give you a better idea of what I'm talking about.  The thing that is really weird is the fact that it is blue and not something like black or white, or whatever was on the texture from the previous frame.

I still need to set up my test machines to see how common the problem is.  I'll try to create a better test case if I can't get the problem solved using your suggestion for synchronization.

[EgonOlsen]

I've watched through the whole video without noticing a single glitch (except Britney herself...  ).

[influt]

That's not specific to one machine. I did notice blue shapes on the Britney Spears music video, too.

[EgonOlsen]

Which graphics card are you both using?

[paulscode]

I discovered that I was using an older version of jPCT (when I tried to find the FrameBuffer getLock() method).  Upgrading to the most recent version and synchronizing on the getLock() Object seems to have corrected the problem in my initial test in hardware rendering mode.  I'll need to run additional tests to make sure, but it looks like the problem has been solved.  Sorry for not checking my jPCT version before posting (this is the second time I've done that).

[paulscode]

I have run some additional tests, and the problem now definitely appears to be solved.  It seems that synchronizing on the getLock() object is the key to correcting the issue in hardware-rendering mode.

Also, after switching to the most recent version of jPCT, I have been unable to replicate the problem in software-rendering mode with or without synchronizing on the getLock() object.  I am a little baffled why I experience the behavior in software-rendering mode at all with the older version of jPCT, since it appears to be a synchronization with the AWT dispatch thread, which shouldn't affect software-rendering   Looking at the test applet code, the only thing I can think of is that I didn't call setVisible( false ) on the GLCanvas bevore removing it from the applet, so perhaps the canvas was still sticking around or something   I've never had to do that before, though.  Oh, well, I guess it is a non-issue now, since the problem seems to be gone with the new jPCT version.

I re-uploaded the music video applet with the ITextureEffect synchronization problem hopefully fixed.  I hate to subject everyone to more of Britney Spears, but could I get some folks to run the applet again to make sure the random flashing blue shapes are gone now?  This applet still has several other problems, but it was the best/worst example I had of the ITextureEffect synchronization issue:

http://www.paulscode.com/demos/jpct/VideoTexture/

[influt]

It runs fine now on my PC.

[EgonOlsen]

Good to know. Too bad that i can't add this synchronization by myself to the applyEffect-method, because there's no reference to the FrameBuffer there. I had the same problem with animate() in Object3D and "solved" it by adding an animateSync-method. Maybe i should do the same thing here. At least it makes more clear that there could be a problem....it's strange though, because texture unloading/uploading happens at one place...either the texture has changed (i.e. applyEffect() has been called) before or after, it shouldn't matter if it does in between, but obviously it does...