Texture Size

[signifer123]

Edit: It appears after some code profiling that TextureManager adding a new texture is the cause of the slowdown not the resizing. Is there a way to bypass TextureManager?

I'm tinkering with the ability to take frames directly out of a video stream via a DirectShow wrapper, and place them as a texture, this works fine when just presenting to a JPanel, but when it's resized to place as a texture on a Plane it causes my CPU to not be able to keep up. Is there anyway to change the size textures to be larger than 256x256 by default? Though the movie i was testing with is 720x540, so i suppose some form of sampling would be required? Would there be a better way to render a video to 3d?

Currently, i use a frame grab, which returns a AWT.Image, then i use that to replace the texture in the textureManager.

Or perhaps a way to resize on the GPU?

[paulscode]

Don't know how fast this would be, but you could buffer the video by drawing your AWT.Image onto another image with a compatible texture size (width and height powers of 2).  Then just set the proper u/v coordinates on your object to use only the part of the texture that contains the video.

[signifer123]

Unfortunately that resulted in being about as slow as scaling it, even without U/V. I went in and timed some stuff the grabbing of the image took between 7 and 25 ms, the clearing and applying to a new image took 7-12 ms, but whole process of replacing the texture would take either about 56 ms and ~207 ms alternating between the two the entire time:

~56,~207,~56,~207,~56,~207


If i do everything except replace the texture i get about 60 ms for the whole render loop, using drawing onto the buffer. So i feel my question should be reformated to can i use a texture without texture manager?

[EgonOlsen]

No, the texture manager is mandatory except for textures that are used for blitting only. However, its a very thin layer anyway.  Just some simple lists and maps. Are you using one of the hardware renderers?

[signifer123]

It says "OpenGL renderer initialized (using 4 texture stages)" shows in stdout and i'm using buffer.displayGLOnly(); to present the data. The rendering only seems to take fairly long when using replaceTexture or addTexture. I guess i'll just leave to my processor not being powerful enough for such a thing yet. The blitting is fast enough but it takes about 70% CPU for the whole render loop that way.

[EgonOlsen]

You are using the hardware renderer. It's not your cpu, at least not completely. It's mainly because a replaceTexture causes two operations when the next frame is being rendered: The old texture gets unloaded (thats not much of a problem) and the new one gets uploaded to the graphics card. And that is very very slow. In the current version of jPCT, there's not much you can do against this. However, some things may help and are worth a try.

- provide the texture in a size that's a power of two to avoid the downscaling. The smaller the texture is, the faster the upload will be.

- if you are using mipmaps, disable them for the texture in question.

- if possible, use the constructors in Texture that take the alpha channel of the texture. That prevents jPCT from creating its own one every time.

- if you have access to the videos pixel data directly, use an ITextureEffect to write them directly into the texture.

[paulscode]

I had an idea I'd like to try for this (using UVs and "mosaic" textures to cut down the rate of adds to the texture manager).  Is there any chance I can see your code for reading the frames of a video stream into AWT.Image's?  I'll post the code if I can get it working at a reasonable framerate.

[paulscode]

The mosaic texture idea didn't solve the problem, because there is still that noticible .2 second blocking every time a texture is swapped in the TextureManager (happens less often, of course, but it doesn't go away).  I also tried performing the swap on a seperate thread, but as expected, that just caused thread synchronization issues and random crashes.

However, I was able to use Egon's suggestion of ITextureEffect:

Demo Applet  (LINK TO SOURCE REMOVED)

I am aware of the random blocking problems with the above applet - they are due to an ugly hack of a JMF streamer class that I threw together for aquiring video frames as instances of awt.Image.  I think there is also a synchronization issue between changing the frame Image and updating the texture (causes light-blue artifacts showing up randomly on the texture - usually coincides with the blocking).  I am sure there are much better ways to acomplish this if I wanted to take the time to learn the JMF API (which I don't) and further debug my code.  However, I think it works well enough to demonstrate that video textures are definitely possible using ITextureEffect.  You can look at the above source code if you like, but it is rather ugly.  Here are the relevant parts (not in any particular order):

Code Select Expand


        textureWidth = nextPowerOf2( movie.getWidth() );
        textureHeight = nextPowerOf2( movie.getHeight() );
        float uw = (float) movie.getWidth() / (float) textureWidth;
        float uh = (float) movie.getHeight() / (float) textureHeight;

        screenTexture = new Texture( textureWidth, textureHeight, Color.BLACK );
        TextureManager.getInstance().addTexture( "TV Screen", screenTexture );
        screenTexture.setEffect( new VideoTextureEffect() );
        SimpleVector screenOffset = new SimpleVector( 80 * tvScale,
                                                      -14 * tvScale,
                                                      12 * tvScale );
        SimpleVector br = new SimpleVector( 0, 0, 0 );
        SimpleVector bl = new SimpleVector( 105 * tvScale, 0, 0 );
        SimpleVector tr = new SimpleVector( 0, -77 * tvScale, 0 );
        SimpleVector tl = new SimpleVector( 105 * tvScale, -77 * tvScale, 0 );
        tvScreen = new Object3D( 2 );
        tvScreen.addTriangle( tl, 0, 0, bl, 0, uh, br, uw, uh,
                          TextureManager.getInstance().getTextureID( "TV Screen" ) );
        tvScreen.addTriangle( br, uw, uh, tr, uw, 0, tl, 0, 0,
                          TextureManager.getInstance().getTextureID( "TV Screen" ) );
        tvScreen.build();
        tvScreen.translate( screenOffset );
        world.addObject( tvScreen );
        tvScreen.setVisibility( true );
        tvPivot.addChild( tvScreen );


...


                currentFrameImage = movie.getCurrentFrame();


...


            synchronized( jPCTSync )
            {
                screenTexture.applyEffect();
            }


...


/**
* The VideoTextureEffect class changes the video texture to the current frame.
*/
    private class VideoTextureEffect implements ITextureEffect
    {
        /**
         * Ignored by VideoTextureEffect.
         * @param tex The video texture.
        */
        @Override
        public void init( Texture tex)
        {}

        /**
         * Changes the contents of the video texture to the current frame.
         * @param dest Pixels to change (change to current frame).
         * @param dest Current pixels (previous frame).
        */
        @Override
        public void apply(int[] dest, int[] source)
        {
            if( lastEffectFrame != currentFrame )
            {
                lastEffectFrame = currentFrame;
                if( currentFrameImage != null )
                {
                    PixelGrabber pg = new PixelGrabber( currentFrameImage, 0, 0,
                                                        movie.getWidth(),
                                                        movie.getHeight(),
                                                        dest, 0, textureWidth );
                    try
                    {
                        pg.grabPixels();
                    }
                    catch( InterruptedException e )
                    {
                        System.err.println( "interrupted waiting for pixels!" );
                        return;
                    }
                    if( ( pg.getStatus() & ImageObserver.ABORT ) != 0 )
                    {
                        System.err.println( "image fetch aborted or errored" );
                        return;
                    }
                }
            }
        }


...


/**
* Returns the next power of two higher than x, or x itself it it is already a
* power of two.
* @param x A positive integer value.
* @return The next power of two.
*/
    private static int nextPowerOf2( int x )
    {
        --x;
        x |= x >> 1;
        x |= x >> 2;
        x |= x >> 4;
        x |= x >> 8;
        x |= x >> 16;
        return ++x;
    }


On a related topic, I added an audio track to the above applet just to see how it looks.  You will notice that the sound and video are definitely not synced in the above applet.  This is due to the following factors:
1) The audio track is an ogg created from an mp3 I downloaded - it wasn't split from the actual video.
2) The random blocking delays (most likely due to a thread synchronization problem) affect both the video frame rate and the audio stream over time.
3) Frames are processed in the main game-loop so moving through the world changes the FPS, which affects the video frame rate over time.

I added in some tweaking with frame offset, framerate, and pitch to make the audio and video match up fairly close on my computer (but these settings won't be the same for every computer).  A much better way to synchronize the audio and video would be to count the frames and milliseconds and plug that into some mathematical formula to calculate tiny pitch changes and/or framerate changes to compensate.  However, that was beyond the scope of the above example.

[EgonOlsen]

Looks cool. It's surprisingly fast...or at least fast enough. It should be possible to improve performance by using pbuffers as storage for the texture data, but that's something the jPCT doesn't support right now. And to be honest, it's not very high on my priority list...