jPCT + Vuforia user defined target integration

[tudor08]

hi all !
So I've bee trying to integrate Vuforia with jPCT for the last couple of days and I have seen a couple of tutorials and answers(including the official wikipedia).
However I am not able to do it on my own.
Integration is on Vuforia User Defined Target sample which basically displays a teapot on the screen when the user takes a picture of something and remembering the picture later so it can track it.
When I follow the tutorial of integrating these two(vuforia+jpct) I can't make the primitive(basic cube) to render onto the screen.
Here is my current code

Code Select Expand


/*===============================================================================
Copyright (c) 2012-2014 Qualcomm Connected Experiences, Inc. All Rights Reserved.

Vuforia is a trademark of QUALCOMM Incorporated, registered in the United States
and other countries. Trademarks of QUALCOMM Incorporated are used with permission.
===============================================================================*/

package com.marked.vifo.UserDefinedTargets;

import android.annotation.SuppressLint;
import android.opengl.GLES20;
import android.opengl.GLSurfaceView;
import android.opengl.Matrix;
import android.util.Log;

import com.marked.vifo.VuforiaApplication.ApplicationSession;
import com.marked.vifo.VuforiaApplication.utils.CubeShaders;
import com.marked.vifo.VuforiaApplication.utils.MeshObject;
import com.marked.vifo.VuforiaApplication.utils.SampleUtils;
import com.marked.vifo.VuforiaApplication.utils.Texture;
import com.qualcomm.vuforia.Matrix44F;
import com.qualcomm.vuforia.Renderer;
import com.qualcomm.vuforia.State;
import com.qualcomm.vuforia.Tool;
import com.qualcomm.vuforia.TrackableResult;
import com.qualcomm.vuforia.VIDEO_BACKGROUND_REFLECTION;
import com.qualcomm.vuforia.Vuforia;
import com.threed.jpct.Camera;
import com.threed.jpct.FrameBuffer;
import com.threed.jpct.Light;
import com.threed.jpct.Object3D;
import com.threed.jpct.Primitives;
import com.threed.jpct.SimpleVector;
import com.threed.jpct.World;
import com.threed.jpct.util.MemoryHelper;

import java.util.Vector;

import javax.microedition.khronos.egl.EGLConfig;
import javax.microedition.khronos.opengles.GL10;

// The renderer class for the ImageTargetsBuilder sample.
public class ImageTargetRenderer implements GLSurfaceView.Renderer {
    // Constants:
    static final float kObjectScale = 3.f;
    private static final String LOGTAG = "ImageTargetRenderer";
    public boolean mIsActive = false;
    Object3D box;
    private ApplicationSession vuforiaAppSession;
    private Vector<Texture> mTextures;
    private int shaderProgramID;
    private int vertexHandle;
    private int normalHandle;
    private int textureCoordHandle;
    private int mvpMatrixHandle;
    private int texSampler2DHandle;
    private MeshObject mTeapot;
    // Reference to main activity
    private ImageTargets mActivity;
    private FrameBuffer fb;
    private World world;
    private float[] modelViewMat;
    private Light sun;
    private Object3D cube;
    private Camera cam;
    private float fov;
    private float fovy;
    private float[] modelViewMatrix;
    private SimpleVector mCameraPosition;
    private SimpleVector mCameraDirection;
    private SimpleVector mCameraUp;

    public ImageTargetRenderer(ImageTargets activity, ApplicationSession session, MeshObject meshObject) {
        this.mActivity = activity;
        this.vuforiaAppSession = session;
        this.mTeapot = meshObject;
        this.world = new World();
        world.setAmbientLight(25, 25, 25);
        this.sun = new Light(world);
        sun.setIntensity(250, 250, 250);
        // Create a texture out of the icon...:-)
//        TextureManager txtMgr = TextureManager.getInstance();
//        if (!txtMgr.containsTexture("texture")) {
//            com.threed.jpct.Texture texture = new com.threed.jpct.Texture(64, 64, RGBColor.BLUE);
//            txtMgr.addTexture("texture", texture);
//        }

        box = Primitives.getCube(10.0f);
        box.setEnvmapped(Object3D.ENVMAP_ENABLED);
        box.translate(5, 0, 12);
//        box.setTexture("texture");
//        box.calcTextureWrapSpherical();
        box.strip();
        box.build();

        world.addObject(box);

        cam = world.getCamera();
        cam.moveCamera(Camera.CAMERA_MOVEOUT, 50);
        cam.lookAt(box.getTransformedCenter());

        SimpleVector sv = new SimpleVector();
        sv.set(box.getTransformedCenter());
        sv.y -= 100;
        sv.z -= 100;
        sun.setPosition(sv);
        MemoryHelper.compact();
    }

    // Called when the surface is created or recreated.
    @SuppressLint("LongLogTag")
    @Override
    public void onSurfaceCreated(GL10 gl, EGLConfig config) {
        Log.d(LOGTAG, "GLRenderer.onSurfaceCreated");
        // Call function to initialize rendering:
        initRendering();
        // Call Vuforia function to (re)initialize rendering after first use
        // or after OpenGL ES context was lost (e.g. after onPause/onResume):
        vuforiaAppSession.onSurfaceCreated();
    }

    // Called when the surface changed size.
    @SuppressLint("LongLogTag")
    @Override
    public void onSurfaceChanged(GL10 gl, int width, int height) {

        // Call function to update rendering when render surface
        // parameters have changed:
        mActivity.updateRendering();
        // Call Vuforia function to handle render surface size changes:
        vuforiaAppSession.onSurfaceChanged(width, height);


        if (fb != null) {
            fb.dispose();
        }
        fb = new com.threed.jpct.FrameBuffer(width, height);

    }

    // Called to draw the current frame.
    @Override
    public void onDrawFrame(GL10 gl) {
        if (!mIsActive) {
            return;
        }
        // Call our function to render content
renderFrame();
updateCamera();
        drawWorld();
    }

    public void drawWorld() {
        try {
//            if (fb==null)
            world.renderScene(fb);
            world.draw(fb);
            fb.display();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private void renderFrame() {
        // Clear color and depth buffer
        GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);

        // Get the state from Vuforia and mark the beginning of a rendering
        // section
        State state = Renderer.getInstance().begin();
        // Explicitly render the Video Background
        Renderer.getInstance().drawVideoBackground();
        GLES20.glEnable(GLES20.GL_DEPTH_TEST);
        GLES20.glEnable(GLES20.GL_CULL_FACE);
        if (Renderer.getInstance().getVideoBackgroundConfig().getReflection() ==
                VIDEO_BACKGROUND_REFLECTION.VIDEO_BACKGROUND_REFLECTION_ON)
            GLES20.glFrontFace(GLES20.GL_CW); // Front camera
        else
            GLES20.glFrontFace(GLES20.GL_CCW); // Back camera

        // Render the RefFree UI elements depending on the current state
        mActivity.refFreeFrame.render();

        // Did we find any trackables this frame?
        for (int tIdx = 0; tIdx < state.getNumTrackableResults(); tIdx++) {
            // Get the trackable:
            TrackableResult trackableResult = state.getTrackableResult(tIdx);
            Matrix44F modelViewMatrix_Vuforia = Tool.convertPose2GLMatrix(trackableResult.getPose());
            modelViewMatrix = modelViewMatrix_Vuforia.getData();
            float[] modelViewProjection = new float[16];
            Matrix.translateM(modelViewMatrix, 0, 0.0f, 0.0f, kObjectScale);
            Matrix.scaleM(modelViewMatrix, 0, kObjectScale, kObjectScale, kObjectScale);
            Matrix.multiplyMM(modelViewProjection, 0, vuforiaAppSession.getProjectionMatrix().getData(), 0, modelViewMatrix,
                    0);

            GLES20.glUseProgram(shaderProgramID);
            mCameraPosition = new SimpleVector(modelViewMatrix[12], modelViewMatrix[13], modelViewMatrix[14]);
            mCameraDirection = new SimpleVector(modelViewMatrix[8], modelViewMatrix[9], modelViewMatrix[10]);
            //LANDSCAPE: 0, 1, 2
            //PORTRAIT: 4, 5, 6
            mCameraUp = new SimpleVector(-modelViewMatrix[0], -modelViewMatrix[1], -modelViewMatrix[2]);



            com.threed.jpct.Matrix mat = new com.threed.jpct.Matrix();
            mat.setDump(modelViewProjection);

//If I comment this out, picking works flawless:
            cam.setBack(mat);

            GLES20.glVertexAttribPointer(vertexHandle, 3, GLES20.GL_FLOAT, false, 0, mTeapot.getVertices());
            GLES20.glVertexAttribPointer(normalHandle, 3, GLES20.GL_FLOAT, false, 0, mTeapot.getNormals());
            GLES20.glVertexAttribPointer(textureCoordHandle, 2, GLES20.GL_FLOAT, false, 0, mTeapot.getTexCoords());

            GLES20.glEnableVertexAttribArray(vertexHandle);
            GLES20.glEnableVertexAttribArray(normalHandle);
            GLES20.glEnableVertexAttribArray(textureCoordHandle);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextures.get(0).mTextureID[0]);
            GLES20.glUniformMatrix4fv(mvpMatrixHandle, 1, false, modelViewProjection, 0);
            GLES20.glUniform1i(texSampler2DHandle, 0);
            GLES20.glDrawElements(GLES20.GL_TRIANGLES, mTeapot.getNumObjectIndex(), GLES20.GL_UNSIGNED_SHORT,
                    mTeapot.getIndices());

            GLES20.glDisableVertexAttribArray(vertexHandle);
            GLES20.glDisableVertexAttribArray(normalHandle);
            GLES20.glDisableVertexAttribArray(textureCoordHandle);
            drawWorld();
            SampleUtils.checkGLError("UserDefinedTargets renderFrame");
        }


        GLES20.glDisable(GLES20.GL_DEPTH_TEST);
        Renderer.getInstance().end();
    }

    public void updateCamera() {
        if (modelViewMatrix != null) {
            com.threed.jpct.Matrix m = new com.threed.jpct.Matrix();
            m.setDump(modelViewMatrix);
            cam.setBack(m);

//            cam.setOrientation(mCameraDirection, mCameraUp);
//            cam.setPosition(mCameraPosition);
        }
    }

    @SuppressLint("LongLogTag")
    private void initRendering() {
        Log.d(LOGTAG, "initRendering");

        // Define clear color
        GLES20.glClearColor(0.0f, 0.0f, 0.0f, Vuforia.requiresAlpha() ? 0.0f : 1.0f);

        // Now generate the OpenGL texture objects and add settings
        for (Texture t : mTextures) {
            GLES20.glGenTextures(1, t.mTextureID, 0);
            GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, t.mTextureID[0]);
            GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
            GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
            GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, t.mWidth, t.mHeight, 0, GLES20.GL_RGBA,
                    GLES20.GL_UNSIGNED_BYTE, t.mData);
        }

        shaderProgramID = SampleUtils.createProgramFromShaderSrc(CubeShaders.CUBE_MESH_VERTEX_SHADER,
                CubeShaders.CUBE_MESH_FRAGMENT_SHADER);

        vertexHandle = GLES20.glGetAttribLocation(shaderProgramID, "vertexPosition");
        normalHandle = GLES20.glGetAttribLocation(shaderProgramID, "vertexNormal");
        textureCoordHandle = GLES20.glGetAttribLocation(shaderProgramID, "vertexTexCoord");
        mvpMatrixHandle = GLES20.glGetUniformLocation(shaderProgramID, "modelViewProjectionMatrix");
        texSampler2DHandle = GLES20.glGetUniformLocation(shaderProgramID, "texSampler2D");

    }


    public void setTextures(Vector<Texture> textures) {
        mTextures = textures;

    }

}


When I run the app I see the cube on the screen for 0.5 sec, then disappears and I'm getting an infinite loop of thrown exception

Code Select Expand


10-02 19:40:11.257 21975-22043/com.marked.vifo I/jPCT-AE: Additional visibility list (43) created with size: 512
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err: java.lang.NullPointerException: Attempt to read from field 'int com.threed.jpct.Texture.glTarget' on a null object reference
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at com.threed.jpct.CompiledInstance.render(CompiledInstance.java:692)
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at com.threed.jpct.GLRenderer.drawVertexArray(GLRenderer.java:2308)
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at com.threed.jpct.World.draw(World.java:1417)
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at com.threed.jpct.World.draw(World.java:1100)
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at com.marked.vifo.UserDefinedTargets.ImageTargetRenderer.drawWorld(ImageTargetRenderer.java:157)
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at com.marked.vifo.UserDefinedTargets.ImageTargetRenderer.onDrawFrame(ImageTargetRenderer.java:150)
10-02 19:40:11.257 21975-22043/com.marked.vifo W/System.err:     at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1522)
10-02 19:40:11.258 21975-22043/com.marked.vifo W/System.err:     at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:1239)

The exception is thrown from this code

Code Select Expand


public void drawWorld() {
        try {
            world.renderScene(fb);
            world.draw(fb);
            fb.display();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

And I get an infinite number of thrown exception because I call this method in renderFrame() and for each frame throws an exception.
To me it seems that somewhere fb == null but I don't know how to fix this because I'm quite new to graphics/opengl/vuforia/jpct .
Also I don't know if I need/should link somehow the framebuffer(fb) to Vuforia's image/buffer/something
I would love some help if anyone knows what's going on.
Thank you very much!

[EgonOlsen]

You've somehow managed get away with an unassigned texture without causing an exception before. I'm not sure how you did this, but try to put the

//        box.setTexture("texture");

back in and see if that helps to get rid of at least this exception.

[tudor08]

Thank you for replying!
That doesn't solve the problem

Code Select Expand


10-03 12:18:19.718 24492-24551/com.marked.vifo W/System.err: java.lang.NullPointerException: Attempt to read from field 'int com.threed.jpct.Texture.glTarget' on a null object reference
10-03 12:18:19.718 24492-24551/com.marked.vifo W/System.err:     at com.threed.jpct.CompiledInstance.render(CompiledInstance.java:692)
10-03 12:18:19.718 24492-24551/com.marked.vifo W/System.err:     at com.threed.jpct.GLRenderer.drawVertexArray(GLRenderer.java:2308)
10-03 12:18:19.718 24492-24551/com.marked.vifo W/System.err:     at com.threed.jpct.World.draw(World.java:1417)
10-03 12:18:19.719 24492-24551/com.marked.vifo W/System.err:     at com.threed.jpct.World.draw(World.java:1100)
10-03 12:18:19.719 24492-24551/com.marked.vifo W/System.err:     at com.marked.vifo.UserDefinedTargets.ImageTargetRenderer.drawWorld(ImageTargetRenderer.java:158)
10-03 12:18:19.719 24492-24551/com.marked.vifo W/System.err:     at com.marked.vifo.UserDefinedTargets.ImageTargetRenderer.renderFrame(ImageTargetRenderer.java:230)
10-03 12:18:19.719 24492-24551/com.marked.vifo W/System.err:     at com.marked.vifo.UserDefinedTargets.ImageTargetRenderer.onDrawFrame(ImageTargetRenderer.java:150)
10-03 12:18:19.719 24492-24551/com.marked.vifo W/System.err:     at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1522)
10-03 12:18:19.719 24492-24551/com.marked.vifo W/System.err:     at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:1239)
10-03 12:18:19.719 24492-24551/com.marked.vifo I/jPCT-AE: Additional visibility list (156) created with size: 512


I said I see the cube for 0.5 seconds an then disappears. The error could be thrown because it tries to apply a texture on a cube that is no longer alive ?

[EgonOlsen]

No idea how you manage to do this. If the texture would be missing, it should exit with an error message much earlier. Looking at the code, I fail to see how you can get to this stage without a texture.
Can you please try to comment out the setEnvmapped() call?

[tudor08]

I commented out setEnvmapped() and surprize ! the cube is on the screen !
It's a good start, now I must find out how to link the cube to appear on picture detection.

Can you give me some insight to what was happening with the texture/model or how did you knew where to look ?

[EgonOlsen]

Can you give me some insight to what was happening with the texture/model or how did you knew where to look ?

I'm not sure myself. I looked at the code that caused the exception and it somehow deals with the environment mapping. But that's not really supported on Android anyway, so I might have overlooked a little problem here. I'll try to fix the issue in the next version.