Problem with object picking and modified camera backbuffer matrix

[redfalcon]

Hi,

I'm currently developing an AR-App and use jPCT-ae for rendering. To affix the model on a marker I receive a pose matrix and apply her to the camera with camera.setBack(). This works for putting my model on the marker, but I can't get picking to work with it. If I just render my model without changing the camera matrix it works fine, but as soon as I change her, it doesn't work anymore (calcMinDistanceAndObject3D() returns everywhere on my object COLLISION_NONE). Since the picking needs the camera position, I assume that something with the position vector is wrong.

My picking code:

Code Select Expand



private void doPicking(int touchX, int touchY){
SimpleVector dir = Interact2D.reproject2D3DWS(camera, frameBuffer, touchX, touchY).normalize();
Object[] res = world.calcMinDistanceAndObject3D(camera.getPosition(), dir, 10000);
  if (res[1] != null) {
    //do something...
 
  }
}



Object properties:

Code Select Expand


Object3D[] serializedObject = Loader.loadSerializedObjectArray(serializedInputStream);

for (int i = 0; i < serializedObject.length; i++) {
serializedObject[i].setCollisionMode(Object3D.COLLISION_CHECK_OTHERS);
if (i > 0) serializedObject[i].addParent(serializedObject[0]);
serializedObject[i].strip();

}



Camera pose setting:

Code Select Expand



public void onDrawFrame(GL10 arg0) {

//...

float[] matrix = getMatrixForJPCT(new float[16]);

Matrix mat = new Matrix();
mat.setDump(matrix);

//If I comment this out, picking works flawless:
camera.setBack(mat);

//...
}



Any idea why that is?

[EgonOlsen]

Does the tweaked matrix include a translation? If so, try to remove it from the matrix and set it as the camera's position instead.

[redfalcon]

Thanks, that brought me on the right track. The AR-engine converted the pose to a OpenGL-compatible matrix beforehand, which additionally screwed my calculations. But now I have some trouble doing the correct conversions between the coordinate systems now.
The AR-engine uses a right handed system (Picture). According to your wiki, I need to rotate the matrix by 180° around x to convert it to the jPCT system. But do I need to do it before or after setting the camera matrix? I currently extract the rotation/translation from the float array, dump it into a Matrix/SimpleVector, rotate the Matrix/SimpleVector by 180° around x and set them as the backbuffer/position. The car is fixed to the marker, but behaves totally unpredictable (it seems as the model sticks to the camera, and not to the marker) as I move around my device, so I guess my conversion is wrong.

[EgonOlsen]

Apart from the picking, was it working before? If so, just do what you did before separate the rotation from the translation afterwards.

[redfalcon]

Yes, besides some unwanted minor movement of the model when I move my device, everything but picking works. But the separating doesn't really work. As soon as I set the position of the camera, the model doesn't stay on the marker.

The GL-like matrix that does the placement correctly (but not the picking) looks like this:

Code Select Expand


-0.996   0.069  -0.063   0.000
0.092   0.808  -0.582   0.000
0.011  -0.586  -0.811   0.000
-75.993 -110.593 1902.010   1.000


I set this directly as the backbuffer and leave the position alone.

The unmodified pose itself is just transposed:

Code Select Expand


-0.996   0.092   0.011 -75.993
0.069   0.808  -0.586 -110.593
-0.063  -0.582  -0.811 1902.010


Looks like a 3x3 rotation matrix and a 3x1 translation vector to me.

If I set this matrix as the cameras backbuffer and this vector as the position, the picking works but the model is somewhere far off and doesn't stay on the marker if I move the device.

[EgonOlsen]

Strange...i don't see why it shouldn't work that way assuming that the rotation matrix doesn't contain the translation anymore when setting it.
If nothing else helps, you can try to do the projection into world space on your own based on the combined matrix. The reason why it doesn't work with the combined matrix is most likely that reproject2D3DWS treats the matrix as a 3x3 matrix for performance reasons. You can try

Code Select Expand


public SimpleVector reproject2D3DWS(Camera camera, FrameBuffer buffer, int x, int y, SimpleVector toFill) {
SimpleVector ray = reproject2D3D(camera, buffer, x, y, VIEWPLANE_Z_VALUE, toFill);
ray.matMul(camera.getBack().invert());
return ray;
}


instead. However, the clean solution would be to separate the two transformations...

[redfalcon]

Just that I get it right:
What should I use for VIEWPLANE_Z_VALUE, the far clipping plane?

Code Select Expand



//...
public SimpleVector reproject2D3DWS(Camera camera, FrameBuffer buffer, int x, int y, SimpleVector toFill) {
SimpleVector ray = Interact2D.reproject2D3D(camera, buffer, x, y, 2000, toFill);
ray.matMul(camera.getBack().invert());
return ray;
}

public void findNearestObject(int touchX, int touchY) {

SimpleVector dir = reproject2D3DWS(camera, frameBuffer, touchX, touchY, new SimpleVector()).normalize();
Object[] res = world.calcMinDistanceAndObject3D(camera.getPosition(), dir, 10000);
              //...
}



Is that correct?

[EgonOlsen]

No, just use 1f.

[redfalcon]

No Luck, that doesn't work either :/

I tried a different approach, and set the GL-Matrix as the rotation matrix of my model. This works fine for placement and picking, but now the rotations (With object.rotateX()) rotate around an axis which seems to be far above the object (so it's pretty unusable). Which is normal I guess, since I modified the rotation matrix. Is there a way to get a proper axis that I can use with rotateAxis() or the like?

[EgonOlsen]

If you use a combined matrix as rotation matrix, you'll change the rotation pivot by this translation. The new rotation pivot should be the negative translation.

[EgonOlsen]

BTW: Is this solution somehow related: http://www.jpct.net/forum2/index.php/topic,1586.msg11938.html#msg11938?

[redfalcon]

If you use a combined matrix as rotation matrix, you'll change the rotation pivot by this translation. The new rotation pivot should be the negative translation.

If I set the negative pivot as pivot, the model will instantly snap back to its previous position when trying to rotate it.

I've tried the other solution, but since it uses a different (geographic) coordinate system again, it doesn't work either. The model gets attached to the camera and doesn't stay where it should.

[EgonOlsen]

Then go back to this: http://www.jpct.net/forum2/index.php/topic,2461.msg18064.html#msg18064

The problem here seems to be that camera.getPosition() doesn't provide the proper position now that you don't set it correctly but hide it in the matrix. Try to multiply the camera's position with the matrix and use that as the new position.

[redfalcon]

I scrapped my approach and now pass the rotation values (from the touchscreen) directly to the pose calculation method. The AR engine provides a method for rotating the pose matrix, so I just pass the values I usually would use for object.rotateX/Y/Z() to the JNI function and rotate the matrix itself.
Thanks anyway!

[Vi_O]

Hello,

I am having the same issue than redfalcon and indeed, separating doesn't work.

I made a bit of code to see the difference between normal and rotation/translation separated :

Code Select Expand



_cameraMat.setDump(matrix);
_camera.setBack(_cameraMat);

Log.d("JpctEngine", "Valeur camera : " + _camera.getPosition().x + ";" + _camera.getPosition().y +
";" + _camera.getPosition().z);
Log.d("JpctEngine", "Valeur camera : " + _camera.getXAxis().x + ";" + _camera.getYAxis().x +
";" + _camera.getZAxis().x);

float x = matrix[12]; float y = matrix[13]; float z = matrix[14];
matrix[12] = matrix[13] = matrix[14] = 0;
_cameraMat.setDump(matrix);
_camera.setBack(_cameraMat);
_camera.setPosition(x, y, z);

Log.d("JpctEngine", "Valeur camera : " + _camera.getPosition().x + ";" + _camera.getPosition().y +
";" + _camera.getPosition().z);
Log.d("JpctEngine", "Valeur camera : " + _camera.getXAxis().x + ";" + _camera.getYAxis().x +
";" + _camera.getZAxis().x);


It is strange because getPosition and get*Axis give the same results for both but the separated one doesn't place the camera well... And still no picking...

From what you were writing there, it is a problem with the getPosition that returns {0.0,0.0,0.0}.

Would it be possible then to extends Camera and Overriding setPosition/getPosition to make the reproject2D3DWS and calcMinDistanceAndObject3D works (assuming it uses getPosition to work) ?

I can't test it now as I don't have my source right now, but i'll try tomorrow if you don't tell me it is useless before