[Tips] Android, augmented reality 3D with JPCT + Camera.

[dl.zerocool]

All the code provide here is free to use and I'm not responsible for what you use it for.

I received a question that I found very interesting to share with everybody.

There is very few information about this subject on android and how to setup a correct working layout.

So in this topic, I'll answer the simple question : How to use Android Camera and JPCT-AE as a render that overlays the camera.
(Augmented reality concept)

== ALL THE SOURCES PROVIDED IN CODE QUOTES ARE NOT COMPLETE ! ==
You have to code your own engines around it to get it fully functional.

First we need to set up an XML layout.
Our minimum requirement is a glSurfaceView that's where we will draw 3D(JPCT engine),
and a SurfaceView to draw the camera preview.

Code Select Expand

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:orientation="vertical" android:layout_width="fill_parent"
android:layout_height="fill_parent">

<android.opengl.GLSurfaceView android:id="@+id/glsurfaceview"
android:layout_width="fill_parent" android:layout_height="fill_parent" />

<SurfaceView android:id="@+id/surface_camera"
android:layout_width="fill_parent" android:layout_height="fill_parent"
android:layout_centerInParent="true" android:keepScreenOn="true" />
</FrameLayout>


This is to Initialize the window and the glSurfaceView.

Code Select Expand


       // It talks from itself, please refer to android developer documentation.
getWindow().setFormat(PixelFormat.TRANSLUCENT);

       // Fullscreen is not necessary... it's up to you.
getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
WindowManager.LayoutParams.FLAG_FULLSCREEN);

setContentView(R.layout.THE_XML_LAYOUT_CREATED_BEFORE);
       // attach our glSurfaceView to the one in the XML file.
glSurfaceView = (GLSurfaceView) findViewById(R.id.glsurfaceview);



Now let's create the camera and the engine.
This is an example of my own code, so perhaps it won't fill exactly your needs,
but you can be inspired by this one.


The following code is pretty easy to understand,
I create a new camera and I give a render to my glSurfaceView
and of course set the Translucent window  (8888) pixel format and depth buffer to it.
(Without that your glSurfaceView will not support alpha channel and you will not see the camera layer.)

So basically :
1) Create the camera view.
2) Set up the glSurfaceView.
3) Set a Render to glSurfaceView.
4) Set the correct pixelformat to the glSurfaceView holder.

Code Select Expand


try{
cameraView = new CameraView(this.getApplicationContext(),
(SurfaceView) findViewById(R.id.surface_camera), imageCaptureCallback);
}
catch(Exception e){
   e.printStackTrace();
}
// Translucent window 8888 pixel format and depth buffer
glSurfaceView.setEGLConfigChooser(8, 8, 8, 8, 16, 0);

       // GLEngine is a class I design to interact with JPCT and with all the basic function needed,
       // create a world, render it, OnDrawFrame event etc.
       glEngine = new GLEngine(getResources());
glSurfaceView.setRenderer(glEngine);

game = new Game(glEngine, (ImageView) findViewById(R.id.animation_screen), getResources(), this
.getBaseContext());

// Use a surface format with an Alpha channel:
glSurfaceView.getHolder().setFormat(PixelFormat.TRANSLUCENT);

// Start game
game.start();


Here is my CameraView class :

Code Select Expand

package com.dlcideas.ARescue.Camera;

import java.io.IOException;

import com.threed.jpct.Logger;

import android.content.Context;
import android.hardware.Camera;
import android.view.SurfaceHolder;
import android.view.SurfaceView;

public class CameraView extends SurfaceView implements SurfaceHolder.Callback {
   /**
    * Create the cameraView and
    *
    * @param context
    * @param surfaceView
    */
   public CameraView(Context context, SurfaceView surfaceView,
   ImageCaptureCallback imageCaptureCallback) {
super(context);

// Install a SurfaceHolder.Callback so we get notified when the
// underlying surface is created and destroyed.
previewHolder = surfaceView.getHolder();
previewHolder.addCallback(this);
previewHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
//previewHolder.setType(SurfaceHolder.SURFACE_TYPE_NORMAL);


// Hold the reference of the caputreCallback (null yet, will be changed
// on SurfaceChanged).
this.imageCaptureCallback = imageCaptureCallback;
   }

   /**
    * Initialize the hardware camera. holder The holder
    */
   public void surfaceCreated(SurfaceHolder holder) {
camera = Camera.open();
try {
   camera.setPreviewDisplay(holder);
} catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
}
   }

   /**
    *
    */
   public void surfaceDestroyed(SurfaceHolder holder) {
this.onStop();
   }

   public void surfaceChanged(SurfaceHolder holder, int format, int width,
   int height) {
if (previewRunning)
   camera.stopPreview();

Camera.Parameters p = camera.getParameters();
p.setPreviewSize(width, height);
// camera.setParameters(p);

try {
   camera.setPreviewDisplay(holder);
} catch (IOException e) {
   e.printStackTrace();
}
previewRunning = true;
Logger.log("camera callback huhihihihih", Logger.MESSAGE);
camera.startPreview();
imageCaptureCallback = new ImageCaptureCallback(camera, width, height);
//camera.startPreview();


   }

   public void onStop() {
// Surface will be destroyed when we return, so stop the preview.
// Because the CameraDevice object is not a shared resource, it's very
// important to release it when the activity is paused.
imageCaptureCallback.stopImageProcessing();
camera.setPreviewCallback(null);
camera.stopPreview();
previewRunning = false;
camera.release();
   }

   public void onResume() {
camera = Camera.open();
camera.setPreviewCallback(imageCaptureCallback);
previewRunning = true;
   }

   private Camera camera;
   private SurfaceHolder previewHolder;
   private boolean previewRunning;
   private ImageCaptureCallback imageCaptureCallback;

}


[Darkflame]

Thanks for the help, I'm sure this will be usefull to a few others as well.

The vital bit I needed was...


      mGLView.getHolder().setFormat(PixelFormat.TRANSLUCENT);

and to change my;

Code Select Expand


            mGLView.setEGLConfigChooser(new GLSurfaceView.EGLConfigChooser() {
public EGLConfig chooseConfig(EGL10 egl, EGLDisplay display) {
// Ensure that we get a 16bit framebuffer. Otherwise, we'll fall
// back to Pixelflinger on some device (read: Samsung I7500)
int[] attributes = new int[] { EGL10.EGL_DEPTH_SIZE, 16, EGL10.EGL_NONE };
EGLConfig[] configs = new EGLConfig[1];
int[] result = new int[1];
egl.eglChooseConfig(display, attributes, configs, 1, result);
return configs[0];
}
});

to

Code Select Expand

mGLView.setEGLConfigChooser(8, 8, 8, 8, 16, 0);

      
Only thing is, thats clearly a fixed solution. Id be worried about other device compatibility.

[fir3d]

I think I saw a while back of how its possible to use camera tracking if you print out a page, then it will look like the object is sitting on the page. Any idea how to do that 

[Darkflame]

You'll need a specific library for that as its quite complex work.
If you google around you should be able to find some open source projects for it though. Theres a lot of rapid AR development at the moment, there's tones of open source projects.
---

Anyone know a good way to sycn camera angle in the code to the real camera's angle on the phone?

I know how to read the sensors and get (rough) angles in the x/y/z from both magnetic and gravitational sensors.

Not sure how to turn this into a SimpleVector for my camera though.
I'm guessing maths is involved

[paulscode]

I know how to read the sensors and get (rough) angles in the x/y/z from both magnetic and gravitational sensors.

Now if only you could get exact GPS coordinates for the phone as well - with that and the angles, you could, for example, place a secret clue somewhere, and create a real-world treasure hunt game that people use their androids to play..

[Darkflame]

Thats exactly my goal.

Or, rather, allowing anyone to place messages tied to real locations and share them with anyone else

This is why I'm using Wave-servers as a back-end, it lets people have a kinda "social" AR. They can share their posts with either individuals, groups, or the public at large.
I already got the system working on PC's with a google map style client;
http://arwave.org/ (see video)
That was more or less to prove the concept. (though as its made in qt, porting later to nokia phones shouldn't be too hard).

Now I want to make a full AR one.

[Darkflame]

Anyone know how to use the orientation sensors to set the JPCT camera to corrispond?

Ive been looking at the sourcecode of Mixare;
http://code.google.com/p/mixare/source/checkout
for guidelines.

The problem comes that they are using their own engine, and thus the rotations arnt in the needed format.

Heres what Ive tried;

Code Select Expand

@Override
public void onSensorChanged(SensorEvent evt) {
try
{


if (evt.sensor.getType() == Sensor.TYPE_ACCELEROMETER) {
grav[0] = evt.values[0];
grav[1] = evt.values[1];
grav[2] = evt.values[2];

arView.postInvalidate();
} else if (evt.sensor.getType() == Sensor.TYPE_MAGNETIC_FIELD) {
mag[0] = evt.values[0];
mag[1] = evt.values[1];
mag[2] = evt.values[2];

arView.postInvalidate();
}

SensorManager.getRotationMatrix(RTmp, I, grav, mag);
//SensorManager.remapCoordinateSystem(RTmp, SensorManager.AXIS_X, SensorManager.AXIS_MINUS_Z, Rt);

Rt=RTmp;

tempR.setRow(0, Rt[0], Rt[1], Rt[2],0);
tempR.setRow(1, Rt[3], Rt[4], Rt[5],0);
tempR.setRow(2, Rt[6], Rt[7], Rt[8],0);
tempR.setRow(3, 0, 0, 0,1);


    Log.i("--", Rt[0] +" "+ Rt[1] +" "+ Rt[2]);
    Log.i("--", Rt[3] +" "+ Rt[4] +" "+ Rt[5]);
    Log.i("--", Rt[6] +" "+ Rt[7] +" "+ Rt[8]);
     
    arView.setCameraOrentation(tempR);


}
        catch (Exception ex)
        {
        Log.e("Sensor", "ProcessingError", ex);
        }
}



The function "setCameraOrentation" just leads to a

world.getCamera().setBack(RotMatrix);

Where RotMatrix is the Matrix passed to it.

What I'm not sure of is how Androids SensorManager.getRotationMatrix()  matrix corresponds to the JCPTs "setBack" matrix :-/

I know one is 3x3 and the other is 4x4...but I think I dealt with that correct, so I'm not sure whats wrong now :?

The jcpt camera moves on rotations, but clearly not correctly. (or any simple angular displacement).

[Darkflame]

Theres also this code here which I've tried with a similar lack of success;
http://mysticlakesoftware.blogspot.com/2009/07/sensor-accelerometer-magnetics.html

This code features a filtering function, which is nice, but I still cant match the v

• output to the Jpct camera.
  Also this code seems very slow compared to the above.
  

[EgonOlsen]

If the matrix from Android is similar to what OpenGL uses, it's most likely column major, while jPCT's matrices are row major. You have to convert them by making rows to cols. The easiest way is to create a float[16]-array for Matrix.setDump() and fill it accordingly. So that

Code Select Expand


a d g
b e h
c f j


becomes

Code Select Expand


a b c 0
d e f 0
g h j 0
0 0 0 1


In addition, you have to convert between the coordinate systems. You can either do this by rotating the matrix 90° around the x-axis or by negating the second and third column of the matrix (can be done when filling the array anyway). The next release will include a method that does this conversion.

[Darkflame]

I thought that at first but.....

Each matrix is returned either as a 3x3 or 4x4 row-major matrix depending on the length of the passed array:
If the array length is 16:
  /  M[ 0]   M[ 1]   M[ 2]   M[ 3]  \
  |  M[ 4]   M[ 5]   M[ 6]   M[ 7]  |
  |  M[ 8]   M[ 9]   M[10]   M[11]  |
  \  M[12]   M[13]   M[14]   M[15]  /
This matrix is ready to be used by OpenGL ES's glLoadMatrixf(float[], int).
Note that because OpenGL matrices are column-major matrices you must transpose the matrix before using it. However, since the matrix is a rotation matrix, its transpose is also its inverse, conveniently, it is often the inverse of the rotation that is needed for rendering; it can therefore be used with OpenGL ES directly.
Also note that the returned matrices always have this form:
  /  M[ 0]   M[ 1]   M[ 2]   0  \
  |  M[ 4]   M[ 5]   M[ 6]   0  |
  |  M[ 8]   M[ 9]   M[10]   0  |
  \      0       0       0   1  /
If the array length is 9:
  /  M[ 0]   M[ 1]   M[ 2]  \
  |  M[ 3]   M[ 4]   M[ 5]  |
  \  M[ 6]   M[ 7]   M[ 8]  /

so that's right isn't it?

I guess its the varying co-ordinate system causing the problem?

Also, where's the most efficient place to put the
world.getCamera().setBack(CameraMatrix);
?

Camera rotation is only every second or more at the moment, so I'm trying to work out what part of my code is slowing it down.

[EgonOlsen]

Then just try to apply a rotateX((float)Math.PI) on the matrix (the 90° i wrote in my former post are of course wrong, it has to 180). Or maybe you have to invert it in addition to be useful? Keep in mind that a camera transformation is actually an inverse world transformation. How are you creating the jPCT Matrix? Have you ensured that the result really looks like

   /  M[ 0]   M[ 1]   M[ 2]   0  \
   |  M[ 4]   M[ 5]   M[ 6]   0  |
   |  M[ 8]   M[ 9]   M[10]   0  |
   \      0       0       0   1  /

?

The place where you set the Matrix shouldn't matter. It's not expensive. Just try to avoid object creation where possible, i.e. don't create a new Matrix each frame if possible.

[Darkflame]

Its basically;

Code Select Expand


SensorManager.getRotationMatrix(Rt, I, accels, mags);

Matrix tempR = new Matrix();

tempR.setRow(0, Rt[0], Rt[1], Rt[2],0);
tempR.setRow(1, Rt[3], Rt[4], Rt[5],0);
tempR.setRow(2, Rt[6], Rt[7], Rt[8],0);
tempR.setRow(3, 0, 0, 0,1);


tempR.rotateX((float)Math.PI);

arView.setCameraOrentation(tempR);


Is this correct?
Rt[] is supposed to be a 3x3 returned by the get RotationMatrix according to google's documentation. (I defined it as Rt[9]).


Something is still very wrong unfortunately. I haven't inverted it yet, but when laid flat on the table using the above code I get this;

The diagonal angle is odd no? (this is basically a simplified version of the demo scene)


[attachment deleted by admin]

[dl.zerocool]

I know how to read the sensors and get (rough) angles in the x/y/z from both magnetic and gravitational sensors.

Now if only you could get exact GPS coordinates for the phone as well - with that and the angles, you could, for example, place a secret clue somewhere, and create a real-world treasure hunt game that people use their androids to play..

Yes that would bee a cool idea But another project XD

[Darkflame]

Not really, if I can get my system working, Treasure hunts will be possible

[Darkflame]

Ok, I decided to give this another go, and I'm determined to get it right
I'm going to go step by step this time and be as methodical as possible.

I'm currently just bliting a rotation matrix onto the screen.
The source of which is "RT" from;

SensorManager.getRotationMatrix(Rt, I, accels, mags);

Where accels and mags are from the sensors.

Question;

How often should this update? It seems to be far too slow to be any use.
I mean, once every 2-3 seconds it gives a new matrix out, even though its triggered hundreds of times inbetween.

Does anyone successfully use getRotationMatrix from here? What speeds does it update? I'm using a HTC Legend, Android 2.1.