How to get world coordinates from mouse coordinates

[tpistor]

I'm not trying to "pick" anything.  I just want to get the world coordinates (at plane z=0) from the mouse coordinates.  Seems like the method I need to use would be Interact2D.reproject2D3DWS, however it doesn't give me ability to choose z=0.  (it assumes z=1).  Even with the assumption of z=1, the result seems incorrect.

From other examples in the forums it seems that the result of reproject2D3DWS gives a "direction" and not a "position".

Here's my code:

    public void mouseMoved(MouseEvent e) {
        double x=currentMouseX=e.getX();
        double y=currentMouseY=e.getY();
        System.out.printf("Mouse screen: %d %d\n",currentMouseX,currentMouseY);
        System.out.println("Mouse position: "+Interact2D.reproject2D3DWS(world.getCamera(), buffer, currentMouseX, currentMouseY).toString());
}

It just gives me nonsense.

Other information:
I adjusted my camera position to be at (6,6,6) with camera orientation pointing to the origin, and camera "up" being the vector normal to the camera orientation such that "z" is up.

Tom

[tpistor]

One thing I tried was to multiply the mouse coordinates by 2.  I think I need to do this, but it didn't solve my problem.

For whatever reason, the project3D2D works fine.  Given a point in 3D, I can figure out where it is on the 2D screen. 

I was thinking I could do the mapping from 2D to 3D myself (knowing the mapping of the origin, a point at 1,0,0 and a point at 0,1,0), but the perspective rendering make it difficult.

Let me try to better explain what I'm trying to do:

Suppose I have a rectangle in the z=0 plane with the 4 points:  (0,0,0),(1,0,0),(1,1,0),(0,1,0).  (in world space)
I could be viewing this rectangle from any direction.  (example, put the camera at position (1,1,1) pointing to the origin)

When the mouse moves over this rectangle, I want to know the coordinates within the rectangle.  (ie. map a point in on the 2D screen to a point in the 3D world space at z=0).

[EgonOlsen]

The question in your case is rather where the z=0-plane is located in camera space and if it's in parallel to the view plane, because otherwise, it won't work anyway using this approach. If that  is know, you can use the reproject-methods without the WS and multiply with the inverse camera matrix afterwards. You should be able to find this code in the forum, because it was the older way of doing picking, i.e. before the ...WS method was introduced.
A more flexible approach would be to use the "normal" calcMinDistance-approach from the wiki. You can calculate the point in world space simply by multiplying the direction vector with that calculated distance. That will work for all camera positions and angles.
BTW: The vector returned by reproject is actually a position vector, but when viewed in this context, it can also be seen as a direction vector from the camera's origin to the point.

[tpistor]

Thanks for the quick reply.  I'm still confused.  In my situation it is possible that there isn't even any objects at z=0 yet I still want to know what the coordinate is.  (ie. the calcMinDistance wouldn't work because there is no object getting hit by the ray).

Are you saying that the reproject2D3DWS method returns a vector who's direction and length is from camera origin to the mouse position (assuming the mouse position is at z=1)?  If this is true, maybe somehow I could use this information to calculation where the mouse is at z=0?

[EgonOlsen]

The non-WS methods actually return a position vector. But because the camera is always located at (0,0,0), it can be viewed as a direction vector in camera space too. Anyway, back to your problem. I'm not sure what exactly you mean with z=0. If you are in camera space, you can't work with that plane, because no projection is defined for it (division by zero!). You might have an object lying in the z=0 plane in world space(!) but then, you don't have to calculate your vector with z=0 but with z=-camera.z. I.e. if the camera has moved 10 units back from your object in the z=0 plane, you can do something like:

Code Select Expand


SimpleVector pos=reproject2D3D(camera, buffer, mouseX, mouseY, 10);


However, this is valid only if the camera doesn't rotate. In that case, your object doesn't lie within the z=-camera.z plane any longer (in camera space) and this simple approach doesn't work anymore.

[tpistor]

My object is a square in the z=0 plane.  (as described in previous post).  My camera is at some arbitrary position (cx,cy,cz).

When I look at this square on the screen, it looks like a diamond shape (in general).

As I move the mouse around on the screen, I want to know where I am in the rectangle.  ie. I want the mapping from 2D mouse coordinates to the 3D world coordinates at z=0.

Since I understand project3D2D method (but don't  understand your reproject2D3D) I am able to accomplish what I want with the following algorithm:

1)  initialize my "guess" of world coordinates to be:  worldGuess=(0,0,0)
2)  calculate the screen position of my "guess" using project3D2D(worldGuess)
3)  calculate the error in screen position between my "guess" and my actual mouse position
3a) if error is less than 1 pixel then I'm done
4)  calculate the gradients in the screen position for "world delta" vectors (0.001,0,0) and (0,0.001,0)  (ie. figure out how my screen coordinates change for a small movement in the world coordinates)
5)  refine my guess based on the gradients and the error
6)  goto 2)

ie. I used Newton's method to find the solution
ie. successive approximation

This works, but is obviously not optimal (because it iterates).

I think the reproject2D3D should be able to do it somehow.  I just don't understand how to do it.

[EgonOlsen]

If i understand this correctly, you just have to find out where in camera space that z=0 plane is located (i.e. something like square.getTransformedCenter().z). Feed that z-value into the reproject2D3D(...) method mentioned above and you should be fine...as long as the square lies in that z=xxx plane in camera space.
Another idea (if you really need the positions outside of the square) is to use a larger square in the same plane that covers the whole screen and is enabled only for picking and pick on that one like described in the wiki. However, as long as you rotate around z only, i.e. don't rotate the xy-plane in camera space, you should be fine with the first solution.

[tpistor]

I put a video describing my question here:

http://www.youtube.com/watch?v=9IcNArtiG_c

I don't know (or understand) where my square is in camera space.  I know that it is at z=0 in world space.  And I never change it (I only move the camera).

in the video, when you see the image moving, I'm only moving the camera position and orientation (not moving the axes which remain fixed in the z=0 plane in world space)

Note:  what I want is the inverse of the the project3D2D(x,y,z=0) function.  It maps a point (x,y,z=0) in the world space to the screen - for any camera position/orientation.  I want the inverse of this.

Sorry for the confusing description of my problem.

Tom

[EgonOlsen]

I can't watch the video with sound ATM, so i'm not 100% sure what's going on. However, it doesn't look like as if your xy-plane in world space is in parallel to the one in camera space, i.e. the reproject(...)-methods alone don't do the trick. As long as your points are all in one known plane, this is simply a set of 3 equations with 2 unknown (x and y in world space, because z is known to be the plane's z...0 in this case).
I made a little test case to show it. With this, you can place a cube in the z=0-plane by clicking the mouse. I hope that this helps:

Code Select Expand


import org.lwjgl.input.Mouse;

import com.threed.jpct.*;


public class PlanePicker
{
  private final static float Z_PLANE = 0;

  private World world;
  private FrameBuffer buffer;
  private Object3D cube;


  public static void main(String[] args)
    throws Exception
  {
    new PlanePicker().loop();
  }

  public PlanePicker()
    throws Exception
  {
    cube = Primitives.getCube(5);
    cube.compile();
    cube.build();

    world = new World();
    world.setAmbientLight(255, 255, 255);

    world.getCamera().setPosition(0, -100, -150);

    world.addObject(cube);
  }


  private void loop()
    throws Exception
  {
    buffer = new FrameBuffer(800, 600, FrameBuffer.SAMPLINGMODE_NORMAL);
    buffer.disableRenderer(IRenderer.RENDERER_SOFTWARE);
    buffer.enableRenderer(IRenderer.RENDERER_OPENGL);

    while (!org.lwjgl.opengl.Display.isCloseRequested())
    {
      SimpleVector newPos = pick();
      if (newPos != null)
      {
        cube.clearTranslation();
        cube.translate(newPos);
      }
      buffer.clear(java.awt.Color.BLACK);
      world.renderScene(buffer);
      world.draw(buffer);
      buffer.update();
      buffer.displayGLOnly();
      Thread.sleep(10);
    }
    buffer.disableRenderer(IRenderer.RENDERER_OPENGL);
    buffer.dispose();
    System.exit(0);
  }


  private SimpleVector pick()
  {
    if (Mouse.isButtonDown(0))
    {
      int x = Mouse.getX();
      int y = buffer.getOutputHeight() - Mouse.getY();

      SimpleVector dir = Interact2D.reproject2D3DWS(world.getCamera(), buffer, x, y).normalize();
      SimpleVector pos = world.getCamera().getPosition();

      float a = (Z_PLANE - pos.z) / dir.z;

      float xn = pos.x + a * dir.x;
      float yn = pos.y + a * dir.y;

      return new SimpleVector(xn, yn, Z_PLANE);
    }
    return null;
  }
}



[tpistor]

Thanks for your code.  I ran the program you wrote and saw that the cube was placed wherever my mouse was clicked.  I'm not sure if this solves my problem or not because I still don't understand it.  Where's the z=1 that the reproject2D3D assume fit in?  I don't see any reference to z=1 in the code.

I was able to accomplish what I wanted using the algorithm I described earlier (iteratively solving for the inverse of the project3D2D).

Thank you for your kind efforts, and thanks for being so quick to respond to the forum questions.

[EgonOlsen]

The z=1 is just the view plane in camera space. It has nothing to do with your plane and you shouldn't have to care about it. Your plane's position is determined by the Z_PLANE constant in that example.