How I can throw a ray between camera (my eyes) and the mouse cursor on a VR Simulation?

Question

I'm trying to click on VR simulation mode in the browser (Will also be used on mobile), and show a yellow ray from the camera (my eyes/center of window) to the mouse cursor, but I can't get the right direction/position.

public function onMouseDown(e:Event): Void
{
    // The camera in VR was created automatically by the threejs engine.
    var camera:WebXRArrayCamera = renderer.xr.getCamera();

    // x and y will get a value from (-1 to +1)
    // I have no clear with z must be in -0.5 and not another value.
    
    var x:Float =  (event.clientX / canvas.parentElement.offsetWidth ) * 2 - 1;
    var y:Float = -(event.clientY / canvas.parentElement.offsetHeight) * 2 + 1;
    var z:Float = -0.5;
    
    var mouse:Vector3 = new Vector3(x, y, z);
    
    var origin:Vector3 = new Vector3();
        origin.copy(camera.position);
    
    var direction:Vector3 = new Vector3();
        direction.copy(mouse);
    
    var arrow:ArrowHelper = new ArrowHelper(direction, origin, 5, 0xffff00);
    scene.add(arrow);
}

If I click in the center of window, it's more or less correct, but if I move my cursor mouse, for example, to the right, the result is wrong as you can see in the screenshoot https://ibb.co/TBnNtnb

https://jsfiddle.net/0t2yczh7/

The yellow arrow shuld be exactly centered under my mouse cursor and it is not. If I go more to the right the offset is bigger.

What I am doing wrong?

By other hand, the unit of measure in my world is the meter, so the values for x and y (-1 to +1) I'm not sure if is wrong. How I can translate pixels (canvas.parentElement.offsetWidth) to meters?? For example, something like 1920x1080 to 7.55x4.25, depending of the Fov.

Answer 1

You can utilize raycasters

Here is how you would set it up:

const raycaster = new THREE.Raycaster();
const pointer = new THREE.Vector2();

function onPointerMove( event ) {
    pointer.x = ( event.clientX / window.innerWidth ) * 2 - 1;
    pointer.y = - ( event.clientY / window.innerHeight ) * 2 + 1;
}

In your animation loop, you would need to add the following:

function render(){
    ...
    raycaster.setFromCamera( pointer, camera );

    const intersects = raycaster.intersectObjects( scene.children );
    for ( let i = 0; i < intersects.length; i ++ ) {
        //intersects[i] was intersected, do something.
    }
}

Lastly, you need to add an event listener:

window.addEventListener( 'pointermove', onPointerMove ); //When the pointer moves, cast a new ray.

That should do it~

Answer 2

I mean raycast directly in your browser with the plugin that simulates the VR Environment.

I think it could be great don't have to use the VR helmet everytime compile and I want to try/debug something. I would like to do directly from my browser with the WebXR emulator or directly from my phone, that it will works like a VR helmet but without any hand controller.

You can install the plugin from chrome from here

https://chrome.google.com/webstore/detail/webxr-api-emulator/mjddjgeghkdijejnciaefnkjmkafnnje

and after installed, take a look to my fiddle, you will understand what I mean.

https://jsfiddle.net/foxfairlane/d3t7khvu/4/