Develop Reference

Trying to transform GTA V coordinates to CEF three.js coordinates

I'm using GTA V with modification which allows me to create multiplayer server. This modification displaying CEF in front of the game and gives me ability to show information for player using chromium.
My goal was to make an editor of objects that can be created with the GTA V engine. For this purpose I wanted to use TransformControl, built in the library three.js. To do this, I need to transform the coordinates coming from GTA V into three.js. Unfortunately, I got lost and currently don't know what's wrong with this.
Based on my research, I noticed that the axes in GTAV are based on the +X+Y+Z pattern, for which the answer in three.js is +X+Z-Y. I tried to translate these coordinates by doing this:
// GTA CODE
function toEditorVector(vector: Vector3): Vector3 {
const xV = 1, yV = -1, zV = 1;
return new Vector3(vector.x * xV, vector.z * zV, vector.y * yV);
}
The effect of this is as follows:
https://streamable.com/72qrui
The GTA system returns camera and object rotation in degrees. I convert them to radians before passing them to three.js.
Building a scene in three.js:
// CEF code
public startScene(fov: any, camPos: any, camRot: any, objPos: any): void {
this.stopScene();
this.objData.pos = objPos;
this.transform.renderer = new WebGLRenderer({
canvas: this.$refs.editorCanvas as HTMLCanvasElement,
alpha: true,
depth: true
});
this.transform.renderer.setPixelRatio(window.devicePixelRatio);
this.transform.renderer.setSize(window.innerWidth, window.innerHeight);
const aspect = window.innerWidth / window.innerHeight;
this.transform.cameraPerspective = new PerspectiveCamera(fov, aspect, 0.01, 50000);
this.transform.cameraPerspective.position.set(camPos.x, camPos.y, camPos.z);
this.transform.cameraPerspective.rotation.set(camRot.x, camRot.y, camRot.z);
this.transform.scene = new Scene();
this.transform.mesh = new Mesh();
this.transform.mesh.position.set(objPos.x, objPos.y, objPos.z);
this.transform.transformControls = new TransformControls(this.transform.cameraPerspective, this.transform.renderer.domElement);
const helper = new CameraHelper(this.transform.cameraPerspective);
this.transform.scene.add(this.transform.mesh);
this.transform.scene.add(helper);
this.transform.transformControls.attach(this.transform.mesh);
this.transform.scene.add(this.transform.transformControls);
this.renderTransform()
this.transform.transformControls.addEventListener("change", this.renderTransform);
}
and when the camera is updating, an event is send from GTAV to CEF:
// CEF code
public updateCamera(data: any): void {
this.camData = JSON.parse(data.detail);
if (this.transform.cameraPerspective) {
this.transform.cameraPerspective.position.set(this.camData.pos.x, this.camData.pos.y, this.camData.pos.z);
this.transform.cameraPerspective.rotation.set(this.camData.rot.x, this.camData.rot.y, this.camData.rot.z);
}
this.renderTransform();
}
Can anyone help me determine the cause of the problem? I think it's a problem with the camera rotation transformation, but I don't know what the correct one should look like.

Three.js - Place a dynamic 2D Canvas on a PlaneGeometry

so I have a school project where I have to remake a GameBoy. For this I wanted to create a GameBoy model with ThreeJS (i'm a beginner) and use a public repo of a GameBoy emulator in JavaScript. So I somewhat finished the GameBoy model (still need some some stuff to be added but i'll make it better later) and I decided to use this repo for the GameBoy emulator https://github.com/alexaladren/jsgameboy. This repo worked perfectly fine when it was given a canvas with an ID "display". But when I tried to change the canvas to the canvas I made in ThreeJS it doesn't display anything, here is the code of when I make the Canvas:
geometry = new THREE.PlaneGeometry( 0.55, 0.45, 0.1 );
for (let index = 0; index < 6; index++) {
let x2 = document.createElement("canvas");
let xc2 = x2.getContext("2d");
x2.width = 320;
x2.height = 288;
xc2.fillStyle = "rgba(0, 0, 200, 0.5)";
x2.style.id = 'display';
screenCanvas = x2;
xc2.fillRect(0, 0, x2.width, x2.height);
let tex2 = new THREE.CanvasTexture(x2);
screen.push(new THREE.MeshBasicMaterial({
map: tex2,
transparent:true,
opacity:0.3
}))
number++;
}
material = new THREE.MeshBasicMaterial({color: 0xA1A935});
mesh = new THREE.Mesh( geometry, screen );
mesh.position.y = 0.25;
mesh.position.z = 0.3;
group.add(mesh);
Here is the code I edited in the emulator where the default "display" canvas was mentioned:
if(window.gb != undefined){
clearInterval(gb.interval);
screenCanvas.getContext("2d").setTransform(1,0,0,1,0,0);
}
gb = new GameBoy(arraybuffer);
gb.displaycanvas = screenCanvas.getContext("2d");
screenCanvas.getContext("2d").scale(2,2);
The PlaneGeometry is correctly displayed (I also tried BoxGeometry but same results) but the game won't display on the Canvas I created.
My thoughts on why it doesn't work:
- Because the Canvas is created in ThreeJS it doesn't seem to be added to the DOM elements and probably to the already existing ThreeJS canvas?
- Maybe the canvas I created isn't updating? But I set it to a CanvasTexture so it should update?
Thank you for your help.
Update: Still looking into it but haven’t found a solution, been trying to find help in 3 different Discord servers that provide threejs Discord but no luck. I might have to make the gameboy static while the game is playing and put the game display ontop of the screen if I don’t find another solution.

(Three.JS) Transform controls for translation (Size Issue)

I am working on customizing the transform controls available in three.js for my project.
I have already changed the rotation part and now working on translation part.
if you notice in translation Gizmo, there is a XYZ octahedron in the center. I have removed all other planes and arrows and wrote all functionality only on that center mesh, which is working fine.
Now I am only stuck at one small problem, that is the size and position of that controller. I changed that Octahedron to boxGeometry and writing the code to make the size of that controller to be exact size of the selected object. for that I get the idea to make the size of the controller, same as the boxHelper size, which act as outline of object.
when I tried this logic in sample code, where I created a box, and getting the size of box helper and creating another box of same size, it was working fine. but when I am writing same code in threejs transform controls, result is not the same.
below is the geometry code init
XYZ: [[ new THREE.Mesh( new THREE.BoxGeometry( 0.1, 0.1, 0.1 ), pickerMaterial )]],
then I am getting the size of box3 when attaching to any object
this.addBoxHelper = function () {
this.removeBoxHelper();
if(this.object.box3) {
**this.object.box3.getSize(selectionBoxSize);**
console.log(selectionBoxSize)
this.objectBoxHelper = new THREE.Box3Helper(this.object.box3, 0xffff00);
this.objectBoxHelper.canSelect = function () {
return false;
}
this.object.add(this.objectBoxHelper);
}
}
then below is my update function of transform controls
this.update = function () {
if ( scope.object === undefined ) return;
scope.object.updateMatrixWorld();
worldPosition.setFromMatrixPosition( scope.object.matrixWorld );
worldRotation.setFromRotationMatrix( tempMatrix.extractRotation( scope.object.matrixWorld ) );
scope.object.box3.getSize(selectionBoxSize);
scope.object.getWorldPosition(selectionBoxPos);
camera.updateMatrixWorld();
camPosition.setFromMatrixPosition( camera.matrixWorld );
camRotation.setFromRotationMatrix( tempMatrix.extractRotation( camera.matrixWorld ) );
**scaleT = selectionBoxSize;**
//below three lines are for dynamic size change based on camera position..for
//next level functionality
//scaleT.x = worldPosition.distanceTo( camPosition ) / 6 * selectionBoxSize.x;
//scaleT.y = worldPosition.distanceTo( camPosition ) / 6 * selectionBoxSize.y;
//scaleT.z = worldPosition.distanceTo( camPosition ) / 6 * selectionBoxSize.z;
this.position.copy( selectionBoxPos );
this.scale.set( scaleT.x, scaleT.y, scaleT.z);
this.updateMatrixWorld();
below is the output of console log
TransformControls.js:526 Vector3 {x: 10.020332336425781, y: 2.621583938598633, z: 3.503500819206238}
TransformControls.js:601 Vector3 {x: 10.020332336425781, y: 2.621583938598633, z: 3.503500819206238}
as you can see, the scale is the same, but the result is different. see the result in below images.
as you see in images, that red color box at bottom is translation controller but smaller than selection box.
another issue is that pivot of my objects are at bottom, and I want this controller to come at the center of the selection box, that is also not happening with getCenter method of box3.
Please help!! let me know if I am unclear in explaining the issue

If you are getting the bounding box of the object from its geometry, that will be wrong because it doesn't take the objects transform into account. You have to use box3.setFromObject (yourObject) instead. does that help?

Three.js - dynamic skeleton animation with BVH and JSON-model (not baked)

Brief description: How can a JSON-character-model be dynamically animated using Three.js and a BVH-motion capture-file? As in: The animation is not baked into the JSON-file but rather loaded and parsed from the BVH-file using BVHLoader.js
I got as far as loading the BVH-file, attaching its data to the skeletonHelper, extracting the bone-positions, animating some boxes based on their position (for testing pursposes) and loading a (hopefully?) rigged JSON-model. See the demo here: https://www.patrik-huebner.com/repo/JSON-BVH/
See a GIF of the animation here: https://www.patrik-huebner.com/repo/JSON-BVH/demo.gif (sadly, I can't yet post images)
I did quite some extensive research on this topic and while there are quite a lot of helpful tips[1] on stackoverflow on exporting a mesh from Blender to JSON using the Three.js-exporter and baking in animations, I was not able to figure out how to
a) set up a JSON-file / rigged model / it's skeleton/bones so that I can dynamically animate it (I used a rigged, low-poly from blendswap.com/blends/view/21948). Are there any recommended settings? In order to be able to load the file at all I have to check the "export scene"-option which I'm not quite certain why that should be necessary.
b) map the BVH-data to the JSON-model
If you have any advice and / or code-samples I can use - or at least some resources or keywords that I can research, I'd be most thankful !
Using the supplied BVHLoader.js I am able to load the animation and attach it to the skeletonHelper
var loader = new THREE.BVHLoader();
loader.load( "models/pirouette.bvh", function( result ) {
skeletonHelper = new THREE.SkeletonHelper( result.skeleton.bones[0] );
skeletonHelper.skeleton = result.skeleton; // allow animation mixer to bind to SkeletonHelper directly
boneContainer = new THREE.Group();
boneContainer.add( result.skeleton.bones[ 0 ] );
scene.add( skeletonHelper );
scene.add( boneContainer );
mixer = new THREE.AnimationMixer( skeletonHelper );
mixer.clipAction( result.clip ).setEffectiveWeight( 1.0 ).play();
} );
Though I am rather certain this is not how its done: I can access the Vector3-data of the skeletonHelper and animate some test-cubes based on those positions (compare code-sample above and my supplied GIF-animation)
for (var i = 0; i < skeletonHelper.bones.length; i++) {
var myPos = skeletonHelper.bones[i].getWorldPosition();
cubeGroup.children[i].position.x = myPos.x;
cubeGroup.children[i].position.y = myPos.y;
cubeGroup.children[i].position.z = myPos.z;
}
Finally, I load the JSON-file and add it to the scene using THREE.ObjectLoader (as the THREE.JSONLoader would not accept my exported model)
// Load the rigged JSON-file based on this character http://www.blendswap.com/blends/view/21948
// converted to JSON with the Three.json-export-plugin
var loader = new THREE.ObjectLoader();
var s = 50;
loader.load(
"models/lowPoly-rigged.json",
function ( obj ) {
var material = new THREE.MeshLambertMaterial({color: 0xff0000});
scene.add( obj );
obj.position.y = 0;
obj.children[0].material = material;
obj.scale.set(s,s,s);
},
function ( xhr ) {
console.log( (xhr.loaded / xhr.total * 100) + '% loaded' );
},
function ( xhr ) {
console.error( 'An error happened' );
}
);
But now: How do I animate the model using the loaded BVH-data. Do I have to match the amount of bones from the JSON-model to the BVH? Can I just supply some data to the model and have it happen "magically"? I have seen some online demos that load BVH-files dynamically and pass the data to a model[2]
[1] Stuff I did research (sadly, I can't post more links) that did not help, is not quite specific to what I am looking to achieve:
stackoverflow.com/questions/20433474/dynamic-bones-animation-in-three-js
stackoverflow.com/questions/29970791/animate-makehuman-character-with-bvh-using-three-js
stackoverflow.com/questions/40719572/three-js-wrong-bones-orientation (this did look the most helpful as it is actually about combining a loaded BVH-file and a model but researching/implementing the code with my own model lead to error-messages about the material which points to some misunderstanding of mine in the export-process from Blender to JSON)
[2] Some resources that were able to handle BVH-files and could attach those animations to a model:
BVH Player by Aki A. on chromeExperiments: chromeexperiments.com/experiment/bvh-player
lo-th.github.io/root/blending2/index_bvh.html
Again, thanks a lot for any input!

Intersection within Object3D

I have some objects added to an Object3D (for grouping elements) and I'm trying to detect clicks on it.
My scene has a size of 600x400, my camera is within a three-object and my event handler code looks like below:
function onDocumentMouseDown( event ) {
event.preventDefault();
var mouse = {};
mouse.x = ( event.clientX / 600 ) * 2 - 1;
mouse.y = - ( event.clientY / 400 ) * 2 + 1;
var vector = new THREE.Vector3( mouse.x, mouse.y, 1 );
projector.unprojectVector( vector, three.camera );
var ray = new THREE.Ray( three.camera.position, vector.subSelf( three.camera.position ).normalize() );
var intersects = ray.intersectObjects( group.children );
alert(intersects.length);
[...]
}
Actually I'm alerting the count of intersected objects. But it stays zero. It couldn't find any intersected objects. I've already played a bit aroud with the x, y and z values of my projection vector - without success.
I've added a stripped down sample for demonstrating this issue on jsfiddle. Maybe someone has a short hint for me what goes wrong with it?

In your fiddle, because you are calling THREE.SceneUtils.createMultiMaterialObject( ), which creates a hierarchical structure, you need to add the recursive flag to ray.intersectObjects().
var intersects = ray.intersectObjects( group.children, true );
EDiT: ray is now an instance of THREE.Raycaster -- not THREE.Ray.
three.js r.58

I had the same problem and WestLangley's answer provides the answer. Great job! For anybody struggling with mouse selection of objects grouped in Object3D wrapper too, I am posting my own solution.
First, I created an array of objects that are selectable - I hope this also saves some performance, as RayCaster doesnt need to search all objects in the scene, but only those you wish to respond to selection. I also attached this array to scene object directly (solely for the fact that it is already accessible from most parts of my app)
scene.selectable = [];
Next step is to push all objects that you wish to make selectable into this array. You will insert only meshes/sprites/etc from your group, not the whole group. Only last line is important here:
var myWrapper = new THREE.Object3D();
var myObject = new THREE.Mesh( something );
myWrapper.add( myObject );
scene.add ( myWrapper );
scene.selectable.push( myObject );
And lastly in your mouse selection routine you will call raycaster like this:
var intersects = ray.intersectObjects( scene.selectable );