Develop Reference

2D face tracking on Spark AR

I've been trying to make the face texture in a 2D canvas/plane move only along the X/Y axis, following the movements of the face without rotating, with the 2D background camera texture reflected accurately on top. Right now, when I connect the canvas to the face tracker I'm getting distorted scale, and the 2D plane rotates in 3D space. See below for the current canvas/camera texture/face tracker set-up. Manual scaling results in poor tracking.
Here is my code:
const FaceTracking = require('FaceTracking')
const Scene = require('Scene')
export const Diagnostics = require('Diagnostics');
// Locate the plane in the Scene
// Enable async/await in JS [part 1]
(async function () {
const [plane] = await Promise.all([
Scene.root.findFirst('blur_plane')
])
// Store a reference to a detected face
const face = FaceTracking.face(0)
// To access scene objects
const planeTransform = plane.transform
const faceTransform = face.cameraTransform
// const blurCanvas = Scene.root.find('canvas0');
// To access class properties
planeTransform.rotationX = faceTransform[0]
planeTransform.rotationY = faceTransform[0]
planeTransform.rotationZ = faceTransform[0]
})()
This is the current look:
I want the 375x667px canvas to look exactly like the camera layer beneath it, so that without adjustments to the camera texture the layer would not be visible.

Turns out Facebook has an example that deals with 2D movement but not scale:
https://sparkar.facebook.com/ar-studio/learn/reference/classes/facetrackingmodule

javascript games ThreeJS and Box2D conflicts?

I've been trying to experiment with box2d and threejs.
So box2d has a series of js iterations, I've been successful at using them so far in projects as well as threejs in others, but I'm finding when including the latest instance of threejs and box2dweb, threejs seems to be mis-performing when just close to box2dweb but maybe I'm missing something really simple, like a better way to load them in together, or section them off from one another?
I've tried a few iterations of the box2d js code now and I always seemed to run into the same problem with later versions of threejs and box2d together! - currently version 91 threejs.
The problem I'm seeing is very weird.
I'm really hoping someone from either the box2d camp or threejs camp can help me out with this one, please?
Below is a very simple example where I don't initialize anything to do with box2d, but just by having the library included theres problems and you can test by removing that resource, then it behaves like it should.
The below demo uses threejs 91 and box2dweb. It is supposed to every couple of seconds create a box or a simple sphere each with a random colour. Very simple demo, you will see the mesh type never changes and the colour seems to propagate across all mesh instances. However if you remove the box2dweb resource from the left tab then it functions absolutely fine, very odd :/
jsfiddle link here
class Main {
constructor(){
this._container = null;
this._scene = null;
this._camera = null;
this._renderer = null;
console.log('| Main |');
this.init();
}
init(){
this.initScene();
this.addBox(0, 0, 0);
this.animate();
}
initScene() {
this._container = document.getElementById('viewport');
this._scene = new THREE.Scene();
this._camera = new THREE.PerspectiveCamera(75, 600 / 400, 0.1, 1000);
this._camera.position.z = 15;
this._camera.position.y = -100;
this._camera.lookAt(new THREE.Vector3());
this._renderer = new THREE.WebGLRenderer({antialias:true});
this._renderer.setPixelRatio( 1 );
this._renderer.setSize( 600, 400 );
this._renderer.setClearColor( 0x000000, 1 );
this._container.appendChild( this._renderer.domElement );
}
addBox(x,y,z) {
var boxGeom = new THREE.BoxGeometry(5,5,5);
var sphereGeom = new THREE.SphereGeometry(2, 5, 5);
var colour = parseInt(Math.random()*999999);
var boxMat = new THREE.MeshBasicMaterial({color:colour});
var rand = parseInt(Math.random()*2);
var mesh = null;
if(rand == 1) {
mesh = new THREE.Mesh(boxGeom, boxMat);
}
else {
mesh = new THREE.Mesh(sphereGeom, boxMat);
}
this._scene.add(mesh);
mesh.position.x = x;
mesh.position.y = y;
mesh.position.z = z;
}
animate() {
requestAnimationFrame( this.animate.bind(this) );
this._renderer.render( this._scene, this._camera );
}
}
var main = new Main();
window.onload = main.init;
//add either a box or a sphere with a random colour every now and again
setInterval(function() {
main.addBox(((Math.random()*100)-50), ((Math.random()*100)-50), ((Math.random()*100)-50));
}.bind(this), 4000);
so the way im including the library locally is just a simple...
<script src="js/vendor/box2dweb.js"></script>
So just by including the box2d library threejs starts to act weird, I have tested this across multiple computers too and multiple version of both box2d (mainly box2dweb) and threejs.
So with later versions of threejs it seems to have some comflicts with box2d.
I found from research that most of the box2d conversions to js are sort of marked as not safe for thread conflicts.
Im not sure if this could be the cause.
I also found examples where people have successfully used box2d with threejs but the threejs is always quite an old version, however you can see exactly the same problems occurring in my example, when I update them.
So below is a demo I found and I wish I could credit the author, but here is a copy of the fiddle using threejs 49
jsfiddle here
.....and then below just swapping the resource of threejs from 49 to 91
jsfiddle here
its quite an odd one and maybe the two libraries just don't play together anymore but would be great if someone can help or has a working example of them working together on latest threejs version.
I have tried a lot of different box2d versions but always found the same problem, could this be a problem with conflicting libraries or unsafe threads?
but also tried linking to the resource include in the fiddles provided.
Any help really appreciated!!

Where does TrackballControls come from?

I am trying to have some camera control in a threejs scene.
I looked at this example and it seems that it is completely handled with those lines :
controls = new THREE.TrackballControls( camera );
controls.rotateSpeed = 1.0;
controls.zoomSpeed = 1.2;
controls.panSpeed = 0.8;
controls.noZoom = false;
controls.noPan = false;
controls.staticMoving = true;
controls.dynamicDampingFactor = 0.3;
Those lines use THREE.TrackballControls which comes from js/controls/TrackballControls.js
My question is : what exactly is TrackballControls.js? I cannot find it in the threejs download bundle. Is it an extension? Where can I find it? (Apart from taking it directly from the example's file)

TrackballControls.js is in the jsm/controls sub-directory of the examples directory.
https://github.com/mrdoob/three.js/tree/master/examples/jsm/controls
It is part of the examples -- not the library. You must include it explicitly in your project. You are free to modify it to your liking.
You may also want to consider OrbitControls, which is appropriate if your scene has a natural "up" direction.
three.js r.147

I noticed that the TrackballControls linked by #WestLangley is much more slow than an old version used by this example.
Fiddle with new code: https://jsfiddle.net/vt8n6dcs/1/
Fiddle with old code: https://jsfiddle.net/vt8n6dcs/
I tested it with Firefox 41.0.2. No benchmarks, the performance degradation is quite evident, since when you start the rotation using the mouse, sometimes the image update lags. It happens also with the old version, but a lot less frequently. Not surprisingly, performance seems quite the same in Chrome 48.0.2564.82.
Furthermore mouse sensitivity is much lower. You have to move it a lot to get an appreciable effect on the image. This happens on both Firefox and Chrome.
The only problem I found with the old version is that the center of the commands are always set at the center of the page. You can fix it by using the code of the newer version for handleResize() function:
this.handleResize = function () {
if ( this.domElement === document ) {
this.screen.offsetLeft = 0;
this.screen.offsetTop = 0;
this.screen.width = window.innerWidth;
this.screen.height = window.innerHeight;
} else {
var box = this.domElement.getBoundingClientRect();
// adjustments come from similar code in the jquery offset() function
var d = this.domElement.ownerDocument.documentElement;
this.screen.offsetLeft = box.left + window.pageXOffset - d.clientLeft;
this.screen.offsetTop = box.top + window.pageYOffset - d.clientTop;
this.screen.width = box.width;
this.screen.height = box.height;
}
this.radius = ( this.screen.width + this.screen.height ) / 4;
};

You can import it directly from the example. Insert the following line into your HTML file's <head>:
<script src="https://threejs.org/examples/js/controls/TrackballControls.js"></script>
Here is a demo.

Three.js skinned animation mesh disappears when material skinning is true

I've exported an animated model from Blender which doesn't seem to have any issue instantiating. I'm able to create the THREE.Animation and model, but I was finding there was no animation. I realized I needed to set skinning true on each material, but when I do that the entire mesh goes missing.
Below is my (quick and messy) code trying to get everything to work.
function loadModel() {
var loader = new THREE.JSONLoader();
loader.load('assets/models/Robot.js', function(geom, mat) {
_mesh = new THREE.Object3D();
_scene.add(_mesh);
geom.computeBoundingBox();
ensureLoop(geom.animation);
THREE.AnimationHandler.add(geom.animation);
for (var i = 0; i < mat.length; i++) {
var m = mat[i];
//m.skinning = true; <-- Uncommenting this makes the model disappear
//m.morphTargets = true; <-- This causes all sorts of WebGL warnings
m.wrapAround = true;
}
var mesh = new THREE.SkinnedMesh(geom, new THREE.MeshFaceMaterial(mat));
mesh.scale.set(400, 400, 400);
mesh.position.set(0, -200, 0);
mesh.rotation.set(Utils.toRadians(-90), 0, 0);
_mesh.add(mesh);
_robot = mesh;
Render.startRender(loop);
var animation = new THREE.Animation(mesh, geom.animation.name);
animation.JITCompile = false;
animation.interpolationType = THREE.AnimationHandler.LINEAR;
animation.play();
});
}
I believe I'm updating the AnimationHandler correctly in my loop
function loop() {
_mesh.rotation.y += 0.01;
var delta = 0.75 * _clock.getDelta();
THREE.AnimationHandler.update(delta);
}

In the section metadata of the exported JSON file the number of morphTargets and bones are both greater than 0?
I think that you followed the example here:
http://threejs.org/examples/#webgl_animation_skinning_morph
in which the animated model uses Morph Target and Skeletal Animation (see Wikipedia for the theoretical concepts).
If the animated model uses only Skeletal Animation as in this example http://alteredqualia.com/three/examples/webgl_animation_skinning_tf2.html
you have to instantiate a THREE.SkinnedMesh Object and then set only the m.skinning property to true.

I was having the same problem just now. What worked for me was to remake the model with applied scale and have keyframes for LocRotScale, not just location.

lately, I've encoutered a similar issue of mesh disapearing while exporting blender skinning animation to json. It turned out, the mesh I was using had double vertex (one vertice hidding another). All looks good While creating the vertex groups and the animations in blender, but when I imported the mesh via three.js, it kept disapearing as soon as the animation started. In other words, If 1 vertice from your mesh is omitted from the vertex groups, you will experience this disapearing behavior. To prevent this issue, I now use the "remove doubles" function from blender to validate the mesh integrity before exporting it to json. You might have encountered the same issue and redoing your mesh work fix it... Anyways, the question is pretty old, but the topic is still valid as of today, so I hope this fresh info will help someone out there...
Peace INF1

How do I use texture on a sphere in three.js

I've downloaded a sphere example from: http://aerotwist.com/lab/getting-started-with-three-js/ and I can see the nice red sphere. I'd like to use a texture on it. I've tried this:
var texture = THREE.ImageUtils.loadTexture("ball-texture.jpg");
texture.wrapS = texture.wrapT = THREE.ClampToEdgeWrapping;
texture.repeat.set( 125, 125 );
texture.offset.set( 15, 15 );
texture.needsUpdate = true;
var sphereMaterial = new THREE.MeshBasicMaterial( { map: texture } );
var sphere = new THREE.Mesh(new THREE.Sphere(radius, segments, rings),sphereMaterial);
but I can't see anything, all is black. Does anyone have a working example for sphere texture?

You might have two problems.
First, try loading it like this:
var texture = THREE.ImageUtils.loadTexture('ball-texture.jpg', {}, function() {
renderer.render(scene, camera);
});
texture.needsUpdate = true;
Make sure that the texture size is a power of two (512x512px for IE).

Are you using Firefox? This could be a problem in your browser. Firefox uses some kind of cross-site-blocker for textures. The result is black instead. Take a look at this site for more info: http://hacks.mozilla.org/2011/06/cross-domain-webgl-textures-disabled-in-firefox-5/

Do you have a rendering loop, or did you render the scene just once?
You need to have a rendering loop so that when the THREE.ImageUtils loads the image and updates the texture, you re-render the scene with the now updated texture.
All the three.js examples seem to rely on this technique. I.e., Fire off several async operations involving a fetch of a remote resource, start rendering loop, let scene be updated as remote resources arrive.
IMHO this is Three.js's biggest gotcha for Javascript newbs (like me) who are not familiar with how async operations work.

I had this problem, but if you are loading the html as a file (i.e. locally not a webserver), many browsers (chrome for e.g.) will not allow you to load images in the standard three.js way as it is a security violation.