Leap Motion Gesture Detection Accuracy with LeapJS - javascript

I am using a leap motion sensor with my node app to detect gestures. With my current setup, the gesture detection is working but not as accurately as I need it to be. When I perform a swipe gesture the leap motion does not always send a swipe gesture object and when it does detect a swipe it often will send multiple gestures. For example, when swiping left, it will return a left swipe as well as an up swipe. I am wondering if this is an issue with my code or my leap motion sensor.
Here is my leap controls code:
var controller = Leap.loop({ inNode: true, enableGestures: true }, function(frame) {
// finger location tracking
if(frame.hands.length > 0) {
var iBox = frame.interactionBox;
var pointable = frame.pointables[0];
if (pointable) {
var leapPoint = pointable.stabilizedTipPosition;
var normalizedPoint = iBox.normalizePoint(leapPoint, true);
let mouseControlY = (1 - normalizedPoint[1]) * appHeight; // with leap facing up
// invert directions for mirror
let invertMouseControlX = (1 - normalizedPoint[0]) * appWidth;
updateMouseCords(invertMouseControlX, mouseControlY)
}
}
// finger gesture tracking
if(frame.valid && frame.gestures.length > 0){
frame.gestures.forEach(function(gesture){
switch (gesture.type){
case "screenTap":
console.log("Click");
break;
case "swipe":
handleSwipe(gesture);
break;
}
});
}
});
function handleSwipe (swipe) {
let io = getIO();
if(swipe.state === 'stop'){
if (swipe.direction[0] > 0){
// user swiped right
swipeDirection = 'right'
} else {
// user swiped left
swipeDirection = 'left'
}
if(swipe.direction[1] > 0) {
// user swiped up
swipeDirection = 'up'
}
// send swipe direction to frontend
io.emit('swipeData', swipeDirection);
}
}
I only need to detect a left, right, or up swipes and possibly the screen tap gesture. Is there anything I can do to optimize my code or is the issue likely my sensor?

I am not an expert myself, but I have prevented multiple detections by using the setTimeout() function, which is set to a few milliseconds where no gesture detection can occur.
The JavaScript documentation also mentions the attributes Gesture.duration and Gesture.type. Perhaps you can achieve better results by using them.
It is also worth noting, that Gestures are unfortunately deprecated in version 3.0. However, I am not sure if this is due to the problems you described or if they will be fundamentally revised in later versions. I would also be interested in an update on this topic.

Related

Make trackpad wheel event run once [duplicate]

Is there any way to detect if the client is using a touchpad vs. a mouse with Javascript?
Or at least to get some reasonable estimate of the number of users that use touchpads as opposed to mice?
Compare e.wheelDeltaY and e.deltaY (or e.deltaMode in Firefox) to detect touchpad mouse device
function handler(e) {
var isTouchPad = e.wheelDeltaY ? e.wheelDeltaY === -3 * e.deltaY : e.deltaMode === 0
// your code
document.body.textContent = isTouchPad ? "isTouchPad" : "isMouse"
}
document.addEventListener("mousewheel", handler, false);
document.addEventListener("DOMMouseScroll", handler, false);
The answer above by Lauri seems to work, but it took me a while to understand why it works. So here I'll provide a slightly more human readable version, along with a conceptual explanation. First, that same code written out in a human readable manner:
function detectTrackPad(e) {
var isTrackpad = false;
if (e.wheelDeltaY) {
if (e.wheelDeltaY === (e.deltaY * -3)) {
isTrackpad = true;
}
}
else if (e.deltaMode === 0) {
isTrackpad = true;
}
console.log(isTrackpad ? "Trackpad detected" : "Mousewheel detected");
}
document.addEventListener("mousewheel", detectTrackPad, false);
document.addEventListener("DOMMouseScroll", detectTrackPad, false);
This works because wheelDeltaY measures the physical distance that the actual hardware mouse wheel has travelled, while deltaY measures the amount of scrolling produced on screen. A conventional mouse typically has a much lower "scroll resolution" than a trackpad. That is to say, with a trackpad you can make a tiny motion and a get a tiny scroll on screen. A conventional mouse scrolls in chunkier, low resolution clicks. To complete a full rotation of the mouse wheel, it might make 10 clicks. There is no such thing as a half click or quarter click.
For a conventional mouse, a single wheel click is reported as 120 wheelDeltaY "units" and results in about ~100px worth of scrolling. The physical wheelDeltaY unit is a completely arbitrary number, it is not measuring inches or degrees or anything like that. The number 120 was selected simply because it has a lot of useful factors. The amount of scrolling on screen is represented by deltaY, and it varies significantly by browser. (Sidenote, deltaY is generally measured in "lines" not pixels, though it's complicated, see previous link).
Interacting with a trackpad is different in two ways. First of all, you can get wheelDeltaY values much smaller than 120, because very subtle finger gestures are detectable. Second, the wheelDeltaY is exactly 3x the deltaY value (at least in every browser I've managed to test). So, for instance, if you make a physical finger gesture equal to 12 click units, it will generally result in 4 pixels worth of scrolling. Lauri's code uses this second property (Y1 = Y2 * 3) to detect the existence of a trackpad, but you could probably also be successful simply by checking if abs(wheelDeltaY) equals 120
I haven't tested this, but I think it would also work:
function detectTrackPad(e) {
var isTrackpad = false;
if (e.wheelDeltaY) {
if (Math.abs(e.wheelDeltaY) !== 120) {
isTrackpad = true;
}
}
else if (e.deltaMode === 0) {
isTrackpad = true;
}
console.log(isTrackpad ? "Trackpad detected" : "Mousewheel detected");
}
document.addEventListener("mousewheel", detectTrackPad, false);
document.addEventListener("DOMMouseScroll", detectTrackPad, false);
This topic may be already solved, but the answer was there is no way to detect it. Well I needed to get a solution, it was very important. So I found a acceptable solution for this problem:
var scrolling = false;
var oldTime = 0;
var newTime = 0;
var isTouchPad;
var eventCount = 0;
var eventCountStart;
var mouseHandle = function (evt) {
var isTouchPadDefined = isTouchPad || typeof isTouchPad !== "undefined";
console.log(isTouchPadDefined);
if (!isTouchPadDefined) {
if (eventCount === 0) {
eventCountStart = new Date().getTime();
}
eventCount++;
if (new Date().getTime() - eventCountStart > 100) {
if (eventCount > 10) {
isTouchPad = true;
} else {
isTouchPad = false;
}
isTouchPadDefined = true;
}
}
if (isTouchPadDefined) {
// here you can do what you want
// i just wanted the direction, for swiping, so i have to prevent
// the multiple event calls to trigger multiple unwanted actions (trackpad)
if (!evt) evt = event;
var direction = (evt.detail<0 || evt.wheelDelta>0) ? 1 : -1;
if (isTouchPad) {
newTime = new Date().getTime();
if (!scrolling && newTime-oldTime > 550 ) {
scrolling = true;
if (direction < 0) {
// swipe down
} else {
// swipe up
}
setTimeout(function() {oldTime = new Date().getTime();scrolling = false}, 500);
}
} else {
if (direction < 0) {
// swipe down
} else {
// swipe up
}
}
}
}
And registering the events:
document.addEventListener("mousewheel", mouseHandle, false);
document.addEventListener("DOMMouseScroll", mouseHandle, false);
It may need some optimization and is maybe less than perfect, but it works! At least it can detect a macbook trackpad. But due to the design i'd say it should work anywhere where the pad introduces a lot of event calls.
Here is how it works:
When the user first scrolls, it will detect and check that in 50ms not more than 5 events got triggered, which is pretty unusual for a normal mouse, but not for a trackpad.
Then there is the else part, which is not for importance for the detection, but rather a trick to call a function once like when a user swipes. Please come at me if I wasn't clear enough, it was very tricky to get this working, and is of course a less than ideal workaround.
Edit: I optimized the code now as much as I can. It detects the mouseroll on the second time and swipe on trackpad instantly. Removed also a lot of repeating and unnecessary code.
Edit 2 I changed the numbers for the time check and numbers of events called from 50 to 100 and 5 to 10 respectively. This should yield in a more accurate detection.
You could detect JS events.
A touch device will fire touch events such as touchstart in addition to mouse events.
A non-touch device will only fire the mouse events.
In the general case, there is no way to do what you want. ActiveX might allow you to see and examine USB devices, but in the best case, even if that is somehow possible, that limits you to IE users. Beyond that, there is no way to know.
You might be able to discern patterns in how (or how often) a touchpad user moves the cursor versus how a mouse user might move the cursor. Differentiating between physical input devices in this way is an absurdly difficult prospect, and may be wholly impossible, so I include here it for completeness only.
Wheel event triggered by touchpad will give much smaller event.deltaY,1 or 2,but trigger by mouse wheel will give like 100,200.
Trust me. This is the easiest and only solution that also works for Safari (as far as I know).
isTrackPad(e) {
const { deltaY } = e;
if (deltaY && !Number.isInteger(deltaY)) {
return false;
}
return true;
}
From testing plugging in a mouse to a mac which does have a touchpad and also a windows machine with one, I can summarize how I got this working.
Detect if the navigator user agent contains "Mobile" or "Mac OS" If either
of these are true, it is likely a touch based system, but work to
eliminate that. Set boolean hasTouchPad to true
If the above is true, detect "mouse" events and run a test like high
numbers, or a frequency of non integers or Kalman filtering.
Keep these in a queue and if that queue's sum passes a threshold,
disable the hasTouchpad variable and disconnect the event.
let isMouseCounts: Array<number> = []
if (Client.hasTouchpad) {
document.addEventListener('wheel', detectMouseType);
}
function detectMouseType(e:WheelEvent) {
if (!Client.hasTouchpad) return
let isMouse = e.deltaX === 0 && !Number.isInteger(e.deltaY)
isMouseCounts.push(isMouse ? 1 : 0)
if (isMouseCounts.length > 5) isMouseCounts.shift()
let sum = isMouseCounts.reduce(function(a, b) { return a + b; });
if (sum > 3 && e.type === "wheel") {
console.log("Touchpad disabled")
document.removeEventListener('wheel', detectMouseType);
Client.hasTouchpad = false;
}
}
You could just check for the device driver softwares installed into the local package as functioning. Like in windows synaptics, elan hardware, as for UNIX(Linux) you could just check for the package installed during the basic installed onto. A lot of packages come in different formats in different versions of Linux and Linux like systems(Not linux entirely) but they use the same package name for all. Just got to know the code to pull it. Still working on it.

Trying to run a HTML5 accelerometer listener in the background on mobile

I'm currently in the process of creating a webapp that detects whether the user is sitting or standing. I have had great success with detecting the tilt of the device, changing a boolean variable and sending the correct data to a graph.
The following code runs perfectly in the background, updating the data every second.
var trackInterval = setInterval(function(){
if(isSitting){
addData(myPieChart, "Sitting");
} else{
addData(myPieChart, "Standing");
}
}, 1000)
My issue is that the listening function that changes the variable 'isSitting' does not continue once the browser is closed. This means that the last value of 'isSitting' gets data added to it, even though the device might be tilted otherwise.
This is the code that creates the accelerometer updates:
window.addEventListener("devicemotion", accelerometerUpdate, true);
var isSitting = true;
function accelerometerUpdate(event) {
var aX = event.accelerationIncludingGravity.x * 100 ;
var aY = event.accelerationIncludingGravity.y * 100 ;
var aZ = event.accelerationIncludingGravity.z * 100 ;
if (aY > 600 || aY < -900 ){
isSitting = false;
} else{
isSitting = true;
}
}
I have tried Chrome, Opera and Firefox on my Android device.
Any help or tips to work around this would be greatly appreciated
This sounds like regular behaviour as your javascript code, to be executed, should run in your browser.
To achieve your goal, a possible guess would be to look into a way to use this api in service workers, which is currently not possible with the devicemotion api but should be (at least with chrome) with the new Generic Sensors API described here : https://developers.google.com/web/updates/2017/09/sensors-for-the-web

Watching a phone's physical orientation with JavaScript

I want to know whether the user is physically holding their screen in portrait or landscape mode, even if they have locked their screen.
I cannot use orientationchange events, as these won't fire if the rotation is locked. I want to follow the actual, real-world rotation of the device, not the logical orientation of the viewport.
I am aware that we should normally never force a display orientation on our users, and that we should respect users' preferences when they lock the orientation - but I really do have a reason to do this.
Is there ever any way?
Yes - if the phone is being held upright. Then we can use DeviceMotionEvents to read the pull of gravity on the phone's X and Y axes:
function watchOrientation(subscriber) {
if (window.DeviceMotionEvent) {
window.addEventListener('devicemotion', eventData => {
const hg = eventData.accelerationIncludingGravity.x;
const vg = eventData.accelerationIncludingGravity.y;
if (vg > 5) {
subscriber('up');
} else if (vg <= -5) {
subscriber('down');
} else if (hg > 0) {
subscriber('left');
} else if (hg <= 0) {
subscriber('right');
}
}, true);
} else {
console.warn('Device does not support "devicemotion" events');
}
}
Obviously, this won't work if the phone is held flat or laid on a table. In that case your only chance is to use the compass data in a DeviceOrientationEvent (but only if you already know whether your user is facing North!)

Flash Player stops animating when out of viewport

I have several flash movies on a site. They all loop through an animation which lasts for approximately 15 seconds (but this is not set in stone). They all begin to play at the same time and are in sync.
However, when I resize the browser window, or scroll the page, if a flash player instance is not in the viewport, it stops playing. When it returns to the viewport, it resumes and subsequently is out of sync with the other flash player instances.
I am under the impression this is a flash player optimisation. Is there anyway of disabling this behaviour, possibly through JS / AS3? It appears to happen in Firefox and Chrome on Windows.
UPDATE
Just to clarify. I have methodologies in place using local connection and ExternalInterface to sync the ads. What I am looking for is a method to disable the "optimisation" of FlashPlayer which results in the frame rate being drastically reduced.
You can't disable this feature. It's put in place to lower the memory and CPU use when the flash application isn't visible.
What is available for you though is something called the Throttle API. This is a dedicated API created to allow creators of Flash applications the ability to be notified exactly when their application is going to be slowed down/throttled.
Here's an example.
addEventListener(ThrottleEvent.THROTTLE, onThrottleEventHandler);
function onThrottleEventHandler(e:ThrottleEvent):void
{
if(e.state == ThrottleType.THROTTLE)
{
/**
* the player is about to be slowed down to a frame rate
* set in e.targetFrameRate
*/
}
else if(e.state == ThrottleType.PAUSE)
{
/**
* the player is going to be paused completely, AKA 0 fps
*/
}
else if(e.state == ThrottleType.RESUME)
{
/**
* the player is now back to normal as it has resumed
* from the paused or throttled state
*/
}
}
Now you can figure out a way that works best for you but my suggestion is to store the current time that has passed whenever being throttled or paused via:
currentTime = getTimer();
Then calculate how much time has passed once your application has resumed using:
passedTime = getTimer() - currentTime;
Then do what you like with this information.
Hopefully this has helped, should offer you a greater degree of control now that you're familiar with the Throttle API. For more information on it, check it out in the documentation here: ThrottleEvent
I belive this kind of behavior its normal, its kind of a bug of flash that has never been fixed.
I belive inha may got u a solution, but not on an enter_frame event. that just to brutal.
what I would do is:
create a timer event.. each X seconds.. so it will call a checkFunction,
in my checkFunction(). I would check if all my movieClips are syncronized.
and if I found 1 that is not.. ill put a simple gotoAndPlay(xFrame);
var aMovieClips:Array; //get all ur movieclips into this array.
var timeToCheck = 1000; //time to check for a unsincronized movieclip
var time:Timer=new Timer(timeToCheck,0);//do inifinite call each 1000 seconds
time.start();
time.addEventListener(TimerEvent.TIMER, checkFunction);
public function checkFunction(e:TimerEvent):void
{
var aCurrentFrames:Array;
for (var n:int = 0;n<aMovieClips.length();n++)
aCurrentFrames.push(aMovieClips[n].currentFrame);
//so now u have all current frames of all movie clips.. so u can add a
//gotoAndPlay(x); to any unsyncronized movieclip
}
If it is very important that each SWF progresses simultaneously, I would control it by JavaScript.
Assuming each SWF file is a simple MovieClip, something like this is how I would go about it:
Set the Document Class of each FLA file to this:
package {
import flash.display.MovieClip;
import flash.external.ExternalInterface;
public class ExternalInterfaceControlledMC extends MovieClip {
public function ExternalInterfaceControlledMC() {
this.stop();
if (ExternalInterface.available) {
try {
ExternalInterface.addCallback("setFrame", jsSetFrame);
} catch (error:Error) {
trace("An Error occurred:", error.message);
}
} else {
trace("ExternalInterface is not available");
}
}
private function jsSetFrame(value:String):void {
var frameNumber = int(value) % this.totalFrames;
this.gotoAndStop(frameNumber);
}
}
}
In the JavaScript, you would add a reference of each instance of the SWFs into an array, then use a timer to tell each SWF to progress to a frame number.
var swfElements; //Array
var currFrame = 1;
function onPageLoad() {
//Init timer
setInterval("progressFrame", 1000 / 30); //30 fps
}
function progressFrame() {
for (var i = 0; i < swfElements.length; i++) {
swfElements[i].setFrame(currFrame);
}
currFrame++;
}
Please beware that nothing about this code is tested and is only meant to be used to illustrate my train of thought.

JQuery - Can I query the MacBook Pro Trackpad?

Can I use JQuery to query the Trackpad?
So I can do something like this:
Pseudo Javascript (JQuery)
$(document).keyInput().trackpadTwoFingersLeft(function() {
$('#div ul').animate({left: "=+1"},1);
});
Is there a plugIn or another framework where I can do this?
Thank you for every response and idea. :)
I've looked around a bit on the web, and so far see that both Chrome and Safari do not expose these events in the browser.
https://superuser.com/questions/27627/three-finger-page-up-page-down-in-safari-chrome
Touch events available in Safari?
Firefox does support something:
https://developer.mozilla.org/En/DOM/Mouse_gesture_events
But I don't see a lot of references to this.
I guess when only one browser supports it, it is a bit useless to use these kind of events.
There is a wheel event which you can use to detect two-finger swipe on Mac.
Your code could look something like this:
$('element').on('wheel', function(e){
var eo = e.originalEvent;
if(Math.abs(eo.wheelDeltaY) < 10 && Math.abs(eo.wheelDeltaX) > 2){
e.preventDefault();
if(eo.wheelDeltaX < -100 && !scope.item.swipedLeft){
// swipe left
}
if(eo.wheelDeltaX > 100 && scope.item.swipedLeft){
// swipe right
}
}
});
This could possibly not work in some older browser and/or Mozilla (as it fires some different event for the wheel movement), but as long as you implement this as an additional/helper feature, this code should suffice.
You could probably use mouse wheel tracking to get the desired effect: http://www.switchonthecode.com/tutorials/javascript-tutorial-the-scroll-wheel
If you want to use jQuery, this mousewheel plugin should help: http://brandonaaron.net/code/mousewheel/docs
We manage to determine trackpad usage by first assuming an OSX machine has it and then working to detect if that user has a mouse plugged in. Excuse me, because we use typescript, but it will work in javascript.
First, just look at the user agent properties, and then if OSX, assume it has one. I just use some basic string expressions on the user agent to determine what the OS is.
if (!props.isMobile && props.operatingSystem === OSType.Darwin) {
props.hasTouchpad = true
props.hasGestureSupport = props.browser === BrowserType.Safari | props.isMobile
}
Now work to eliminate that device by listening to the wheel event. A trackpad device will create really small deltaY values that are non-integer. I have tried this and it does a very good job.
const detectMouseType = useCallback((e:WheelEvent) => {
if (e.ctrlKey || !browserProperties.hasTouchpad) {
//Didn't detect originally, or pinch zoom OSX and surface, ignore
return
}
let isMouse = e.deltaX === 0 && !Number.isInteger(e.deltaY)
isMouseCounts.push(isMouse ? 1 : 0)
if (isMouseCounts.length > 5) isMouseCounts.shift()
let sum = isMouseCounts.reduce(function(a, b) { return a + b; });
if (sum > 3 && e.type === "wheel") {
console.log("Touchpad disabled")
//detected a mouse not a touchpad, maybe a mouse plugged into a mac
document.removeEventListener('wheel', detectMouseType);
props.hasTouchpad = false
props.hasGestureSupport = false
}
}, [])
Now onto using trackpad events and detecting pinch in browsers that normally don't support it (HammerJS for example):
On OSX Safari and iOS Safari the "gesturestart" and "gesturechange" events are fired which are sufficient for detecting pinch and rotate. There are scale and rotate parameters you can use to get the data you need.
Here's some code to get you started:
https://medium.com/#auchenberg/detecting-multi-touch-trackpad-gestures-in-javascript-a2505babb10e
On OSX Chrome, you can use the "wheel" event to detect pinch, scroll, and two finger tap. Most solutions you'll see will not support pinch or two finiger tap on OSX Chrome, but this will. Here's how two finger tap and pinch are done for Chrome and also mice on all other browsers:
let scale = 1.0;
let posX = 0.0;
let posY = 0.0;
element.addEventListener('wheel', (e) => {
e.preventDefault();
if (e.ctrlKey) {
//using two fingers
if (e.deltaY === 0 && e.deltaX === 0) {
console.log("Chrome two finger tap detected")
} else {
console.log("pinch zoom")
scale = scale - e.deltaY * 0.01
}
} else {
console.log('scrolling')
posX = posX - e.deltaX * 2
posY = posY - e.deltaY * 2
}
});
There's some info here but I wasn't able to test it
function setupForceClickBehavior(someElement)
{
// Attach event listeners in preparation for responding to force clicks
someElement.addEventListener("webkitmouseforcewillbegin", prepareForForceClick, false);
someElement.addEventListener("webkitmouseforcedown", enterForceClick, false);
someElement.addEventListener("webkitmouseforceup", endForceClick, false);
someElement.addEventListener("webkitmouseforcechanged", forceChanged, false);
}
https://developer.apple.com/library/archive/documentation/AppleApplications/Conceptual/SafariJSProgTopics/RespondingtoForceTouchEventsfromJavaScript.html#//apple_ref/doc/uid/TP40016162-SW6

Categories

Resources