Develop Reference

iOS application fails to respond to mouse and touch events

I am developing an iOS application using Swift that incorporates web content by using WKWebView.
[URL Loading and Goal] Once a URL has been loaded inside this web view, I want to open a keyboard whenever the user wants to type.
[JavaScript Events] As browsing is based on eye coordinates, I want a JavaScript event to be triggered when the user's gaze is focused on an input field. On the basis of eye gaze coordinates, I can determine the underlying element of the webpage document. After this, I want to trigger a mouse event/touch event for clicks within the input element.
However, the system-level keyboard does not appear if I follow these steps.
Code:
touchPoint variable contains the current x and y coordinates of the eye gaze.
#IBOutlet var webView : WKWebView!
let jsStyle = """
function sendTouchEvent(x, y, element, eventType) {
const touchObj = new Touch({
identifier: Date.now(),
target: element,
clientX: x,
clientY: y,
radiusX: 2.5,
radiusY: 2.5,
rotationAngle: 10,
force: 0.5,
});
const touchEvent = new TouchEvent(eventType, {
cancelable: true,
bubbles: true,
touches: [touchObj],
targetTouches: [],
changedTouches: [touchObj],
shiftKey: true,
});
element.dispatchEvent(touchEvent);
}
function click(x, y)
{
var ev = new MouseEvent('click', {
'view': window,
'bubbles': true,
'cancelable': true,
'screenX': x,
'screenY': y
});
var el = document.elementFromPoint(x, y);
if(el.nodeName == "INPUT") {
el.autofocus = true;
el.click();
el.focus();
sendTouchEvent(x, y, el, 'touchstart');
}
try {
el.dispatchEvent(ev);
} catch(error) {
console.log(error);
}
}
click(\(touchPoint.x), \(touchPoint.y));
"""
webView.evaluateJavaScript(jsStyle, completionHandler : { result, error in
print("### Evaluate JavaScript for current click!")
} => Called when eye gaze coordinates are received.
[Temporary Solution] In order to make this work, I created a native UITextField that acts as a proxy for the webpage element and opens the keyboard by becoming the first responder when I determine using JS that the eye gaze is focused on an input element.
My goal, however, is to remove the proxy logic and replace it with JavaScript code that opens the system-level keyboard.

Chrome extension click bot in webGL

I am trying to develop a chrome plugin that will record a series of click and play them back
I have seen that you can simulate a click on a DOM element but the problem is in my case, I only have a webGL canvas and the buttons are not directly accessible
I have managed to get the user's click position using this :
document.onclick = function(e)
{
var x = e.pageX;
var y = e.pageY;
console.log(x + " " + y)
//browser.runtime.sendMessage({"message": "open_new_tab", "url": "aze"});
};
But I haven't found anything to use these positions to perform a click action on the page

If I understand your question properly, you want to simulate a click on the webpage?
JavaScript has the click() method exposed for all elements:
element.click()
If you don't know the element, but just the positions:
document.elementFromPoint(x, y).click()
You can dispatch a mouse click at a specific position:
const = (x, y) => {
const ev = new MouseEvent('click', {
'view': window,
'bubbles': true,
'cancelable': true,
'screenX': x,
'screenY': y
})
const el = document.elementFromPoint(x, y);
el.dispatchEvent(ev);
}
All the above should be done via Content Script within the extension.

How to open mulitple tabs at once [JavaScript]

I have a button that is purposely designed to open multiple pages at once, it is the main feature of the button.
I tried using:
(1)
urls.forEach(url => {
window.open(url);
});
(2) promises with a delay on them but that did not work either.
(3) multiple a tags and trying to simulate a human click however, that did not work either.
var element = document.createElement("a");
element.href = tempUrl;
element.innerHTML = "temp";
element.id = "tempAtag";
element.target = "_blank";
document.getElementById("dashboardID").appendChild(element);
element = document.getElementById("tempAtag");
var box = element.getBoundingClientRect(),
coordX = box.left + (box.right - box.left) / 2,
coordY = box.top + (box.bottom - box.top) / 2;
var simulateMouseEvent = function (element, eventName, coordX, coordY) {
element.dispatchEvent(
new MouseEvent(eventName, {
view: window,
bubbles: true,
cancelable: true,
clientX: coordX,
clientY: coordY,
//button: 0,
})
);
};
simulateMouseEvent(element, "mousedown", coordX, coordY);
simulateMouseEvent(element, "mouseup", coordX, coordY);
simulateMouseEvent(element, "click", coordX, coordY);
This does work for the first link but I get a warning in the console saying I am trying to open multiple tabs with only one interaction. So it works for one link but not for the rest.
I got the code from Simulate a REAL HUMAN mouse click in pure javascript?
(Note: I did delete my a tags after each iteration and I did test it out, there are no duplicates)
I ran out of ideas. Any ideas?
(I have looked at other solutions and none of what I came across has worked for me)

The following works with a big IF attached.
That if is you need to click the "allow pop-up" warning that will appear in the address bar and continue to allow pop-ups from that site.
You also need to have pop-ups to load in a new tab.
I still think this is a duplicate question with Open a URL in a new tab (and not a new window)
var id = 0, u = ['12','34','56','78'];
function openNextTab(){
if (u[id]){
var x = window.open("https://some.site/id:"+ u[id],"id"+u[id]);
id++;
setTimeout(openNextTab,2000);
}
}
function openTabs(e){
e.stopPropagation();e.preventDefault();
openNextTab();
}
window.onload = function(){
var b = document.getElementById("openLinks");
b.addEventListener("click",openTabs,false);
}

How to create a new element, then drag it with interactjs?

What I'm trying to do, is create an element from a none-draggable element. Then start dragging the new element from there.
I've seen the examples, and managed to replicate the examples from Interact.js where you clone and drag an already draggable element in Interact.js. But that's not what I'm doing here.
interact('.drag-drop')
.draggable({
inertia: true,
modifiers: [
interact.modifiers.restrictRect({
restriction: 'parent',
endOnly: true
})
],
autoScroll: true,
onmove: dragMoveListener
})
function createDragDropElement(event){
var element = document.createElement('div');
element.className = 'drag-drop';
element.text = 'Drop me!';
element.style.position = 'absolute';
element.style.left = event.pageX + 'px';
element.style.top = event.pageY + 'px';
document.body.appendChild(element);
//here is where I'd like to transfer the drag over to the newly created element.
}
I've managed to create the element just fine. However, I'm having trouble automatically dragging it. After creation, I need to release the mouse and start a new drag on the element. What I'd like to happen though, is that I start dragging the element immediately rather than having to re-click on it.

javascript: How can find the location in screen/webpage to click

I need to build and automate task where I need to click a button in a web page but I don't know how can I get the location on screen. Any of you can be so kind and help me to figure out how to get position on screen to click ?
I'll really appreciate your help.

Here is a javascript function I wrote to simulate clicking a link. I think you can adapt it pretty easily to click any object in a page that has a valid ID:
function ClickLink(AnchorName)
{
var evt = document.createEvent("MouseEvents");
var Link = document.getElementById(AnchorName);
var LinkForJquery = '#' + AnchorName
var x = $(LinkForJquery).offset().left;
var y = $(LinkForJquery).offset().top;
evt.initMouseEvent("click", true, true, window, 1, 0, 0, x, y, false, false, false, false, 0, null);
var allowDefault = Link.dispatchEvent(evt);
}