We have a web application that that works with user camera and keystrokes, collects data on the client side using JS. The data contains the users head movements captured by headtracker.js and keystrokes that are captured as an onscreen test runs in realtime.
We are planning to write end to end test cases for this test flow and the first thing we plan to do is to mock live streaming using recorded videos which is supported by headtracker (it supports ogv, mp4 etc).
Our next challenge is to mock and supply keystrokes to the web application as the test proceeds. So, each action on the UI may (or may not) require a user action through a keystroke which we are planning to automate.
Or is there a tool or technology that allows us to create such a virtual environment which could provide us real time response?
I am able to provide more information as required.
You will want to look at Selenium WebDriver. As for a reliable web browser to driver it from tests, look at phantomjs (and the PhantomJSDriver to control it).
Whatever you do, avoid IE at all cost when using Selenium. If you want to do unit testing with IE, buy a commercial testing framework.
Related
In an HTML5 web app, I'm building a feature that relies on client-to-client communication (with pusher). It's made of PHP on the server-side and Javascript with Vue on the client side.
The typical scenario is: a window popup is opened, and from there it communicates directly with some other windows opened into any another browser on the same computer. Let's say you have 2 browsers installed, you open the web app popup with Firefox and it communicates with its web app sister page you did open previously into Chrome.
The only (half-)way we have found so far is to use the public IP address to build a private channel named with the IP address… It's basic and efficient.
However, if there is more than 1 computer connected to the same router, all of them will share the same public IP, and that's where things become difficult!
A solution could be to add the computer's local IP to the channel name (that was already built with the public IP), but despite a few nice workarounds I found to get this info from an initiated RTC Connection, this looks quite unreliable and often goes against browsers privacy rules…
Obviously, I cannot use session information with PHP on the server-side, nor cookies / local storage on the client-side, as all those solutions are tightly coupled with the browser itself (thank God Chrome won't share its cookies with Firefox on your computer). Those solutions would be perfect (and no need for a pusher) if we were using 1 single browser, but we need to handle multiple browsers on the same machine.
That's where I'm wondering if anyone would have already dealt with this design challenge and shared some tips, it would be awesome! Thanks for reading so far!
You can check the user agent of the browser.
You can check a combination of the request headers coming from different browsers.
You can explicitly throw and catch an error in the user's browser and send it in the request header/body to determine what browser they're using.
You can do canvas drawings to see the user's GPU/CPU information (since you're already using html5 that's a bonus).
You can directly use webgl to do the same with perhaps different metrics (since canvas uses webgl anyways).
You can check their typing speed or even build up a profile of their vocabulary and use of language.
If you ask for permissions you can see all of their connected media devices like headphones, even just asking for audio permissions will show you all of them.
You can benchmark their CPU with things like the time it takes to find primes or encrypt a key.
You can use audio fingerprinting, which is almost as unique as your voice, since each browser and CPU architecture slightly differ in the digital pattern and oscillations created from audio, which can be captured.
You can check their window size and screen size and screen resolution.
There's probably even more I didn't think of now, you can also use any of them in combination to fingerprint a device.
For more information research browser sniffing and digital fingerprinting. What's more is that you can uniquely identify the user across their own browsers on the same computer and also different users from different devices using a combination of browser sniffing/digital fingerprinting.
In your specific case you can't use all the browser sniffing techniques but you can still use some of them, like the user agent since it will still give you the user device information even if they're using a different browser.
The idea with digital fingerprinting is that you want to build up a probability high enough that you can be fairly certain it's the same user, you can't ever be truly sure, but sure enough. Something like screen size doesn't mean much by itself, there's millions of devices using i.e. a size 1600 screen, however consider the following hypothetical example:
User's device has screen size of 1600, that's i.e. ~1/8 users.
User's device took 20ms to encrypt a 4096 key, that's i.e. ~1/8 users.
User's device took 40ms to draw a canvas image, that's i.e. 1/8 users.
Now you already have a 8 * 8 * 8 = 1/512 probability of knowing what user it is and that value goes up way higher very quickly, based on 3 fundamentally unidentifiable things.
However it should be noted that using any browser sniffing or digital finger printing techniques like above fall under privacy regulations (at least in some countries). A lot of things like the user agent is being deprecated and if you do things like this on a site you'll get into trouble with things like GDPR. I believe you can get around that if you explicitly ask the user for their permission and let them know that i.e. their browser is being fingerprinted. However you have to be careful because doing this can get you in trouble if it's malicious, doing things like this without a user's knowledge is unethical.
I would like to access a special hardware (a penta scanner, for the wisest ;) ) from a web page.
The aim is to retrieve the information from the scanner and compute it on server side.
The problem here, is that I have to summon functions from a DLL that has to be on the client side. I have two leads for that:
Develop a COM DLL and use ActiveX
Try to get through a homemade extension for a web browser in order to communicate with the local DLLs.
I've tried the first option, and I got stuck and I've posted another thread about it. Anyway, even if it works, there are too many constraints about it (as to use IE or the fact that even Microsoft is not fond of this feature and banned it from edge).
The second method is something that I found on another forum but, I do not understand how I can interact with a browser extension (whatever the browser).
So what I am asking is:
Is it possible to use a web browser extension as a medium to a local DLL and if so, would you be so kind as to give a hint about how to do ar anything that might look like a start about how to do it (even just some key words to use on google, since mine didn't get anything)....
Thanks.
Based on my understanding, the motive to develop an Extensions is to enhance the feature and functionality for particular web browser.
You cannot control any hardware devices like printer or scanner with it.
So if your goal is to control the scanner from your web page with the help of any kind of extension than I think you cannot do this with Extension.
I know this is not a specific question, but I just want to get design ideas about a screen sharing web site with SignalR.
We want to add a link on our website which is called "Share My Screen" and then our support team be able to see the content of the browser (not whole desktop) and even they be able to click or type on customer browser.
I was thinking to do it as
Taking screen shot from browser by js (by http://html2canvas.hertzen.com for example)
send taken screen shot to server constantly (I don't know how yet)
Server sends the received screen shot to our support team browser
Capturing mouse move and key press on support team browser
Sending this captured data to customer browser
Since each part of this needs a lot of work I just want to gather all possible ideas to find a tested solution
First of all, I do not think your idea of capturing screen is really doable with javascript technology. Security would be a huge issue, you would need to process a high amount of data, and syncing events would be a nightmare no matter how you approach it. Capturing and sharing tab content and events is a much more manageable goal.
If you are aiming to use this for people who can not manage to install a remote control app, then we can count newer technologies such as WebRTC out due to browser compatibility issues.
There is a good blog post discussing this issue here - (Screensharing a browser tab in HTML5?).
I especially like the first method, using Mutation Observer (browser support) and Web Sockets (browser support). It basically syncs two html documents through the use of mutation observer and uses web sockets for communication. You could use SignalR instead of standard web socket API for communication if you prefer.
I have an application on Samsung Smart TV. I'm fetching data from web service and display this data in my application.
I'm fetching the data periodically and updating the application accordingly.
I want to run this operation as a background process, so it'll download the data and if there is an update, I warn the user when s/he watching TV.
The other thing I want is to start application on TV launch, is it possible?
Only 'ticker' type application can run in background. It is not supported in Europe and will not pass certification for Samsung Apps store.
Samsung Smart TV suppots push-notifications (samsungdforum.com/Guide/art00080/index.html).
It will allow you to show popup-message on screen with ability to launch your application.
You cannot auto-start application on TV launch.
The only way is to use custom firmware like SamyGo (http://www.samygo.tv/)
About the "background process"... as far as we assume that JavaScript's setTimeout or setInterval can be used to execute application's "internal" background process, there is no problem - Just DO it! :)
But if you were thinking about system's background process - for ex. crontab of device - it's impossible.
for your second doubt
Auto launch of the application is supported in ticker applications only and there is property in config file of the ticker widget :
"<ticker itemtype="boolean">y</ticker>"
if u mark it as y(as above) than you can set up auto launch of the app from
Menu->smart Hub->Apps settings->auto ticker. in 2014 Samsung Smart TV models.
Since Tv has limited memory I wont recommend background process like setInterval or setTimout. You may have refresh button for the same.
And They must have some js api for controlling internal method of tv application (not sure). Anyway if you really want to use interval/timeout dont forget to clear it.
As it shown here, WebSocket is supported by Samsung Smart TV. Your server application can trigger TV application when an update is appear. It is more efficient way than polling server with ajax for your case.
(Edit) I wrote a sample application to demonstrate:
https://github.com/aksakalli/socket.io-hello-world
(Edit 2) I see that your problem is about platform limits. My answer is just about approaches that can be applied then I have very limited experience on Samsung Smart TV.
I have a system tray icon that receives incoming phone calls. When a call comes in, I want to send the phone number into a web app that does stuff on that phone number. We can launch the web app with the phone number in a query string, but if I launch a new window for every call, it will junk up the user's computer with lots of browser instances, would not preserve the location/size the user moved the browser window to, and would take longer than just refreshing the page. Instead, I'd like to make the Win32 app re-use the same IE browser window and just send the web app the new phone number every time a new call comes in. I'm envisioning somehow sending a Windows message, or somehow instructing the IE browser to run a certain javascript event with some data? I can see why doing the reverse (javascript out to Win32) would be a security issue, but this would be just sending a message from Win32 to javascript.
So I'm specifically NOT asking how to do what's been answered in this question: How to Interact Between Web App and Windows Form Application
That user was asking how to launch a Win32 app from Javascript and pass data to the win32 app. Roughly, I need to do the opposite. I need to send data from a Win32 app into a running javascript program.
Note also that I'm not asking how to launch one IE window with arguments to Javascript one time; I can easily do that with query strings. What I'm asking is how can I / is it possible to pass data from a running Win32 app outside the browser to a running Javascript app inside a browser window.
Since you can send the phone number to the site through a querystring, all you really need to do is tell IE to navigate to a URL of your choosing.
To that end, you can use what is in this KB article to help you find the instance of IE you want to connect to.
http://support.microsoft.com/kb/299356/de
Granted, its in Visual J++ of all things, but since you are interacting through COM Automation, the calls should be easy to translate.
Once you have the instance of IE (in the form of an IWebBrowser2 interface implementation) you can simply call the Navigate or Navigate2 method with the URL you need (with the phone number in the query string of course).
You should be able to find the handle of the IE window, and then send messages (keypresses perhaps) to it. Use a bit of javascript to capture all keypress activity and you have a very simple method to transfer information.
It won't be easy (the devil is in the details - issues of focus, etc) but it should be possible.
-Adam
What you need is a way to How To Connect to a Running Instance of Internet Explorer
(caveat: this one is in Java, but you should be able to translate it to C# quite easy).
Or you could enumerate the top-level windows to find a particular instance of IE and then Get IHTMLDocument2 from a HWND (this one is in C++, so you might need to do some Win32 interop; you can find all necessary declarations on PInvoke.net).
If you search on your friendly local search engine for How to connect to a running instance of Internet Explorer, you will find a lot more info.
Once you get to the document, you can either invoke the JavaScript function through the document scripting interface, or you can just navigate it to your page and pass the phone number as a parameter.
you could possibly get around the security and other issues with this method by using a web service on the web server, and have the win32 app update the web service and have the web page poll the same web service every however many seconds. Then you the the option of mapping that number to a database and getting additional information.
The only draw back is that instead of being immediate, there is a delay to get the information displayed in the browser.
How about you write an active-x control that you create in the browser using Javascript. This is effectively an Explorer browser plugin. Same idea for Mozilla, etc., except they use a different plug-in structure. This lets you support other browsers in the future.
The control can talk to your win32 app using a pipe or a socket or whatever type of inter-process communications and is then accessible as a Javascript object.