Is there any way to consistently detect PhantomJS/CasperJS? I've been dealing with a spat of malicious spambots built with it and have been able to mostly block them based on certain behaviours, but I'm curious if there's a rock-solid way to know if CasperJS is in use, as dealing with constant adaptations gets slightly annoying.
I don't believe in using Captchas. They are a negative user experience and ReCaptcha has never worked to block spam on my MediaWiki installations. As our site has no user registrations (anonymous discussion board), we'd need to have a Captcha entry for every post. We get several thousand legitimate posts a day and a Captcha would see that number divebomb.
I very much share your take on CAPTCHA. I'll list what I have been able to detect so far, for my own detection script, with similar goals. It's only partial, as they are many more headless browsers.
Fairly safe to use exposed window properties to detect/assume those particular headless browser:
window._phantom (or window.callPhantom) //phantomjs
window.__phantomas //PhantomJS-based web perf metrics + monitoring tool
window.Buffer //nodejs
window.emit //couchjs
window.spawn //rhino
The above is gathered from jslint doc and testing with phantom js.
Browser automation drivers (used by BrowserStack or other web capture services for snapshot):
window.webdriver //selenium
window.domAutomation (or window.domAutomationController) //chromium based automation driver
The properties are not always exposed and I am looking into other more robust ways to detect such bots, which I'll probably release as full blown script when done. But that mainly answers your question.
Here is another fairly sound method to detect JS capable headless browsers more broadly:
if (window.outerWidth === 0 && window.outerHeight === 0){ //headless browser }
This should work well because the properties are 0 by default even if a virtual viewport size is set by headless browsers, and by default it can't report a size of a browser window that doesn't exist. In particular, Phantom JS doesn't support outerWith or outerHeight.
ADDENDUM: There is however a Chrome/Blink bug with outer/innerDimensions. Chromium does not report those dimensions when a page loads in a hidden tab, such as when restored from previous session. Safari doesn't seem to have that issue..
Update: Turns out iOS Safari 8+ has a bug with outerWidth & outerHeight at 0, and a Sailfish webview can too. So while it's a signal, it can't be used alone without being mindful of these bugs. Hence, warning: Please don't use this raw snippet unless you really know what you are doing.
PS: If you know of other headless browser properties not listed here, please share in comments.
There is no rock-solid way: PhantomJS, and Selenium, are just software being used to control browser software, instead of a user controlling it.
With PhantomJS 1.x, in particular, I believe there is some JavaScript you can use to crash the browser that exploits a bug in the version of WebKit being used (it is equivalent to Chrome 13, so very few genuine users should be affected). (I remember this being mentioned on the Phantom mailing list a few months back, but I don't know if the exact JS to use was described.) More generally you could use a combination of user-agent matching up with feature detection. E.g. if a browser claims to be "Chrome 23" but does not have a feature that Chrome 23 has (and that Chrome 13 did not have), then get suspicious.
As a user, I hate CAPTCHAs too. But they are quite effective in that they increase the cost for the spammer: he has to write more software or hire humans to read them. (That is why I think easy CAPTCHAs are good enough: the ones that annoy users are those where you have no idea what it says and have to keep pressing reload to get something you recognize.)
One approach (which I believe Google uses) is to show the CAPTCHA conditionally. E.g. users who are logged-in never get shown it. Users who have already done one post this session are not shown it again. Users from IP addresses in a whitelist (which could be built from previous legitimate posts) are not shown them. Or conversely just show them to users from a blacklist of IP ranges.
I know none of those approaches are perfect, sorry.
You could detect phantom on the client-side by checking window.callPhantom property. The minimal script is on the client side is:
var isPhantom = !!window.callPhantom;
Here is a gist with proof of concept that this works.
A spammer could try to delete this property with page.evaluate and then it depends on who is faster. After you tried the detection you do a reload with the post form and a CAPTCHA or not depending on your detection result.
The problem is that you incur a redirect that might annoy your users. This will be necessary with every detection technique on the client. Which can be subverted and changed with onResourceRequested.
Generally, I don't think that this is possible, because you can only detect on the client and send the result to the server. Adding the CAPTCHA combined with the detection step with only one page load does not really add anything as it could be removed just as easily with phantomjs/casperjs. Defense based on user agent also doesn't make sense since it can be easily changed in phantomjs/casperjs.
Related
I've tried on getting browser info from javascript - navigator.userAgent, but cannot find any related string with "Vivaldi".
Please help to suggest if there are any other programming method or javascript can be used to detect Vivaldi browser?
The correct syntax in JavaScript is navigator.userAgent.
You can check window size, but it may be incorrect.
Vivaldi has its own sidebar and bottom bar that the others not.
But remember, user can change the window size to prevent detection.
According to Vivaldi, the userAgent will not have Vivaldi in it for a better user experience and mimic Chrome in all sites except a few known ones.
For the next release of Vivaldi, we have decided to try something different. The problem with our current approach is that with the web being almost infinite, we can’t possibly discover all the websites who have blocks set against us. Thus maintaining a list of sites where we present a non-Vivaldi User Agent is difficult. Instead, we will try doing the reverse. For a handful of sites where we know the label Vivaldi (and our version number) is responsibly used, we will present our full User Agent. Those sites being our own and a handful of interesting alternative search engines: duckduckgo.com, ecosia.org, qwant.com, and startpage.com. Every other site will get a User Agent that appears to be identical to Chrome.
src: https://vivaldi.com/blog/user-agent-changes/
Since you haven't mentioned your use case, assuming you are building sites, the best solution would be to do what everyone does. I recommend building websites agnostic to browsers and devices, but responsive to resolutions and feature support. Almost all of the browsers are running on Chromium these days, except Safari and Firefox (to my knowledge). So they are all going to work consistently across the board. For any given feature, one could always refer to w3c or mdn docs for compatibility and build around that.
Browser detection is also not quite reliable, as there are extensions to spoof the information. Like this extension that mimics a TV agent so one can run youtube console version on it.
On the other hand, once again I don't know your use case, one might benefit from platform detection over browser detection, like catering to mobile and TV users. Even then, if you have a site responsive enough the platform becomes non-essential information.
Windows 7 / Firefox Latest versions / Preferrably 64bit beta versions
Whenever a Firefox Profile is updated, the user is presented with a dialog to check for updates to the addons. I would like to interact with tha dialog programmatically so that it is automatically ok-ed.
So far my research has brought me to:
https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIWindowMediator
For that I need a windowtype to enumerate. The window type correspons to the attribute in the XUL definitions. On my installation of Firefox, these seem to be compressed inside "c:\Program Files\Mozilla Firefox\browser\omni.ja\chrome\browser\content\browser".
I have limited experience of the firefox API. I'm executing code through userchrome.js.
Any pointer would be gladly received.
UPDATE 1 20151125 0748: I believe this is the xul - https://archive.is/NyGBS - any pointer on how to "overlay" it within userchrome.js? I can also try to enumerate windows of windowtype="Addons:Compatibility".
UPDATE 2 20151125 0748: It looks like the extensions update dialog is not showed if I have:
user_pref("xpinstall.signatures.required", false);
Which marks the addon as unsafe but allows you to continue using it.
[Which solves my immediate problem, but I am sure I will have this requirement at another point for another dialog, so would still like to get to the bottom of this]
TL;DR version with info about the problem follows.
I currently have a vbs script that:
Deletes FIREFOX_RUNNING_PROFILE folder.
Copies FIREFOX_BASE_PROFILE folder-> FIREFOX_RUNNING_PROFILE folder.
Runs FIREFOX_RUNNING_PROFILE.
Whenever a firefox update takes place, a number of dialogs need to be confirmed by the user in relation to checking/updating extensions.
The outcome of this process will not stick to the profile as obviously given my setup the RUNNING_PROFILE is recreated every time.
Note that for the purposes of this discussion it doesn't matter whether the udpate is performed automatically. I have the ability to rewrite the prefs.js so i can change behaviour dynamically just by doing that and restarting the browser externally - it's vbs right now but I'm ready to move over to something else if I hit some limitations with process / window management, or I want this to be portable (e.g. python). Note these are not dealbreakers right now and I'd be happy for a reasaonbly sleek windows-only solution.
I have a few of this BASE_PROFILES, hence I would like to handle the updates programmatically.
I have not found a way of achieving this that does not involve interacting with the dialogs firefox presents in the GUI. And I have noticed that the dialogs popping up after each update can change.
Options:
1) userchrome.js -> I already use this to manage some imacros autostart features, including grabbing the profile name from Components.Interfaces. I am reporting snippets below to help the reader understand what is the most sophisticated snippet I've written against the Mozilla xpcom components:
var strProfileFilePath=Components.classes["#mozilla.org/file/directory_service;1"].getService(Components.interfaces.nsIProperties).get("ProfD", Components.interfaces.nsIFile).path;
I have had a look at the documentation and it looks rather impressive. It feels like it will let me do pretty much everything, so my next move here would be to undersrtand how to enumerate dialogs and somehow select the right one based on some properties, then push the "ok" button. I think that's pretty much all I need to do.
2) Autohotkey. I can try and send keys to the firefox window (not sure about the exact dialog inside the window). The dialogs normally have shortcuts that would respond to those to this keysends (nice theory). In autohotkey I can also "position-click", although it is less desirable for obvious reasons.
3) One of the avenues through Autohotkey involves using the MozRepl Firefox addon behind the scenes. Library discussed here https://archive.is/73u4f, github seems out of date https://github.com/arbiter34/FF-Control. I could then use mozrepl directly, no need for Autohotkey. I would rather code in python, java, or even javascript, powershell, vbs than autohotkey. Ar a first look it looks like it would give me the same power that userchrome.js currently offers.
4) Selenium. There are some profile manipulation capabilities I could exploit - although Selenium was not really designed to interact with Firefox own's dialogs, and I don't see an API to automate that. Rather one would synthesise a profile on the fly, installing on top all the required extensions from a known superprofile (that would contain a superset of all the extensions to be used), and applying whatever mods to prefs.js also dynamically. This sounds a bit ugly for my original requirements but experience has told me that ugly is still better than manual.
5) There is also the option of writing an addon, but again, I think I'ld be interacting with the same interfaces that I can get in userchrome.js (actually I believe userchrome.js is even better as it seems to have unfettered access to the core components, which is exactly what I sense I might need for my case). I would rather avoid having to write an addon, although I might be lost for words trying to explain why exactly.
I appreciate I have not "fully" investigated further, but I am at a point now where I need to elicit the opinion of the community before I start spending substantial time on either of these.
The below isn't strictly required to address the question above, but I feel it might be useful to share this as there must be a few others undergoing the same predicament:
Additional Points:
a. As a general principle, I have zero desire to update anything (firefox.exe, addons, language packs, plugins and what not) (plugins are actually completely disabled). I am ok with the way the system works, and I don't want to change it one millimitre. Yes, NOTHING, not even the security updates. This is because of the particular circumstances of my app, my particular estimation of the particular risks I'm taking, and my general aversion to risk. YMMV. NEVERTHELESS I'm still forced to upgrade whenever (1) I "make a mistake" (all it takes is for one profile to have the update settings wrong for the exe to be updated for all the profiles on the machine - doh! there's only one exe); (2) Firefox is crashing too much / unusable (happened twice); (3) I need functionality offered by the upgrade (hasn't happened); I would also like to reserve the option for the "very critical security updates" (sigh! sometimes even I can bow).
b. The arcane art of keeping your firefox profile integer across updates also involves search engines - it doesn't matter if I say I don't want to update them, whenever there's a firefox.exe update I will get the new standard fluff installed in the program folder. I currently have a way of resetting that by (1) emptying the firefox program subfolder containing the xml files (in %programfiles%\Mozilla...), and (2) deleting search-metadata.json and the searchplugins folder (in the profile folder), then copying them over from a baseline afresh. It all feels very greasy. I have yet to try this on the 64bit version I'm running right now as this is a minor issue and I would rather not introduce more entropy. The firefox search engines mechanism might have changed in the meantime.
I need to do some changes in my HTML file based on whether it is opened in browser or Some Desktop Application.
Till now I've tried this in my script:
alert(navigator.appName);
alert(navigator.appCodeName);
alert(navigator.platform);
But the values are coming same whether the HTML file is opened in browser or some application Software.
How can i set a variable which toggle its value from 0 to 1 when opened in application Software and vice versa.
PS: Application Software like Matlab, MS Office , Britanica Encyclopedia etc.
Short answer, you can't.
Longer answer, to some extent. What you need to do is make a list of differences in each desktop apps implementation, based on known flaws, missing/existing properties, user agent flavours and so on (some of this called "browser spoofing"), to be able to sort them out. It will still be possible to trick this if one want to though.
Normally there is another way of dealing with the differences, the question is what is the different behaviour you want between the 2?
As a sample, and if you still need/insists to detect this, there is device detection libraries which can help as a start, like https://51degrees.com/device-detection
I still recommend to find another way to solve it.
UPDATE
As it is easy to create a desktop app and return the values needed for your page to believe it is a normal browser, I think the best solution is to ask the user on first page hit if they are on a normal browser or not (the one who is not will normally know) and then store in a cookie/set a flag and act upon what the user selected.
I mean it is easy to cheat your page either way so better trust on user selection.
You can use 2 step verification for such cases as devices are getting quite varied. First you can detect devices by media queries strings using innerWidth property. And then you can apply second filter matched by protocol it supports.
Secondly, this SO question might help Frame Buster Buster ... buster code needed
Trying out Windows Universal apps with JavaScript I noticed the WinJS.Utilities.isPhone property is no longer available, which makes sense since there would be no reason to ask for that at runtime.
I do want to know just for testing purposes if there is a proper way of detecting the device my app is running in.
EDIT: My question is NOT about detecting a mobile browser. I'm talking about about a brand new Universal Windows App for Window 10 that can run on phones, desktops, tablets, Xbox, HoloLEns, IoT devices et all. WinJS had a property that would tell me whether I was running on the phone or not. That property is now gone. Please don't close this question due to duplicate with "detecting mobile browser". That is NOT what I need.
Caveat: Any form of device detection is fragile due to the dynamic nature of hardware - a new device could come along tomorrow that breaks your app's logic. It is best to use these APIs only for telemetry / analytics rather than to trigger runtime behaviour.
More often than not, what you really want to know is some attribute of the device or the app that is not inherently tied to the device family (does this device support SystemTray API? Is there a keyboard attached? Is the window more than 500px wide? etc.).
That said, in Windows 10 you can query the DeviceFamily via AnalyticsInfo.VersionInfo.DeviceFamily and it will tell you things like "Mobile" or "Desktop" or "Xbox" etc. (where "Mobile" could be any class of device - phone, tablet, etc.). There is also a property DeviceForm that might be helpful, but again you can't really rely on it at runtime to deterministically say you're running on "a phone."
So basically the answer is "you can use these APIs for telemetry but don't hard-code any values into your app lest it break when a new device arrives on the market." At the very least, always make sure you handle the case where the returned value isn't one you know about a-priori.
You can also check out the following links
http://www.abeautifulsite.net/detecting-mobile-devices-with-javascript/
http://www.sitepoint.com/detect-mobile-devices-jquery/
and of course a similar post here on stackoverflow with a good answer
Detecting a mobile browser
And talking about Windows 10, extracting from Winjs Github repo, here is the answer.
https://github.com/winjs/winjs/issues/601#issuecomment-87137485
There are numerous JS libs to detect which platform/device is used.
I personally love using this lib: https://github.com/kaimallea/isMobile
You will then be able to detect mobile device in such a way:
isMobile.apple.tablet
isMobile.android.phone
and so on.
In case you have an idea to implement such lib yourself, keep in mind that it takes some efforts to keep it up-to-date, because ways of detecting mobile device may change over time.
In general, common way of detecting user device is checking query headers.
Keep in mind, though, that you can't absolutely rely on this information - headers may be easily modified. Google for User Agent for more info.
So "omitting auth process for users with phones" is extremely bad idea
Browsers allow extensions to inject code, manipulate the DOM, etc.
Over the years, I have noticed lots and various uncaught errors (using window.onerror) on a website (app) I am watching, generated by unknown browser extensions on Firefox, Chrome and Internet Explorer (all versions).
These errors didn't seem to be interrupting anything. Now I want to increase the security of this website, because it will start processing credit cards. I have seen with my own eyes malware/spyware infecting browsers with modified browser extensions (innocent browser extension, modified to report to attackers/script kiddies) working as keyloggers (using trivial onkey* event handlers, or just input.value checks).
Is there a way (meta tag, etc.) to inform a browser to disallow code injection or reading the DOM, standard or non-standard? The webpage is already SSL, yet this doesn't seem to matter (as in give a hint to the browser to activate stricter security for extensions).
.
Possible workarounds (kind of a stretch vs. a simple meta tag) suggested by others or off the top of my head:
Virtual keyboard for entering numbers + non textual inputs (aka img for digits)
remote desktop using Flash (someone suggested HTML5, yet that doesn't solve the browser extension listening on keyboard events; only Flash, Java, etc. can).
Very complex Javascript based protection (removes non white listed event listeners, in-memory input values along with inputs protected with actual asterix characters, etc.) (not feasible, unless it already exists)
Browser extension with the role of an antivirus or which could somehow protect a specific webpage (this is not feasible, maybe not even possible without creating a huge array of problems)
Edit: Google Chrome disables extensions in Incognito Mode, however, there is no standard way to detect or automatically enable Incognito Mode and so a permanent warning must be displayed.
Being able to disable someone's browser extension usually implies taking over the browser. I don't think it's possible. It would be a huge security risk. Your purpose maybe legit, but consider the scenario of webmasters programatically disabling addblockers for users in order to get them to view the advertisments.
In the end it's the user's responsability to make sure they have a clean OS when making online banking transactions. It's not the website's fault that the user is compromised
UPDATE
We should wrap things up.
Something like:
<meta name="disable-extension-feature" content="read-dom" />
or
<script type="text/javascript">
Browser.MakeExtension.MallwareLogger.to.not.read.that.user.types(true);
</script>
doesn't exist and i'm sure there won't be implemented in the near future.
Use any means necessary to best use the current up to date existing technologies and design your app as best as you can security wise. Don't waste your energy trying to cover for users who souldn't be making payments over the internet in the first place
UPDATE (2019-10-16): This isn't a "real" solution - meaning you should not rely on this as a security policy. Truth is, there is no "real" solution because malicious addons can hijack/spoof JavaScript in a way which in not detectable. The technique below was more of an exercise for me to figure out how to prevent simple key logging. You could expand on this technique to make it more difficult for hackers... but Vlad Balmos said it best in his answer below - Don't waste your energy trying to cover for users who souldn't be making payments over the internet in the first place.
You can get around the key logging by using a javascript prompt. I wrote a little test case (which ended up getting a little out of hand). This test case does the following:
Uses a prompt() to ask for the credit card number on focus.
Provides a failsafe when users check "prevent additional dialogs" or if the user is somehow able to type in the CC field
Periodically checks to make sure event handlers haven't been removed or spoofed and rebinds/ warns the user when necessary.
http://jsfiddle.net/ryanwheale/wQTtf/
prompt('Please enter your credit card number');
Tested in IE7+, Chrome, FF 3.6+, Android 2.3.5, iPad 2 (iOS 6.0)
Your question is interesting, and thoughtful (+1'd), however unfortunately the proposed security does not provide real security, thus no browser will ever implement it.
One of the core principle on browser/web/network security is to resist from the desire of implementing a bogus security feature. Web will be less secure with the feature than without!
Hear me out:
Everything execute on the client-side can be manipulated. Browsers are just another HTTP clients that talks to server; server should never ever trust the computation result, or checks done in front-end Javascript. If someone can simply bypass your "security" check code executed in a browser with a extension, they can surely fire the HTTP request directly to your server with curl to do that. At least, in a browser, skilled users can turn to Firebug or Web Inspector and bypass your script, just like what you do when you debug your website.
The <meta> tag stopping extensions from injection does make the website more robust, but not more secure. There are a thousand ways to write robust JavaScript than praying for not having an evil extension. Hide your global functions/objects being one of them, and perform environment sanity check being another. GMail checks for Firebug, for example. Many websites detects Ad block.
The <meta> tag does make sense in terms of privacy (again, not security). There should be a way to tell the browser that the information currently present in the DOM is sensitive (e.g. my bank balance) and should not be exposed to third parties. Yet, if an user uses OS from vender A, browser from vender B, extension from vender C without reading through it's source code to know exactly what they do, the user have already stated his trust to these venders. Your website will not be at fault here. Users who really cares about privacy will turn to their trusted OS and browser, and use another profile or private mode of the browser to check their sensitive information.
Conclusion: If you do all the input checks on sever-side (again), your website is secure enough that no <meta> tag can make it more secure. Well done!
I saw something similar being done many times, although the protection was directed in the other way: quite a few sites, when they offer sensitive information in a form of text would use a Flash widget to display the text (for example, e-mail addresses, which would be otherwise found by bots and spammed).
Flash applet may be configured to reject any code that comes from the HTML page, actually, unless you specifically expect this to be possible, it will not work out of the box. Flash also doesn't re-dispatch events to the browser, so if the keylogger works on the browser level, it won't be able to log the keys pressed. Certainly, Flash has its own disadvantages, but given all other options this seems the most feasible one. So, you don't need remote desktop via Flash, simple embedded applet will be just as good. Also, Flash alone can't be used to make a fully-functional remote desktop client, you'd be looking into NaCl or JavaFX, which would make this only usable by corporate users and only eventually by private users.
Other things to consider: write your own extension. Making Firefox extension is really easy + you could reuse a lot of your JavaScript code since it can also use JavaScript. I never wrote a Google Chrome or MSIE extension, but I would imagine it's not much more difficult. But you don't need to turn it into an antivirus extension. With the tools available, you could make it so no other extension can eavesdrop on what's going on inside your own extension. I'm not sure how friendly your audience will greet that, but if you are targeting corporate sector, then that audience is, in a way, a very good one, as they don't get to choose their tools... so you can just obligate them to use the extension.
Any more ideas? - well, this one is very straight-forward and efficient: have users open a pop-up window / separate tab and disable JavaScript in it :) I mean, you could decline to accept a credit card info if the JavaScript is enabled in the browser - obviously, it is very easy to check. This would require some mental effort from the users to find the setting, where they can disable it + they will be raging over a pop-up window... but almost certainly this will disable all code injection :)
This wont work, but i'll try something around document.createElement = function(){};
That should affect client side scripts (greasemonkey)
You can also try to submit the current DOM using an hidden input
myform.onsubmit=function(){myform.hiddeninput.value=document.body.innerHTML;} and check server side for unwanted DOM elements. I guess using a server side generated id/token on every element can help here (as injected DOM node will surely miss it)
=> page should look like
<html uniqueid="121234"> <body uniqueid="121234"><form uniqueid="121234"> ...
So finding un-tracked elements in the POST action should be easy (using xpath for example)
<?php
simplexml_load_string($_POST['currentdom'])->xpath("*:not(#uniqueid)") //style
Something around that for the DOM injection issue.
As for the keylogging part, i don't think you can do anything to prevent keylogger from a client side perspective (except using virtual keyboard & so), as there is no way to discern them from the browser internals. If you are paranoid, you should try a 100% canvas generated design (mimicking HTML element & interaction) as this might protect you (no DOM element to be bound to), but that would mean creating a browser in a browser.
And just that we all know we cannot explicitly block the extensions from our code,
one another way can be to find the list of event listeners attached to key fields like password, ssn and also events on body like keypress, keyup, keydown and verify whether the listener belongs to your code, if not just throw a flash message to disable addons.
And you can attach mutation events to your page and see if there are some new nodes being created / generated by a third party apart from your code.
ok its obvious that you will get into performance issues, but thats a trade off for your security.
any takers ?