Develop Reference

Eliminate: ISP Injects Pages with Iframe Script for Ads

So my ISP (Smartfren; Indonesia) has decided to start injecting all non-SSL pages with an iframing script that allows them to insert ads into pages. Here's what's happening:
My browser sends a request to the server. ISP intercepts it and instead returns a javascript that loads the requested page inside an iframe.
Aside being annoying in principle, this injection also breaks any number of standard page functionality; and presents possible security hazards.
What I've tried to do so far:
Using a GreaseMonkey script to nix away the injected code and redirect to the original URL. Result: Breaks some legitimate iframes. Also, the ISP's code gets executed, because GreaseMonkey only kicks in after the page is loaded.
Using Privoxy for a local proxy and setting up a filter to clean up the injection and replace it with a plain javascript redirect to the original URL. Result: Breaks some legitimate iframes. ISP's code never gets to the browser.
You can view the GreaseMonkey and Privoxy fixes I've been working on at the following paste: http://pastebin.com/sKQTvgY2 ... along with a sample of the ISP's injection.
Ideally I could configure Privoxy to immediately resend the request when the alteration is detected, instead of filtering out the injected JS and replacing it with a JS redirection to the original URL. (The ISP-injection gets switched off when the same request is resent without delay.) I'm yet to figure out how to accomplish that. I believe it'd fix the iframe-breaking problem.
I know I could switch to a VPN or use the Tor browser. (Or change the ISP.) I'm hoping there's another way around. Any suggestions on how to eliminate this nuisance?

Actually now I have a solution:
The ISP proxy react on the Accept: header that the browser sends.
So this is the default for firefox:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Now we are going to change this default:
And set it to: Accept: */*
Here is how to setup header hacker for google chrome
Set the title to anything you like:NO IFRAME
Append/replace select replace with
String */*
And Match string to .* and then click add.
In the permanent header switches
Set domain to .* and select the rule you just created
PS: changing it in the firefox settings does not work 100% because some request like ajax seem to bypass it so a plugin is the only way as it literally intercepts every outgoing browser request
That's it no more iframes!!!
Hope this helps!

UPDATE: Use DNSCrypt is the best solution 😁
OLD ANSWER
Im using this method
Find resource that contain iframe code (use chrome dev tool)
Block the url with proxy or host file
I'm using linux, so i edited my hosts file on
/etc/hosts
Example :
127.0.0.1 ibnads.xl.co.id

A couple of requests with user# in URL lead to "Policy breach notice" from Google AdSense

I've recently got an email from Google, saying that they are going to ban my AdSense account because I'm sending Personally Identifiable Information to them with my Google AdSense tag requests. It says that around 1% of requests from my website have a referrer of:
some_user#my_website.com/some/subpage
and they consider some_user#my_website.com to be PII (even though it can be completely made up abcd1234#my_website.com). More on this here: https://support.google.com/adsense/answer/6163366?hl=en .
I never link to this kind of URLs (the only form I use is my_website.com/some/subpage), but I guess my users sometimes enter it manually (since product-wise my website is providing an email service, it may seem reasonable by some logic).
I figured URI of some_user#my_website.com/some/subpage is legal since http basic auth allows for specifying user like this. When I entered it manually to Firefox, some_user# disappears from the location bar but in the Net panel of Firebug I can see all files are indeed requested from some_user#my_website.com/some/subpage and that's how Google sees it too.
I though that as a brute-force solution even something like:
if uri contains '#':
redirect to my_website.com
would do.
I'm using NGINX/UWSGI/Python Paste + JS. I've tried to implement the above condition both on server side and in JS, but my URI always says my_website.com/some/subpage even if I manually put some_user#my_website.com/some/subpage in the browser address bar.
I've also tried configuring basic_auth in NGINX to disallow providing any user but with no effect.
How do I get rid of these requests?
How do I get the FULL URI (with some_user#) in JS? I tried document.URI and window.location.href but they didn't contain the user part...

Apparently presence of user# part in the URI can be detected by examining window.location.href. I haven't noticed it before since window.location.href only contains user# in Webkit-based browsers (e.g. Chrome, Opera, Safari) but not in Firefox!
To resolve the problem I've added a check on that in JS + a JS redirect to an URL without user[:password]#.
Hopefully Google uses the same variable to figure out referrer for the ad requests, so it get PII only from Webkit browsers & fixing it for Webkit suffices. Will keep you posted.

How to handle basic authentication with protractor?

I'm trying protractor to write a few tests in a non angular application. I have to login in a page trough basic authentication in google chrome, but i have no idea how.
I already tried baseUrl: 'https://username:password#url' and capabilities: {
'browserName': 'chrome',
'chromeOptions' : {
args: ['--login-user=foo', '--login-password=bar']
}
}
But none if these worked for me. Anyone knows how to do it? I'm having some hard time on it.

You can set the URL as http://username:password#yourdomain.example. Chrome will handle it!

The short answer is there is no easy way of doing it on chrome because they do not support modifying request headers -- see https://code.google.com/p/selenium/issues/detail?id=141 (title says response headers, but if you read it, it's for all headers).
That being said, there are ways to do it, albeit difficult.
1) Find a chrome extension/plugin that allows you to modify header. A simple search bring up many of them: https://chrome.google.com/webstore/search/modify%20header. You'll need to add the plugin to webdriver: see Is it possible to add a plugin to chromedriver under a protractor test?.
2) You can use browsermob-proxy (https://github.com/lightbody/browsermob-proxy); this way you route your traffic through the proxy, which would add the headers for you.
From the docs:
POST /proxy/[port]/auth/basic/[domain] - Sets automatic basic authentication for the specified domain
Payload data should be json encoded username and password name/value pairs (ex: {"username": "myUsername", "password": "myPassword"}
There's a node project that may help you, https://github.com/zzo/browsermob-node, but you would still need to set up your proxy server yourself.
Both ways for chrome are complex, but would get you what you want. (or you can stick with firefox and follow Robert's answer)

As of version 59 Chrome no longer supports URLs with embedded credentials.
To work around this I wrote the authenticator-browser-extension Node module, which might be useful if you're using Protractor, WebDriver.io or similar test runners.
To use the module install it from npm:
npm install --save-dev authenticator-browser-extension
And import in the protractor.conf.js:
const { Authenticator } = require('authenticator-browser-extension');
exports.config = {
capabilities: {
browserName: 'chrome',
chromeOptions: {
extensions: [
Authenticator.for('username', 'password').asBase64()
]
}
},
}
Pro tip: remember not to commit your credentials with your code, consider using env variables instead.
Hope this helps!
Jan

It's because Firefox doesn't trust any site by default with sending the Windows auth info over. Even if you change it in the configurations manually, it won't affect protractor because it opens Firefox with an isolated configuration each time you run your end to end tests.
You'll need to programatically set up a Firefox profile and set its preferences such that it would trust localhost (or some other website, depending where the pages are loaded from)
First, check out this example. It shows how you can set up the profile and how you can set preferences.
https://github.com/juliemr/protractor-demo/tree/master/howtos/setFirefoxProfile
What it does is that it modifies the homepage for each new tab. In the same manner (with the firefoxProfile.setPreference method) you can change the preferences responsible for trusting websites. They're called "network.automatic-ntlm-auth.trusted-uris" and "network.negotiate-auth.delegation-uris". You'll need to set them both to "localhost". (Again, if they're at some other place, it's obviously that URL)

hankduan's browsermob-proxy solution worked for me on Chrome - but the latest revisions of browsermob are using a thing called littleproxy which does not support auth headers. Thusly I had to do browsermob-proxy -port 9090 --use-littleproxy false, which got things working.

You may use Windows Credentials Manager to avoid this pop-up being constantly shown on every attempt to log in.
Add your credentials to the 'Generic' category there, restart browser (including background apps running).
Some explanation I currently have: this pop-up is not 'browser' specific, it is 'in the middle', between browser and domain credentials verification. Thus browser features (save password, autofill) do not work completely. By the same reason Protractor / Selenium etc. do not have complete control over that pop-up - it is by design of the domain authentication.
As not completely sure if it is the only reason there are some other hints:
- you may also need to add your site to the IE (IE, not Chrome) list of trusted sites (Chrome grabs information from there);
- check "Automatic logon with current user name and password" in IE (not Chrome) - may not work if credentials you are using for the site are different from those you use to login to the machine.

If you're reading this in 2019, with Angular 7/8, consider this:
https://www.npmjs.com/package/authenticator-browser-extension
I find it much easier than the solutions suggested above.

jQuery ajax on internal network (phonegap android)

I am working on a phonegap application which runs simple ajax requests to a rendered website, pulls down the html and reskins content locally.
A requirement is that websites on an internal network can be requested. I have connections via global ip addresses working fine, but cannot get the ajax to work on internal networks.
For example connecting to a 10.0.0.xx address will always fail. The device is connected to the same network and I can access the websites on the internal network fine in browser.
Is there any way around this?

Make sure you modified your domain whitelist file correctly to allow external access.
For more information about domain whitelist, check the online doc: http://docs.phonegap.com/en/2.1.0/guide_whitelist_index.md.html#Domain%20Whitelist%20Guide
Hope this helps. Let me know if this works for after modifying your domain whitelist file.
Example of domain whitelist configuration for iOS:
Open your file Cordova.plist and check that you made the following:
1 - Set OpenAllWhitelistURLsInWebView to YES
2 - Right click on ExternalHosts -> Add Row
3 - Set the String value of your new added row to *, so you should have your new line look like this:
Item0 String *
Normally, you would replace * with the external URL you want provide access to. But, to make sure that the problem really comes from the whitelist, we will use *.

RESTful cookie path fails in IE without trailing slash

I have been asked to build a tabbed section on pages that have RESTful URLs without a trailing slash (.NET MVC), for example http://myhost/books/the-amber-spyglass
For the tabbed sections to be a bit more user friendly I want them to remember which tab was open on each page as the user moves around, so that if they return to a book they were previously on the same tab remains opened. This is achieved by setting a cookie named "tab" with value set to the tabid and path set to the page they are on (so it doesn't affect tabs on other pages). So far pretty basic stuff you'd think, and it does work quite nicely too.
Enter Internet Explorer.
In IE it turns out a cookie with path /books/the-amber-spyglass will NOT match the above URL and consequently won't get set properly. If I add a trailing slash so the path is /books/the-amber-spyglass/ instead it works fine - but I cannot change our URL schema, and even if I could "the-amber-spyglass" is a resource, NOT a folder.
Does anyone have a recommended solution to this, surely exceedingly common, problem?
Many thanks in advance,
JS

See http://blogs.msdn.com/ieinternals/archive/2009/08/20/WinINET-IE-Cookie-Internals-FAQ.aspx
Due to an obscure bug in the
underlying WinINET InternetGetCookie
implementation, IE’s document.cookie
will not return a cookie if it was set
with a path attribute containing a
filename.
For instance, if a page sets a cookie
on itself like so:
Set-Cookie:
HTTPSet-PathCookie=PASS;path=/check.htm
…the cookie will be sent with HTTP
requests but will not appear in the
document.cookie collection.