I am looking to monitor all outgoing HXR requests on the page in order to intercept a specific one that I wish to extract its data to log.
Here's the specific request I am trying to intercept:
Request header from the console
Request body I want to extract
Here's what I tried so far:
(function(send) {
XMLHttpRequest.prototype.send = function(body) {
var info="send data\r\n"+body;
console.log(info);
send.call(this, body);
};
})(XMLHttpRequest.prototype.send);
This seems to be somehow working because I get some requests logged in the console but not the ones I am looking for.
I also get some send data [object FormData] logged.
I am probably missing something about which HXR requests I am intercepting with my script and would love some help!
Thanks for your time!
This may just be the reason for the implicit conversion, you can try the JSON.Stringify(body) method.
I'm trying to load a cross-domain HTML page using AJAX but unless the dataType is "jsonp" I can't get a response. However using jsonp the browser is expecting a script mime type but is receiving "text/html".
My code for the request is:
$.ajax({
type: "GET",
url: "http://saskatchewan.univ-ubs.fr:8080/SASStoredProcess/do?_username=DARTIES3-2012&_password=P#ssw0rd&_program=%2FUtilisateurs%2FDARTIES3-2012%2FMon+dossier%2Fanalyse_dc&annee=2012&ind=V&_action=execute",
dataType: "jsonp",
}).success( function( data ) {
$( 'div.ajax-field' ).html( data );
});
Is there any way of avoiding using jsonp for the request? I've already tried using the crossDomain parameter but it didn't work.
If not is there any way of receiving the html content in jsonp? Currently the console is saying "unexpected <" in the jsonp reply.
jQuery Ajax Notes
Due to browser security restrictions, most Ajax requests are subject to the same origin policy; the request can not successfully retrieve data from a different domain, subdomain, port, or protocol.
Script and JSONP requests are not subject to the same origin policy restrictions.
There are some ways to overcome the cross-domain barrier:
CORS Proxy Alternatives
Ways to circumvent the same-origin policy
Breaking The Cross Domain Barrier
There are some plugins that help with cross-domain requests:
Cross Domain AJAX Request with YQL and jQuery
Cross-domain requests with jQuery.ajax
Heads up!
The best way to overcome this problem, is by creating your own proxy in the back-end, so that your proxy will point to the services in other domains, because in the back-end not exists the same origin policy restriction. But if you can't do that in back-end, then pay attention to the following tips.
**Warning!**
Using third-party proxies is not a secure practice, because they can keep track of your data, so it can be used with public information, but never with private data.
The code examples shown below use jQuery.get() and jQuery.getJSON(), both are shorthand methods of jQuery.ajax()
CORS Anywhere
2021 Update
Public demo server (cors-anywhere.herokuapp.com) will be very limited by January 2021, 31st
The demo server of CORS Anywhere (cors-anywhere.herokuapp.com) is meant to be a demo of this project. But abuse has become so common that the platform where the demo is hosted (Heroku) has asked me to shut down the server, despite efforts to counter the abuse. Downtime becomes increasingly frequent due to abuse and its popularity.
To counter this, I will make the following changes:
The rate limit will decrease from 200 per hour to 50 per hour.
By January 31st, 2021, cors-anywhere.herokuapp.com will stop serving as an open proxy.
From February 1st. 2021, cors-anywhere.herokuapp.com will only serve requests after the visitor has completed a challenge: The user (developer) must visit a page at cors-anywhere.herokuapp.com to temporarily unlock the demo for their browser. This allows developers to try out the functionality, to help with deciding on self-hosting or looking for alternatives.
CORS Anywhere is a node.js proxy which adds CORS headers to the proxied request.
To use the API, just prefix the URL with the API URL. (Supports https: see github repository)
If you want to automatically enable cross-domain requests when needed, use the following snippet:
$.ajaxPrefilter( function (options) {
if (options.crossDomain && jQuery.support.cors) {
var http = (window.location.protocol === 'http:' ? 'http:' : 'https:');
options.url = http + '//cors-anywhere.herokuapp.com/' + options.url;
//options.url = "http://cors.corsproxy.io/url=" + options.url;
}
});
$.get(
'http://en.wikipedia.org/wiki/Cross-origin_resource_sharing',
function (response) {
console.log("> ", response);
$("#viewer").html(response);
});
Whatever Origin
Whatever Origin is a cross domain jsonp access. This is an open source alternative to anyorigin.com.
To fetch the data from google.com, you can use this snippet:
// It is good specify the charset you expect.
// You can use the charset you want instead of utf-8.
// See details for scriptCharset and contentType options:
// http://api.jquery.com/jQuery.ajax/#jQuery-ajax-settings
$.ajaxSetup({
scriptCharset: "utf-8", //or "ISO-8859-1"
contentType: "application/json; charset=utf-8"
});
$.getJSON('http://whateverorigin.org/get?url=' +
encodeURIComponent('http://google.com') + '&callback=?',
function (data) {
console.log("> ", data);
//If the expected response is text/plain
$("#viewer").html(data.contents);
//If the expected response is JSON
//var response = $.parseJSON(data.contents);
});
CORS Proxy
CORS Proxy is a simple node.js proxy to enable CORS request for any website.
It allows javascript code on your site to access resources on other domains that would normally be blocked due to the same-origin policy.
CORS-Proxy gr2m (archived)
CORS-Proxy rmadhuram
How does it work?
CORS Proxy takes advantage of Cross-Origin Resource Sharing, which is a feature that was added along with HTML 5. Servers can specify that they want browsers to allow other websites to request resources they host. CORS Proxy is simply an HTTP Proxy that adds a header to responses saying "anyone can request this".
This is another way to achieve the goal (see www.corsproxy.com). All you have to do is strip http:// and www. from the URL being proxied, and prepend the URL with www.corsproxy.com/
$.get(
'http://www.corsproxy.com/' +
'en.wikipedia.org/wiki/Cross-origin_resource_sharing',
function (response) {
console.log("> ", response);
$("#viewer").html(response);
});
The http://www.corsproxy.com/ domain now appears to be an unsafe/suspicious site. NOT RECOMMENDED TO USE.
CORS proxy browser
Recently I found this one, it involves various security oriented Cross Origin Remote Sharing utilities. But it is a black-box with Flash as backend.
You can see it in action here: CORS proxy browser
Get the source code on GitHub: koto/cors-proxy-browser
You can use Ajax-cross-origin a jQuery plugin.
With this plugin you use jQuery.ajax() cross domain. It uses Google services to achieve this:
The AJAX Cross Origin plugin use Google Apps Script as a proxy jSON
getter where jSONP is not implemented. When you set the crossOrigin
option to true, the plugin replace the original url with the Google
Apps Script address and send it as encoded url parameter. The Google
Apps Script use Google Servers resources to get the remote data, and
return it back to the client as JSONP.
It is very simple to use:
$.ajax({
crossOrigin: true,
url: url,
success: function(data) {
console.log(data);
}
});
You can read more here:
http://www.ajax-cross-origin.com/
If the external site doesn't support JSONP or CORS, your only option is to use a proxy.
Build a script on your server that requests that content, then use jQuery ajax to hit the script on your server.
Just put this in the header of your PHP Page and it ill work without API:
header('Access-Control-Allow-Origin: *'); //allow everybody
or
header('Access-Control-Allow-Origin: http://codesheet.org'); //allow just one domain
or
$http_origin = $_SERVER['HTTP_ORIGIN']; //allow multiple domains
$allowed_domains = array(
'http://codesheet.org',
'http://stackoverflow.com'
);
if (in_array($http_origin, $allowed_domains))
{
header("Access-Control-Allow-Origin: $http_origin");
}
I'm posting this in case someone faces the same problem I am facing right now. I've got a Zebra thermal printer, equipped with the ZebraNet print server, which offers a HTML-based user interface for editing multiple settings, seeing the printer's current status, etc. I need to get the status of the printer, which is displayed in one of those html pages, offered by the ZebraNet server and, for example, alert() a message to the user in the browser. This means that I have to get that html page in Javascript first. Although the printer is within the LAN of the user's PC, that Same Origin Policy is still staying firmly in my way. I tried JSONP, but the server returns html and I haven't found a way to modify its functionality (if I could, I would have already set the magic header Access-control-allow-origin: *). So I decided to write a small console app in C#. It has to be run as Admin to work properly, otherwise it trolls :D an exception. Here is some code:
// Create a listener.
HttpListener listener = new HttpListener();
// Add the prefixes.
//foreach (string s in prefixes)
//{
// listener.Prefixes.Add(s);
//}
listener.Prefixes.Add("http://*:1234/"); // accept connections from everywhere,
//because the printer is accessible only within the LAN (no portforwarding)
listener.Start();
Console.WriteLine("Listening...");
// Note: The GetContext method blocks while waiting for a request.
HttpListenerContext context;
string urlForRequest = "";
HttpWebRequest requestForPage = null;
HttpWebResponse responseForPage = null;
string responseForPageAsString = "";
while (true)
{
context = listener.GetContext();
HttpListenerRequest request = context.Request;
urlForRequest = request.RawUrl.Substring(1, request.RawUrl.Length - 1); // remove the slash, which separates the portNumber from the arg sent
Console.WriteLine(urlForRequest);
//Request for the html page:
requestForPage = (HttpWebRequest)WebRequest.Create(urlForRequest);
responseForPage = (HttpWebResponse)requestForPage.GetResponse();
responseForPageAsString = new StreamReader(responseForPage.GetResponseStream()).ReadToEnd();
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Send back the response.
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseForPageAsString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
response.AddHeader("Access-Control-Allow-Origin", "*"); // the magic header in action ;-D
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
//listener.Stop();
All the user needs to do is run that console app as Admin. I know it is way too ... frustrating and complicated, but it is sort of a workaround to the Domain Policy problem in case you cannot modify the server in any way.
edit: from js I make a simple ajax call:
$.ajax({
type: 'POST',
url: 'http://LAN_IP:1234/http://google.com',
success: function (data) {
console.log("Success: " + data);
},
error: function (e) {
alert("Error: " + e);
console.log("Error: " + e);
}
});
The html of the requested page is returned and stored in the data variable.
To get the data form external site by passing using a local proxy as suggested by jherax you can create a php page that fetches the content for you from respective external url and than send a get request to that php page.
var req = new XMLHttpRequest();
req.open('GET', 'http://localhost/get_url_content.php',false);
if(req.status == 200) {
alert(req.responseText);
}
as a php proxy you can use https://github.com/cowboy/php-simple-proxy
Your URL doesn't work these days, but your code can be updated with this working solution:
var url = "http://saskatchewan.univ-ubs.fr:8080/SASStoredProcess/do?_username=DARTIES3-2012&_password=P#ssw0rd&_program=%2FUtilisateurs%2FDARTIES3-2012%2FMon+dossier%2Fanalyse_dc&annee=2012&ind=V&_action=execute";
url = 'https://google.com'; // TEST URL
$.get("https://images"+~~(Math.random()*33)+"-focus-opensocial.googleusercontent.com/gadgets/proxy?container=none&url=" + encodeURI(url), function(data) {
$('div.ajax-field').html(data);
});
<div class="ajax-field"></div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
You need CORS proxy which proxies your request from your browser to requested service with appropriate CORS headers. List of such services are in code snippet below. You can also run provided code snippet to see ping to such services from your location.
$('li').each(function() {
var self = this;
ping($(this).text()).then(function(delta) {
console.log($(self).text(), delta, ' ms');
});
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<script src="https://cdn.rawgit.com/jdfreder/pingjs/c2190a3649759f2bd8569a72ae2b597b2546c871/ping.js"></script>
<ul>
<li>https://crossorigin.me/</li>
<li>https://cors-anywhere.herokuapp.com/</li>
<li>http://cors.io/</li>
<li>https://cors.5apps.com/?uri=</li>
<li>http://whateverorigin.org/get?url=</li>
<li>https://anyorigin.com/get?url=</li>
<li>http://corsproxy.nodester.com/?src=</li>
<li>https://jsonp.afeld.me/?url=</li>
<li>http://benalman.com/code/projects/php-simple-proxy/ba-simple-proxy.php?url=</li>
</ul>
Figured it out.
Used this instead.
$('.div_class').load('http://en.wikipedia.org/wiki/Cross-origin_resource_sharing #toctitle');
I'm running up against a very frustrating bug. I'm not exactly sure what is happening, but I think xhr is doing some kind of cache on the response headers.
My app is using devise_token_auth for the backend authentication service. We're using it with rotating access-tokens, and so I have written a function that runs after every request.
function storeAndGetResponseHeaders(xhr) {
const headersObj = {};
headerKeys.filter((key) => xhr.getResponseHeader(key))
.forEach((key) => {
headersObj[key] = xhr.getResponseHeader(key);
window.sessionStorage.setItem(key, xhr.getResponseHeader(key));
});
return headersObj;
}
where headerKeys is ['access-token', 'client', 'expiry', 'uid', 'token-type']. So any response that has these headers it should save them into sessionStorage and then return them in an object which gets stored within my AJAX service that I wrote and added to every request. We're using rxjs, and this service is just a thin wrapper around it. This is what RxAjax.ajax looks like.
ajax(urlOrRequest) {
const request = typeof urlOrRequest === 'string' ? { url: urlOrRequest } : urlOrRequest;
request.headers = Object.assign({}, this.headers, urlOrRequest.headers);
request.url = `${this.baseUrl}${request.url}`;
return Observable.ajax(request).map(this.afterRequest, this);
}
where this.headers is the stored headers from last request (or the loaded headers from sessionStorage). this.afterRequest is what sets the headers from the response xhr.
My problem is that I'm getting bad values into my headers object (specifically old access tokens). What I've noticed is that when I add a logging statement of headersObj after assignment, sometimes it will have old response headers from a past request. However when I look at the request itself in the dev console Network tab, it doesn't show any of the auth headers in the response headers ('access-token', 'client', etc...). This gets fixed for a little while if I do a hard refresh on the browser, but comes back seemingly inexplicably.
Note we're using rxjs to make our requests, which might be relevant (but I don't think it is the cause of this problem, as I'm trying to read the headers from the original xmlhttprequest object). Thanks!
As Barmar suggested in the comments, it was a caching issue. There may be a bug in the chrome console, where it wasn't showing the cached headers that were on the cached request. Hence even though it looked like there were no auth headers there really were.
It looks like if you're using jQuery you can add the option cache: false to the request in order to prevent caching. Because I'm not, the first thing I did was try adding ?cache=${new Date().toJSON} to each request, which successfully busted the cache and fixed my problem (that is what cache: false in jQuery does).
Our backend is in rails, and so I ended up adding
before_action :set_cache_headers
...
private
def set_cache_headers
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
end
to my application controller. Now no requests are cached by the browser. Not sure if this will be our long term solution or not
I need to automatically perform authorization with a proxy server using an addon.
I also checked out the source of other extensions that do that, and they do it like this:
var httpRequestObserver =
{
observe: function(subject, topic, data)
{
if (topic === "http-on-modify-request") {
var channel = subject.QueryInterface(Components.interfaces.nsIHttpChannel);
channel.setRequestHeader("Proxy-Authorization", "Basic myauthorizationtoken" , false);
}
}
};
A request observer adds the authorization header to every outgoing request. That works fine if a regular proxy is set, however in my case i have a proxy pac url where only specific requests are done using the proxy, and others not!
In that case the authorization header which basically includes credentials in the clear is transmitted to every website not accessed via proxy.
Obviously that cannot be, therefore I need to find out whether the current request is a proxy request, and only then set the header.
Or find another way alltogether...
In chrome I did it like this:
chrome.webRequest.onAuthRequired.addListener(handleAuthRequest,
{urls: ["<all_urls>"]}, ["asyncBlocking"]);
function handleAuthRequest(details, callback) {
if (details.isProxy === true){
callback({authCredentials: {username: localStorage['login'],
password: localStorage['pass']}});
}
callback();
}
Clearly this is optimal! However I cannot see a nice way in firefox to do it.
turns out that subject has an attribute proxyInfo which is null or set, also proxyInfo has further attributes to examine
I know that you can't, when using an XMLHttpRequest, intercept a redirect or prevent it, as the browser will transparently follow it, but is it possible to either
A. Determine whether a request redirected, or
B. Determine where it redirected to? (assuming that the response gives no hints)
Example code:
$.post("/my-url-that-redirects/", {},
function(response, statusCode, xmlHttpRequest){
//Somehow grab the location it redirected to
}
);
In my case, firebug will first show a POST to the url, then a GET to the redirected url. Can that GET location be captured?
1) Use different status code than 301 (2**) (if request by ajax) and handle redirection on client side:
var STATUS = {
REDIRECT: 280
};
$.post('/redirected', {}, function(response, status, request) {
if (status == STATUS.REDIRECT) {
// you need to return the redirect url
location.href = response.redirectUrl;
} else {
$('#content').html(request.responseText);
}
});
2) DO NOT REDIRECT:
I use that in "redirect pattern" = redirecting after post request (you don't want to allow user to refresh the post request, etc..)
With ajax request, this is not necessary, so when the post request is ajax, I do forward instead (just forward to different controller - depends on your server-side framework, or what you are using...). POST requests are not cached by browsers.
Actually, I don't know what's the reason you need that, so this might not be so useful for you. This is helpful when server returns different responses for ajax requests than common requests, because when browser redirect ajax request, the redirected request is not XMLHttpRequest...
[updated]
You can access headers (of redirected request) like that:
$.post('redirected', {}, function(r, s, req) {
req.getAllResponseHeaders();
req.getResponseHeader('Location');
});
There should be 'Location' header, but it depends on the server, which headers are sent back...
After 4 years now it's possible to find the last redirect location using responseURL from XHR instance in Chrome 42+ (Opera 29+) and Firefox 39+ but it's not available in IE, Edge or safari yet.