I've tried use chrome.webRequest API and finally found out that it looks like google don't allow us to modify requestBodies of POST requests? I could only cancel it or modify its headers.
So is there any other way to modify the raw (not form) body of a post request? I know that a proxy server could do that but I want to deal with it using extension.
This works for some cases: first, save the body of the request in a variable in the onBeforeRequest listener. Then, in onBeforeSendHeaders you can either cancel or redirect the original request (sorry, Chrome gives you only two options to deal with the original). Also in onBeforeSendHeaders, you issue a new request (say, jquery ajax) to which you attach the old body from the variable, and the old headers - both can be modified/rewritten as needed.
(Minor catch: it won't let you set all of the headers for "security reasons", so you may add one more onBeforeSendHeaders listener to add the sensitive headers to the new request as well).
Works for cases when the request issuer is happy with a redirect or cancel as a response. If the request issuer expects the full actual response, intact, with no redirects, then it gets harder.
Chrome extension Netify allows request modification, including POST body
Related
I am trying to create a chrome extension that navigates to a webpage, lets the user click a button on the webpage that sends an asynchronous post request, and to read that response and use it in the extension.
Everything that I am finding from my research is telling me to create the request in the extension itself, which I do not want to do, because I need the web page to make the request itself.
Is there a way to listen to a post request on the page itself on my background script?
You can use chrome.webRequest and onBeforeRequest to fire when a request is about to occur chrome.webRequest.onBeforeRequest.addListener(function callback). Parameters contains the HTTP request data. You also need to supply an extraInfoSpec of requestBody to the listener.
Below is a sample code snippet how to use onBeforeRequest:
const WEB_REQUEST = chrome.webRequest;
WEB_REQUEST.onBeforeRequest.addListener(
function(details) {
if(details.method == "POST")
console.log(JSON.stringify(details));
},
);
For more information regarding POST request, try to read the Chrome Extensions developer guide: https://developer.chrome.com/extensions/getstarted
I want to capture the HTTP request header fields, primarily the Referer and User-Agent, within my client-side JavaScript. How may I access them?
Google Analytics manages to get the data via JavaScript that they have you embed in you pages, so it is definitely possible.
Related:
Accessing the web page's HTTP Headers in JavaScript
If you want to access referrer and user-agent, those are available to client-side Javascript, but not by accessing the headers directly.
To retrieve the referrer, use document.referrer.
To access the user-agent, use navigator.userAgent.
As others have indicated, the HTTP headers are not available, but you specifically asked about the referer and user-agent, which are available via Javascript.
Almost by definition, the client-side JavaScript is not at the receiving end of a http request, so it has no headers to read. Most commonly, your JavaScript is the result of an http response. If you are trying to get the values of the http request that generated your response, you'll have to write server side code to embed those values in the JavaScript you produce.
It gets a little tricky to have server-side code generate client side code, so be sure that is what you need. For instance, if you want the User-agent information, you might find it sufficient to get the various values that JavaScript provides for browser detection. Start with navigator.appName and navigator.appVersion.
This can be accessed through Javascript because it's a property of the loaded document, not of its parent.
Here's a quick example:
<script type="text/javascript">
document.write(document.referrer);
</script>
The same thing in PHP would be:
<?php echo $_SERVER["HTTP_REFERER"]; ?>
Referer and user-agent are request header, not response header.
That means they are sent by browser, or your ajax call (which you can modify the value), and they are decided before you get HTTP response.
So basically you are not asking for a HTTP header, but a browser setting.
The value you get from document.referer and navigator.userAgent may not be the actual header, but a setting of browser.
One way to obtain the headers from JavaScript is using the WebRequest API, which allows us to access the different events that originate from http or websockets, the life cycle that follows is this:
WebRequest Lifecycle
So in order to access the headers of a page it would be like this:
browser.webRequest.onHeadersReceived.addListener(
(headersDetails)=> {
console.log("Request: " + headersDetails);
},
{urls: ["*://hostName/*"]}
);`
The issue is that in order to use this API, it must be executed from the browser, that is, the browser object refers to the browser itself (tabs, icons, configuration), and the browser does have access to all the Request and Reponse of any page , so you will have to ask the user for permissions to be able to do this (The permissions will have to be declared in the manifest for the browser to execute them)
And also being part of the browser you lose control over the pages, that is, you can no longer manipulate the DOM, (not directly) so to control the DOM again it would be done as follows:
browser.webRequest.onHeadersReceived.addListener(
browser.tabs.executeScript({
code: 'console.log("Headers success")',
});
});
or if you want to run a lot of code
browser.webRequest.onHeadersReceived.addListener(
browser.tabs.executeScript({
file: './headersReveiced.js',
});
});
Also by having control over the browser we can inject CSS and images
Documentation: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onHeadersReceived
I would imagine Google grabs some data server-side - remember, when a page loads into your browser that has Google Analytics code within it, your browser makes a request to Google's servers; Google can obtain data in that way as well as through the JavaScript embedded in the page.
var ref = Request.ServerVariables("HTTP_REFERER");
Type within the quotes any other server variable name you want.
I'm implementing my own http module.
As I'm reading the official node.js http module api, I couldn't understand a few things:
If the user is using the response.writeHead(statusCode, [reasonPhrase], [headers]) function, are the headers should be written immidiatly to the socket or are they first supposed to be saved as a member to the object? and then written only after .end() function?
What is the meaning of implicit headers that should be used whenever the user didn't use writeHead()? are they supposed to be set ahead? and if the user didn't set them? what should be the behavior? thanks
Answers:
Anything that you write into response either headers with writeHead or body with write is buffered and sent. You see they use socket buffers. They can only hold fixed amount of data, before being sent. The important fact to remember is that you can only set headers before you start writing the body. If you do, some headers will set for you by the http server itself.
Implicit headers are ones which you don't write specifically but are still sent. Setup a simple http server, by responding a request without setting any header. Then view the request headers opening the site in browser. There will be headers like Date, Server, Host etc which are added to every request automatically without user's volition.
I found answer for the first question, but still don't understand the second one.
The first time response.write() is called, it will send the buffered header information and the first body to the client. The second time response.write() is called, Node assumes you're going to be streaming data, and sends that separately. That is, the response is buffered up to the first chunk of body.
I want to capture the HTTP request header fields, primarily the Referer and User-Agent, within my client-side JavaScript. How may I access them?
Google Analytics manages to get the data via JavaScript that they have you embed in you pages, so it is definitely possible.
Related:
Accessing the web page's HTTP Headers in JavaScript
If you want to access referrer and user-agent, those are available to client-side Javascript, but not by accessing the headers directly.
To retrieve the referrer, use document.referrer.
To access the user-agent, use navigator.userAgent.
As others have indicated, the HTTP headers are not available, but you specifically asked about the referer and user-agent, which are available via Javascript.
Almost by definition, the client-side JavaScript is not at the receiving end of a http request, so it has no headers to read. Most commonly, your JavaScript is the result of an http response. If you are trying to get the values of the http request that generated your response, you'll have to write server side code to embed those values in the JavaScript you produce.
It gets a little tricky to have server-side code generate client side code, so be sure that is what you need. For instance, if you want the User-agent information, you might find it sufficient to get the various values that JavaScript provides for browser detection. Start with navigator.appName and navigator.appVersion.
This can be accessed through Javascript because it's a property of the loaded document, not of its parent.
Here's a quick example:
<script type="text/javascript">
document.write(document.referrer);
</script>
The same thing in PHP would be:
<?php echo $_SERVER["HTTP_REFERER"]; ?>
Referer and user-agent are request header, not response header.
That means they are sent by browser, or your ajax call (which you can modify the value), and they are decided before you get HTTP response.
So basically you are not asking for a HTTP header, but a browser setting.
The value you get from document.referer and navigator.userAgent may not be the actual header, but a setting of browser.
One way to obtain the headers from JavaScript is using the WebRequest API, which allows us to access the different events that originate from http or websockets, the life cycle that follows is this:
WebRequest Lifecycle
So in order to access the headers of a page it would be like this:
browser.webRequest.onHeadersReceived.addListener(
(headersDetails)=> {
console.log("Request: " + headersDetails);
},
{urls: ["*://hostName/*"]}
);`
The issue is that in order to use this API, it must be executed from the browser, that is, the browser object refers to the browser itself (tabs, icons, configuration), and the browser does have access to all the Request and Reponse of any page , so you will have to ask the user for permissions to be able to do this (The permissions will have to be declared in the manifest for the browser to execute them)
And also being part of the browser you lose control over the pages, that is, you can no longer manipulate the DOM, (not directly) so to control the DOM again it would be done as follows:
browser.webRequest.onHeadersReceived.addListener(
browser.tabs.executeScript({
code: 'console.log("Headers success")',
});
});
or if you want to run a lot of code
browser.webRequest.onHeadersReceived.addListener(
browser.tabs.executeScript({
file: './headersReveiced.js',
});
});
Also by having control over the browser we can inject CSS and images
Documentation: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/onHeadersReceived
I would imagine Google grabs some data server-side - remember, when a page loads into your browser that has Google Analytics code within it, your browser makes a request to Google's servers; Google can obtain data in that way as well as through the JavaScript embedded in the page.
var ref = Request.ServerVariables("HTTP_REFERER");
Type within the quotes any other server variable name you want.
On an HTML page, while clicking the link of an Image ("img") or anchor ("a") tags, I would like to add custom headers for the GET request. These links are typically for downloading dynamic content. These headers could be SAML headers or custom application specific headers.
Is it possible to add these custom headers through JavaScript? Or if I add these through XMLHttpRequest, how can I achieve the download functionality?
This requirement is for IE6 or 7 only.
If you're using XHR, then setRequestHeader should work, e.g.
xhr.setRequestHeader('custom-header', 'value');
P.S. You should use Hijax to modify the behavior of your anchors so that it works if for some reason the AJAX isn't working for your clients (like a busted script elsewhere on the page).
I think the easiest way to accomplish it is to use querystring instead of HTTP headers.
The only way to add headers to a request from inside a browser is use the XmlHttpRequest setRequestHeader method.
Using this with "GET" request will download the resource. The trick then is to access the resource in the intended way. Ostensibly you should be able to allow the GET response to be cacheable for a short period, hence navigation to a new URL or the creation of an IMG tag with a src url should use the cached response from the previous "GET". However that is quite likely to fail especially in IE which can be a bit of a law unto itself where the cache is concerned.
Ultimately I agree with Mehrdad, use of query string is easiest and most reliable method.
Another quirky alternative is use an XHR to make a request to a URL that indicates your intent to access a resource. It could respond with a session cookie which will be carried by the subsequent request for the image or link.
In 2021, this can be accomplished with Service Workers.
First, you have to register a Service Worker on your page.
// index.html
window.addEventListener('load', function () {
navigator
.serviceWorker
.register('/service-worker.js');
});
In the Service Worker a request is captured, modified and forwarded by calling the WindowOrWorkerGlobalScope.fetch() method.
// service-worker.js
self.addEventListener('fetch', function (event) {
event.respondWith(async function () {
let headers = new Headers()
headers.append("X-Custom-Header", "Random value")
return fetch(event.request, {headers: headers})
}());
});
Please note that some web browsers do not show modified requests in the developer console. In that case requests can be observed with tools like tcpdump, Wireshark or in a server's logs.