Develop Reference

How do I do a POST request using the Google Chrome Puppeteer library?

Good Day All,
I'm trying to do a POST request using the puppeteer headless chrome library. I can't seem to get the below code to work.
// Get csrf token
let token = await page.evaluate(() => document.querySelector('[name="CSRFToken"]').value);
let postResponse = await page.evaluate(async(token, cookies) => {
let response = fetch("/loyalty/points", {
method : 'POST',
cookie : cookies,
postData : 'CSRFToken=' + token,
}).then(response => response.text()).catch(error => console.log(error));
return response;
});
console.log('Final response');
console.log(postResponse);
I keep on getting an error that the CSRF token has not been set or that the cookie is invalid.
My question is, am I using the correct method in puppeteer to do a POST? If so, is there any way for me to do some debugging that I can see the actual POST request that was sent?
I appreciate any advice or help. Thanks

You are not creating a request body: hence the error. The postData attribute you set on the request object is not any known attribute, so it won't be set on the request either, meaning that the server will never see your CSRF token. You should look into the MDN docs on fetch().
I believe you should be all good by simply replacing postData with body, but it's hard to know without access to your endpoint. For all we know it might require special headers.
Given that you only post normal form data (which is implied by your key=value code), I would also start using the FormData objects provided by your browser to avoid manual coding of implementation details.
const formData = new FormData();
formData.append("CSRFToken", token);
const response = fetch("/loyalty/points", {
method : 'POST',
cookie : cookies,
body : formData,
headers : {
'cookie' : cookies,
/* other headers you need, possibly content-type (see below) */
},
}).then(response => response.text()).catch(error => console.log(error));
return response;
});
Caveat: using the FormData API will always set the content-type of the data to multipart/form-data. If your server for some reason doesn't support that encoding, and you need to use application/x-www-form-urlencoded (see here for difference),
you can't blindly change the Content-Type: you also need to url encode the content.
For debugging I would simply use a normal Chrome instance to see this. You should be able to run the code there and see the network requests in DevTools (where it would be immediately noticeable that you POST an empty request).

How to send client side cookies (javascript) to server side (node.js) using Microsoft Bot Framework Directline API? [duplicate]

I am working on an internal web application at work. In IE10 the requests work fine, but in Chrome all the AJAX requests (which there are many) are sent using OPTIONS instead of whatever defined method I give it. Technically my requests are "cross domain." The site is served on localhost:6120 and the service I'm making AJAX requests to is on 57124. This closed jquery bug defines the issue, but not a real fix.
What can I do to use the proper http method in ajax requests?
Edit:
This is in the document load of every page:
jQuery.support.cors = true;
And every AJAX is built similarly:
var url = 'http://localhost:57124/My/Rest/Call';
$.ajax({
url: url,
dataType: "json",
data: json,
async: true,
cache: false,
timeout: 30000,
headers: { "x-li-format": "json", "X-UserName": userName },
success: function (data) {
// my success stuff
},
error: function (request, status, error) {
// my error stuff
},
type: "POST"
});

Chrome is preflighting the request to look for CORS headers. If the request is acceptable, it will then send the real request. If you're doing this cross-domain, you will simply have to deal with it or else find a way to make the request non-cross-domain. This is why the jQuery bug was closed as won't-fix. This is by design.
Unlike simple requests (discussed above), "preflighted" requests first
send an HTTP request by the OPTIONS method to the resource on the
other domain, in order to determine whether the actual request is safe
to send. Cross-site requests are preflighted like this since they may
have implications to user data. In particular, a request is
preflighted if:
It uses methods other than GET, HEAD or POST. Also, if POST is used to send request data with a Content-Type other than
application/x-www-form-urlencoded, multipart/form-data, or text/plain,
e.g. if the POST request sends an XML payload to the server using
application/xml or text/xml, then the request is preflighted.
It sets custom headers in the request (e.g. the request uses a header such as X-PINGOTHER)

Based on the fact that the request isn't sent on the default port 80/443 this Ajax call is automatically considered a cross-origin resource (CORS) request, which in other words means that the request automatically issues an OPTIONS request which checks for CORS headers on the server's/servlet's side.
This happens even if you set
crossOrigin: false;
or even if you ommit it.
The reason is simply that localhost != localhost:57124. Try sending it only to localhost without the port - it will fail, because the requested target won't be reachable, however notice that if the domain names are equal the request is sent without the OPTIONS request before POST.

I agree with Kevin B, the bug report says it all. It sounds like you are trying to make cross-domain ajax calls. If you're not familiar with the same origin policy you can start here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Same_origin_policy_for_JavaScript.
If this is not intended to be a cross-domain ajax call, try making your target url relative and see if the problem goes away. If you're really desperate look into the JSONP, but beware, mayhem lurks. There really isn't much more we can do to help you.

If it is possible pass the params through regular GET/POST with a different name and let your server side code handles it.
I had a similar issue with my own proxy to bypass CORS and I got the same error of POST->OPTION in Chrome. It was the Authorization header in my case ("x-li-format" and "X-UserName" here in your case.) I ended up passing it in a dummy format (e.g. AuthorizatinJack in GET) and I changed the code for my proxy to turn that into a header when making the call to the destination. Here it is in PHP:
if (isset($_GET['AuthorizationJack'])) {
$request_headers[] = "Authorization: Basic ".$_GET['AuthorizationJack'];
}

In my case I'm calling an API hosted by AWS (API Gateway). The error happened when I tried to call the API from a domain other than the API own domain. Since I'm the API owner I enabled CORS for the test environment, as described in the Amazon Documentation.
In production this error will not happen, since the request and the api will be in the same domain.
I hope it helps!

As answered by #Dark Falcon, I simply dealt with it.
In my case, I am using node.js server, and creating a session if it does not exist. Since the OPTIONS method does not have the session details in it, it ended up creating a new session for every POST method request.
So in my app routine to create-session-if-not-exist, I just added a check to see if method is OPTIONS, and if so, just skip session creating part:
app.use(function(req, res, next) {
if (req.method !== "OPTIONS") {
if (req.session && req.session.id) {
// Session exists
next();
}else{
// Create session
next();
}
} else {
// If request method is OPTIONS, just skip this part and move to the next method.
next();
}
}

"preflighted" requests first send an HTTP request by the OPTIONS method to the resource on the other domain, in order to determine whether the actual request is safe to send. Cross-site requests
https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS

Consider using axios
axios.get( url,
{ headers: {"Content-Type": "application/json"} } ).then( res => {
if(res.data.error) {
} else {
doAnything( res.data )
}
}).catch(function (error) {
doAnythingError(error)
});
I had this issue using fetch and axios worked perfectly.

I've encountered a very similar issue. I spent almost half a day to understand why everything works correctly in Firefox and fails in Chrome. In my case it was because of duplicated (or maybe mistyped) fields in my request header.

Use fetch instead of XHR,then the request will not be prelighted even it's cross-domained.

$.ajax({
url: '###',
contentType: 'text/plain; charset=utf-8',
async: false,
xhrFields: {
withCredentials: true,
crossDomain: true,
Authorization: "Bearer ...."
},
method: 'POST',
data: JSON.stringify( request ),
success: function (data) {
console.log(data);
}
});
the contentType: 'text/plain; charset=utf-8', or just contentType: 'text/plain', works for me!
regards!!

Can’t access cross-origin response header from frontend JavaScript

I am building a simple web app using ReactJS and create-react-app.
I have a backend API set up on Heroku where I can make POST requests. Everything works fine, except:
When I make a POST request using fetch API, the response is 100% correct but it only gives me 2 standard headers. I want to get my custom header. I have added expose header in my response and here's the plot twist: When I view the headers from Chrome Inspection Tool or Postman (API tool), it shows all the headers, including my custom one. Here is the fetch code I'm using -
fetch(theUrl, {
method: 'POST',
body: JSON.stringify({
"placeholder": "placeholder"
})
})
.then(function(res) {
console.log(res.headers.get('CUSTOM_HEADER_NAME'));
})
If it makes any difference, this fetch method is called from a function outside the main body of the ReactJS component.
The name of the custom header is Image-Identification-Path, and the header in my response header is Access-Control-Expose-Headers for Image-Identification-Path.
Summary: How do I get my custom header using fetch?

You must configure the server to which the request is sent, such that its response has an Access-Control-Expose-Headers header that has the name of your custom response header.
Otherwise, if your browser doesn’t see the name of your custom header in that Access-Control-Expose-Headers header, it won’t let you access the value of your custom header.
In such a case it’s expected that you’d still be able to see the custom header if you look at the response in Postman or even in your browser devtools.
But just because the browser gets the custom header in the response doesn’t mean the browser will expose it to your frontend JavaScript code.
For cross-origin requests, browsers will only expose that custom response header to your frontend code if that header name is in the Access-Control-Expose-Headers value.

I know this question is old but I ran into this problem yesterday, and the given answer didn't work for me.
The solution I found was given in this article. Basically:
You can’t directly access the headers on the response to a fetch call - you have to iterate through after using the entries() method on the headers.
So, in your particular case you should be able to achieve the goal by using this code:
fetch(theUrl, {
method: 'POST',
body: JSON.stringify({
"placeholder": "placeholder"
})
})
.then(response => {
for (var pair of response.headers.entries()) { // accessing the entries
if (pair[0] === 'CUSTOM_HEADER_NAME') { // key you're looking for, in your case Image-Identification-Path
let imagePath = pair[1]; //// saving that value
}
}
.... })

JavaScript: parsing JSONP callback response from GitHub API

I'm creating an Express route that calls the GitHub API with a ?callback=foo pattern added to the endpoint so that it will return the Link headers which I'll need to parse out the Link: header because it contains the link that I'll have to call to get the next page of the response.
The problem is that the response has the expected pattern, but when I try to create a function to tease out the meta and data portions of the function, they turn up undefined.
My code:
app.get('/populate', function(req, res, next) {
console.log('/populate route hit');
var token = "<something>";
var options = {
url: 'https://api.github.com/users?callback=resp',
headers: {
'User-Agent': 'Our-App',
'Authorization': 'token '+ token
}
};
api(options) // 'api' is request-promise module, makes http requests
.then(function(response) {
console.log(response); // Note 1
function resp(res) {
var meta = res.meta;
var data = res.data;
console.log('meta ', meta); // Note 2
console.log('data ', data);
}
resp(response);
Note 1: The response looks like:
/**/resp({"meta":{"X-RateLimit-Limit":"5000","X-RateLimit-Remaining":"4993",
"X-RateLimit-Reset":"1435297775","X-OAuth-Scopes":"public_repo, user:email",
"X-Accepted-OAuth-Scopes":"repo","Cache-Control":"private, max-age=60, s-maxage=60",
"Vary":"Accept, Authorization, Cookie, X-GitHub-OTP",
"ETag":"\"0cbbd180648a54f839a237b0302025db\"",
"X-GitHub-Media-Type":"github.v3; format=json",
"Link":[["https://api.github.com/users?callback=resp&since=46",
{"rel":"next"}],["https://api.github.com/users{?since}",
{"rel":"first"}]],"status":200},"data":[{"login":"mojombo","id":1,
...etc etc...
}]})
The response looks like it's been JSON.stringified but when I JSON.parse(response) it returns an error. I don't know how to access the deeply-embedded Link: headers and even the data, which looks like JSON, too.
Note 2 The res.meta and res.data log as undefined.

The response isn't JSON, it's JSONP. JSONP is a cross-domain mechanism for retrieving data. You don't use XHR (e.g., app.get) to request JSONP, you use a script tag. (Because XHR is limited by the Same Origin Policy; script tags aren't.)
If your call retrieving that data via XHR works, it means cross-domain XHR calls are allowed in your situation (the server supports Cross-Origin Resource Sharing with your page's origin, and the browser supports CORS). You can get JSON instead of JSONP by removing the ?callback=resp in the URL.

getjson - not returning object to console

I am trying to retrieve a json feed every 1 second. The URL that I am trying to retrieve displays JSON in the browser but will not be retrieved via a jquery getJSON
http://www.ridestreamline.com/Services/JSONPRelay.svc/GetMapVehiclePoints
function getBusLoc() {
$.getJSON('http://www.ridestreamline.com/Services/JSONPRelay.svc/GetMapVehiclePoints?callback=?', function(data) {
console.log(data);
setTimeout(getBusLoc, 1000);
})
}
getBusLoc()
It has something to do with the above link. What am I missing? Fiddle here

This is because of same origin policy, you can't sent ajax request from host A to host B, you can use jsonp instead (if your service supports this) , or if you has control to server side and you don't mind to old browsers you can use x-access-control-allow-origin http header in response to OPTIONS request (more info here https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS)