Develop Reference

JavaScript, Making POST and GET requests simultaneously using Promises

Background: I have a function that needs a small debugging which is related to API requests(POST,GET) and promises. For the past few days, I have been trying to figure how asynchronous functions work and it turns out it's impossible (or so I heard) to directly return a value from such functions to synchronous functions.
My initial idea was to make a POST request function that returns an access token which I could plug to the authorization header of my GET function (which works perfectly so long as a token is provided). From my discovery, however, I can't do this on two separate methods because the POST request returns undefined outside the function. So my idea now is to include the GET request within the POST so that the access token can be passed directly without any method call. I tried to do this using Promises which I'm not very familiar with but manage to make some sense. My expectation was that I can call the httpGetData() function which would first get token, then pass it to a .then() portion which is supposed to return actual data or [object Promise] instead of undefined.
Question: The issue is that when I call the httpGetData() function it returns undefined but my console.log(result[0]) shows clearly that the GET request has fetched the data I want. I was wondering if there is a tweak that would allow my code to return this data (console.log is useless for me other than to see what's happening on background of api request). I saw some SO questions like these answered by returning the values in the .then() but my attempts all return undefined.
I've attached screenshots of the outputs. Keep in mind this is an Alexa skill Lambda code so the return appears after .speak() and the console.log shows below.
function httpGetData() {
var promise1 = new Promise(((resolve, reject) => {
var options = {
method: "POST", // POST request intended to get access token
host: 'hostinfo',
path: "pathinfo",
headers: {
"content-type": "stuff here",
"cache-control": "no-cache",
}
};
var request = http.request(options, function (response) {
var returnData = [];
response.on("data", function (chunk) {
returnData += chunk;
});
response.on("end", function () {
var data = JSON.parse(returnData);
var accessToken = data.access_token; // the access token value of the POST api request like "039509544932n3249fc21232nc32423nci"
resolve(accessToken);
});
response.on('error', (error) => {
reject(error);
});
});
request.write("------stuff: form-data; name=\"client_secret\"\r\n\r\stuff here\r\n------stuff: form-data; name=\"client_id\"\r\n\r\stuff here\r\n------stuff: form-data; name=\"grant_type\"\r\n\r\nclient_credentials\r\n------stuff");
request.end();
}));
promise1.then(function(value) {
return new Promise(((resolve, reject) => {
var options = {
method: "GET", //this is a GET request so it's intended to get usable data using access token
host: 'hostinfo',
path: "pathinfo",
headers: {
"content-type": "stuff here",
"Authorization": "Bearer " + value, // value is the access token that was resolved earlier ("039509544932n3249fc21232nc32423nci") so I'm trying to concatonate this to the authorization header
"cache-control": "no-cache",
}
};
var request = http.request(options, function (response) {
var returnData = [];
response.on("data", function (chunk) {
returnData += chunk;
});
response.on("end", function () {
console.log(value); //this is the line that prints access token in screenshot
resolve(JSON.parse(returnData));
});
response.on('error', (error) => {
reject(error);
});
});
request.write("------stuff: form-data; name=\"client_secret\"\r\n\r\stuff here\r\n------stuff: form-data; name=\"client_id\"\r\n\r\stuff here\r\n------stuff: form-data; name=\"grant_type\"\r\n\r\nclient_credentials\r\n------stuff");
request.end();
}))
}).then((result) => {
console.log('hello');
return result; //I'm not sure here but wanted to return the result
}).then((result) => {
console.log(result[0]); // just a check which shows that result[0] actually executes and can be printed as shown in screenshot
return result[0]; //other desperate attempt
});
}// httpGetData() function ends here
return image
Console log image

I've finally solved the problem thanks to #Bergi. Although this answer can be shortened, I want to post it so that it can serve as an option. What Bergi suggested was that instead of having both the POST and GET requests done in one function, we can separate them and use another function to do execute them simultaneously. In the code below, httpToken() performs the POST request and returns an access token. The httpResource(accessToken) function performs a GET request where it will take an accessToken parameter. The other function, finalData() will help us make the POST and GET requests simultaneously hence returning an [object Promise] instead of undefined, which we can access. Our last step will be to have an async/await function (retrieveInfo()), which will use finalData() to get the response from our requests. Enjoy!
function httpToken() {
return new Promise(((resolve, reject) => {
var options = {
method: "POST",
host: 'hostinfo',
path: "pathinfo",
headers: {
"content-type": "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW",
"cache-control": "no-cache",
}
};
var request = http.request(options, function (response) {
var returnData = [];
response.on("data", function (chunk) {
returnData += chunk;
});
response.on("end", function () {
var data = JSON.parse(returnData);
var accessToken = data.access_token;
resolve(accessToken); //resolving access token (can be changed to resolve(data) to include timestamp etc)
});
response.on('error', (error) => {
reject(error);
});
});
request.write("------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"client_secret\"\r\n\r\n"client secret here"\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"client_id\"\r\n\r\n"client id here"\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"grant_type\"\r\n\r\n"grant type here"\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--");
request.end();
}));
}
function httpResource(accessToken){
return new Promise(((resolve, reject) => {
var options = {
method: "GET",
host: 'hostinfo',
path: "pathinfo",
headers: {
"content-type": "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW",
"Authorization": "Bearer " + accessToken,
"cache-control": "no-cache",
}
};
var request = http.request(options, function (response) {
var returnData = [];
response.on("data", function (chunk) {
returnData += chunk;
});
response.on("end", function () {
resolve(JSON.parse(returnData));
});
response.on('error', (error) => {
reject(error);
});
});
request.write("------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"client_secret\"\r\n\r\n"client secret here"\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"client_id\"\r\n\r\n"client id here"\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"grant_type\"\r\n\r\n"grant type here"\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--");
request.end();
}));
}
function finalData()
{
return httpToken().then(token => httpResource(token));
}
async function retrieveInfo()
{
response = await finalData(); //response will have the data returned from our GET request using finalData()
//do stuff with response...
}

How to call a function only after getting response from restcall node js

I have one request.post call and another function.Where I need to pass the response of restcall as paramaters to the function.
The current issue which Iam facing here is that the function is getting called even before i get response from the rest call and null values are getting passed.I know that we need to use some callabcks for this issue.But I dont know how to do it.can someone help.
app.post('/verifycreds',function(req,res) {
var reqdata = req.body;
var data = {};
data.custid = reqdata.custid;
request.post({
url:'https://database.mybluemix.net/verifycredentials',
headers:{
'Content-Type':'application/json'
},
body:data,
json:true
}, function(err,response) {
verifycreds(response.body);
});
function verifycreds(data) {
if((datareq.customerid === data.customerid ) && (datareq.password == data.password)){
res.send("valid");
} else {
res.send("invalid");
}
}
So how can I call verifycreds function only after I get response from the request .post call..Any help!

Your callback is valid, the problem in callback parameters. It should be defined with three parameters:
error
response
body
So correct code is:
request.post({
url: 'https://database.mybluemix.net/verifycredentials',
headers: {
'Content-Type': 'application/json'
},
body: data,
json: true
}, function(err, res, body) {
// TODO: process possible errors
// if (err) { ...
verifycreds(body);
});

Node.JS Request Module Callback Not Firing

I'm running this code using the request module for node.js
var hsKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
var hsForm = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
var hsHost = "https://docs.google.com/"
var url = hsHost + "forms/d/" + hsForm + "/formResponse"
var form = {
"entry.129401737": pointsAvg,
"entry.2000749128": hiddenNeurons,
"submit": "Submit",
"formkey": hsKey
};
request.post({
url: url,
form: form
}, function (err, res, body) {
console.log("Sent data");
});
I have tried running the above code just using standard Node.JS libraries, to no avail. The callback function is never fired and the request doesn't go through. I don't know why.

I believe I've found the answer to my own problem. The issue seems to be that I'm not allocating any time in the Node.js event loop to allow the request to be executed.

Have a look at this question:
your code should look something like
var hsKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
var hsForm = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
var hsHost = "https://docs.google.com/"
var url = hsHost + "forms/d/" + hsForm + "/formResponse"
var form = {
"entry.129401737": pointsAvg,
"entry.2000749128": hiddenNeurons,
"submit": "Submit",
"formkey": hsKey
};
request.post({
url: url,
form: form
}, function (response) {
response.setEncoding('utf8');
response.on('data', function(chunk){
//do something with chunk
});
});
The data event should get fired on receiving a response.
So if you read the docs for the request module at npm
request
.get('http://google.com/img.png')
.on('response', function(response) {
console.log(response.statusCode) // 200
console.log(response.headers['content-type']) // 'image/png'
});
There is a response event that should get fired.

I ran into this as well. I ended up creating a separate js file containing only the request, without the describe and it methods, and running it with 'mocha mynewbarebonesreq.js'. suddenly I could see that there was an exception being thrown and swallowed by mocha (with the standard reporter, spec).
I finally installed and enabled mocha_reporter which shows the exceptions
now it looks like this:
describe('CMSLogin', function () {
it('should log in as user ' + JSON.stringify(USER_PASS), function (done) {
request({
url: "http://cms.lund.multiq.com:3000/api/CMSUsers/login",
method: "POST",
headers: {
'Content-Type': 'application/json',
'Accept': 'application/json',
},
json: false,
body: JSON.stringify(USER_PASS)
}, (err, res, body) => {
var parsedBody = JSON.parse(body);
this.token = parsedBody.id;
console.log(this.token)
assert.equal(USER_PASS.userId, parsedBody.userId);
assert.doesNotThrow(() => Date.parse(parsedBody.created));
if (err) { done.fail(err); }
done();
});
});
}

Node JS, make HTTPS request synchronously from two links

I want to make a HTTPS request to an external link through Node JS. On my first call, I need to fetch user id by looping through several users. On my second call, I need to input that user id in the URL link and fetch user properties. Keep repeating this process till I go through all users. The end goal is to store data of every user in a JSON format. There is no front-end involved. Any direction/advice is much appreciated.
I can't share the actual link due to api keys. But here is the hypothetical scenario. I only show 2 users here. I have about 10,000 users in my actual data set.
Link 1
https://www.google.com/all_users
JSON Output
{
"name": "joe",
"uri": "/id/UserObject/User/1234-1234",
},
{
"name": "matt",
"uri": "/id/UserObject/User/5678-5678",
}
Link 2
https://www.google.com//id/UserObject/User/1234-1234
JSON Output
{
"name": "joe",
"uri": "/id/UserObject/User/1234-1234",
"Property Values": {
"height": "2",
"location": "canada"
},
"Other Values": {
"work": "google",
"occupation": "developer"
}
}
Nested JSON
{
"PropertySetClassChildrenResponse": {
"PropertySetClassChildren": {
"PropertySetInstances": {
"totalCount": "1",
"Elements": [
{
"name": "SystemObject",
"uri": "/type/PropertySetClasses/SystemObject"
}
]
}
}
}
}

Not tested, but this should point you in the right direction. It uses Promises and assumes that run in an ES6 environment:
const rp = require('request-promise');
const Promise = require('bluebird');
fetchAllUsers()
.then(extractUserUris)
.then(extractUserIds)
.then(buildUserDetailRequests)
.then(Promise.all) // run all the user detail requests in parallel
.then(allUserData => {
// allUserData is an array of all users' data
});
function fetchAllUsers() {
return rp('https://api.whatever.com/all_users');
}
function extractUserUris(users) {
return users.map(user => user.uri);
}
function extractUserIds(userUris) {
return userUris.map(userUri => userUri.split('/').pop());
}
function buildUserDetailRequests(userIds) {
return userIds.map(userId => rp("https://api.whatever.com/user/" + userId));
}

I'd suggest using the request package to make your HTTP requests easier.
> npm install request
Then you would obtain a list of all users with something like this:
var request = require('request');
request.get({url: "https://example.org/all_users"}, handleUsersResponse);
You'd handle the request response like this:
function(err, response, body) {
if (!err && response.statusCode == 200) {
// parse json (assuming array of users)
var users = JSON.parse(body);
// iterate through each user and obtain user info
for(var i = 0; i < users.length; i++) {
var userUri = users[i].uri;
obtainUserInfo(userUri)
}
}
}
obtainUserInfo function would be similar to the above code.
One important thing to keep in mind is that since the HTTP requests are being made asynchronously, when you make the requests in a loop, the next iteration of the loop does not wait until the work is finished before moving to the next iteration and starting the next request. So in effect, your loop would start all the HTTP requests nearly in parallel. This can easily overwhelm both your client and the server. One way to get around this is to use a worker queue to enqueue the work and ensure that only a maximum number of HTTP requests are being executed at any given time.

You don't want to do synchronous calls, it defeats the purpose of using Node. So by the Node powers invested in me by the State of Texas I hereby cast that synchronous way I thinking out of you!
Just kidding :), but let's do this the Node way.
Install these two libraries:
sudo npm install Promise
sudo npm install request
And set your code to look like:
var Promise = require('promise');
var request = require('request');
//Get your user data, and print the data in JSON:
getUserData()
.then(function(userData) {
console.log(JSON.stringify(userData));
}).catch(function(err) {
console.log('Error: ' +err);
});
/**
* Prepares an Object containing data for all users.
* #return Promise - Contains object with all user data.
*/
function getUserData() {
return new Promise(function(fulfill, reject) {
// Make the first request to get the user IDs:
var url1 = 'https://www.google.com/all_users';
get(url1)
.then(function(res) {
res = JSON.parse(res);
// Loop through the object to get what you need:
// Set a counter though so we know once we are done.
var counter = 0;
for (x=0; x<res.users.length; x++) {
var url2 = 'https://www.google.com//id/UserObject/User/';
url2 = url2 + res.users.id; //Wherever the individual ID is stored.
var returnDataArr = [];
get(url2)
.then(function(res2) {
// Get what you need from the response from the 2nd URL.
returnDataArr.push(res2);
counter++;
if (counter === res.users.length) {
fulfill({data: returnDataArr}); //Return/Fulfill an object containing an array of the user data.
}
}).catch(function(err) {
// Catch any errors from the 2nd HTTPS request:
reject('Error: ' +err);
});
}).catch(function(err) {
// Catch any errors from the 1st HTTPS request:
reject('Error: ' +err);
});
}
/**
* Your HTTPS GET Request Function
* #param url - The url to GET
* #return Promise - Promise containing the JSON response.
*/
function get(url) {
return new Promise(function(fulfill, reject) {
var options = {
url: url,
headers: {
'Header Name': 'Header Value',
'Accept': 'application/json',
'Content-Type': 'application/json'
};
request(options, function(err, res, body) {
if (err) {
reject(err);
} else {
fulfill(body);
}
});
});
}
So what this Promise does, is that it returns the value once we actually have it. In the code above, we are first getting that list of users, and then as we parse through it, we are making a new asynchronous HTTP request to get the additional data on it. Once we get the user data, we push it to an array.
Finally, once our counter hits its endpoint, we know that we have gotten all the user data, and so we call fulfill which essentially means return, and it returns an object containing an array of the user data.
Let me know if this makes sense.

The answers above helped me go further with my solution and get the desired outcome. However, I spent a lot of time trying to understand node, promises in node, making an API call, etc. Hopefully, this will help to a beginner level node developer.
NODE
Node.js® is a JavaScript runtime built on Chrome's V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. Node.js' package ecosystem, npm, is the largest ecosystem of open source libraries in the world.
If you are a JavaScript developer, you would prefer to use Node as you wouldn't have to spend time learning a new language like Java or Python.
GOAL
Make a HTTPS call to an external link to fetch all server URIs. Pass in the URI as a param to create a second link to fetch all server properties. Loop through to all server uris and properties. Refer the original post on the top for the data structure. The external link also required basic auth and headers.
CODE
Install NPM modules request (https call), bluebird (promises) and lodash(utility) and express(node framework).
/
********************** MODULES/DEPENDENCIES **********************/
var express = require('express');
var request = require('request');
var Promise = require('bluebird');
var _ = require("lodash");
/********************** INITIATE APP **********************/
var app = express();
console.log("Starting node server...");
/**
* Your HTTPS GET Request Function
* #param url - The url to GET
* #return Promise - Promise containing the JSON response.
*/
function get(url) {
return new Promise(function(resolve, reject) {
// var auth = "Basic " + new Buffer(username + ':' + password).toString("base64");
var options = {
url: url,
headers: {
// 'Authorization': auth,
'Content-Type': 'application/json',
'Accept': 'application/json'
}
};
console.log("Calling GET: ", url);
if ('development' == app.get('env')) {
console.log("Rejecting node tls");
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
}
request(options, function(error, response, body) {
if (error) {
reject(error);
} else {
// console.log("THIS IS BODY: ", body);
resolve(body);
}
});
});
}
/********************** GET DATA FUNCTION **********************/
function getServerData() {
/********************** URI VARIABLES **********************/
var username = 'username',
password = 'password',
role = 'Read-Only',
url_host = 'https://link.com:10843';
/********************** URL 1 **********************/
var url1 = url_host + '/type/PropertySetClasses/SystemObject/Server/?maxResults=1000&username=' + username + '&password=' + password + '&role=' + role;
console.log("Getting server data...", url1);
/********************** GET REQUEST 1 **********************/
return get(url1)
.then(function(res) {
console.log("Got response!");
res = JSON.parse(res);
res = res.PropertySetClassChildrenResponse.PropertySetClassChildren.PropertySetInstances.Elements;
// console.log("THIS IS RES: ", res);
/********************** FETCH URI FROM RES NESTED OBJECT **********************/
var server_ids = _.map(res, function(server) {
return server.uri;
});
console.log("Calling server urls", server_ids);
// Loop through the object to get what you need:
// Set a counter though so we know once we are done.
return Promise.map(server_ids, function (id) {
var url2 = url_host + id + '?username=' + username + '&password=' + password + '&role=' + role;
console.log("Calling URL", url2);
return get(url2)
.then(function(res2) {
res2 = JSON.parse(res2);
var elements = res2.PropertySetInstanceResponse.PropertySetInstance.PropertyValues.Elements;
console.log("Got second response", res2, elements);
return elements;
});
})
.then(function (allUrls) {
console.log("Got all URLS", allUrls);
return allUrls;
});
})
.catch(function(err) {
console.error(err);
throw err;
});
};
app.listen(8080, function() {
console.log("Server listening and booted on: " + 8080);
app.get("/serverInfo", function (req, res) {
console.log("Calling server info");
return getServerData()
.then(function(userData) {
var userData = JSON.stringify(userData, null, "\t");
console.log("This is USERDATA Data: ", userData);
res.send(userData);
})
.catch(function(err) {
console.error(err);
res.send({
__error: err,
message: err.message
});
});
});
});

Firebase REST API - how to set character encoding?

I'm calling the Firebase REST API from a Node.js process. The problem I'm seeing is that POSTS fail when the post body contains non-ASCII characters. This is despite the request returning a "200" status, and the name of a node (which doesn't actually get created).
I'm currently trying something like this:
function push(path, object, callback) {
console.log("Pushing to "+path+" on: "+firebase.host);
var fullPath=firebase.basePath+path;
console.log("fullPath="+fullPath);
var body = JSON.stringify(object);
var options = {
host: firebase.host,
port: 80,
method: "POST",
path: fullPath, //gamma.firebase.com/...
agent: false,
headers: {
'content-type': 'application/json',
'Content-Length': body.length,
}
};
var req = http.request(options, function(response) {
var result = "";
console.dir(response.headers);
response.on('data', function(chunk) {
result+=chunk;
});
response.on('end', function() {
console.error("POST response result: "+result);
try {
callback(JSON.parse(result));
} catch(e) {
callback({ error: e });
}
});
response.on('error', function(e) {
console.error("POST response error: "+error);
callback({error: e});
});
});
req.on('error', function(error) {
console.error("POST request error: "+error);
});
req.write(body);
req.end();
}
The contents of "object" can be as simple as:
{"text": "test\u00a0text"}
The result I get back is status 200, and an reasonable-looking child name, which doesn't actually get created.
I've tried setting content-type to a bunch of different things (adding ; charset="UTF-8", for example), and it doesn't seem to affect the results at all.

There is an error in the way we are handling certain types of input which is yielding the erroneous 200 status. We will roll out a fix shortly. To work around the problem in the meantime you can omit sending the Content-Length header. This will allow you to post ASCII and non-ASCII data.