Develop Reference

How does one access, process and pass along response data outside of the request?

In the snippet below, I want to be able to access locationArray outside of the request function, I understand that in my code below why it will not work, however, I have tried many different methods to access the array. I have tried using promises, callback functions etc, however, none of them seem to be working.
Any other ideas on how to do this? Even open to ways I've tried as at this point everything is worth a try.
request(process.env.RESOURCE_SHEET, (error, response, html) => {
var locationArray = new Array
if(!error && response.statusCode == 200) {
const $ = cheerio.load(html);
$("h3").each((i, lle) => {
const location = $(lle).text();
if(location.includes("Kansas")) return;
if(location.includes("In Stock")) {
var level = location + " ✅";
} else {
var level = location + " ❌";
}
locationArray.push(level);
});
}
console.log(locationArray) // Output 1: [level1,level2,level3,leveletc]
});
console.log(locationArray) // Output 2: []

One not only might consider a Promise based approach, as it got already suggested, but also a code refactoring which separates the different concerns into tasks and implements the latter as functions which can be fed/passed to the promise chain. As an advantage the refactoring pays back with human readable code which also might be easier to maintain ...
function createRequest(src) {
return new Promise((resolve, reject) => {
request(src, (error, response, html) => {
if (!error && response.statusCode === 200) {
resolve(html);
} else {
reject({ error, response });
}
});
};
}
function handleFailedRequest(reason) {
const { error, response } = reason;
// proceed with failure handling based on
// either request data `error` and/or `response`.
}
function createLocationArray(html) {
const locationArray = [];
const $ = cheerio.load(html);
$('h3').each((i, lle) => {
const location = $(lle).text();
if (!location.includes('Kansas')) {
const isInStock = location.includes('In Stock');
locationArray.push(
`${ location } ${ isInStock && '✅' || '❌' }`
);
}
});
return locationArray;
}
function processLocationArray(array) {
console.log('locationArray ... ', array);
}
const promisedResponse = createRequest(process.env.RESOURCE_SHEET);
promisedResponse
.then(createLocationArray)
.then(processLocationArray)
.catch(handleFailedRequest);

#StackSlave was right, it just needed a promise, I believe I messed up the syntaxing when I first tried resolving it using a promise, but this seemed to work.
const promise = new Promise((resolve,reject) => {
request(process.env.RESOURCE_SHEET, (error, response, html) => {
var locationArray = new Array
if(!error && response.statusCode == 200) {
const $ = cheerio.load(html);
$("h3").each((i, lle) => {
const location = $(lle).text();
if(location.includes("Kansas")) return;
if(location.includes("In Stock")) {
var level = location + " ✅";
} else {
var level = location + " ❌";
}
locationArray.push(level);
resolve(locationArray);
});
}
console.log(locationArray) // Output 1: [level1,level2,level3,leveletc]
});
});
promise.then(array => {
console.log(array);
});

How do I make sure a function is completely executed before its value is passed to res.render?

I want to parse the address query and return the addresses and their titles such that
http://localhost:3000/I/want/title/?address=google.com&address=youtube.com shall return:
google.com - 'google', youtube.com - 'youtube'
I'm using cheerio.js to extract the title from the URLs but it takes time and the res.render line is executed before the variable titles is filled with the URL titles. How do I make sure that my code for retrieving the titles is executed before the res.render?
As of now, I'm not getting any errors but the titles[] array is sent without data to my .ejs file. I've tried solving this through callbacks, step.js, async.js but nothing seems to work. I've tried solving it using rsvp.js (promise) as shown below (from app.js) but it doesn't work either and titles[] remains empty:
app.get("/I/want/title/", function(req,res){
if (typeof req.query.address === "string"){
query = [req.query.address];
}
else {
query = req.query.address;
}
var titles=[];
var promise = new RSVP.Promise(function(resolve, reject) {
for (i=0;i<(query.length);i++){
if (!((query[i]).startsWith("https://www."))){
var url = "https://www." + query[i];
}else{
url=query[i];
}
request(url, function (err, resp, body) {
if (err) {
var title = "NO RESPONSE"
} else {
var $ = cheerio.load(body);
var title = $("title").text();
}
titles.push(title);
});
}
resolve(titles);
reject();
});
promise.then(function(titles) {
res.render("title", {url: query, siteName: titles});
}).catch(function() {
console.log("oh no");
});
});
Is there something wrong with my syntax or logic? How should I execute this with either callbacks or promises?

You have to put reject & resolve of promise inside request callback, then your code should work fine (as shown below).
app.get('/I/want/title/', function(req, res) {
if (typeof req.query.address === 'string') {
query = [req.query.address]
} else {
query = req.query.address
}
var titles = []
var promise = new RSVP.Promise(function(resolve, reject) {
for (i = 0; i < query.length; i++) {
if (!query[i].startsWith('https://www.')) {
var url = 'https://www.' + query[i]
} else {
url = query[i]
}
request(url, function(err, resp, body) {
if (err) {
var title = 'NO RESPONSE'
reject()
} else {
var $ = cheerio.load(body)
var title = $('title').text()
}
titles.push(title)
resolve(titles)
})
}
})
promise
.then(function(titles) {
res.render('title', {url: query, siteName: titles})
})
.catch(function() {
console.log('oh no')
})
})
Does this answer your question ?

Your resolve should be inside the else part of the request callback, and the reject should be inside the if (err).

How do you get javascript promises to work with node + express and display on an EJS file?

Before saying to look at the docs, I have and they were not helpful in the slightest.
I have an web page with node as the backbone. On one page I need to request the past 10 images from NASA's Astronomy Picture of the Day (APOD) API and then after that, I need to request the next 5 upcoming launches from the Launch Library API (https://launchlibrary.net/docs/1.3/api.html). My problem is that not all the APODs will load (which I understand because of the nature of asynchronous requests).
Here is my concise app.js file for the Node backend:
app.get("/index", function(req, res) {
/**Requesting NASA's Astronomy Picture of the Day**/
var apod_url = "https://api.nasa.gov/planetary/apod?api_key=[My Key]"
var apod_img_urls = [];
var curr_moment = moment();
for(var i = 0; i < 10; i++) {
var appended_url = apod_url + "&date=" + curr_moment.subtract(i, "days").format("YYYY-MM-DD");
request(appended_url, function(error, reponse, body) {
if(!error && reponse.statusCode == 200) {
var img_json = JSON.parse(body);
if(img_json.media_type == "image") {
var apod_promise = new Promise(function(resolve, reject){
resolve(img_json.hdurl);
});
apod_img_urls.push(apod_promise);
}
} else {
console.log(error);
}
});
}
/**************************************************/
var url = "https://launchlibrary.net/1.3/launch?next=20&mode=verbose";
request(url, function(error, response, body) {
if(!error && response.statusCode == 200) {
var data = JSON.parse(body);
res.render("index", {data: data, apod_img_urls: apod_img_urls});
} else {
console.log(error);
}
});
});
Here is an EJS snippet
<% apod_img_urls.forEach(function(promise, index) { %>
<div class="carousel-item <%= (index == 0 ? 'active' : '') %>">
<div class="w-100 home-image" style="background-image: url('<%= promise.then(function(url) {return url}); %>')"></div>
</div>
<% }); %>
When I check in the source it shows that the background image urls for the divs are [object Promise]. The way I have it, no images show up. Also the number of divs displayed (i.e. the number of images I should have) is variable; sometimes it's 5, sometimes it's 3, and other times it's none. Could my problem be that I'm rendering the page inside of another request? Also how can I get the actual image URL to show up in the EJS file?

You are creating the promise too late, inside the asynchronous callback to reequest - fairly simple reorganisation of code required
Once all the promises are in an array, you then need to wait for them to complete, using Promise.all
app.get("/index", function(req, res) {
/**Requesting NASA's Astronomy Picture of the Day**/
var apod_url = "https://api.nasa.gov/planetary/apod?api_key=[My Key]"
var promises = [];
var curr_moment = moment();
for(var i = 0; i < 10; i++) {
var appended_url = apod_url + "&date=" + curr_moment.subtract(i, "days").format("YYYY-MM-DD");
promises.push(new Promise((resolve, reject) => {
request(appended_url, function(error, reponse, body) {
if(!error && reponse.statusCode == 200) {
var img_json = JSON.parse(body);
resolve(img_json.hdurl);
} else {
reject(error);
console.log(error);
}
});
}));
}
Promise.all(promises).then(apod_img_urls => {
var url = "https://launchlibrary.net/1.3/launch?next=20&mode=verbose";
request(url, function(error, response, body) {
if(!error && response.statusCode == 200) {
var data = JSON.parse(body);
res.render("index", {data, apod_img_urls});
} else {
console.log(error);
}
});
});
});
Rather than using request-promise and all the cruft it requires, you can always make your own promisified request function, and in this case, it rejects if status is anything other than 200
const requestP = url => new Promise((resolve, reject) => request(url, (error, response, body) => {
if (error) {
reject({error, response, body});
} else if (resonse.statusCode != 200) {
reject(resonse.statusCode);
} else {
resolve({response, body});
}
}));
now, your code can be written as:
app.get("/index", function(req, res) {
/**Requesting NASA's Astronomy Picture of the Day**/
const apod_url = "https://api.nasa.gov/planetary/apod?api_key=[My Key]"
const curr_moment = moment();
Promise.all(Array.from({length:10}, (_, i) => {
const appended_url = apod_url + "&date=" + curr_moment.subtract(i, "days").format("YYYY-MM-DD");
return requestP(appended_url).then(({response, body}) => JSON.parse(body).hdurl);
})).then(apod_img_urls => {
const url = "https://launchlibrary.net/1.3/launch?next=20&mode=verbose";
return requestP(url).then(({response, body}) => {
const data = JSON.parse(body);
return res.render("index", {data, apod_img_urls});
});
});
});
Note: there's a lot of ES2015+ going on in there

If you use a promise-returning request library like request-promise then you can do something like:
app.get("/index", function(req, res) {
var apod_url = "https://api.nasa.gov/planetary/apod?api_key=[My Key]"
var curr_moment = moment();
var urls = [];
for(var i = 0; i < 10; i++) {
urls[i] = apod_url + "&date=" + curr_moment.subtract(i, "days").format("YYYY-MM-DD");
}
Promise.all(urls.map(url => rp({ url, json: true})).then((results) => {
// here you have all results with JSON already parsed for you
// ...
}).catch((err) => {
// handle error
// make sure to return response to client
// ...
});
// ...
});
If you want to work with promises then use a promise-returning module like request-promise or axios instead of the standard request, see:
https://www.npmjs.com/package/request-promise
https://www.npmjs.com/package/axios
Or, alternatively, use bluebird.promisify or built-in util.primisify (since 8.0) to promisify the request module, see:
http://bluebirdjs.com/docs/api/promise.promisify.html
https://nodejs.org/dist/latest-v8.x/docs/api/util.html
For more options of promisifying request see:
https://www.npmjs.com/package/request#promises--asyncawait
Then, when you have a promise-returning request library, make an array of URLs however you want, then map that array with the request functions, like this:
let rp = require('request-promise');
let urls = ['http://...', 'http://...'];
let promises = urls.map(rp);
and then use Promise.all to wait for all of them to finish, while being done concurrently:
Promise.all(promises)
.then(...)
.catch(...);
or, if you're using async/await, then:
try {
let x = await Promise.all(promises);
...
} catch (err) {
...
}
Many of modules like axios or request-promise-json or request-promise will parse JSON for you if you run it correctly, see:
https://github.com/request/request-promise#get-something-from-a-json-rest-api
Avoid parsing JSON yourself but if you do then always put JSON.parse() inside of try/catch (or use my little tryjson module) - see those answers to know why:
Calling a JSON API with Node.js
Reading requests body Nodejs
Node JS ignores undefined check

Node.js web scraping with loop on array of URLs

I'm trying to build a little script to scrap some data. I'm some basics knowledge in javascript however I'm kind of lost with all the async callback or promises stuff. Here is what I have now :
url = "http://Blablablabla.com";
var shares = function(req, res) {
request(url, function (error, response, body) {
if (!error) {
var $ = cheerio.load(body),
share = $(".theitemIwant").html();
return res.send(url + ":" + share);
} else {
console.log("We've encountered an error: " + error);
}
})
}
So everything is fine with this piece of code. What I would like to do is :
Using an array of url var urls = [url1,url2,url3,etc...]
Storing my scrapped data into another array, something like this data = [{url: url1, shares: share},{url: url2, shares: share},etc...]
I know I need to use something like this data.push({ urls: url, shares: share})})
and I understand that I need to loop over my first url array to push data into my second data array.
however I'm kind of lost with the request method and the way I should deal with async issue in my situation.
thanks !
edit#1 :
I tried this to use promises :
var url = "www.blablabla.com"
var geturl = request(url, function (error, response, body) {
if (!error) { return $ = cheerio.load(body) } else
{ console.log("We've encountered an error: " + error); }
});
var shares = geturl.then( function() {
return $(".nb-shares").html();
})
but got the following error geturl.then is not a function

I think you should use async:
var async = require('async');
var urls = ["http://example.com", "http://example.com", "http://example.com"];
var data = [];
var calls = urls.map((url) => (cb) => {
request(url, (error, response, body) => {
if (error) {
console.error("We've encountered an error:", error);
return cb();
}
var $ = cheerio.load(body),
share = $(".theitemIwant").html();
data.push({ url, share })
})
})
async.parallel(calls, () => { /* YOUR CODE HERE */ })
You could do the same with promises, but I don't see why.

I took a stab at it. You need to install the q library and require it to
var Q = require('q');
//... where ever your function is
//start with an array of string urls
var urls = [ "http://Blablablabla.com", '...', '...'];
//store results in this array in the form:
// {
// url: url,
// promise: <will be resolved when its done>,
// share:'code that you wanted'
// }
var results = [];
//loop over each url and perform the request
urls.forEach(processUrl);
function processUrl(url) {
//we use deferred object so we can know when the request is done
var deferred = Q.defer();
//create a new result object and add it to results
var result = {
url: url,
promise: deferred.promise
};
results.push(result);
//perform the request
request(url, function (error, response, body) {
if (!error) {
var $ = cheerio.load(body),
share = $(".theitemIwant").html();
//resolve the promise so we know this request is done.
// no one is using the resolve, but if they were they would get the result of share
deferred.resolve(share);
//set the value we extracted to the results object
result.share = share;
} else {
//request failed, reject the promise to abort the chain and fall into the "catch" block
deferred.reject(error)
console.log("We've encountered an error: " + error);
}
});
}
//results.map, converts the "array" to just promises
//Q.all takes in an array of promises
//when they are all done it rull call your then/catch block.
Q.all(results.map(function(i){i.promise}))
.then(sendResponse) //when all promises are done it calls this
.catch(sendError); //if any promise fails it calls this
function sendError(error){
res.status(500).json({failed: error});
}
function sendResponse(data){ //data = response from every resolve call
//process results and convert to your response
return res.send(results);
}

Here is another solution I like a lot :
const requestPromise = require('request-promise');
const Promise = require('bluebird');
const cheerio = require('cheerio');
const urls = ['http://google.be', 'http://biiinge.konbini.com/series/au-dela-des-murs-serie-herve-hadmar-marc-herpoux-critique/?src=konbini_home']
Promise.map(urls, requestPromise)
.map((htmlOnePage, index) => {
const $ = cheerio.load(htmlOnePage);
const share = $('.nb-shares').html();
let shareTuple = {};
shareTuple[urls[index]] = share;
return shareTuple;
})
.then(console.log)
.catch((e) => console.log('We encountered an error' + e));

Async / Callstack confusion

Alrighty, I'm pretty sure I know what the issue is, but I can't for the life of me figure out how to resolve it.
The way the below code works is the front-end sends two words back to the server, some sanitization happens and breaks the string into an array. That array is then iterated over, an async request is made for each word to the Wordnik API for synonyms. The resulting data structure sent back to the client is an object with {word1: [...synonyms], word2: [...synonyms]}.
With two words, this works exactly how I want 4 out of 5 times. That fifth time, the synonyms for the second word get applied to the first word and the second word has no data. Obviously, send it more words and the data confusion occurs more often.
So, I'm pretty sure this is a call stack issue, but I can't figure out how to resolve it. I keep thinking if I wrap the wordnikClient in setTimeout(..., 0); it's a step in the right direction, but feel like I'm misapplying the pattern. Any words of wisdom out there?
EDIT: https://github.com/ColinTheRobot/tweetsmithy-node/blob/master/server.js This was the prior version it has the same async issue. I had initially designed it with a Promise, but realized over the last couple days, that it wasn't really doing anything/I had also probably misapplied it so took it out for now.
app.get('/get-synonyms', (req, res) => {
var tweetWords = sanitizeTweet(req.query.data);
getDefs(tweetWords, res);
});
var getDefs = function(tweetWords, res) {
var i = 0;
var serialized = {};
tweetWords.forEach((word) => {
wordnikClient(word, (body) => {
var wordToFind = tweetWords[i];
var shortenedWords = [];
i++;
if (body[0]) {
shortenedWords = _.filter(body, (syn) => {
return syn.length < wordToFind.length;
});
serialized[wordToFind] = shortenedWords;
}
if (tweetWords.length == i) {
res.send(serialized);
}
});
});
}
var sanitizeTweet = function(tweet) {
var downcasedString = tweet.toLowerCase();
var punctuationless = downcasedString.replace(/[.,-\/#!$%\^&\*;:{}=\-_`~()]/g,"");
var finalString = punctuationless.replace(/\s{2,}/g," ");
return finalString.split(' ');
}
var wordnikClient = function(word, callback) {
var url = `http://api.wordnik.com:80/v4/word.json/${word}/relatedWords?useCanonical=false&relationshipTypes=synonym&limitPerRelationshipType=10&api_key=${process.env.WORDNIK_API_KEY}`
console.log('calling client');
request(url, (err, response, body) => {
if (!err && response.statusCode == 200 && response.body != '[]') {
callback(JSON.parse(body)[0].words);
} else if (!err && response.statusCode == 200 && response.body == '[]') {
callback([false]);
}
});
}

Yes, what is happening is that your second async call is completing first and because fo
if (tweetWords.length == i) {
res.send(serialized);
}
});
is returning to the client. One alternative is to use https://github.com/caolan/async to cooridnate your async calls, but I would suggest you convert wordnikClient to promises and then use Promise.all to control res.send
var wordnikClient = function(word) {
var url = `http://api.wordnik.com:80/v4/word.json/${word}/relatedWords?useCanonical=false&relationshipTypes=synonym&limitPerRelationshipType=10&api_key=${process.env.WORDNIK_API_KEY}`
console.log('calling client');
return new Promise( (resolve, reject) => {
request(url, (err, response, body) => {
if (!err && response.statusCode == 200 && response.body != '[]') {
resolve(JSON.parse(body)[0].words);
} else if (!err && response.statusCode == 200 && response.body == '[]') {
reject([false]);
}
});
});
and
Promise.all(tweetWords.map((word) => wordnikClient(word)))
.then(serialized => res.send(serialized))
.catch(err => res.status(500).send(err))
I've probably lost a little functionality along the way but you can re-add that

What the asynchronous callbacks do inside getDefs is not clear. The i variable counts the order of the replies, so I don't see why to use that to index tweetWords. I suggest you to use just word instead. A somewhat clearer solution could be made using Promises:
function getDefs(tweetWords, res) {
var serialized = {};
Promise.all(tweetWords.map(word => {
return wordnikClientAsync(word).then(body => {
if (body[0]) {
serialized[word] = _.filter(body, syn => syn.length < word.length);
}
});
})).then(() => {
res.send(serialized);
}, () => {
res.send("Error");
});
function wordnikClientAsync(word) {
return new Promise(resolve => wordnikClient(word, resolve));
}
}

Change tweetWords[i]; for word because the variable is outside the callback and iteration may don't run currently.