Related
I'm learning to use MongoDB by creating a simple blog app. However, a portion of my code that saves a given post seems to give problems with promises occasionally, but not always, and whether the code succeeds simply seems to be luck.
Each post in my database is stored with the following schema:
{
title: String,
author: String,
body: String,
slug: String,
baseSlug: String,
published: { type: Boolean, default: false }
}
The slug defines the link used to access the blog post, and is automatically generated based upon the title of the blog post. However, if article titles are duplicates, the slug will have a number added to the end to differentiate it from similar articles, while the baseSlug will remain the same. For example:
I create the post "My first post", and it is assigned the baseSlug of "my-first-post". Because no other posts have the same baseSlug, the slug is also set to be "my-first-post".
I create another post called "My first post", and it is assigned the baseSlug of "my-first-post". However, because another post has the same baseSlug, it is assigned the slug "my-first-post-1".
To create this behavior, I wrote the following addpost route in Express:
app.post("/addpost", (req, res) => {
let postInfo = req.body;
for (key of Object.keys(postInfo)) {
if (postInfo[key] == "true") postInfo[key] = true;
}
let slug = postInfo.title
.toLowerCase()
.split(" ")
.filter(hasNumber) // return /\d/.test(str);
.slice(0, 5)
.join("-");
postInfo.slug = slug;
var postData;
Post.find({ baseSlug: postInfo.slug }, (error, documents) => {
if (documents.length > 0) {
let largestSlugSuffix = 0;
for (let document of documents) {
var fullSlug = document.slug.split("-");
var suffix = fullSlug[fullSlug.length - 1];
if (!isNaN(suffix)) {
if (parseInt(suffix) > largestSlugSuffix) {
largestSlugSuffix = suffix;
}
}
}
largestSlugSuffix++;
postInfo.baseSlug = postInfo.slug;
postInfo.slug += "-" + largestSlugSuffix;
} else {
postInfo.baseSlug = postInfo.slug;
}
postData = new Post(postInfo);
})
.then(() => {
postData
.save()
.then(result => {
res.redirect("/");
})
.catch(err => {
console.log(err);
res.status(400).send("Unable to save data");
});
})
.catch(err => {
console.log(err);
res.status(400).send("Unable to save data");
});
});
This code seems to work most of the time, but sometimes it fails, and outputs the following:
TypeError: Cannot read property 'save' of undefined
at C:\Users\User\BlogTest\app.js:94:18
at processTicksAndRejections (internal/process/task_queues.js:94:5)
(For reference, line 94 in my file is postData.save())
I suspect it is because the main body of the function takes longer than it should to execute, and the postData variable is not yet defined. However, postData.save() should not be executed until the promise finishes, because of the .then() callback function.
Why is my code behaving like this? Is there any way to fix it?
The issue is that you are mixing promises with callbacks and closures. That's not how this is intended to work.
When you chain promises, whatever you return in the first promise handler will be added as an input to the next one. And if you return a promise, that promise will be resolved first before being sent to the next thenable.
So you need to return promises from your promises, like this:
app.post("/addpost", (req, res) => {
let postInfo = req.body;
for (key of Object.keys(postInfo)) {
if (postInfo[key] == "true") postInfo[key] = true;
}
let slug = postInfo.title
.toLowerCase()
.split(" ")
.filter(hasNumber) // return /\d/.test(str);
.slice(0, 5)
.join("-");
postInfo.slug = slug;
// var postData; <-- Don't do that
Post.find({ baseSlug: postInfo.slug })
.then((documents) => {
if (documents.length > 0) {
let largestSlugSuffix = 0;
for (let document of documents) {
var fullSlug = document.slug.split("-");
var suffix = fullSlug[fullSlug.length - 1];
if (!isNaN(suffix)) {
if (parseInt(suffix) > largestSlugSuffix) {
largestSlugSuffix = suffix;
}
}
}
largestSlugSuffix++;
postInfo.baseSlug = postInfo.slug;
postInfo.slug += "-" + largestSlugSuffix;
} else {
postInfo.baseSlug = postInfo.slug;
}
return new Post(postInfo);
// We could actually have called postData.save() in this method,
// but I wanted to return it to exemplify what I'm talking about
})
// It is important to return the promise generated by postData.save().
// This way it will be resolved first, before invoking the next .then method
.then( (postData) => { return postData.save(); })
// This method will wait postData.save() to complete
.then( () => { res.redirect("/"); })
.catch( (err) => {
console.log(err);
res.status(400).send("Unable to save data");
});
});
It can be greatly simplified with async/await:
app.post("/addpost", async (req, res) => {
try {
let postInfo = req.body;
for (key of Object.keys(postInfo)) {
if (postInfo[key] == "true") postInfo[key] = true;
}
let slug = postInfo.title
.toLowerCase()
.split(" ")
.filter(hasNumber)
.slice(0, 5)
.join("-");
postInfo.slug = slug;
let documents = await Post.find({ baseSlug: postInfo.slug });
if (documents.length > 0) {
let largestSlugSuffix = 0;
for (let document of documents) {
var fullSlug = document.slug.split("-");
var suffix = fullSlug[fullSlug.length - 1];
if (!isNaN(suffix)) {
if (parseInt(suffix) > largestSlugSuffix) {
largestSlugSuffix = suffix;
}
}
}
largestSlugSuffix++;
postInfo.baseSlug = postInfo.slug;
postInfo.slug += "-" + largestSlugSuffix;
} else {
postInfo.baseSlug = postInfo.slug;
}
let postData = new Post(postInfo);
await postData.save();
res.redirect("/");
} catch (err) {
console.log(err);
res.status(400).send("Unable to save data");
};
});
You are mixing callbacks and promises and while it may do something, I'm not sure what it will do exactly. You should pick one or the other and not mix them as much as possible. I would recommend picking promises if you are using a language that supports async/await, otherwise callbacks.
So for example your outter handler could be an async function
app.post("/addpost", async (req, res) => {
//...
})
Your real bug is in handling Post.find you are handling it somewhat with a callback and somewhat with a promise, and probably whats happening is that its random which one will get called first the callback or the promise resolution. Instead of both you should just do this now that you have an async function:
try {
const posts = await Post.find({ baseSlug: postInfo.slug });
// stuff you were doing in the callback
const post = new Post(postInfo)
// Now the promise code
await post.save()
// success!
res.redirect("/");
} catch (err) {
// With an async function you can just catch errors like normal
console.log(err);
res.status(400).send("Unable to save data");
}
If you're not using webpack or typescript and cannot target es7 then and thus cannot use async/await then I would recommend just using callbacks, do not use .then or .catch and that would look more like:
function error(err) {
console.log(err)
res.status(400).send("Unable to save data")
}
Post.find({ baseSlug: postInfo.slug }, (err, documents) => {
if (err) return error(err)
// stuff you're doing in the callback now
const post = new Post(postInfo)
post.save((err) => {
if (err) return error(err)
// success!
res.redirect("/");
})
})
So I have a method, which I want to call multiple times in a loop. This is the function:
function PageSpeedCall(callback) {
var pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${websites[0]}&strategy=mobile&key=${keys.pageSpeed}`;
// second call
var results = '';
https.get(pagespeedCall, resource => {
resource.setEncoding('utf8');
resource.on('data', data => {
results += data;
});
resource.on('end', () => {
callback(null, results);
});
resource.on('error', err => {
callback(err);
});
});
// callback(null, );
}
As you see this is an async function that calls the PageSpeed API. It then gets the response thanks to the callback and renders it in the view. Now how do I get this to be work in a for/while loop? For example
function PageSpeedCall(websites, i, callback) {
var pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${websites[i]}&strategy=mobile&key=${keys.pageSpeed}`;
// second call
var results = '';
https.get(pagespeedCall, resource => {
resource.setEncoding('utf8');
resource.on('data', data => {
results += data;
});
resource.on('end', () => {
callback(null, results);
});
resource.on('error', err => {
callback(err);
});
});
// callback(null, );
}
var websites = ['google.com','facebook.com','stackoverflow.com'];
for (let i = 0; i < websites.length; i++) {
PageSpeedCall(websites, i);
}
I want to get a raport for each of these sites. The length of the array will change depending on what the user does.
I am using async.parallel to call the functions like this:
let freeReportCalls = [PageSpeedCall, MozCall, AlexaCall];
async.parallel(freeReportCalls, (err, results) => {
if (err) {
console.log(err);
} else {
res.render('reports/report', {
title: 'Report',
// bw: JSON.parse(results[0]),
ps: JSON.parse(results[0]),
moz: JSON.parse(results[1]),
// pst: results[0],
// mozt: results[1],
// bw: results[1],
al: JSON.parse(results[2]),
user: req.user,
});
}
});
I tried to use promise chaining, but for some reason I cannot put it together in my head. This is my attempt.
return Promise.all([PageSpeedCall,MozCall,AlexaCall]).then(([ps,mz,al]) => {
if (awaiting != null)
var areAwaiting = true;
res.render('admin/', {
title: 'Report',
// bw: JSON.parse(results[0]),
ps: JSON.parse(results[0]),
moz: JSON.parse(results[1]),
// pst: results[0],
// mozt: results[1],
// bw: results[1],
al: JSON.parse(results[2]),
user: req.user,
});
}).catch(e => {
console.error(e)
});
I tried doing this:
return Promise.all([for(let i = 0;i < websites.length;i++){PageSpeedCall(websites, i)}, MozCall, AlexaCall]).
then(([ps, mz, al]) => {
if (awaiting != null)
var areAwaiting = true;
res.render('admin/', {
title: 'Report',
// bw: JSON.parse(results[0]),
ps: JSON.parse(results[0]),
moz: JSON.parse(results[1]),
// pst: results[0],
// mozt: results[1],
// bw: results[1],
al: JSON.parse(results[2]),
user: req.user,
});
}).catch(e => {
console.error(e)
});
But node just said it's stupid.
And this would work if I didn't want to pass the websites and the iterator into the functions. Any idea how to solve this?
To recap. So far the functions work for single websites. I'd like them to work for an array of websites.
I'm basically not sure how to call them, and how to return the responses.
It's much easier if you use fetch and async/await
const fetch = require('node-fetch');
async function PageSpeedCall(website) {
const pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${website}&strategy=mobile&key=${keys.pageSpeed}`;
const result = await fetch(pagespeeddCall);
return await result.json();
}
async function callAllSites (websites) {
const results = [];
for (const website of websites) {
results.push(await PageSpeedCall(website));
}
return results;
}
callAllSites(['google.com','facebook.com','stackoverflow.com'])
.then(results => console.log(results))
.error(error => console.error(error));
Which is better with a Promise.all
async function callAllSites (websites) {
return await Promise.all(websites.map(website => PageSpeedCall(website));
}
Starting on Node 7.5.0 you can use native async/await:
async function PageSpeedCall(website) {
var pagespeedCall = `https://www.googleapis.com/pagespeedonline/v4/runPagespeed?url=https://${website}&strategy=mobile&key=${keys.pageSpeed}`;
return await promisify(pagespeedCall);
}
async function getResults(){
const websites = ['google.com','facebook.com','stackoverflow.com'];
return websites.map(website => {
try {
return await PageSpeedCall(website);
}
catch (ex) {
// handle exception
}
})
}
Node http "callback" to promise function:
function promisify(url) {
// return new pending promise
return new Promise((resolve, reject) => {
// select http or https module, depending on reqested url
const lib = url.startsWith('https') ? require('https') : require('http');
const request = lib.get(url, (response) => {
// handle http errors
if (response.statusCode < 200 || response.statusCode > 299) {
reject(new Error('Failed to load page, status code: ' + response.statusCode));
}
// temporary data holder
const body = [];
// on every content chunk, push it to the data array
response.on('data', (chunk) => body.push(chunk));
// we are done, resolve promise with those joined chunks
response.on('end', () => resolve(body.join('')));
});
// handle connection errors of the request
request.on('error', (err) => reject(err))
})
}
Make PageSpeedCall a promise and push that promise to an array as many times as you need, e.g. myArray.push(PageSpeedCall(foo)) then myArray.push(PageSpeedCall(foo2)) and so on. Then you Promise.all the array.
If subsequent asynch calls require the result of a prior asynch call, that is what .then is for.
Promise.all()
Promise.all([promise1, promise2, promise3]).then(function(values) {
console.log(values);
});
Please forgive the fairly case-specific question, though I think the general end goal could be of use to other people.
Goal: Populate a MongoDB with data requested from multiple JSON API URLs.
Short question: So far I've had some success with request-promise, which uses Bluebird:
var rp = require('request-promise');
var options = {
uri: 'http://www.bbc.co.uk/programmes/b006qsq5.json',
headers: {
'User-Agent': 'Request-Promise'
},
json: true
};
rp(options)
.then(function (body) {
// Mongoose allows us query db for existing PID and upsert
var query = {pid: body.programme.pid},
update = {
name: body.programme.title,
pid: body.programme.pid,
desc: body.programme.short_synopsis
},
options = { upsert: true, new: true };
// Find the document
Programme.findOneAndUpdate(query, update, options, function(err, result) {
if (err) return res.send(500, { error: err });
return res.send("succesfully saved");
});
})
.catch(function (err) {
return res.send(err);
})
But how do I loop over an array of URLs, without the program failing if any of the promises are rejected?
Something like this for example, using Bluebird, fails if any of the URLs errors.
const urls = ['http://google.be', 'http://google.uk']
Promise.map(urls, rp)
.map((htmlOnePage, index) => {
return htmlOnePage;
})
.then(console.log)
.catch((e) => console.log('We encountered an error' + e));
As I want to write to the DB with successful requests, and ignore those that might not be responding right then, I need something that skips over rejected promises, which .all does not do.
Long question:
I've been reading up about promises all day and it's making my head hurt! But I've found some good resources, such as https://pouchdb.com/2015/05/18/we-have-a-problem-with-promises.html, which mentions the use of a Promise factory. Would this work for my case? I initially thought I should make each request, process the result and add it to the DB, then move on to the next request; but having seen .all I thought I should do all the requests, save the results in an array and loop over that with my DB saving function.
Should I even be using Promises for this? Maybe I should just make use of something like async.js and run my requests in series.
Thanks very much for any help or ideas.
But how do I loop over an array of URLs, without the program failing if any of the promises are rejected?
if you return a value from .catch other than a rejected promise, you will return a resolved promise
So, your .then for each individual request could return an object like
{
success: true,
result: whateverTheResultIs
}
and your catch returns
{
success: false,
error: whateverTheErrorIs
}
Really you don't NEED the success property, it's a convenience though
So the code would be - assuming process(url) returns a Promise
Promise.map(urls, url =>
process(url)
.then(result => ({result, success:true}))
.catch(error => ({error, success:false}))
)
.then(results => {
let succeeded = results.filter(result => result.success).map(result => result.result);
let failed = results.filter(result => !result.success).map(result => result.error);
});
Or, in ES5
Promise.map(urls, function (url) {
return process(url).then(function (result) {
return { result: result, success: true };
}).catch(function (error) {
return { error: error, success: false };
});
}).then(function (results) {
var succeeded = results.filter(function (result) {
return result.success;
}).map(function (result) {
return result.result;
});
var failed = results.filter(function (result) {
return !result.success;
}).map(function (result) {
return result.error;
});
});
I don't know if this fit your case, but I think You can use a counter to check when all promises has returned, regardless of the fact that each one has been resolved or rejected
var heroes = [
'Superman',
'Batman',
'Spiderman',
'Capitan America',
'Ironman',
];
function getHero(hero) {
return new Promise((resolve, reject) => {
setTimeout(() => {
return Math.round(Math.random()) ? resolve(hero + ' lives') : reject(hero + ' dead');
}, Math.random() * 3000)
})
}
function checkHeroes() {
var checked = heroes.length;
heroes.forEach((hero) => {
getHero(hero)
.then((res) => {
checked --;
console.log(res);
if (!checked) done();
})
.catch((err) => {
checked --;
console.log(err);
if (!checked) done();
});
})
}
function done() {
console.log('All heroes checked');
}
checkHeroes();
I think your issue is less about the bluebird api than structuring your promise chain.
const reducePropsToRequests = (props) => Promise.resolve(Object
.keys(props)
.reduce((acc, key) => {
acc[key] = request(sources[key]);
return acc;
}, {}));
const hashToCollection = (hash) => Promise.resolve(Object
.keys(hash)
.reduce((acc, k) => {
return [...acc, {source: k, data: hash[k]}];
}, []));
const fetchFromSources = (sources) => Promise.props(sources);
const findSeveralAndUpdate = (results) => Promise
.each(results.map(obj => {
// you have access to original {a: 'site.com'}
// here, so use that 'a' prop to your advantage by abstracting out
// your db config somewhere outside your service
return Programme.findOneAndUpdate(someConfig[obj.source], obj.data);
}))
const requestFromSeveralAndUpdate = (sources) => reducePropsToRequests(sources)
.then(fetchFromSources)
.then(hashToCollection)
.then(findSeveralAndUpdate)
.catch(/* some err handler */);
requestFromSeveralAndUpdate({ a: 'site.com', b: 'site.net' });
I'd just use request and write my own promise with try catch inside that only resolves. Pseudo example below
var request = require('request')
var urls = ['http://sample1.com/json', 'http://sample2.com/json']
var processUrl = (url) => {
return new Promise((resolve,reject)=> {
var result;
try {
var myRequest = {
uri: url,
method: 'GET',
header: {...}
};
request(option, (res,body,err)=> {
if(err) {
result = err;
return;
}
result = body;
})
}
catch(e) {
result = e;
}
finally {
resolve(result)
}
})
}
Response.json should execute after foreach loop completes its execution
var todoarr = (req.body.data) ? req.body.data : undefined
todoarr.forEach(function(element) {
if(element.done == true) {
TodoService.removeTodo(element, function(success) {
});
}
});
res.json("success");
You can try to use async.js http://caolan.github.io/async/ .
each method http://caolan.github.io/async/docs.html#each
Or you can try use Promise.all.
For example:
let promiseArr = [];
todoarr.forEach(function(element) {
if(element.done == true) {
promiseArr.push(somePromiseMethod(element));
}
});
//now execute promise all
Promise.all(promiseArr)
.then((result) => res.send("success"))
.catch((err) => res.send(err));
More info here. https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
Some promise example:
function somePromiseMethod(element) {
return new Promise((resolve,reject) => {
TodoService.removeTodo(element, function(success) {
resolve();
});
});
}
Hope this helps.
You can't send multiple responses on single request, the only thing you can do it's a single response with the array of results:
es with async:
const async = require('async')
// FIX ME: this isn't correctly handled!
const todoarr = (req.body.data) ? req.body.data : undefined
let results = []
async.each(todoarr, function(element, callback) {
console.log('Processing todo ' + element)
if(element.done == true) {
TodoService.removeTodo(element, function(err, success) {
if(err){
callback(err)
} else {
results.push(success)
callback(null, success)
}
})
}
}, function(err) {
if(err) {
console.log('A element failed to process', err)
res.status(500).json(err)
} else {
console.log('All elements have been processed successfully')
// array with the results of each removeTodo job
res.status(200).json(results)
}
})
You can send the response inside the callback function of forEach.
Modify your function so that it will call res.json() on the last iteration only.
Example:
var todoarr = (req.body.data) ? req.body.data : undefined
todoarr.forEach(function(element,index) {
if(element.done == true) {
TodoService.removeTodo(element, function(success) {
});
}
if(index==todoarr.length-1){
res.json("success");
}
});
However, it may not be according to coding standards but it can definitely solve the problem.
I'm using Express for routing and Sequelize for DB management.
app.get('/api/users/:username', (req, res) => {
let username = req.params.username;
findChattersPerRole()
.then(chattersPerRole => {
console.log('instakbot should\'ve been added by now...');
});
});
The function findChattersPerRole returns an object with each user's username and role as another object.
const findChattersPerRole = () => {
return fetch('https://tmi.twitch.tv/group/user/instak/chatters')
.then(parseJSON)
.then(r => {
let chatters = r.chatters;
let chattersPerRole = Object.keys(chatters).map(role => {
return chatters[role].map(username => {
console.log('findOrCreateViewer will be executed after this');
findOrCreateViewer(username, role);
return {
username: username,
role: role
};
});
});
return Promise.resolve(flattenDeep(chattersPerRole));
}).catch(err => {
console.log(`Error in fetch: ${err}`);
});
};
The problem is, in my route, I expect the console.log('instakbot should\'ve been added by now...'); to be executed AFTER my viewers got inserted into the database because in my function findChattersPerRole I already insert them with the function findOrCreateViewer. I expect this to happen because in my route I write the console.log when findChattersPerRole() is resolved...
const findOrCreateViewer = (username, role) => {
return Viewer.findOrCreate({
where: {
username
},
defaults: {
instakluiten: 5,
role
}
}).spread((unit, created) => {
console.log('unit is: ', unit.dataValues.username);
if(created){
return `created is ${created}`;
}else{
return unit;
}
});
};
However, in my terminal you can see that this is not the way it's happening... Why aren't my promises being executed at the expected time?
Screenshot of my terminal
The return {username: ...} after findOrCreateViewer(username, role); happens immediately after the function is called and before any data has been inserted. That also means that return Promise.resolve(flattenDeep(chattersPerRole)); happens before any data has been inserted, etc.
You said findOrCreateViewer returns a promise, so you need to wait until that promise is resolved (i.e. wait until after the data was inserted) before continuing with something else.
You want chattersPerRole to be an array of (arrays of) promises and only proceed after all the promises are resolved.
This is easy to do with Promise.all:
const findChattersPerRole = () => {
return fetch('https://tmi.twitch.tv/group/user/instak/chatters')
.then(parseJSON)
.then(r => {
let chatters = r.chatters;
let chattersPerRole = Object.keys(chatters).map(
role => chatters[role].map(username => {
console.log('findOrCreateViewer will be executed after this');
return findOrCreateViewer(username, role).then(
() => ({username, role})
);
});
);
return Promise.all(flattenDeep(chattersPerRole));
}).catch(err => {
console.log(`Error in fetch: ${err}`);
});
};
Now the promise returned by findChattersPerRole will be resolved after all the promises returned by findOrCreateViewer are resolved.
Promises are doing no magic. Returning a promise doesn't mean that calling the function will block, but rather that you can easily chain callbacks to do something with the result. You'll need to use
function findChattersPerRole() {
return fetch('https://tmi.twitch.tv/group/user/instak/chatters')
.then(parseJSON)
.then(r => {
let chatters = r.chatters;
let chattersPerRole = Object.keys(chatters).map(role => {
return chatters[role].map(username => {
console.log('findOrCreateViewer will be executed after this');
return findOrCreateViewer(username, role).then(() => {
// ^^^^^^ ^^^^^
return {
username: username,
role: role
};
});
});
});
return Promise.all(flattenDeep(chattersPerRole));
// ^^^ get a promise for an array of results from an array of promises
}).catch(err => {
console.log(`Error in fetch: ${err}`);
});
}