I am using the below piece of code to execute some set of queries and send the response of doing some validations on query results. For this scenario, I am using async module in Node JS.
async.series([
function(callback){
common.commonValidations(db);
callback();
},
function(callback){
console.log('second function');
res.end(JSON.stringify(gErrors));
callback();
}
], function(err){
console.log('sending res to client');
console.log(err);
});
common.commonValidations(db) function is used to execute few db2 queries.
Here my issue is, though I am using async module, the response is sent to the client while the query execution is going on. As per my understanding of async module, the second function in the array is executed once the first function is done with it's job.
Can someone help me on this? Thanks in advance.
Looks like common.commonValidations(db) is an asynchronous function, but you are not waiting for it to be done. you are calling callback() function before the answer for commonValidations comes back.
one possible change might be like
common.commonValidations(db,function(err,data) {
//check error
//process data
//and then
callback()
})
Related
fs.readFileSync(process.env.id_to_name, 'utf-8', function(err, data) {
if (err) throw err;
/*
a lot of stuff
*/
fs.mkdirSync(`clips`);
fs.writeFileSync(`clips/recap.json`, '{"players":[]}', 'utf8');
});
fs.readFileSync(`clips/recap.json`, 'utf-8', function(err, data) {
var info = JSON.parse(data);
info.players.push(/* stuff */);
fs.writeFileSync(`clips/recap.json`, JSON.stringify(info), 'utf8', function (err) { });
});
I don't know what I'm doing wrong here.
The second fs.readFileSync just doesn't wait for the first one to end so it doesn't find the file he should read.
You're using fs.readFileSync() incorrectly. It does not accept a callback as an argument and does not call a callback. See doc here.
I don't know if you meant to show us fs.readFile() that does accept a callback or not.
fs.readFileSync() returns its result directly (in a synchronous fashion) from the function call as in:
let data = fs.readFileSync(someFileName, someEncoding);
It does not use a callback and it throws an exception if there's an error reading the file.
If you meant to show us an example using fs.readFile(), then it's a non-blocking, asynchronous call. If you want your second file read to wait until the first one is done, you would have to put the second file read INSIDE the completion callback of the first.
Also, please never write code like if (err) throw err; inside an asynchronous callback. That's a pointless piece of code that does nothing useful as nothing can catch that throw and you have no way to communicate back errors. It is most unfortunate that nodejs documentation shows that regularly in its examples as real world code should never do that. You will need to write real error handling where you either handle the error in some way and continue or you abort the process or you communicate back the error (probably with a callback) so the calling code can handle and see the error. Exceptions throw in asynchronous callbacks do NOT propagate back to the caller. They end up going back to the bowels of the file system code where the callback was triggered from where you cannot catch or handle them.
If you really mean to be using all synchronous calls, then you would write your code like this:
try {
let data1 = fs.readFileSync(process.env.id_to_name, 'utf-8');
// other code here
fs.mkdirSync(`clips`);
fs.writeFileSync(`clips/recap.json`, '{"players":[]}', 'utf8');
let data2 = fs.readFileSync(`clips/recap.json`, 'utf-8');
var info = JSON.parse(data2);
info.players.push(/* stuff */);
fs.writeFileSync(`clips/recap.json`, JSON.stringify(info));
} catch(err) {
// handle errors here
console.log(err);
}
Note that this code can likely only be run once without error because fs.mkdirSync('clips') will fail unless you set the recursive flag.
Hint, you can use require() to load and parse a JSON file in one step so you don't have to read it and then parse it into a Javascript object.
I'm struggling with a javascript/Nodejs8 Google Cloud Function to publish payloads to Google PubSub.
So I have a Cloud Function triggered by HTTP requests and the request body is then published to a pubsub topic (configured for pull mode).
Here is my code:
const {PubSub} = require('#google-cloud/pubsub');
const pubsub = new PubSub();
const topic = pubsub.topic('my-fancy-topic');
function formatPubSubMessage(reqObj){
// the body is pure text
return Buffer.from(reqObj.body);
};
exports.entryPoint = function validate(req, res) {
topic.publish(formatPubSubMessage(req)).then((messageId) => {
console.log("sent pubsub message with id :: " + messageId)
});
res.status(200).json({"res":"OK"});
};
My issue is that the cloud function finishes executing before the pubsub message being published (in logs, the log "Function execution took X ms, finished with status code: 200" shows up around 30 or 40 seconds before the my pubsub log. I also had several times a log with "Ignoring exception from a finished function" and I dont get my pubsub log)
I'm not a javascript or nodejs specialist and I don't master javascript promises neither but I was wondering if I could make the publish synchronous. I'm thinking as well that I might be doing something wrong here !
Thank you in advance for your help.
In your logic, your callback / event handler function is being called when the HTTP message arrives. You then execute a publish() function. Executing a publish is an asynchronous activity. This means that it make take some time for the publish to complete and since JavaScript (intrinsically) doesn't want to block, it returns immediately with a promise that you can then use to be notified when the asynchronous work has completed. Immediately after executing the publish() your logic executes a res.status(....) which sends a response to the HTTP request and that is indeed the end of the flow request from the HTTP client. The asynchronous publish is still cooking and when it itself completes, then the callback for the publish occurs and you log a response.
Unfortunately, this is not a good practice as documented by Google here ...
https://cloud.google.com/functions/docs/bestpractices/tips#do_not_start_background_activities
In this last story, the function you call validate will still end prior to the publish being completed. If you want to block while the publish() executes (effectively making it synchronous), you can use the JavaScript await key word. Loosely, something like:
try {
let messageId = await topic.publish(....);
console.log(...);
catch(e) {
...
}
You will also need to flag the functions as being async. For example:
exports.entryPoint = async function validate(req, res) {
...
See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
You can also simply return a Promise from the function and the callback function will not be considered resolved until the Promise as a whole is resolved.
The bottom line is to study Promises in depth.
Right now, this code is sending a response before the publish is complete. When the response is sent, the function is terminated, and ongoing async work might not complete.
What you should do instead is send the response only after the publish is complete, which means putting that line of code in the then callback.
exports.entryPoint = function validate(req, res) {
topic.publish(formatPubSubMessage(req)).then((messageId) => {
console.log("sent pubsub message with id :: " + messageId)
res.status(200).json({"res":"OK"});
});
};
I suggest spending some time learning about how promises work, as this is crucial to building functions that work correctly.
i am trying to save something to a database using mongoose. The thing is i need to make sure the save was completed before i move on in the program and close the connection. Knowing that save is async in mongoose i tried using this code:
saveFunction = function(song){
song.save(function(err, userObj){
if(err){
console.log('Error' + err);
} else{
console.log('saved successfully:', userObj);
}
});
};
database.prototype.add= function(newSong){
mongoose.connect(url);
var song = new songModel({id : newSong.getId(),
title : newSong.getTitle(),
artist : newSong.getArtist,
genre : newSong.getGenre(),
rating : newSong.getRating(),
link : newSong.getLink()});
console.log("before async");
async.parallel([function (callback){
saveFunction(song);
callback();
}],function(){
mongoose.connection.close();
console.log('closed connection');
});
console.log("after async");
nextFreeId++;
};
^songModel is defined globally.
I tried lots of different methods and changed lots of thing but i always get and error of somekind. With this code i get a process.nexttick(function() throw err ) error. I just can't get it to work. Can someone tell me whats wrong or provide me with working code?
I think optimally the console should look like this:
before async
saved successfully
closed connection
after async
Thanks!
EDIT: Open to other alternatives to async aswell. I just want to get this code to work any way possible. I just need to save/find something/remove something and it needs to wait with the rest of the execution of the program until the save/find/removal is done. I'm getting pretty desperate, lost nearly a day on this problem alone on a tight shedule :(
You need to return a callback from your save function.
saveFunction = function(song,callback){
song.save(function(err, userObj){
if(err){
console.log('Error' + err);
return callback(true,err)
} else{
console.log('saved successfully:', userObj);
return callback(null);
}
});
};
Edit
From your comment, the behavior you are expecting will never occur. You are expecting
console.log("before async");
async.parallel -> do your bits
console.log('closed connection');
console.log("after async");
However this will never happen because async.parallel is an asynchronous call, which means that the execution does not wait for it to finish before moving onto the next command. The behavior you are seeing is
console.log("before async");
async.parallel -> starts
console.log("after async");
async.parallel -> console.log('closed connection');
Node is doing the first log, starting the async.parallel, then console.logging "after async". Then when async.parallel gets to its callback function, it prints "closed connection", so it appears after "after async" because it was executed after.
Any logic you want to perform that relies on the result of async.parallel must happen in the callback function. Furthermore, async.parallel is used when you want to run 2 or more functions asynchronously, then execute a callback once they are all finished. Your solution does not require async.parallel. You can replace that with:
saveFunction(song,function(err){
if(err){
//failed to save song
}
mongoose.connection.close(); //you do not need to do this anyway
console.log('closed connection');
nextFreeId++;
//anything else you need to do here
});
I have a node.js server. When a user requests a page I call a function that pulls some info from db and services the request. Simple function with callback then execute response.send
I need to perform secondary computation/database updates which are not necessary for rendering the page request. I don't want the user to wait for these secondary ops to complete (even though they take only 200 ms.)
Is there a way to call a function and exit gracefully without callback?
You can simply do something like this
app.get('/path', function(req, res){
getInfoFromDatabase(); // get info from the database
res.render('myview', {info: data});
// perform post render operations
postRenderingCode();
return;
});
If I understand your problem correctly you can use setTimeout with a value of 0 to place the maintenance code at the end of the execution queue.
function service(user, callback) {
// This will be done later
setTimeout(function() {
console.log("Doing some maintenance work now...");
}, 0);
// Service the user
callback("Here's your data " + user);
}
service("John", function(data) { console.log(data); });
service("Jane", function(data) { console.log(data); });
The output will be:
Here's your data John
Here's your data Jane
Doing some maintenance work now...
Doing some maintenance work now...
You can call your extra ASYNCHRONOUS function before, or after your actual response; for example:
yourCoolFunction() // does awesome stuff...
response.writeHead(200, 'OK');
response.write('some cool data response');
response.end();
Note that the "yourCoolFunction" mentioned must be asynchronous, else the rest of the code will wait for it to complete.
Assuming you're using express.js:
function(req, res, next) {
doSomeAsyncWork(function(e, d) {
// Some logic.
doSomeMoreAsyncWork(function() {})
res.send(/* some data*/)
})
}
Basically you don't really care about the response of the additional async work so you can put in a function that does nothing for the callback.
since I can see none of the answers so far are even somehow helpful, and in order to avoid confusing. What I suggest is use on the object you are working on the following:
function doStuff() {
myObj.emit('myEvent', param);
}
function callback(param) {
do stuff;
}
myObj.on('myEvent', callback);
well, just do what you said, render the page, respond to the request and do whatever you have to do, your code isn't suddenly going to die because you responded to the request.
with express:
function handleTheRequest(req, res) {
res.status(200).send("the response")
// do whatever you like here
}
I have a small problem with nested callbacks in javascript. Apparently im doing something wrong, but i did my research and tried to follow the tutorials avaialble throughout the web. I know that my code works, since query returns proper data, but i have no idea why my code doesnt "wait" within executeQuery method till the res is fetched from database, it just goes straight to "oh noes" section.
DatabaseConnection.prototype.executeQuery = function(query, executeQueryDone){
var activeConnection;
console.log("YEAAA, executing Query: " + query);
this.pool.getConnection(function (err, connection){
console.log("Got Connection, we are ready to go!");
if (err){
console.log("Error, DAMMNIT! " + err);
executeQueryDone(err);
}
activeConnection = connection;
activeConnection.connect();
activeConnection.query(query, function(error, res){
console.log("Connection from pool is executing Query");
if(error){
console.log("Error during executing query");
executeQueryDone(error);
}
else {
console.log(" OK now release connection (dont be selfish)! ");
activeConnection.release();
executeQueryDone(null, res);
}
});
});
console.log("oh noes! IM AFTER CONNECTION, why dude? WHY???? ");
};
I'd be grateful for any hints since im struggling with that since yesterday.
=====================
PROBLEM SOLVED:
generally all was OK, the "issue" was mistakenly written test:
i made it like that:
describe('testDB2', function () {
it('should return proper STUFF', function (done) {
assert.equal(1, someService.getStuff(function(err, result){
if (err === null){
console.log("err is null, as it should be!");
}
console.log(" result from DB " + result[1].NUMBERS);
}));
});
});
while is should be like that:
describe('testDB2', function () {
it('should return proper STUFF', function (done) {
someService.getStuff(function(err, result){
assert.equal(err, null);
assert.equal(result[1].NUMBERS, 43637654);
done();
});
});
});
as a result (in the incorrect case), i didnt fetch the result the way i wanted as assert couldnt "catch up"
thanks to all for the enlightment ;)
Your console.log call isn't part of a callback, so it will be called as soon as the getConnection call is made. If you want it to be called only after your callback to getConnection fires, you either need to call it at the end of that call, or you need to use some form of promises.
Javascript is single threaded, but it does use a task queue. When the database connection is instantiated and the pool connected to, the response to that is placed into the task queue to be executed or actioned when complete.
Directly after placing that in the task queue the next piece of execution is the log that does the "oh noes" - nice messaging btw lol.
So essentially what happens is the db call gets placed in the task queue for later execution, and then the log occurs, and then the task queue executes at a later time with the db response.