I'm struggling with a javascript/Nodejs8 Google Cloud Function to publish payloads to Google PubSub.
So I have a Cloud Function triggered by HTTP requests and the request body is then published to a pubsub topic (configured for pull mode).
Here is my code:
const {PubSub} = require('#google-cloud/pubsub');
const pubsub = new PubSub();
const topic = pubsub.topic('my-fancy-topic');
function formatPubSubMessage(reqObj){
// the body is pure text
return Buffer.from(reqObj.body);
};
exports.entryPoint = function validate(req, res) {
topic.publish(formatPubSubMessage(req)).then((messageId) => {
console.log("sent pubsub message with id :: " + messageId)
});
res.status(200).json({"res":"OK"});
};
My issue is that the cloud function finishes executing before the pubsub message being published (in logs, the log "Function execution took X ms, finished with status code: 200" shows up around 30 or 40 seconds before the my pubsub log. I also had several times a log with "Ignoring exception from a finished function" and I dont get my pubsub log)
I'm not a javascript or nodejs specialist and I don't master javascript promises neither but I was wondering if I could make the publish synchronous. I'm thinking as well that I might be doing something wrong here !
Thank you in advance for your help.
In your logic, your callback / event handler function is being called when the HTTP message arrives. You then execute a publish() function. Executing a publish is an asynchronous activity. This means that it make take some time for the publish to complete and since JavaScript (intrinsically) doesn't want to block, it returns immediately with a promise that you can then use to be notified when the asynchronous work has completed. Immediately after executing the publish() your logic executes a res.status(....) which sends a response to the HTTP request and that is indeed the end of the flow request from the HTTP client. The asynchronous publish is still cooking and when it itself completes, then the callback for the publish occurs and you log a response.
Unfortunately, this is not a good practice as documented by Google here ...
https://cloud.google.com/functions/docs/bestpractices/tips#do_not_start_background_activities
In this last story, the function you call validate will still end prior to the publish being completed. If you want to block while the publish() executes (effectively making it synchronous), you can use the JavaScript await key word. Loosely, something like:
try {
let messageId = await topic.publish(....);
console.log(...);
catch(e) {
...
}
You will also need to flag the functions as being async. For example:
exports.entryPoint = async function validate(req, res) {
...
See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
You can also simply return a Promise from the function and the callback function will not be considered resolved until the Promise as a whole is resolved.
The bottom line is to study Promises in depth.
Right now, this code is sending a response before the publish is complete. When the response is sent, the function is terminated, and ongoing async work might not complete.
What you should do instead is send the response only after the publish is complete, which means putting that line of code in the then callback.
exports.entryPoint = function validate(req, res) {
topic.publish(formatPubSubMessage(req)).then((messageId) => {
console.log("sent pubsub message with id :: " + messageId)
res.status(200).json({"res":"OK"});
});
};
I suggest spending some time learning about how promises work, as this is crucial to building functions that work correctly.
Related
Im' trying to understand using promises with Google Cloud Functions a bit better. I just learned about the 'finally' method on promises, which is called after all promises in the chain are fully resolved or rejected. In a http function is it good practice to put response.send() inside of the finally method?
The below code uses request-promise-native for the http request. In the first .then() I call parseSchedule, which uses the cheerio web scraping api to loop through some data and on a website, and add it to the scheduledGames array (synchronously, I think).
I return from that and the then log that data to the console in writeDB, but one thing I noticed is that I see response.send() log 'execution finished' before I see the data from scheduleGames in the log. Is that correct?
Should I be using the 'finally' block like this?
Thanks,
const options = {
uri: 'https://www.cbssports.com/nba/schedule/' + urlDate,
Connection: 'keep-alive',
transform: function (body) {
return cheerio.load(body);
}
};
return request(options)
.then(parseSchedule)
.then(writeSchedule)
.catch((err) => console.log("there was an error: " + err))
.finally(res.send("execution finished"));
function parseSchedule($){
const scheduledGames = [];
$('tbody').children('tr').each((i, element) => {
const gameTime = $(element).children('td').eq(2).find('a').text()
const scheduledGame = { gameTime: gameTime};
scheduledGames.push(scheduledGame);
});
return scheduledGames;
}
function writeDB(scheduledGames){
console.log(scheduledGames);
}
}
It typically makes more sense to send a success response at the time in the promise chain when everything is successful, or send an error response in a catch handler. If you do these two things, it doesn't make sense to use finally at all, since success and error are the only two cases you really need to handle. Unless you have some special case, stick to just success and error.
I am trying to build a messaging web app using socket io and redux, and this post comes across. I can't comment on the post to ask questions, so I am posting a new question asking about the answer in that post(sorry if I violate stackoverflow quideline, but there seems to be no way for me to comment.) The part I don't understand is
In socketClient.js
emit(event, data) {
return new Promise((resolve, reject) => {
if (!this.socket) return reject('No socket connection.');
return this.socket.emit(event, data, (response) => {
// Response is the optional callback that you can use with socket.io in every request. See 1 above.
if (response.error) {
console.error(response.error);
return reject(response.error);
}
return resolve();
});
});
}
I don't quite understand the callback funciton in this.socket.emit. From what I understand, the call back function will be executed on server side, when the server receive the event, data and callback funciton from client. If that's the case, what does the return do in the callback function? what should response be? and what should this.socket.emit return? and how can the server resolve a promise on client side?
I don't quite understand the callback function in this.socket.emit.
When you pass a callback as the third argument to socket.io's .emit() method, it tells socket.io that you want an acknowledgement that the server has received the message and an optional response back from the server. That callback will be called after the server has received your message and called a callback on its end and the server has the option of sending a response back with that.
The return statements inside that callback are only for flow of control. They just cause code to stop executing in the callback. People tend to forget that reject() and resolve() are just function calls. They don't cause the rest of the function to stop executing.
The code you show could have been written like this with only the one return statement to return the promise:
emit(event, data) {
return new Promise((resolve, reject) => {
if (!this.socket) {
reject('No socket connection.');
} else {
this.socket.emit(event, data, (response) => {
if (response.error) {
console.error(response.error);
reject(response.error);
} else {
resolve();
}
});
}
});
}
Here, it uses if/else to control program flow rather than using return to stop execution of the function early. Either can work - personal preference.
If that's the case, what does the return do in the callback function?
It just stops further code from executing inside the callback function. Just a control flow thing. There is no meaningful value to return from the callback.
what should response be?
response will be whatever the server sent as part of the acknowledgement.
what should this.socket.emit return?
this.socket.emit() happens to return the socket itself (allows for method chaining). But, that's not relevant here because there's no point in returning any value from a Promise executor function. If you do return a value, it is not used by the Promise in any way. Basically, there's too many return statements in that original code. They lead one to think the return value is relevant when it isn't. They do no harm, but do cloud up the proper intent and meaning of the code.
and how can the server resolve a promise on client side?
The server doesn't actually resolve a promise on the client side. The client defines a promise that will resolve when it gets a response back from the server. So, when that response comes back from the server, the client's own callback code that receives that response, then resolves its own promise.
What I don't get still is the callback function. So the client sends data and a callback function A to the server, and the server can call this function A in it's own callback. Function A will NOT be executed on client side, is this understanding correct? If so, then why is there resolve and reject in callback function A?
No. That's not what's happening. The client specifies a callback function to socket.io when it calls the .emit(). That tells the client socket.io library that the client would LIKE to get an acknowledgement for this message. That causes the socket.io library to set an extra bit in the message packet it sends to the server to TELL the server that the client wants an acknowledgement. No callback is sent to the server, only a request for acknowledgement bit. Then, when the server receives the message, it can call a server-side socket.io callback to say "yes, I acknowledge that I got this message" and it can also send a response. When the server calls that socket.io callback, socket.io will send an acknowledgement packet (with a matching message ID embedded in it) to the client. The socket.io client library will see the incoming acknowledgement packet, find the message ID in that packet and any server response data, find the client-side acknowledgement callback that goes with message ID and call it. The client will see its function getting called which tells it that the server did receive its message. Then, in your client code, that callback will resolve the promise.
Question: Would you consider dangling callbacks as bad node.js style or even dangerous? If so under which premise?
Case: as described below, imagine you need to make calls to a DB in an express server that updates some data. Yet the client doesn't need to be informed about the result. In this case you could return a response immediately, not waiting for the asynchronous call to complete. This would be described as dangling callback for lack of a better name.
Why is this interesting?: Because tutorials and documentation in most cases show the case of waiting, in worst cases teaching callback hell. Recall your first experiences with say express, mongodb and passport.
Example:
'use strict'
const assert = require('assert')
const express = require('express')
const app = express()
function longOperation (value, cb) {
// might fail and: return cb(err) ...here
setTimeout(() => {
// after some time invokes the callback
return cb(null, value)
}, 4000)
}
app.get('/ping', function (req, res) {
// do some declartions here
//
// do some request processesing here
// call a long op, such as a DB call here.
// however the client does not need to be
// informed about the result of the operation
longOperation(1, (err, val) => {
assert(!err)
assert(val === 1)
console.log('...fired callback here though')
return
})
console.log('sending response here...')
return res.send('Hello!')
})
let server = app.listen(3000, function () {
console.log('Starting test:')
})
Yeah, this is basically what called a "fire and forget" service in other contexts, and could also be the first step in a good design implementing command-query response separation.
I don't consider it a "dangling callback", the response in this case acknowledges that the request was received. Your best bet here would be to make sure your response includes some kind of hypermedia that lets clients get the status of their request later, and if it's an error they can fix have the content at the new resource URL tell them how.
Think of it in the case of a user registration workflow where the user has to be approved by an admin, or has to confirm their email before getting access.
I've been working on integrating Google Recaptcha into a Meteor and AngularJS web application. Everything was smooth sailing until I had to validate the recaptcha response -- for some bizarre reason, I can't get an async response from the backend to the frontend.
I've tried a lot of different variations and have read many, many posts on SO and the internet in general, but with no luck -- so I opted to post my own question.
Here's what I'm doing:
Client:
Meteor.call('recaptcha.methods.validateRecaptcha', { 'response' : this.recaptcha.getResponse(this.id) }, function(error, result) {
// error and result are both undefined
console.log('Do something with the ' + error + ' or ' + result + '.');
}
So, I'm calling a Meteor method and passing in a callback that is run when the method is done. However, the error and result parameters are both undefined.
Server:
run: function(data) {
if (this.isSimulation) {
/*
* Client-side simulations won't have access to any of the
* Meteor.settings.private variables, so we should just stop here.
*/
return;
}
return Meteor.wrapAsync(HTTP.post)(_someUrl, _someOptions);
}
That last line is a shortened version of the sync/async structure that I've found in several Meteor guides (I also tried this version), namely:
var syncFunc = Meteor.wrapAsync(HTTP.post);
var result = syncFunc(Meteor.settings.private.grecaptcha.verifyUrl, _options);
return result;
I've also tried a version using Futures:
var Future = Npm.require( 'fibers/future' );
var future = new Future();
var callback = future.resolver();
HTTP.post(Meteor.settings.private.grecaptcha.verifyUrl, _options, callback);
return future.wait();
Now, the intention here is that I use Meteor.call() to call this method from the client, the client-side stub runs (to prevent simulation errors since we use private Meteor.settings variables in the real non-SO server-side code) and returns immediately (which happens), and the server hits Google's Recaptcha API (which happens and the server receives a response) before returning the result to the client (which doesn't happen -- the callback occurs but with no error/success data).
My thought is that one of two things are happening:
I'm just doing something wrong and I'm not properly sending the data back to the client.
The synchronous client stub (which returns immediately) is telling the client that the server response isn't important, so it never waits for the proper asynchronous response.
Could any of the Meteor gurus weigh in here and let me know what's going on and how to get async requests to play nicely in a Meteor application?
Thanks!
From the documentation for HTTP.call, which is the generic version of HTTP.post, it says
Optional callback. If passed, the method runs asynchronously, instead of synchronously, and calls asyncCallback. On the client, this callback is required.
So, on server, you can run it asynchronously like this
run: function(data) {
if (this.isSimulation) {
/*
* Client-side simulations won't have access to any of the
* Meteor.settings.private variables, so we should just stop here.
*/
return;
}
// No need to pass callback on server.
// Since this part is not executed on client, you can do this
// Or you can use Meteor.isClient to run it asynchronously when the call is from client.
return HTTP.post(Meteor.settings.private.grecaptcha.verifyUrl, _options);
}
I am studying Node.js. For this I am using useful nodeschool.io workshops. I am reading the learnyounode workshop now.
In the http client and http collect parts I have a problem. Despite I have read them and the api docs on node.js. Challenge in the first one was; "Write the String contents of each "data" event from the response to a new line on the console (stdout).". It says in the https collect part that "Collect all data from the server (not just the first "data" event)..." And refer the first one as it does not collects all data. Isn't a
var allData = "";
response.setEncoding('utf8');
response.on('data', function(data){
allData = concat(allData, data);
}
capable of collecting all data from the response.
As far as I have understood async nature so far, following first callback will be called when the async http.get() is completed. Or I am wrong the http.get() is not async.
var http = require('http');
var urlString = process.argv[2];
http.get(urlString, function callback (response) {
response.setEncoding('utf8');
response.on('data', console.log);
response.on('error', console.error);
})
What happens while this code is executing. Does callback waits to http.get() to finish and supply a response, so http.get() is async? If so when the events are fired that response.on(..) lines are listening for, after its creation.
Isn't that listening a continuous process that is performed while response is being created, in that case callback should not wait for the http.get() to complete execution so that event listener be available for events fired by response object?
An explanation can be that; what http.get() does to create a response object to server write on it and finish. So it finished its job by creating response object and from now on possibly server will start to write to this response stream, and callback has already taken on, on to listening the response stream for 'data' events fired when server writes any bit of data to it. That would be logically possible.
This is an example of a common feature of node.js called streaming.
Http.get DOES finish execution (by creating the stream) before the callback is called. It is the stream that is still being processed in the callback. The end result of http.get is the stream object that is still updating until the complete response is received.
The way to think about callbacks is not as a function that executes when the parent is done executing, but as a function that is an argument of another function. Theoretically, there is nothing to stop the parent from executing the callback at any time in its execution cycle. The convention in node.js just happens to be that callbacks get executed after the parent is done.
You are correct that function callback is passed as a parameter into function http.get and so it can have access to the response object created during the execution of http.get. However, what is also happening is that the response is a stream, meaning that it is continually updated until it is complete.
Here is the order of operation
http.get calls the external resource.
http.get creates the response object as a stream and updates it as the data comes in from the external resource
Upon each update of the response object, it emits a "data" event.
function callback contains a listener (response.on) that is activated whenever the "data" event is thrown.
Here's an example of a function creating a stream and passing it into a callback
function myAsyncFunction(callback){
var result = createStream(); // perform processing to create a streaming object
// at this point, the parent function is done so lets execute the callback
callback(result);
}
function processStream(example){
example.on('data', function(chunk){
console.log('chunk received' + chunk);
});
example.on('end', function(){
console.log('streaming is complete');
});
}
myAsyncFunction(processStream);