Develop Reference

Would giving response to client while letting asynchronous operation continue to run a good idea?

So I need to implement an "expensive" API endpoint. Basically, the user/client would need to be able to create a "group" of existing users.
So this "create group" API would need to check that each users fulfill the criteria, i.e. all users in the same group would need to be from the same region, same gender, within an age group etc. This operation can be quite expensive, especially since there are no limit on how many users in one group, so its possible that the client requests group of 1000 users for example.
My idea is that the endpoint will just create entry in database and mark the "group" as pending, while the checking process is still happening, then after its completed, it will update the group status to "completed" or "error" with error message, then the client would need to periodically fetch the status if its still pending.
My implementation idea is something along this line
const createGroup = async (req, res) => {
const { ownerUserId, userIds } = req.body;
// This will create database entry of group with "pending" status and return the primary key
const groupId = await insertGroup(ownerUserId, 'pending');
// This is an expensive function which will do checking over the network, and would take 0.5s per user id for example
// I would like this to keep running after this API endpoint send the response to client
checkUser(userIds)
.then((isUserIdsValid) => {
if (isUserIdsValid) {
updateGroup(groupId, 'success');
} else {
updateGroup(groupId, 'error');
}
})
.catch((err) => {
console.error(err);
updateGroup(groupId, 'error');
});
// The client will receive a groupId to check periodically whether its ready via separate API
res.status(200).json({ groupId });
};
My question is, is it a good idea to do this? Do I missing something important that I should consider?

Yes, this is the standard approach to long-running operations. Instead of offering a createGroup API that creates and returns a group, think of it as having an addGroupCreationJob API that creates and returns a job.
Instead of polling (periodically fetching the status to check whether it's still pending), you can use a notification API (events via websocket, SSE, webhooks etc) and even subscribe to the progress of processing. But sure, a check-status API (via GET request on the job identifier) is the lowest common denominator that all kinds of clients will be able to use.
Did I not consider something important?
Failure handling is getting much more complicated. Since you no longer create the group in a single transaction, you might find your application left in some intermediate state, e.g. when the service crashed (due to unrelated things) during the checkUser() call. You'll need something to ensure that there are no pending groups in your database for which no actual creation process is running. You'll need to give users the ability to retry a job - will insertGroup work if there already is a group with the same identifier in the error state? If you separate the group and the jobs into independent entities, do you need to ensure that no two pending jobs are trying to create the same group? Last but not least you might want to allow users to cancel a currently running job.

How to throw a custom message using Dialogflow after three times of fallback

I am developing a chatbot using Dialogflow, I would like to throw a message to user when the chatbot doesn't understand the user input for three times in a row and for the forth time respond with a custom message (not the one of the options declared on the dialogflow interface)
One idea that I have is to make a counter within the input unknown action like this:
var counter = 1;
// The default fallback intent has been matched, try to recover (https://dialogflow.com/docs/intents#fallback_intents)
'input.unknown': () => {
// Use the Actions on Google lib to respond to Google requests; for other requests use JSON
if (requestSource === googleAssistantRequest) {
sendGoogleResponse('I\'m having trouble, can you try that again?'); // Send simple response to user
} else {
if (counter == 3) {
counter = 1;
sendResponse('Custom message');
} else {
counter++;
sendResponse('I\'m having trouble, can you try that again?'); // Send simple response to user
}
}
},
This would work, but idk if this will work for multiple user at the same time, I was thinking to create a storage for storing requests attached by a unique id and have a different counter for each request!
Do you have any better idea of achieving such thing in Dialogflow?

This will not work the way you've designed it. Not quite for the reason you think, but close.
You don't show the rest of your code (that's ok), but the counter variable is probably in a function that gets called each time it processes a message. When that function is finished, the counter variable goes out of scope - it is lost. Having multiple calls at the same time won't really be an issue since each call gets a different scope (I'm glossing over some technical details, but this should be good enough).
One solution is that you could store the variable in a global context - but then you do have the issue of multiple users ending up with the same counter. That is very very bad.
Your solution about keeping a counter in a database, keyed against the user, does make sense. But for this need, it is overkill. It is useful for saving data between conversations, but there are better ways to save information during the same conversation.
The easiest solution would be to use a Dialogflow Context. Contexts let you save state in between calls to your webhook fulfillment during the same conversation and for a specific number of messages received from the user (the lifespan).
In this case, it would be best if you created a context named something like unknown_counter with a lifespan of 1. In the parameters, you might set val to 1.
The lifespan of 1 would mean that you'll only see this context the next time your webhook is called. If they handle it through some other Intent (ie - you understood them), then the context would just vanish after your fulfillment runs.
But if your input.unknown handler is called again, then you would see the context was there and what the value is. If it doesn't meet the threshold, send the context again (with a lifespan of 1 again), but with the value being incremented by 1. If it did meet the threshold - you'd reply with some other answer and close the connection.
By "send the context", I mean that the context would be included as part of the reply. So instead of sending just a string to sendGoogleResponse() or sendResponse() you would send an object that included a speech property and an outputContexts property. Something like this:
var outputContexts = [
{
name: 'unknown_counter',
lifespan: 1,
parameters: {
'val': counterValue,
}
}
];
sendResponse({
speech: "I'm confused. What did you say?",
outputContexts: outputContexts
});

JavaScript - Promise fulfilled too early?

I created a small sample application using VueJs and created a C# REST API to store and retrieve data in a SQL Server back end.
For testing, I created a simple web page with a form to create a "note". The note is stored by the following function, 'saveData()':
saveData()
{
let promiseStack = [];
var jsondata = JSON.stringify(this.note);
promiseStack.push(this.$http.post('REST_API/note', jsondata));
Promise.all(promiseStack).then(data =>
{
this.$http.get('REST_API/note');
this.$router.push({ name: 'viewnotes', params: { id: data[0].body.id }})
}, error =>
{
console.log(error);
});
}
I tried to use a promise to wait until the 'store' operation in the backend is complete, and issue a GET request to retrieve all notes once the promise is fulfilled.
However, the get request inside the promise doesn't return any data. If I issue the get request manually later one, I retrieve the data that was stored previously.
So I had look into the C# REST API. There are currently two functions: createNote(...), getAllNotes(...). I used a StreamWriter to log to the filesystem when these functions are called, using milisecond precision. What I see is that 'createNote' is called after 'getAllNotes'. So I suspect that the API is working correctly, but something with the way I'm using promises seems to be awfully wrong.
Maybe somebody has a hint?
UPDATE
I know that the GET request doesn't return any data by using the developer toolbar in Chromium. The response is empty
The developer toolbar in the network tab shows that the requests are submitted in the correct order, so the "POST" request is issued first

It seems I found the problem. I had a 'href' tag in my 'Save' link, which triggered an early routing. The intended 'POST' and 'GET' were fired correctly, but there was another 'GET' inbetween somewhere because of the 'href' tag in the link, even though it was empty.
I removed the tag, now it works as intended.

MEAN / AngularJS app check if object already posted

I have thig angularJS frontend and I use express, node and mongo on the backend.
My situation looks like:
//my data to push on server
$scope.things = [{title:"title", other proprieties}, {title:"title", other proprieties}, {title:"title", other proprieties}]
$scope.update = function() {
$scope.things.forEach(function(t) {
Thing.create({
title: t.title,
//other values here
}, function() {
console.log('Thing added');
})
})
};
//where Thing.create its just an $http.post factory
The HTML part looks like:
//html part
<button ng-click="update()">Update Thing</button>
Then on the same page the user has the ability to change the $scope.things and my problem is that when I call update() again all the things are posted twice because literally thats what I'm doing.
Can someone explain me how to check if the 'thing' its already posted to the server just to update the values ($http.put) and if its not posted on server to $http.post.
Or maybe its other way to do this?

I see a few decisions to be made:
1) Should you send the request after the user clicks the "Update" button (like you're currently doing)? Or should you send the request when the user changes the Thing (using ngChange)?
2) If going with the button approach for (1), should you send a request for each Thing (like you're currently doing), or should you first check to see if the Thing has been updated/newly created on the front end.
3) How can you deal with the fact that some Thing's are newly created and others are simply updated? Multiple routes? If so, then how do you know which route to send the request to? Same route? How?
1
To me, the upside of using the "Update" button seems to be that it's clear to the user how it works. By clicking "Update" (and maybe seeing a flash message afterwards), the user knows (and gets visual feedback) that the Thing's have been updated.
The cost to using the "Update" button is that there might be unnecessary requests being made. Network communication is slow, so if you have a lot of Thing's, having a request being made for each Thing could be notably slow.
Ultimately, this seems to be a UX vs. speed decision to me. It depends on the situation and goals, but personally I'd lean towards the "Update" button.
2
The trade-off here seems to be between code simplicity and performance. The simpler solution would just be to make a request for each Thing regardless of whether it has been updated/newly created (for the Thing's that previously existed and haven't changed, no harm will be done - they simply won't get changed).
The more complex but more performant approach would be to keep track of whether or not a Thing has been updated/newly created. You could add a flag called dirty to Thing's to keep track of this.
When a user clicks to create a new Thing, the new Thing would be given a flag of dirty: true.
When you query to get all things from the database, they all should have dirty: false (whether or not you want to store the dirty property on the database or simply append it on the server/front end is up to you).
When a user changes an existing Thing, the dirty property would be set to true.
Then, using the dirty property you could only make requests for the Thing's that are dirty:
$scope.things.forEach(function(thing) {
if (thing.dirty) {
// make request
}
});
The right solution depends on the specifics of your situation, but I tend to err on the side of code simplicity over performance.
3
If you're using Mongoose, the default behavior is to add an _id field to created documents (it's also the default behavior as MongoDB itself as well). So if you haven't overridden this default behavior, and if you aren't explicitly preventing this _id field from being sent back to the client, it should exist for Thing's that have been previously created, thus allow you to distinguish them from newly created Thing's (because newly created Thing's won't have the _id field).
With this, you can conditionally call create or update like so:
$scope.things.forEach(function(thing) {
if (thing._id) {
Thing.update(thing._id, thing);
}
else {
Thing.create(thing);
}
});
Alternatively, you could use a single route that performs "create or update" for you. You can do this by setting { upsert: true } in your update call.
In general, upsert will check to see if a document matches the query criteria... if there's a match, it updates it, if not, it creates it. In your situation, you could probably use upsert in the context of Mongoose's findByIdAndUpdate like so:
Thing.findByIdAndUpdate(id, newThing, { upsert: true }, function(err, doc) {
...
});
See this SO post.

#Adam Zemer neatly addressed concerns I raised in a comment, however I disagree on some points.
Firstly, to answer the question of having an update button or not, you have to ask yourself. Is there any reason why the user would like to discard his changes and not save the work he did. If the answer is no, then it is clear to me that the update should not be place and here is why.
To avoid your user from loosing his work you would need to add confirmations if he attempts to change the page, or close his browser, etc. On the other if everything is continuously saved he has the peace of mind that his work is always saved and you dont have to implement anything to prevent him from loosing his work.
You reduce his workload, one less click for a task may seem insignificant but he might click it many time be sure to have his work save. Also, if its a recurrent tasks it will definitely improve his experience.
Performance wise and code readability wise, you do small requests and do not have to implement any complicated logic to do so. Simple ng-change on inputs.
To make it clear to him that his work is continuously save you can simply say somewhere all your changes are saved and change this to saving changes... when you make a request. For exemple uses, look at office online or google docs.
Then all you would have to do is use the upsert parameter on your mongoDB query to be able to create and update your things with a single request. Here is how your controller would look.
$scope.update = function(changedThing) { // Using the ng-change you send the thing itself in parammeter
var $scope.saving = true; // To display the saving... message
Thing.update({ // This service call your method that update with upsert
title: changedThing.title,
//other values here
}).then( // If you made an http request, I suppose it returns a promise.
function success() {
$scope.saving = false;
console.log('Thing added');
},
function error() {
//handle errors
})
};

Meteor.user() alternatives

I am writing an application which needs to display some user information.
Because Meteor.user() is not immediately available I wrapped every user information with an handlerbar helper
Handlebars.registerHelper('isLoggingIn', function() {
return Meteor.loggingIn();
})
This worked for me until I needed to create an admin page and custom content for every user/user role.
Waiting for Meteor.user() to be available or showing general information first while waiting for the roles to load are options I would like to avoid.
I then tried an alternative way and published the currentUser with a new Collection.
Meteor.publish('currentUser', function() {
var sub = this;
var handle = Meteor.users.find({_id: this.userId}).observe({
added: function (user) {
sub.added('currentUser', user._id, user);
}
});
sub.ready();
sub.onStop(function() { handle.stop(); });
});
and
CurrentUser = new Meteor.Collection('currentUser');
In this way I can access the logged in user with CurrentUser.findOne(), and it's available at the same time as the other collections.
What I fear is that this alternative is not as secure and problem free as the common Meteor.user(), and I was wondering if my method is correct and if there are better ways to obtain the same result (user detail information immediately available) without reinventing the wheel.

Just a note you can use {{loggingIn}}, {{#if loggingIn}}.. without writing your own helper.
The option to publish the user who is logged in with a custom publish function adds an unnecessary complexity.
When it comes to security you have to assume if its from the client side, in any scenario it is untrustworthy. This means you publish relevant data for the role, etc only when they are logged in to that user.
On the server the data is immediately available as soon as the user logs in, all you have to do is publish only the data for that users role. On the client it may take some time to adjust to this, which is why you can use placeholder until the subscriptions are complete.
What might be a better option would be to use either a helper that checks for when subscriptions are completed and displays a 'loading message'. Or use a router such as iron-router (github.com/EventedMind/iron-router) that can let you wait for a subcription to complete for a particular page.
This way you can use Meteor.user(), {{#currentUser}} and roles in way you intend.
One thing to keep in mind, is if you want to check if the user is logged in, not to use:
if(Meteor.user())
but instead
if(Meteor.user() && Meteor.user().profile && Meteor.user().profile.name)
(You will have to insert a name property in your profile, though). While logging in the user gets more and more data. I've noticed if you wait for the profile field, then the user is 'ready'. It seems initially the profile field is empty (still loggin in), but it would return true if you used if(Meteor.user())