I am new to nodejs/Express.js development.
I have built my backend service with Express.js / Typescript and I have multiple routes / api endpoints defined. One is like this:
app.post('/api/issues/new', createNewIssue);
where browser will send a post request when a user submits a new photo (also called an issue in my app).
The user can send an issue to another user, and the backend will first query the database to find the number of issues that matches the conditions of "source user" and "destination user", and then give the new issue an identifying ID in the form srcUser-dstUser-[number], where number is the auto-incremented count.
The createNewIssue function is like this:
export const createNewIssue = catchErrors(async (req, res) => {
const srcUser = req.header('src_username');
const dstUser = req.header('dst_username');
// query database for number of issues matching "srcUser" and "dstUser"
...
const lastIssues = await Issue.find( {where: {"srcUser": srcUser, "dstUser": dstUser}, order: { id: 'DESC'}});
const count = lastIssues.length;
// create a new issue Entity with the ID `srcUser-dstUser-[count+1]`
const newIssue = await createEntity(Issue, {
...
id: `srcUser-dstUser-${count+1}`,
...
});
res.respond({ newIssue: newIssue});
})
Say the backend receives multiple requests with the same srcUser and dstUser attributes at the same time, will there be collisions where multiple new issues are created with the same id?
I have read some documentation about nodejs being single-threaded, but I'm not sure what that means definitely for this specific scenario.
Besides business logic in this scenario, I have some confusions in general about Express JS / Node JS:
When there is only one cpu core, Express JS process multiple concurrent requests asynchronously: it starts processing one and does not wait for it to finish, instead continues to process the next one. Is this understanding accurate?
When there are multiple cpu cores, does Express JS / Node Js utilize them all in the same manner?
Node.js will not solve this problem for you automatically.
While it will only deal with one thing at a time, it is entirely possible that Request 2 will request the latest ID in the database while Request 1 has hit the await statement at the same point and gone to sleep. This would mean they get the same answer and would each try to create a new entry with the same ID.
You need to write your JavaScript to make sure that this doesn't happen.
The usual ways to handle this would be to either:
Let the database (and not your JavaScript) handle the ID generation (usually by using a sequence.
Use transactions so that the request for the latest ID and the insertion of the new row are treated as one operation by the database (so it won't start the same operation for Request 2 until the select and insert for Request 1 are both done).
Test to make sure createEntity is successful (and doesn't throw a 'duplicate id' error) and try again if it fails (with a limit in case it keeps failing in which case it should return an error message to the client).
The specifics depend on which database you use. I linked to the Postgresql documentation for the sake of example.
Related
So I need to implement an "expensive" API endpoint. Basically, the user/client would need to be able to create a "group" of existing users.
So this "create group" API would need to check that each users fulfill the criteria, i.e. all users in the same group would need to be from the same region, same gender, within an age group etc. This operation can be quite expensive, especially since there are no limit on how many users in one group, so its possible that the client requests group of 1000 users for example.
My idea is that the endpoint will just create entry in database and mark the "group" as pending, while the checking process is still happening, then after its completed, it will update the group status to "completed" or "error" with error message, then the client would need to periodically fetch the status if its still pending.
My implementation idea is something along this line
const createGroup = async (req, res) => {
const { ownerUserId, userIds } = req.body;
// This will create database entry of group with "pending" status and return the primary key
const groupId = await insertGroup(ownerUserId, 'pending');
// This is an expensive function which will do checking over the network, and would take 0.5s per user id for example
// I would like this to keep running after this API endpoint send the response to client
checkUser(userIds)
.then((isUserIdsValid) => {
if (isUserIdsValid) {
updateGroup(groupId, 'success');
} else {
updateGroup(groupId, 'error');
}
})
.catch((err) => {
console.error(err);
updateGroup(groupId, 'error');
});
// The client will receive a groupId to check periodically whether its ready via separate API
res.status(200).json({ groupId });
};
My question is, is it a good idea to do this? Do I missing something important that I should consider?
Yes, this is the standard approach to long-running operations. Instead of offering a createGroup API that creates and returns a group, think of it as having an addGroupCreationJob API that creates and returns a job.
Instead of polling (periodically fetching the status to check whether it's still pending), you can use a notification API (events via websocket, SSE, webhooks etc) and even subscribe to the progress of processing. But sure, a check-status API (via GET request on the job identifier) is the lowest common denominator that all kinds of clients will be able to use.
Did I not consider something important?
Failure handling is getting much more complicated. Since you no longer create the group in a single transaction, you might find your application left in some intermediate state, e.g. when the service crashed (due to unrelated things) during the checkUser() call. You'll need something to ensure that there are no pending groups in your database for which no actual creation process is running. You'll need to give users the ability to retry a job - will insertGroup work if there already is a group with the same identifier in the error state? If you separate the group and the jobs into independent entities, do you need to ensure that no two pending jobs are trying to create the same group? Last but not least you might want to allow users to cancel a currently running job.
I am having some issues trying to connect to a matrix server using the matrix-js-sdk in a react app.
I have provided a simple code example below, and made sure that credentials are valid (login works) and that the environment variable containing the URL for the matrix client is set. I have signed into element in a browser and created two rooms for testing purposes, and was expecting these two rooms would be returned from matrixClient.getRooms(). However, this simply returns an empty array. With some further testing it seems like the asynchronous functions provided for fetching room, member and group ID's only, works as expected.
According to https://matrix.org/docs/guides/usage-of-the-matrix-js-sd these should be valid steps for setting up the matrix-js-sdk, however the sync is never executed either.
const matrixClient = sdk.createClient(
process.env.REACT_APP_MATRIX_CLIENT_URL!
);
await matrixClient.long("m.login.password", credentials);
matrixClient.once('sync', () => {
debugger; // Never hit
}
for (const room of matrixClient.getRooms()) {
debugger; // Never hit
}
I did manage to use the roomId's returned from await matrixClient.roomInitialSync(roomId, limit, callback), however this lead me to another issue where I can't figure out how to decrypt messages, as the events containing the messages sent in the room seems to be of type 'm.room.encrypted' instead of 'm.room.message'.
Does anyone have any good examples of working implementations for the matrix-js-sdk, or any other good resources for properly understanding how to put this all together? I need to be able to load rooms, persons, messages etc. and display these respectively in a ReactJS application.
It turns out I simply forgot to run startClient on the matrix client, resulting in it not fetching any data.
I just wanted to know if there's a way to count how many times a message has been sent in my Discord server, so the bot can send a message. I'm new with coding, so I don't know many things. Thank you in advance!
Explanation
To store the amount of messages sent in a guild, you'll have to keep track of a count somehow. Each time a message is sent, you can increment it by 1. Then, upon a user's request, you can display that number.
One easy option would be to store this "message count" for each guild inside of a JSON file. However, this would greatly impact performance. Consider a database for much better speeds and reliability.
Example Setup
Before using this system, create a guilds.json file with a blank object ({}).
Declaring the necessary variables...
const fs = require('fs'); // fs is the built-in Node.js file system module.
const guilds = require('./guilds.json'); // This path may vary.
Adding the system to the message event listener...
client.on('message', message => {
// If the author is NOT a bot...
if (!message.author.bot) {
// If the guild isn't in the JSON file yet, set it up.
if (!guilds[message.guild.id]) guilds[message.guild.id] = { messageCount: 1 };
// Otherwise, add one to the guild's message count.
else guilds[message.guild.id].messageCount++;
// Write the data back to the JSON file, logging any errors to the console.
try {
fs.writeFileSync('./guilds.json', JSON.stringify(guilds)); // Again, path may vary.
} catch(err) {
console.error(err);
}
}
});
Using the system in a command...
// Grab the message count.
const messageCount = guilds[message.guild.id].messageCount;
// Send the message count in a message. The template literal (${}) adds an 's' if needed.
message.channel.send(`**${messageCount}** message${messageCount !== 1 ? 's' : ''} sent.`)
.catch(console.error);
JSON is highly prone to corruption if a queue system is not created that will make sure multiple reads and writes are not happening to a file all at the same time. For the purpose of what you want, I would use something like SQLite that requires minimal setup, is easy to learn, and has helper frameworks to make it easier to use such as Keyv and Sequelize.
Here is a good guide on how to use sqlite in the nodejs runtime environment.
I am writing an express app, where I'm pushing data from my views to a database. But most of the data is mapped to some other data in database tables.
For example, is a choose student name drop down- once you choose the student by his name , a drop down below - will show all roles that he is allowed for.
So I'm following this pattern of
app.post('\action1', function(req,res){
function querySomething(){
var defered = Q.defer();
connection.query(some_select_query,defered.makeNodeResolver());
return defered.promise;
}
function querySomethingElse(){
var defered = Q.defer();
connection.query(some_other_select_query,defered.makeNodeResolver());
return defered.promise;
}
Q.all([querySomething(), querySomethingElse()]).then((results,err) => {
connection.release()
if(results){
res.render('some_view.ejs', {
result1:results[0][0],
result2:results[1][0]
});
}
else{
res.render('error.ejs',{});
}
})
})
Now the problem is that I have to follow this pattern of selecting something from multiple tables, pass all these function to a promise- and when the results is passed back, goto my view with all those result objects - so that I can use them in my view - as a means of doing drop downs dependent on one another.
Sometimes I have to re-write this multiple times.
Doing a select query like this would be performance intensive especially if all views are using the result of the same query.
Is there any way I can build a cached data store on my express server side code and query that instead of the actual database??
If there is an insert or an update - i will refresh this store and just do a new select * that one time.
What libraries are there on top of express which will help me do this??
Does mysql-cache does the same thing?? I'm also using connection pooling with createPool.
How do I achieve this - or do I just restore to using big mvc's like sails to rewrite my app?
You can try apiCache npm module.
"Sometimes I have to re-write this multiple times."
Based on the business need, you may want to handle each use case separately and this scenario doesn't deal with caching.
Doing a select query like this would be performance intensive especially if all views are using the result of the same query.
This is a classic example for the need of server-side caching.
I have been trying to implement a RESTFul API with NodeJS and I use Mongoose (MongoDB) as the database backend.
The following example code registers multiple users with the same username when requests are sent at the same time, which is not what I desire. Although I tried to add a check!
I know this happens because of the asynchronous nature of NodeJS, but I could not find a method to do this properly. It looks like "findOne" method immediately returns, causing registerUser to return and then another request is processed.
By the way, I don't want to check for existing users with a separate API function, I need to check at the registration stage. Is there any way to do this?
Controller.prototype.registerUser = function (req, res) {
Users.findOne({'user_name': req.body.user_name}, function(err, user) {
if(!user) {
new User({user_name: req.body.user_name}).save(function(err) {
if(!err) {
res.send("User saved");
} else {
res.send("DB Error: Could not save user!");
}
});
} else {
res.send("User exists");
}
});
}
You should consider setting the user_name to be unique in the Schema. That would ensure that the user_name stays unique even if simultaneous requests are made to set an identical user name.
Yes, the reason this is happening is as you suspected because multiple requests can execute the code simultaneously and therefore the User.fineOne can return false multiple times. Incidentally this can happen with other stacks as well, even ones that use one thread per request.
To solve this, you need a way to somehow either control that just one user is being worked on at the time, you can accomplish this by adding all registerUser requests to a queue and then pulling them off the queue one by one and calling res.Send only after it's processed form the queue.
Alternatively, maybe you can keep a local array of user names, and each time a new request comes in and check the array if it's already there. If it isn't add it to the array and work on it. If it is in the array, send the response "User exists". Then, once the user has been successfully created, you can remove it from that array. (I haven't thought this one through 100% but I think it should work as well.)