Long story short, I have been developing a Discord Bot that requires a query to the database every time a message is sent in a server. It will then perform an action depending on the message etc. The query is asynchronous, therefore it will not block another message from being handled.
However in terms of scalability, I do not believe querying a database every time a message is sent is very speedy and could become a problem. Is there a better solution? I am unaware of a way to store data within a particular discord server, which would likely solve my issue.
My main idea is to have heap storage, where the most recently active servers (ie sent messages recently), their data is queried into the heap, and when they are inactive, it is removed from the heap. Is this a good solution? Or is it better to just keep querying every time?
You could create a cache and every time you fetch or insert something into your database you can write this into the cache.
Then, if you need some data you can check if it's in the cache and if not, get it from the database and store it in the cache right after.
This prevents unnecessary access to the database because the database is only accessed if your bot does not have the required data stored locally.
Note:
The cache will only be cleared when you restart the bot. But of course, you can also clear it after a certain amount of time or by other triggers.
If you need an example, you can take a look at my guildMemberAdd event and the corresponding config command
Related
Hi so first I apologize if my query may seem unclear, it’s first trying to do what I’m doing and I haven’t full idea around the intricacies and lingo lol.
So basically I’m running a NodeJs web server with React handling my front end. I’ve got Express to help manipulate user sessions and I just came by Server-Sent-Events as a way to send one-way messages(which is what I need to do). So far I’m able to send updates and messages via cURL on the terminal and running JS scripts, however these updates/messages go to every active client session but I want/need to be able to send these messages to specific active client sessions/connections.
Example: 5 client connections are established (session IDs A,B,C,D,E), now I want to send an alert message to session E only and manually.
I’m still green with NodeJs/Express and the concept of SSEs however I’m learning as I go for this pet project.
Send help
What you want is how SSE works. It is a dedicated connection between a client and a server process.
however these updates/messages go to every active client session
If that is what you see then your node script is running the exact same code for each client.
I think your question might be higher up - how to organize the data messaging? That is too big a topic for a single StackOverflow question, because it will depend on so many factors specific to your use case.
But one way would be to have an SQL database, with one record for each user. The node script polls that database table and if the record for the current user changes, it sends the new data to them. Then to send data to user E, you just edit the database record for user E.
After logging into an app (React.js), I am caching the member data in localStorage as a lot of my components are using it and request only needs to be done upon log-in, ideally.
However, a few properties in this member object may be changed in the backend manually so the frontend doesn't have a way to know whether the member object has changed at all. (Again, ideally, any change to the member object should go through some form submission that directly changes the DB, with which an update can be triggered for the localStorage, but this is not an option at this time.)
Example scenario: There's a generic form in the app to request for additional credits. Customer service will receive an email regarding the request. Credits would be manually updated for Customer A (in DB). If Customer A doesn't re-login (where the get request for member is done), localStorage will still show the old no. of credits.
If this is the situation, what's the best way to go about it?
Don't store member data in localStorage at all so as to keep the data fresh. Just call the endpoint whenever it's needed.
Use sessionStorage instead?
Trigger a refetch when the user refreshes the page / app (although user may not know that they need to do this to update the data).
Suggestions?
Calling the endpoint whenever its needed is ideal if the data is going to change based on things outside of the user's control.
Session Storage is just local storage that gets wiped when the browsing session ends, you'll still have the exact same issue
This doesn't really solve the problem, and it's typically a bad user experience to require the user to perform regular maintenance tasks in order to use your application to the best of its ability
I'd go with just getting the data fresh.
At a high level, you have two choices:
Poll (periodically call the back end to refresh the data)
Open a persistent connection (like a web socket) to the server, and have the server push updates to clients.
The latter option would require a lot of changes, and it changes the scalability of your app, so the former choice seems like the most reasonable option for you.
It's smart to keep using localStorage so you have an offline copy of the data and aren't blocking rendering during page load; you can have a background periodic refresh process that doesn't disrupt the user in the meantime. If your data is mirrored in something like redux or context, then your UI could seemlessly update if/when the data changes.
If you do not know when member has been updated, don't store it. You should query the back end every time you need member. That is the only way to keep the data sync with your database.
I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.
If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.
Okay, let me start by saying that I know this is weird. I do.
But here goes:
Let's say I have an SQL database which stores my data. And let's say I don't have a choice in this, it has to be SQL. The application I'm building has somewhere in the region of 100,000 records in its database, and once every single record has been processed by the users of the application, they all go off and get sent to a different application entirely. So for a short period of time, this application will be in use, and then stops being used until the same time next year. While the application is in use, no external sources will be touching the database at all.
When the (Node) server starts up, it loads everything from the database, into an object literal on the server.
The client-side of this application, on a very basic level, makes requests (to an API on the server) for data, and sends updated versions of records back to the server once they've been processed.
So here's where it gets weird: Let's say I don't want to have the client-side application have to directly retrieve records from the database, nor do I want it to be able to write to them. So the data from the entire database already exists in memory on the server. There's a module on the server that can handle changing the representation of that data already (again, because the client application only interacts with APIs on the server, the database module exists to facilitate this).
Multiple users access the system at once, but due to the way the system works, it is not possible for two users to be sent the same record, so two users will never be sending an update back for the same record (records are processed individually, and sequentially).
So, let's say that I decided that, since I was already managing all of this data in memory on the server, I would just send an updated version of the current data, in its entirety, back to the database, every time it changed.
The question is, where does this rank on the crazy scale?
Performance, writing an entire database rather than single records, would obviously suffer. But, in a database that is only read from once (on start-up of the application), is that even a concern? If every operation other than "Write all the stuff when any of the stuff changes" happened in memory on the server, does it matter how long those updates actually take? If a new update to the database comes in whilst it's being updated, surely SQL will take care of this?
It feels like the correct way to do this of course, is to have each user directly getting their info from the database, and directly making updates to the database too (or at least interacting with API endpoints to make this happen), but, is just...not doing that, utter lunacy?
Like I said, I know it's weird, but other than the fact that "it feels kind of wrong", I'm not sure I'm convinced that it is in fact entirely wrong. So I figured that this place would have an opinion.
The way that I think it currently works is:
[SQL DB] is updated whenever a change happens on {in-memory DB}
{in-memory DB} is updated in various ways based on API calls to the server
makes requests for data, and sends updates to data, both of which are processed on the in-memory DB
Multiple requests can happen at the same time from the application, but mutliple users can not see the same record, because records are allocated to a given user before they're sent
Multiple updates can come from multiple users, each of which ultimately ends in the entire SQL database being saved to with the contents of the in-memory DB.
(Note: I'm not saying "is this the best way to do this". I'm just asking, is there a significant argument for caring about the performance of a database being written to, if it's not going to be read from again unless the server needs to be restarted)
What I think that I would do, in this situation, is to add an attribute to each cached record to indicate that the record is "dirty." In other words, that something has been done to it, by someone, since it was originally read from the database.
(You could also add an attribute that indicates that someone "has this particular record 'checked-out,'" so that you can be sure that two users are no updating the same record at the same time.)
At some convenient moment, you can then walk through the collection, posting the "dirty" records back to the database. Use an SQL Transaction, not only for efficiency but also to be sure that the final update to the database is atomic.
You will need to be very mindful of the possibility of race-conditions. One possible strategy is to use a Unix timestamp as a "dirty" indicator. A record is selected for posting to the database only if its "dirty-time" is greater-than or equal-to the timestamp when the commit-process was last run.
(And, P.S.: "no, I've seen even 'weirder' things than this, in all my crazy years in this crazy business ...)
This question isn't a code-level issue but merely a functionality question / brainstorm.
Within my PHP script I want to send a notification to another user in real time, there's 1 way I've thought of to do this, if you know any better ones be sure to leave them in the comments!
My idea for this functionality is to insert into a databse table with the user's id and the message, then on the user's end constantly loop a select request looking for notifications corresponding to their id within $_SESSION, if it's found a message then delete it from the table and display it to the user.
This seems like it could "strain" my database and I'm wondering if there's a cleaner way to do this, it would also be much appreciated if somebody could post a javascript loop with a 3 second delay and an ajax post to a php file within it,
Thanks all,
James
The cleaner way to do this would be with websockets. Polling, long polling, and streaming are going to have exactly the problem you thought you were going to have.
The message recipient needs to be listening via broadcast also through websockets. The server will notify all websockets listening for that particular event.
You don't want to block with database read and writes. Just take the action from one user and send it to all the other websockets listening for that event (the other user's client side instances)
For event history you would consider persisting to a database with a message queue.
With a properly indexed, well-structured table, it won't be a strain to the db at all. Of course, this assumes your interval is reasonable (3 seconds you mentioned is great). That's how all real-time session-checking websites work. Those that need more than that, such as chat systems, basically anything that passes data more frequently and/or in larger packets, they use websockets.
Use Websockets or Server-Sent-Events
just an idea:
User-1 send message to the Server {"message" : "hello", "target" : "User-2"}
Server checks the message and redirect it to the target User
User-2 listening for events from Websocket or Server-Sent-Events