I'm currently using AJAX and PHP to send updates to a postgreSQL database.
Say I had 1000 users all sending one ajax post request per second to a php script. Say that php script opened a connection, executed two SQL update commands every time it was run, and then closed the connection.
That would be 1000 connections per second - I'm guessing that isn't going to work out very well for me.
If it's not, how should I deal with it? I've read that node.js is a good solution - If it is are there any good guides for dealing with updating a postgreSQL from a webpage using javascript?
I already have data (some json, some other) in the postgreSQL database and it needs to stay in there, so ideally I would be able to just change the way the handshake between javascript and the database works and leave the rest the same.
As a side question: How many connections per second should I expect to be able to handle if that's my only bottleneck? And if there are more than the max 150 connections does it just queue the connection or does it do something obnoxious like post a message saying 'max connections hit' and not allow page loads?
Connection pooling or a "connection proxy". Try a search combining postgres and connection-pooling
There is a current survey on the Postgres project page:
What Connection Proxy do you use for PostgreSQL?
haproxy and pgbouncer are the most popular ATM.
I would start with pgBouncer at the Postgres Wiki.
Of course, the same connection being kept open and reused only works for the same session user. You can switch context within the same connection with SET role, though:
How to check role of current PostgreSQL user from Qt application?
Related
Hi so first I apologize if my query may seem unclear, it’s first trying to do what I’m doing and I haven’t full idea around the intricacies and lingo lol.
So basically I’m running a NodeJs web server with React handling my front end. I’ve got Express to help manipulate user sessions and I just came by Server-Sent-Events as a way to send one-way messages(which is what I need to do). So far I’m able to send updates and messages via cURL on the terminal and running JS scripts, however these updates/messages go to every active client session but I want/need to be able to send these messages to specific active client sessions/connections.
Example: 5 client connections are established (session IDs A,B,C,D,E), now I want to send an alert message to session E only and manually.
I’m still green with NodeJs/Express and the concept of SSEs however I’m learning as I go for this pet project.
Send help
What you want is how SSE works. It is a dedicated connection between a client and a server process.
however these updates/messages go to every active client session
If that is what you see then your node script is running the exact same code for each client.
I think your question might be higher up - how to organize the data messaging? That is too big a topic for a single StackOverflow question, because it will depend on so many factors specific to your use case.
But one way would be to have an SQL database, with one record for each user. The node script polls that database table and if the record for the current user changes, it sends the new data to them. Then to send data to user E, you just edit the database record for user E.
I would like to allow a user via browser to receive updates if a database record has been changed in (near) real time without reloading the web page.
The idea is:
User opens url /dbrecord.php?id=12345
Registers for SSE (or something else) to receive updates to the record
Received on screen notification of what has been changed in the record without reloading the page
However this needs to be performant as there are a lot of DB Change events and a lot of potential users. Can you please let me know if the approach is good or are there better technologies / approaches to build something like this?
UPDATE: Found this - Server Side Events with PHP & MySQL But it requires a while(true) loop in the PHP script and this cannot be good to have the SQL Query in there?! Any feedback?
Thanks,
fj
This question isn't a code-level issue but merely a functionality question / brainstorm.
Within my PHP script I want to send a notification to another user in real time, there's 1 way I've thought of to do this, if you know any better ones be sure to leave them in the comments!
My idea for this functionality is to insert into a databse table with the user's id and the message, then on the user's end constantly loop a select request looking for notifications corresponding to their id within $_SESSION, if it's found a message then delete it from the table and display it to the user.
This seems like it could "strain" my database and I'm wondering if there's a cleaner way to do this, it would also be much appreciated if somebody could post a javascript loop with a 3 second delay and an ajax post to a php file within it,
Thanks all,
James
The cleaner way to do this would be with websockets. Polling, long polling, and streaming are going to have exactly the problem you thought you were going to have.
The message recipient needs to be listening via broadcast also through websockets. The server will notify all websockets listening for that particular event.
You don't want to block with database read and writes. Just take the action from one user and send it to all the other websockets listening for that event (the other user's client side instances)
For event history you would consider persisting to a database with a message queue.
With a properly indexed, well-structured table, it won't be a strain to the db at all. Of course, this assumes your interval is reasonable (3 seconds you mentioned is great). That's how all real-time session-checking websites work. Those that need more than that, such as chat systems, basically anything that passes data more frequently and/or in larger packets, they use websockets.
Use Websockets or Server-Sent-Events
just an idea:
User-1 send message to the Server {"message" : "hello", "target" : "User-2"}
Server checks the message and redirect it to the target User
User-2 listening for events from Websocket or Server-Sent-Events
I have a local website running that will query a mysql database via inputs given by a html form. The form values are sent to php via jQuery to call the query. Is there a way to send a command to the mysql server to kill the query if the website is refresh?
Currently if I refresh, the last mysql call will still run, so the website will be locked up until that query is finished.
Edit: To give a little more background information on the website, it is a local website that is solely running d3 visualizations based on a series of involved queries (lots of full text searches) on a large database (3 tables all in the 1 - 5 million record range). The "lockout" I'm referring to is not being able to abandon a query and try a more efficient query (i.e. something that will limit my results more).
Perhaps there is (it generally involves selecting the last process ID from the MySQL Processlist and issuing KILL QUERY on that process ID), but there are some problems that I think you should try to address first.
To begin, why in the world does one query "lock up" a website? I might be smelling a design flaw.
JavaScript could be used to make the browser "hit" the server on a refresh, but that's just adding another AJAX call and, presumably, another MySQL query (having to do with the processlist) and more PHP to write to handle the AJAX call, the processlist lookup, and the KILL QUERY query ...
I would recommend you try and make the server/MySQL query more efficient, so that you don't have to get a flying flip whether the browser is refreshed or not. As for browser security, you could probably use either PHP or JavaScript to enforce some sort of "flood limit" on repeated refreshing ....
I'm interning for a company and have been given an assignment. I'm to write a Javascript script (it's for internal use only; security is not a concern) that accesses an RDS database on an AWS instance, grabs a list of email addresses, and uses the server's smtp to send emails to the whole list. The problem is that I know nothing about AWS and RDS. Here are the things I was provided:
--Server address, port, and credentials file of smtp server
--Address of AWS DB, and its username, password, database name, and table
--The company's server url, security key, and I was also given SSH and SCP commands.
Where do I start learning how to do this? I feel like it's within my grasp, but I just don't know the overall process of what I need to do to get this information. I've never used SQL or RDS before. Any direction whatsoever would be appreciated!
Try to break the problem down into chunks:
Write your Database Query as SQL
If the RDS database instance in question is publicly available, try to connect to it with your preferred (GUI?) SQL client, and inspect its tables. Figure out how to write the SQL query you need in order to grab the e-mail addresses you'll need in the next step, e.g. select email_address from users where user_type='Internal'. The w3schools SQL Introduction is a good place to start.
If the RDS database instance isn't publicly available (i.e. if you can only connect to the database from within AWS - from the server on which you plan to run your code), then figure out how to log in to that server using SSH or RDP, and then use a database client on your AWS server to write and test your SQL query.
Check first that you're not logging into a production machine that you could break by mistake.
Use a Server-Side program to run your SQL query
You must then connect to the RDS database instance using some code that runs server-side. If you're using a server-side JavaScript environment such as NodeJS, this is fine, but whilst it's potentially feasible to use JavaScript directly from a browser to connect to a database, it's not good practice, even for your assignment, so make sure you're using some server-side code.
Find out what your company's usual platform is (Java? Python? PHP? NodeJS?) and go with that. Search the internet for example code - you'll easily find some, because querying a database is an extremely common programming task. Since you've already got your SQL query, you'll be able to write some simple code that connects to the database and grabs your desired email addresses. For now, just have it print them out to a log file or to the console.
Send a test e-mail
Separately, in your server-side program, figure out how to connect to your company's SMTP server and send an e-mail to yourself, as a start. Again, search the internet for examples in your chosen programming language.
Once you've done that, define two or three e-mail addresses in an array or list, and figure out how to send the same message to these users simultaneously.
Connect your Database Query Code and SMTP Code
You're able to query the database, and you're able to send an e-mail to multiple users. Now add to your code to join those two functions together. Don't spam everyone in your firm and get yourself fired!
You mentioned JavaScript
Make sure you are clear about the requirements of your assignment. Should the program be started from a browser? Does a user need to click a button on a webpage saying 'Send Email'? If so, figure out how to submit a request to your server to have the server-side code execute. Search the internet for examples of POSTing HTML forms from JavaScript.
Good luck!