Basically I'm working on an application that posts huge amounts of data to web API(third party API).
I am working on nodejs to connect to MsSQL server and fetch data, process it and post it in desired format to the web API.
Scenario is: In nodejs, I have script that does the above job. It is currently initiated (or should I say triggered?) by a button on web page using axios POST with all necessary parameters. Eg below:
axios.post('/api/v1/fetch-new-labors', {
startDate: 'somedate',
endDate: 'someDate'
}).then(...handles further processing & posting to api)
The process takes around 2-3 mins to finish. Meanwhile, if the page is refreshed, it obviously needs to restart the whole process by clicking the button.
Question: I am sure there is a way to let the process run(on serverside I presume) that takes care of fetching, processing and posting even if the client side page is refreshed/reloaded and at the same time, keep the client side informed with progressbar or x out of y records posted kind of thing. I thought of web sockets but I wasn't sure if there's a prefered way to achieve this. I'm not looking for the whole code/process, I am looking for someone to guide me towards overall concept/idea.
tl;dr: Long-running jobs usually avoid using the traditional request-response cycle, opting instead for some variation of the Pub/Sub pattern described below:
Accept request, then start processing
You should respond to the user immediately with an HTTP 202: Accepted, signalling you accepted the request, then start the processing.
You can perform some initial checks before responding (Does the user have other jobs? Does the request pass basic validation checks?) but you should not start processing the actual long-running job before responding.
You can use a simple HTTP request to create jobs.
Push status updates from server to subscribed clients
On page load the client subscribes to updates from the server.
Using WebSockets, push server-to-client status notifications regarding the progress. Don't forget to also handle and display errors to the client.
At this point, you'll probably need a way to uniquely identify each client across refreshes. You can easily solve this by storing a UUID via LocalStorage when a user first visits your app/website. If your app requires logins, then you can use the logged-in user's ID instead.
Check if user has already running jobs before accepting a new one
When the user refreshes you can send an initial message via WebSockets again notifying the user if he has any running jobs and what their progress is.
Based on your OP, I think you might want to disable the "Create Job" button if there's a pending job.
You can use other mechanisms for bidirectional server-client communication (such as long-polling/Server-sent Events) if you want, although WebSockets should be the most straightforward and flexible solution.
I'd personally go for a batteries-included WebSocket library such as socket.io.
Related
I've seen how to make a post request from JavaScript to get data from the server, but how would I do this flipped. I want to trigger a function in the flask server that will then dynamically update the variable on the JavaScript side to display. Is there a way of doing this in a efficient manner that does not involve a periodic iteration. I'm using an api and I only want to the api to be called once to update.
There are three basic options for you:
Polling - With this method, you would periodically send a request to the server (maybe every 5 seconds for example) and ask for an update. The upside is that it is easy to implement. The downside is that many requests will be unnecessary. It sounds like this isn't a great option for you.
Long Polling - This method means you would open a request up with the server and leave the request open for a long period of time. When the server gets new information it will send a response and close the request - after which the client will immediately open up a new "long poll" request. This eliminates some of the unnecessary requests with regular polling, but it is a bit of a hack as HTTP was meant for a reasonably short request response cycle. Some PaaS providers only allow a 30 second window for this to occur for example.
Web Sockets - This is somewhat harder to setup, but ultimately is the best solution for real time server to client (and vice versa) communication. A socket connection is opened between the server and client and data is passed back and forth whenever either party would like to do so. Javascript has full web socket support now and Flask has some extensions that can help you get this working. There are even great third party managed solutions like Pusher.com that can give you a working concept very quickly.
I discovered SSE (Server Sent Events) pretty late, but I can't seem to figure out some use cases for it, so that it would be more efficient than using setInterval() and ajax.
I guess, if we'd have to update the data multiple times per second then having one single connection created would produce less overhead. But, except this case, when would one really choose SSE?
I was thinking of this scenario:
A new user comment from the website is added in the database
Server periodically queries DB for changes. If it finds new comment, send notification to client with SSE
Also, this SSE question came into my mind after having to do a simple "live" website change (when someone posts a comment, notify everybody who is on the site). Is there really another way of doing this without periodically querying the database?
Nowadays web technologies are used to implmement all sort of applications, including those which need to fetch constant updates from the server.
As an example, imagine to have a graph in your web page which displays real time data. Your page must refresh the graph any time there is new data to display.
Before Server Sent Events the only way to obtain new data from the server was to perform a new request every time.
Polling
As you pointed out in the question, one way to look for updates is to use setInterval() and an ajax request. With this technique, our client will perform a request once every X seconds, no matter if there is new data or not. This technique is known as polling.
Events
Server Sent Events on the contrary are asynchronous. The server itself will notify to the client when there is new data available.
In the scenario of your example, you would implement SSE such in a way that the server sends an event immediately after adding the new comment, and not by polling the DB.
Comparison
Now the question may be when is it advisable to use polling vs SSE. Aside from compatibility issues (not all browsers support SSE, although there are some polyfills which essentially emulate SSE via polling), you should focus on the frequency and regularity of the updates.
If you are uncertain about the frequency of the updates (how often new data should be available), SSE may be the solution because they avoid all the extra requests that polling would perform.
However, it is wrong to say in general that SSE produce less overhead than polling. That is because SSE requires an open TCP connection to work. This essentially means that some resources on the server (e.g. a worker and a network socket) are allocated to one client until the connection is over. With polling instead, after the request is answered the connection may be reset.
Therefore, I would not recommend to use SSE if the average number of connected clients is high, because this could create some overhead on the server.
In general, I advice to use SSE only if your application requires real time updates. As real life example, I developed a data acquisition software in the past and had to provide a web interface for it. In this case, a lot of graphs were updated every time a new data point was collected. That was a good fit for SSE because the number of connected clients was low (essentially, only one), the user interface should update in real-time, and the server was not flooded with requests as it would be with polling.
Many applications do not require real time updates, and thus it is perfectly acceptable to display the updates with some delay. In this case, polling with a long interval may be viable.
I need to refresh a part of my view without refreshing the whole page.
At my index.html page I have three panels, wich one shows the number of Tickets by it's status, I need to refresh this number every time a new ticket is created or updated. I used Java with Spring Boot and Thymelaf to build my application.
This is my view:
This is the way I'm doing it now:
model.addAttribute("resolvedTickets", atendimentoService.findAllTicketsByStatus(STATUS_RESOLVED).size());
I have tried to use web sockets but i can't figure out how to get this and refresh the panels.
In a standard web interaction, the client (i.e. your web browser) sends a request to your server. Your server receives the request, and sends back the information to show in your browser and then terminates the connection.
WebSockets are a way to create a persistent, two-way connection between the client and the server, but it requires cooperation from both. A lot of shared servers don't allow WebSockets, so you first have to make sure your server is capable of providing WebSockets. (I see from your screenshot that you're running on Heroku, which should have no problem running WebSockets.)
On the server side, you need to set up handling for incoming WebSocket requests. I don't know what language you've coded your server in, so I can't provide any guidance, but there are plenty of libraries that do the server-side part of WebSockets in most languages.
On the client side, you need to set up your WebSocket client. MDN has a great guide on WebSockets that explains what you'll need to do. Basically, all you'll have to do is listen for incoming messages and increment your counter.
var count = 0;
var exampleSocket = new WebSocket("ws://example.com/socket");
exampleSocket.onmessage = function(event) {
count++;
document.getElementById('myTicketCounter').innerHTML = count;
}
For some things, WebSockets are overkill. If you find that this is too much work for too little reward, you can also just set up an AJAX call to fire every few minutes that pings another page on your server and returns the number of tickets and updates accordingly. It won't be instantaneous, but if you don't need down-to-the-second resolution, it'll probably suffice. You can adjust the interval to be as long or as short as you want (to an extent; bombarding your server with constant requests will slow you down a bit).
I have a 3rd party service to implement, that provides restful API with long polling... they are serving live events, and displaying it as quick as possible is crucial.
Sadly that service does not provide push notifications, so we need to deal with what we have....
so one of the APIs has a long-polling functionality, so in theory the idea is, I query the API, establish an open channel for 30 seconds and wait for the changes... (and publish the changes on the FE)... and then timeout the connection, establish a new one and repeat the process.
Should be pretty straight forward....
But so far, I couldn't find anything in angular's docs about long polling... the only thing I found was polling in connection with rxJS... so setting an inteval on how often I query the API... (which in my case would be every 30s)... but nothing about on leaving the channel open and listen for the changes...
found this thred:
How to implement http long polling in Angular 2
but it is not talking about this problem.
I don't want to end up querying the API every second.
Any ideas?
I considered implementing signalR (which might not really help here), but it relies on jQuery... which I don't want to add to the bundle unless is 100% necessary!
I think you misunderstood the concept of long-pulling..
Long-pulling where the client polls the server requesting new information. The server holds the request open until new data is available. Once available, the server responds and sends the new information. When the client receives the new information, it immediately sends another request, and the operation is repeated. This effectively emulates a server push feature.
It you want to keep the connection alive you need to use webSoket.
WebSockets provide a persistent connection between a client and server that both parties can use to start sending data at any time. The client establishes a WebSocket connection through a process known as the WebSocket handshake. This process starts with the client sending a regular HTTP request to the server
But I didn't understood from your question why you can't send another fetch request when the long-pulling completes, something like:
myLongPullingFunc(){
this.http.get(longPullingURL)
.subscribe(res=>{
if(sholdFetchData){
useFetchedData(res);
this.myLongPullingFunc();
}else
doSomethingElse()
})
}
You can read more about long-pulling here and on webSokects here
Can I have a value automatically pushed onto a website (using AJAX = JQuery?), when something is done on the server side and python is ready to send it, rather than just in a response to a request by the website.
How do I make Jquery ready to accept this adhoc data?
Tutorial I'm learning from:
http://flask.pocoo.org/docs/patterns/jquery/#the-html
Walk's answer is correct, but these days there is another option. Web Sockets can push data from a server to a browser. For some languages (notably Node.js) there are sophisticated libraries for handling web sockets and these often deal with fallbacks for older browsers.
Python has a number of libraries depending on what you need. These two are some of the most popular:
https://github.com/abourget/gevent-socketio
https://github.com/stephenmcd/django-socketio
You can have jquery make the AJAX request and wait for a period of time while the server processes the response and returns it, you may risk the transaction timing out. This is optimal if the responses are quick to return.
Another method would be to have a timed script that polls the server for updates. The server will place the updates for a specific session in, for instance a database or in memory, and then return the results. This is optimal if it's a long running process.