I think this should be fairly simple, but I think I am looking into things too much and it's not making sense.
What I am currently doing
I am creating a web app using Node + React to record audio in the browser. I'm using RecordRTC on the client side to record the audio from the user's microphone. All is fine and dandy, but sometimes it takes a long time to upload the audio file after the user is finished singing. I want to process this file before sending it back to the user in the next step, so speed is critical here as they are waiting for this to occur.
In order to make the experience smoother for my users, I want to kick off the audio upload process as soon as I begin to receive the audio blobs from RecordRTC. I can get access to these blobs as RecordRTC allows me to pass a timeslice value (in ms) and an 'ondatavailable' function, that will get passed a blob every timeslice amount of milliseconds.
What I have tried
Currently I have it all easily working with FormData() as I only send the file once the user has finished singing.
My first idea was to find an example of something like the Fetch API being used in a manner that resembles what I'm after. There are plenty of examples, but all of them treat the source file as already being available, but as I want to continually add blobs as they come (without being able to pre-determine when these might stop coming, as a user may decide to stop the singing process early) this doesn't look promising.
I then considered a 'write my own' process whereby many request are made instead of attempting one long continuous style one. This would involve attaching a unique identifier to each request, and having the server concatenate each chunk together where the ids match. However, I'm not sure how flexible this would be in the future in say a multi-server environment, not to mention handling dropped connections etc, and no real way to tell the server to scrap everything in the case of a user aborted event such as closing the tab/webpage etc.
Finally, I looked into what was available through the likes of NPM etc without success, before conceding that perhaps my Google Fu was letting me down.
What I want
Ideally, I want to create a SINGLE new request once the recording begins, then take the blob every time I receive it in 'ondataavailable', send it to my request (which will pump it through to my server once it receives something) indefinitely. Once the audio stops (I get this event from RecordRTC as well so can control this), I want to finish/close up my request so that the server knows it can now begin to process the file. As part of the uploading process, I also need to pass in a field or two of text data in the body, so this will need to be handled as well. On the server side, each chunk should be immediately accessible once the server receives it, so that I can begin to create the audio file/append to the audio file on the server side and have it ready for processing almost immediately after the user has finished their singing.
Note: The server is currently set to look for and process multi-part uploads via the multer library on npm, but I am more than happy to change this in order to get the functionality I want.
Thanks!
Providing an update for anyone that may stumble upon this question in their own search.
We ended up 'rolling our own' custom uploader which, on the client side, sends the audio blobs in chunks of up to 5 1-second blobs to the server. Each request contains a 'request number' which is simply +1 of the previous request number, starting at 1. The reason for sending 5 1-second blobs is RecordRTC, at least at the time, would not capture the final X number of seconds. EG. If using 5 second blobs instead, a 38 second song would lose the final 3 seconds. Upon reaching the end of the recording, it sends a final request (marked with an additional header to let the server know it's the final request). The uploader works in a linked list style to ensure that each previous request has been processed before sending the next one.
On the server side, the 5 blobs are appended into a single 5 second audio blob via FFMPEG. This does introduce an external dependency but we were already using FFMPEG for much of our application so it was an easy decision. The produced file has the request number appended to its filename. Upon receiving the final request, we use FFMPEG again to do a final concatenation of all the received files to get our final file.
On very slow connections, we're seeing time savings upwards of 60+ seconds, so it has significantly improved the app's usability for slower internet connections.
If anyone wants the code to use for themselves please PM through here. (It's fairly unpolished but I will clean it up before sending)
Related
I have website with backend in Python (Django) and JavaScript hosted on heroku. Also, I have code in python that does image classification with EfficientNet, so I want to integrate this code into my website.
The logical sequence of ideas is as follows:
The user upload an image on the site;
This image will be classified with the Python code;
The algorithm will return an image;
The returned image should be posted on the site.
Does anyone know what would be the best way to do this?
First of all, yes, if it is possible to implement what you are mentioning, I would implement the following:
Use celery to implement asynchronous tasks where when the photo is uploaded, Django tells celery that it has to do the asynchronous task (in this case, use the CNN) and can leave a pending status for the photo and once the task is complete, it changes the status and would appear published on the platform.
I recommend using asynchronous tasks for this because of the following:
The use of the convolutional neural network can take a certain time, let us remember that the default maximum response time of an HTTP request is 30 seconds and it could cut the request, the user would see it as an error and he can also complain because uploading a photo must wait a while and for user purposes they would think that the site is slow. The implementation of asynchronous tasks allows first in the HTTP request to indicate to the user that the image is being analyzed and secondly you do not have a limit of 30 seconds to analyze, in case of having many image uploads at the same time it can crash the server. That is why with celery you can even implement queues to solve this (Using redis or rabbitMQ).
If you want to implement knowing the status of the image in real time, you could add the use of a websocket, where when uploading the image in the response you get a URL that is the one of the websocket where you would receive information about the image once processed. You can use django-channels for it
I have written an application in node.js which takes input from user and generates pdfs file based on few templates.
I am using pdfkit npm for this purpose. My application is running in production. But my application is very slow, below are the reasons :
What problem I am facing :
It is working in sync manner. I can explain it by giving an example- Suppose a request come to the application to generate a pdf, is starts processing and after processing it returns back the response with generated pdf url. But if multiple request comes to the server it process each request one by one(in sync manner).
All request in queue have to wait untill the previous one is finished.
Maximum time my application gives Timeout or Internal Server Error.
I can not change the library, why ?
There are 40 templates I have written in js for pdfkit. And each template is of 1000 - 3000 lines.
If I will change the lib, i have to rewrite those templates according to new library.
It will take many months to rewrite and test it properly.
What solution I am using now :
I am managing a queue now, once a request come it got queued and a satisfactory message send back in response to the user.
Why this solution is not feasible ?
User should be provided valid pdf url upon success of request. But in queue approach, user is getting only a confirmation message. And pdf is being processed later in queue.
What kind of solution I am seeking now ?
Any way through which I can make this application multi-threaded/asynchronous, So that it will be capable of handling multiple request on a time without blocking the resource?
Please save my life.
I hate to break it to you, but doing computation in the order tasks come in is a pretty fundamental part of node. It sounds like loading these templates is a CPU-bound task, and since Node is single-threaded, it knocks these off the queue in the order they come in.
On the other hand, any framework would have a similar problem. Node being single-threading means its actually very efficient, because it doesn't lose cycles to context switching.
How many PDF-generations can your program handle at once? What type of hardware are you running this on? If it's failing on a few requests a second, then there's probably a programming fix.
For node, the more things you can make asynchronous the better. For example, any time you're reading a file in, it should be asynchronous.
Can you post the code for one of your PDF-creating request functions?
I want to build a little script in nodejs which goes through a list of URLs and checks if there is an audio livestream actually running.
The URLs can either be direct mp3 livestreams or HLS livestream URLs which link to .m3u files.
How can I do this check in JavaScript? Simply checking if there is no 404 is not enough.
The tricky thing here is that even though the source may be down, the server may still return its buffered data. In fact, this is very common. Servers will buffer 20-30 seconds of data or so, and will send that to you on connect. With HLS, the problem is even worse as usually a large number of HLS segments will be on disk.
You will need to connect and read from the stream, and then have a sort of timer where you determine if you're getting any more data. You can put this behind your HLS or HTTP progress clients, and handle both the same. Connect, and begin streaming data. Every time you receive a chunk of data, note the time. Periodically check the time between now and when you last received data. If it's been 20 seconds or so, it's likely that the stream isn't up and running.
I'm using HTML, JQuery and PHP/MySQL. I understand for the most part that if I want to make this game safe then the server needs to do practically everything, but in some situations the game must tell the server to do things. In my case this is a RPG type setup, it will need to at times send a POST request to a PHP script via an Ajax call that updates, inserts or deletes from the database. Such as a player wins a battle and he's exp needs to be appended to, or a player takes a turn in a battle and it needs to work out the amount of HP taken off the other enemy and return it as well as updating the enemy's HP.
Lets say when the player clicks "Attack" and it runs a JavaScript function called playerMove('attack'), what stops the user going into their browser developer tools and running this function manually? Or using similar code on an alternative server and running cross site Ajax calls to the same public scripts on my server?
Is there any way around this problem? Even if I had a game that was made as a client side application (Like C# or whatever) wouldn't these problems still exist, but just harder for users to execute. Or would connecting to MySQL directly through C# be mostly safe if done correctly. But what about C# sending POST requests to PHP scripts, wouldn't that bring you back to the problem that as the scripts are public they could be POSTed to from other sources?
Basically, every time a player "attack" request comes in, the server needs to do at least the following:
Do basic validation on the data in the request.
Check that the subject player attack request is actually coming from the correct account.
Check that the player is currently in battle, it's his turn, the target of the attack is valid, the player doesn't have any status effects preventing attacking, etc. Any game logic that prescribes that the player can't attack right now.
Calculate the attack's effects.
Update the database.
Return the results to the client.
Return the results to all other clients, through long polling, WebSockets, etc.
Now, if the player tries to make the AJAX call when they shouldn't be able to, your server validation should prevent it. Remember, the client is all input/output and can't do any game logic without tons of code duplication between your JS and PHP.
Making a game in PHP, especially multiplayer, requires a ton of overhead and boilerplate. I'd honestly think about using another language/framework. For example, Meteor hooks up a lot of this stuff for you. Client, server, and database data are automatically synchronized. It also has latency compensation so the client can run the server code to get an expected result, and then update to the actual result whenever the server eventually responds. This would make your game feel like it's working in real time, while still giving the server the last say on whatever game logic happens.
I'd like some opinions on the practical implications of moving processing that would traditionally be done on the server to be handled instead by the client in a node.js web app.
Example case study:
The user uploads a CSV file containing a years worth of their bank statement entries. We want to parse the file, categorise each entry and calculate cumulative values for each category so that we can store the newly categorised statement in a db and display spending analysis to the user.
The entries are categorised by matching strings in the descriptions. There are many categories and many entries and it takes a fair amount of time to process.
In our node.js server, we can happily free up the event loop whilst waiting for network responses and so on, but if there is any data crunching or similar processing, the server will be blocked from responding to requests, and this seems unavoidable.
Traditionally, the CSV file would be passed to the server, the server would process, save in db, and send back the output of the processing.
It seems to make sense in our single threaded node.js server that this processing is handled by the browser, and the output displayed and sent to server to be stored. Of course the client will have to wait while this is done, but their processing will not be preventing the server from responding to requests from other clients.
I'm interested to see if anyone has had experience build apps using this model.
So, the question is.. are there any issues in getting browsers rather than the server to handle, wherever possible, any processing that will block the event loop? Is this a good/sensible/viable approach to node.js application development?
I don't think trusting client processed data is a good idea.
Instead you should look into creating a work queue that a separate process listens on, separating the CPU intensive tasks from your node.js process handling HTTP requests.
My proposed data flow would be:
HTTP upload request
App server (save raw file somewhere the worker process can access)
Notification to 'csv' work queue
Worker processes uploaded csv file.
Although perfectly possible, simply shifting the processing to the client machine does not solve the basic problem.
Now the client's event loop is blocked, preventing the user from interacting with the browser. Browsers tend to detect this problem and stop execution of the page's script altogether. Something your users will certainly hate.
There is no way around either delegating or splitting up the work-load.
Using a second process (for example a 2nd node instance) for doing the number crunching server-side has the added benefit of allowing the operating system to use a 2nd CPU core. Ideally you run as many Node instances as you have CPU cores in the server and balance your work-load between them. Have a look at the diode module for some inspiration on how to implement multi-process communication in node.