I'm working on an app where I need to record audio using a microphone and send it to a backend app (tomcat server).
It seems that sending too big streams drives angular crazy and freezes my browser.
To record my audio file, I use the native function RecorderWorkerFactory.getUserMedia() which allow me to get a RecordBlob object.
After that, still in Angular, I extract the audio content in base64 enconding, and I send it to the backend app using $resource.
The backend app correctly receives the data and process it, but the callback of this call is never executed, as Firefox detects an infinite loop and freezes.
However, if I keep running the program, after a very long time the page refresh will pass.
This is the code where I extract the audio content into base64 String, to send it:
var blob = $scope.audio.recordBlob;
if (blob) {
var reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = function() {
$scope.audioContent = reader.result;
$scope.sendMessage();
}
}
$scope.sendMessage = function(){
var outputStream = {
"audio": $scope.audioContent
};
$scope.sendIM(outputStream);
}
Here I send outputStream via POST to the back, and in callback I launch loadData() function that reload my view.
services.FileCreation= $resource(URI_SERVICE_CREATION, {}, {
'post' : urlEncodedFormPost
});
$scope.sendIM = function(fluxSortie) {
$services.FileCreation.post(angular.toJson(outputStream)).$promise.then(function(result){
$scope.loadData();
});
}
And this is the Java code for the creation of the audio file:
private void createAudioFile(File file, byte[] content) throws IOException {
FileOutputStream stream = null;
try {
stream = new FileOutputStream(file.getPath());
IOUtils.write(content, stream);
} catch (IOException e) {
LOGGER.debug("creation failed");
} finally {
if (stream != null) {
stream.flush();
stream.close();
}
}
}
Where content is the conversion of the base64 string sent.
After research I found that the infinite loop is in a native Angular function named shallowClearAndCopy() that occured after the Java execution but just before the callback. In this function the code apparently transforms each character of the audio string (base64 encoded) into an object property and do a loop on these to delete them. But this lead to a very long treatment that Firefox consider as an infinite loop.
function shallowClearAndCopy(src, dst) {
dst = dst || {};
angular.forEach(dst, function(value, key) { // This is where it freezes, as dst contains all my base64 encoded data and iterate over each character of it (which is veeeeeery long !)
delete dst[key];
});
for (var key in src) {
if (src.hasOwnProperty(key) && !(key.charAt(0) === '$' && key.charAt(1) === '$')) {
dst[key] = src[key];
}
}
return dst;
}
Is it because of angularjs performance (and there is nothing else to be done) ?
Or am I missing something that creates an infinite loop ? Or is something wrong in my callback definition ?
Cheers !
I found, the problem!
It was the angular.toJson(outputStream) that transformed the object without need.
Related
I am sending Int16Array buffer to server while audio processing
var handleSuccess = function (stream) {
globalStream = stream;
input = context.createMediaStreamSource(stream);
input.connect(processor);
processor.onaudioprocess = function (e) {
var left = e.inputBuffer.getChannelData(0);
var left16 = convertFloat32ToInt16(left);
socket.emit('binaryData', left16);
};
};
navigator.mediaDevices.getUserMedia(constraints)
.then(handleSuccess);
and in server i am trying to save the file as follows
client.on('start-audio', function (data) {
stream = fs.createWriteStream('tesfile.wav');
});
client.on('end-audio', function (data) {
if (stream) {
stream.end();
}
stream = null;
});
client.on('binaryData', function (data) {
if (stream !== null) {
stream.write(data);
}
});
But this is not working so how i save this array buffer as wav file?
At the O/P question, there is an attempt to write and add data directly to an existing file which it can't work because WAVE files need to have a header and there can't be a header by just creating a write file stream using createWriteStream. You can check about that header format here "WAVE PCM soundfile format".
There is the wav NPM package which it can help to handle the whole process of writing the data to the server. It has the FileWriter class which creates a stream that will properly handle WAVE audio data and it will write the header when the stream ends.
1. Create a WAVE FileWriter stream on start-audio event:
// Import wav package FileWriter
const WavFileWriter = require('wav').FileWriter;
...
// Global FileWriter stream.
// It won't handle multiple client connections; it's just for demonstration
let outputFileStream;
// The UUID of the filename that's being recorded
let id;
client.on('start-audio', function() {
id = uuidv4();
console.log(`start-audio:${id}`);
// Create a FileWriter stream using UUID generated for filename
outputFileStream = new WavFileWriter(`./audio/recs/${id}.wav`, {
sampleRate: 16000,
bitDepth: 16,
channels: 1
});
});
2. Use the stream we created to write audio data on binaryData event:
client.on('binaryData', function(data) {
console.log(`binaryData:${id}, got ${data ? data.length : 0} bytes}`);
// Write the data directly to the WAVE file stream
if (outputFileStream) {
outputFileStream.write(data);
}
});
3. End the stream when we receive end-audio event:
client.on('end-audio', function() {
console.log(`end-audio:${id}`);
// If there is a stream, end it.
// This will properly handle writing WAVE file header
if (outputFileStream) {
outputFileStream.end();
}
outputFileStream = null;
});
I have created a Github repository with this example which you can find here: https://github.com/clytras/node-wav-stream.
Keep in mind that handling data like this will result in issues because using this code, there is only one FileWriter stream variable for every client that connects. You should create an array for each client stream and use client session ID's to store and identify each stream item that belongs to the corresponding client.
I'm consuming a JSON stream and am trying to use fetch to consume it. The stream emits some data every few seconds. Using fetch to consume the stream gives me access to the data only when the stream closes server side. For example:
var target; // the url.
var options = {
method: "POST",
body: bodyString,
}
var drain = function(response) {
// hit only when the stream is killed server side.
// response.body is always undefined. Can't use the reader it provides.
return response.text(); // or response.json();
};
var listenStream = fetch(target, options).then(drain).then(console.log).catch(console.log);
/*
returns a data to the console log with a 200 code only when the server stream has been killed.
*/
However, there have been several chunks of data already sent to the client.
Using a node inspired method in the browser like this works every single time an event is sent:
var request = require('request');
var JSONStream = require('JSONStream');
var es = require('event-stream');
request(options)
.pipe(JSONStream.parse('*'))
.pipe(es.map(function(message) { // Pipe catches each fully formed message.
console.log(message)
}));
What am I missing? My instinct tells me that fetch should be able to mimic the pipe or stream functionality.
response.body gives you access to the response as a stream. To read a stream:
fetch(url).then(response => {
const reader = response.body.getReader();
reader.read().then(function process(result) {
if (result.done) return;
console.log(`Received a ${result.value.length} byte chunk of data`);
return reader.read().then(process);
}).then(() => {
console.log('All done!');
});
});
Here's a working example of the above.
Fetch streams are more memory-efficient than XHR, as the full response doesn't buffer in memory, and result.value is a Uint8Array making it way more useful for binary data. If you want text, you can use TextDecoder:
fetch(url).then(response => {
const reader = response.body.getReader();
const decoder = new TextDecoder();
reader.read().then(function process(result) {
if (result.done) return;
const text = decoder.decode(result.value, {stream: true});
console.log(text);
return reader.read().then(process);
}).then(() => {
console.log('All done!');
});
});
Here's a working example of the above.
Soon TextDecoder will become a transform stream, allowing you to do response.body.pipeThrough(new TextDecoder()), which is much simpler and allows the browser to optimise.
As for your JSON case, streaming JSON parsers can be a little big and complicated. If you're in control of the data source, consider a format that's chunks of JSON separated by newlines. This is really easy to parse, and leans on the browser's JSON parser for most of the work. Here's a working demo, the benefits can be seen at slower connection speeds.
I've also written an intro to web streams, which includes their use from within a service worker. You may also be interested in a fun hack that uses JavaScript template literals to create streaming templates.
Turns out I could get XHR to work - which doesn't really answer the request vs. fetch question. It took a few tries and the right ordering of operations to get it right. Here's the abstracted code. #jaromanda was right.
var _tryXhr = function(target, data) {
console.log(target, data);
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function () {
console.log("state change.. state: "+ this.readyState);
console.log(this.responseText);
if (this.readyState === 4) {
// gets hit on completion.
}
if (this.readyState === 3) {
// gets hit on new event
}
};
xhr.open("POST", target);
xhr.setRequestHeader("cache-control", "no-cache");
xhr.setRequestHeader("Content-Type", "application/json");
xhr.send(data);
};
I need to capture microphone audio in IE10. So far I have two semi-working solutions:
getUserMedia from Microsoft's experimental WebRTC plugin:
http://www.html5labs.com/prototypes/media-capture-api-(2nd-updated)/media-capture-api-(2nd-update)/info
The issue with this is that while I can capture and replay the audio in the browser, I cannot send the audio to the server. In particular, it is not clear how to extract the audio data from the "blob" object:
function msStopRecordCallback(blob) {
console.log(blob) // outputs {}
console.dir(blob) // outputs {}
playMediaObject.Play(blob); // This works!
}
jRecorder: http://www.sajithmr.me/jrecorder-jquery The issue with this is that it relies on Flash to capture the audio, which is something I would like to avoid.
Are there any other ways to capture audio in IE10?
I recognize that my answer a bit late, but...
You may upload a blob to a server as following (Javascript):
function saveBlob(blob)
{
var uploader = new CustomXMLHttpRequest();
uploader.onpartreceived = function (response)
{
// TODO: handle the server response here
};
var base = window.location.toString();
var uploadService = base.substr(0, base.lastIndexOf("/")) + "/api/upload";
uploader.open("POST", uploadService, true);
uploader.responseType = "text";
var form = new FormData();
form.append("fname", blob, "audio.wav");
uploader.send(form);
}
On the server side, you may treat this blob as a file attachment, e.g. (C#):
public class UploadController : ApiController
{
public async Task<HttpResponseMessage> PostFile()
{
// Check if the request contains multipart/form-data.
if (!Request.Content.IsMimeMultipartContent())
{
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}
var root = HttpContext.Current.Server.MapPath("~/App_Data");
var provider = new MultipartFormDataStreamProvider(root);
try
{
// Read the form data and return an async task.
await Request.Content.ReadAsMultipartAsync(provider);
var fileName = "";
// get the uploaded files.
foreach (var data in provider.FileData)
{
var file = new FileInfo(data.LocalFileName);
// TODO: handle received file here
}
if (string.IsNullOrEmpty(fileName))
{
return Request.CreateResponse(HttpStatusCode.UnsupportedMediaType);
}
return Request.CreateResponse(HttpStatusCode.OK);
}
catch (System.Exception e)
{
return Request.CreateErrorResponse(HttpStatusCode.InternalServerError, e);
}
}
}
Hope this will help.
I have a C++ function which once called consumes input from stdin. Exporting this function to javascript using emscripten causes calls to window.prompt.
Interacting with browser prompt is really tedious task. First of all you can paste only one line at time. Secondly the only way to indicate EOF is by pressing 'cancel'. Last but not least the only way (in case of my function) to make it stop asking user for input by window.prompt is by checking the checkbox preventing more prompts to pop up.
For me the best input method would be reading some blob. I know I can hack library.js but I see some problems:
Reading blob is asynchronous.
To read a blob, first you have to open a file user has to select first.
I don't really know how to prevent my function from reading this blob forever - there is no checkbox like with window.prompt and I'm not sure if spotting EOF will stop it if it didn't in window.prompt case (only checking a checkbox helped).
The best solution would be some kind of callback but I would like to see sime hints from more experienced users.
A way would be to use the Emscripten Filesystem API, for example by calling FS.init in the Module preRun function, passing a custom function as the standard input.
var Module = {
preRun: function() {
function stdin() {
// Return ASCII code of character, or null if no input
}
var stdout = null; // Keep as default
var stderr = null; // Keep as default
FS.init(stdin, stdout, stderr);
}
};
The function is quite low-level: is must deal with one character at a time. To read some data from a blob, you could do something like:
var data = new Int8Array([1,2,3,4,5]);
var blob = new Blob([array], {type: 'application/octet-binary'});
var reader = new FileReader();
var result;
reader.addEventListener("loadend", function() {
result = new Int8Array(reader.result);
});
var i = 0;
var Module = {
preRun: function() {
function stdin() {
if (if < result.byteLength {
var code = result[i];
++i;
return code;
} else {
return null;
}
}
var stdout = null; // Keep as default
var stderr = null; // Keep as default
FS.init(stdin, stdout, stderr);
}
};
Note (as you have hinted), due to the asynchronous nature of the reader, there could be a race condition: the reader must have loaded before you can expect the data at the standard input. You might need to implement some mechanism to avoid this in a real case. Depending on your exact requirements, you could make it so the Emscripten program doesn't actually call main() until you have the data:
var fileRead = false;
var initialised = false;
var result;
var array = new Int8Array([1,2,3,4,5]);
var blob = new Blob([array], {type: 'application/octet-binary'});
var reader = new FileReader();
reader.addEventListener("loadend", function() {
result = new Int8Array(reader.result);
fileRead = true;
runIfCan();
});
reader.readAsArrayBuffer(blob);
var i = 0;
var Module = {
preRun: function() {
function stdin() {
if (i < result.byteLength)
{
var code = result[i];
++i;
return code;
} else{
return null;
}
}
var stdout = null;
var stderr = null;
FS.init(stdin, stdout, stderr);
initialised = true;
runIfCan();
},
noInitialRun: true
};
function runIfCan() {
if (fileRead && initialised) {
// Module.run() doesn't seem to work here
Module.callMain();
}
}
Note: this is a version of my answer at Providing stdin to an emscripten HTML program? , but with focus on the standard input, and adding parts about passing data from a Blob.
From what I understand you could try the following:
Implement selecting a file in Javascript and access it via Javascript Blob interface.
Allocate some memory in Emscripten
var buf = Module._malloc( blob.size );
Write the content of your Blob into the returned memory location from Javascript.
Module.HEAPU8.set( new Uint8Array(blob), buf );
Pass that memory location to a second Emscripten compiled function, which then processes the file content and
Deallocate allocated memory.
Module._free( buf );
Best to read the wiki first.
I am using websockets for file transfer, while i am downloading a file i am recieving the data as it is, but when I open an image file it was corrupted. Data files are downloading fine, the code goes as follows.
try {
fileEntry = fs.root.getFile(filename, { create : creat_file });
var byteArray = new Uint8Array(data.data.length);
for (var i = 0; i < data.data.length; i++) {
byteArray[i] = data.data.charCodeAt(i) & 0xff;
}
BlobBuilderObj = new WebKitBlobBuilder();
BlobBuilderObj.append(byteArray.buffer);
if (!writer) {
writer = fileEntry.createWriter();
pos = 0;
}
//self.postMessage(writer.position);
writer.seek(pos);
writer.write(BlobBuilderObj.getBlob());
pos += 4096;
}
catch (e) {
errorHandler(e);
}
It looks like you are reading data from a WebSocket as a string, converting it to a Blob, and then writing this to a file.
If you have control of the WebSocket server then the best thing would be to send the data as binary frames instead of UTF-8 text data. If you can get the server to send the data as binary frames then you can just tell the WebSocket to deliver the data as Blobs:
ws.binaryType = "blob";
ws.onmessage = function (event) {
if (event.data instanceof Blob) {
// event.data is a Blob
} else {
// event.data is a string
}
}
If that is not an option and you can only send text frames from the server, then you will need to encode the binary data to text before sending it from the server and then decode the text on the other end. If you try and send binary data directly as text frames over WebSockets then doing charCodeAt(x) && 0xff will result in corrupt data.
For example you could base64 encode the data at the server and then base64 decode the data in the client:
ws.onmessage = function (event) {
raw = window.atob(event.data);
}
Update:
There is a very well performing pure Javascript base64 decode/encode contained in websockify. It decodes to an an array of numbers from 0-255 but could be easily modified to return a string instead if that is what you require (Disclaimer: I made websockify).