I have a js script that records audio from microphone and sends it via a websocket to a python REST API.
Part of js script (works in console):
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
var options = {mimeType: "audio/webm;codecs=opus", audioBitsPerSecond:16000};
mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.addEventListener("dataavailable", function(event) {
var reader = new FileReader();
reader.addEventListener("loadend", function() {
var int8View = new Int8Array(reader.result);
console.log(int8View);
});
reader.readAsArrayBuffer(event.data);
});
mediaRecorder.start(200);
If I understand correctly the int8array that this code outputs is opus-encoded audio.
How can I decode it into a raw waveform (something that soundfile.read would return, for example) using python?
I can use ffmpeg to decode opus, but it works on files, how can I turn this array into a proper .opus or .ogg file on python side (since I don't want to send files via a websocket)?
how can I turn this array into a proper .opus or .ogg file?
You're capturing a Webm file stream via audio/webm;codecs=opus, so that would need to be converted from Webm to Ogg Opus. Instead, you could simply download or send the Webm Opus stream as a file as-is:
const recordedBuffers = []
reader.addEventListener('loadend', () => {
recordedBuffers.push(reader.result)
})
function download() {
const a = document.createElement('a')
a.href = URL.createObjectURL(new Blob(recordedBuffers))
a.download = 'my-recorded-audio.webm'
a.click()
}
function saveToServer(url) {
return fetch(url, {
method: 'post',
body: new Blob(recordedBuffers),
headers: {
'content-type': 'application/octet-stream', // optional
'content-type': 'audio/webm', // optional
},
})
}
Related
I've been trying to find a clear answer on this for a while now, since some have only been for big CSV files for example.
The following code works perfectly for smaller files, whenever size exceeds 100 MB, upload process takes a big chunk of memory apparently.
I'm already slicing the files into smaller chunks as well, however apparently the initial reading of the file is causing this issue.
Question: How should I change the following code to prevent the crashes to happen?
axios.post(`/URL_GOES_HERE`, {
name: file.name,
size: file.size,
md5,
extension: file.type.split('/')[1]
})
.then((response) => {
const reader = new FileReader();
reader.onload = (e) => {
const { chunks, type, status } = response.data;
if (status === 'uploading') {
let start = 0;
chunks.forEach(async ({ materialId, size, index }) => {
window.file0 = file;
const buf = reader.result.slice(start, start + size);
const arrayBuf = new Uint8Array(buf);
start = start + size;
const url = `URL_FOR_EACH_CHUNK`;
let xhr = new XMLHttpRequest();
xhr.open('PUT', url, false);
xhr.setRequestHeader(
'Content-Type',
'application/octet-stream'
);
xhr.setRequestHeader(
'Authorization',
`Bearer TOKEN_GOES_HERE`
);
xhr.send(arrayBuf);
});
}
};
reader.readAsArrayBuffer(file);
})
This is a bad way to upload files by slicing and reading the data.
just send the file as it's without reading the content
const res = await fetch('URL_GOES_HERE', {
method: 'post',
body: file
})
await res.text()
if you really need to upload them partially with chunks then just slice the blob and upload that without sending arraybuffer
chunk_to_send = file.slice(start, end)
xhr.send(chunk_to_send)
Browser will be able to pipe the data as a stream and upload them without any memory concern
I have a solution to stream using MediaRecorder API:
var socket = new WebSocket("ws://127.0.0.1:8765");
socket.binaryType = "blob";
socket.onopen = function (event) {
const video = document.querySelector('audio');
video.onplay = function() {
mediaStream = video.captureStream();
mediaRecorder = new MediaRecorder(mediaStream, {
mimeType: 'audio/webm'
});
mediaRecorder.addEventListener('dataavailable', (e) => {
socket.send(e.data);
});
mediaRecorder.start(1000);
};
};
But it doesn't play on my server (For example I use ffmpeg to record stream to file) because MediaRecorder API puts headers only to the first chunk. How can I put webm headers to every chunk?
I'm trying to upload a Blob object into S3, which does get uploaded but in a corrupted way. The application is about recording audio on a web page and saving it to S3.
HTML + Javascript Code:
<p>
<button id=startRecord>START</button>
<button id=stopRecord disabled>Submit</button>
</p>
<p id="recording"></p>
<p>
<a id=audioDownload></a>
</p>
<script
src="https://code.jquery.com/jquery-3.4.1.min.js"
integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo="
crossorigin="anonymous"></script>
<script type="text/javascript">
var audioContent;
navigator.mediaDevices.getUserMedia({audio:true})
.then(stream => {
rec = new MediaRecorder(stream);
rec.ondataavailable = e => {
audioChunks.push(e.data);
if (rec.state == "inactive"){
let blob = new Blob(audioChunks);
// audioContent = blob
// audioContent = URL.createObjectURL(new Blob(audioChunks));
// console.log(audioContent);
$.ajax({
type: "POST",
url: 'https://aws-api-url/prod/audio',
data: new Blob(audioChunks),
crossDomain: true,
processData: false,
headers: {"x-api-key": 'someKey'},
contentType: false
});
// audioDownload.href = audioContent;
// audioDownload.download = 'test';
// audioDownload.innerHTML = 'download';
}
}
})
.catch(e=>console.log(e));
startRecord.onclick = e => {
startRecord.disabled = true;
stopRecord.disabled=false;
document.getElementById("recording").innerHTML = "Listening...";
audioChunks = [];
rec.start();
}
stopRecord.onclick = e => {
document.getElementById("recording").innerHTML = "";
startRecord.disabled = false;
stopRecord.disabled=true;
rec.stop();
}
</script>
AWS Lambda that dumps into S3
import json
import boto3
def lambda_handler(event, context):
# TODO implement
s3_client = boto3.client('s3', aws_access_key_id='',aws_secret_access_key='')
s3_client.put_object(Body=event['body'], Bucket='bucket', Key='incoming/test.wav')
return {
'statusCode': 200,
'headers': {
'Access-Control-Allow-Origin': '*'
},
'body': json.dumps(event)
}
What changes can I possibly make into my Javascript to send this data safely
MediaRecorder does not generate the .wav data type. By default it probably generates data of the MIME type audio/webm; codecs=opus. or audio/ogg; codec=vorbis. Your lambda function looks like it faithfully stores the incoming data. But it's not .wav data, it's something else.
Your sample code lets MediaRecorder choose its own MIME type. In this case you should ask it what it used. For example
rec = new MediaRecorder(stream);
console.log (rec.mimeType);
Or, you can (try to) tell it the MIME type you want. In this case you should still ask it what it actually used. (Browsers vary in the MIME types they generate, and in the ways they respond when they can't deliver the exact type you want.) If your browser can do it, this code will probably generate mp3 (aka MPEG Layer III) audio.
rec = new MediaRecorder(stream, {mimeType: "audio/mpeg"});
console.log (rec.mimeType);
Or, you can try the audio/mp4 MIME type, and see what audio codec you get. It may vary from browser to browser.
You can generally use ffmpeg to convert any MIME type to another once you've recorded it. This is handy if require .wav output, or some other particular format. It takes some hacking, but you can do it in your lambda function.
I am sending Int16Array buffer to server while audio processing
var handleSuccess = function (stream) {
globalStream = stream;
input = context.createMediaStreamSource(stream);
input.connect(processor);
processor.onaudioprocess = function (e) {
var left = e.inputBuffer.getChannelData(0);
var left16 = convertFloat32ToInt16(left);
socket.emit('binaryData', left16);
};
};
navigator.mediaDevices.getUserMedia(constraints)
.then(handleSuccess);
and in server i am trying to save the file as follows
client.on('start-audio', function (data) {
stream = fs.createWriteStream('tesfile.wav');
});
client.on('end-audio', function (data) {
if (stream) {
stream.end();
}
stream = null;
});
client.on('binaryData', function (data) {
if (stream !== null) {
stream.write(data);
}
});
But this is not working so how i save this array buffer as wav file?
At the O/P question, there is an attempt to write and add data directly to an existing file which it can't work because WAVE files need to have a header and there can't be a header by just creating a write file stream using createWriteStream. You can check about that header format here "WAVE PCM soundfile format".
There is the wav NPM package which it can help to handle the whole process of writing the data to the server. It has the FileWriter class which creates a stream that will properly handle WAVE audio data and it will write the header when the stream ends.
1. Create a WAVE FileWriter stream on start-audio event:
// Import wav package FileWriter
const WavFileWriter = require('wav').FileWriter;
...
// Global FileWriter stream.
// It won't handle multiple client connections; it's just for demonstration
let outputFileStream;
// The UUID of the filename that's being recorded
let id;
client.on('start-audio', function() {
id = uuidv4();
console.log(`start-audio:${id}`);
// Create a FileWriter stream using UUID generated for filename
outputFileStream = new WavFileWriter(`./audio/recs/${id}.wav`, {
sampleRate: 16000,
bitDepth: 16,
channels: 1
});
});
2. Use the stream we created to write audio data on binaryData event:
client.on('binaryData', function(data) {
console.log(`binaryData:${id}, got ${data ? data.length : 0} bytes}`);
// Write the data directly to the WAVE file stream
if (outputFileStream) {
outputFileStream.write(data);
}
});
3. End the stream when we receive end-audio event:
client.on('end-audio', function() {
console.log(`end-audio:${id}`);
// If there is a stream, end it.
// This will properly handle writing WAVE file header
if (outputFileStream) {
outputFileStream.end();
}
outputFileStream = null;
});
I have created a Github repository with this example which you can find here: https://github.com/clytras/node-wav-stream.
Keep in mind that handling data like this will result in issues because using this code, there is only one FileWriter stream variable for every client that connects. You should create an array for each client stream and use client session ID's to store and identify each stream item that belongs to the corresponding client.
I am using a plugin jsPDF which generates PDF and saves it to local file system. Now in jsPDF.js, there is some piece of code which generates pdf data in blob format as:-
var blob = new Blob([array], {type: "application/pdf"});
and further saves the blob data to local file system. Now instead of saving I need to print the PDF using plugin node-printer.
Here is some sample code to do so
var fs = require('fs'),
var dataToPrinter;
fs.readFile('/home/ubuntu/test.pdf', function(err, data){
dataToPrinter = data;
}
var printer = require("../lib");
printer.printDirect({
data: dataToPrinter,
printer:'Deskjet_3540',
type: 'PDF',
success: function(id) {
console.log('printed with id ' + id);
},
error: function(err) {
console.error('error on printing: ' + err);
}
})
The fs.readFile() reads the PDF file and generates data in raw buffer format.
Now what I want is to convert the 'Blob' data into 'raw buffer' so that I can print the PDF.
If you are not using NodeJS then you should know that the browser does not have a Buffer class implementation and you are probably compiling your code to browser-specific environment on something like browserify. In that case you need this library that converts your blob into a Buffer class that is supposed to be as perfectly equal to a NodeJS Buffer object as possible (the implementation is at feross/buffer).
If you are using node-fetch (not OP's case) then you probably got a blob from a response object:
const fetch = require("node-fetch");
const response = await fetch("http://www.stackoverflow.com/");
const blob = await response.blob();
This blob is an internal implementation and exists only inside node-fetch or fetch-blob libraries, to convert it to a native NodeJS Buffer object you need to transform it to an arrayBuffer first:
const arrayBuffer = await blob.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
This buffer object can then be used on things such as file writes and server responses.
For me, it worked with the following:
const buffer=Buffer.from(blob,'binary');
So, this buffer can be stored in Google Cloud Storage and local disk with fs node package.
I used blob file, to send data from client to server through ddp protocol (Meteor), so, when this file arrives to server I convert it to buffer in order to store it.
var blob = new Blob([array], {type: "application/pdf"});
var arrayBuffer, uint8Array;
var fileReader = new FileReader();
fileReader.onload = function() {
arrayBuffer = this.result;
uint8Array = new Uint8Array(arrayBuffer);
var printer = require("./js/controller/lib");
printer.printDirect({
data: uint8Array,
printer:'Deskjet_3540',
type: 'PDF',
success: function(id) {
console.log('printed with id ' + id);
},
error: function(err) {
console.error('error on printing: ' + err);
}
})
};
fileReader.readAsArrayBuffer(blob);
This is the final code which worked for me. The printer accepts uint8Array encoding format.
Try:
var blob = new Blob([array], {type: "application/pdf"});
var buffer = new Buffer(blob, "binary");