I have developed an electron app and for the first time ever, my antivirus (ESET) has raised "Webcam access attempt" when the app loads. Has anyone else experienced this?
My app does not use the webcam and I have no code that requires the webcam.
I do have code that accesses audio for audio recording. I have denied access to the webcam in antivirus and the app does function as designed. However, antivirus warning messages appear on every load of the app. As you can imagine, this is not cool.
This has surfaced immediately after updating ESET (v14.2.10.0), so they have some new rule that gets triggered. I have to assume that this is not an ESET over sensitivity to something (I have no idea how AV's function and ‘blaming’ the antivirus doesn’t seem like a sound response to provide users), so I am left questioning my deployment of web-apis in my code.
My audio access uses the native web-apis: AudioContext, Navigator, MediaDevices, MediaRecorder. The key lines of code are below:
// getting list of all AUDIO devices:
// const audioSources = await navigator.mediaDevices.enumerateDevices({ audio: true });
// ^ above does NOT filter by audio only
const audioSources = await navigator.mediaDevices.enumerateDevices();
// creating a recorder object:
const audioContext = new AudioContext();
const dest = audioContext.createMediaStreamDestination();
const audioParams = {
deviceId: "6e5fc2d7ffa5c6c04e06d282a5aa743e983e585a7e12118c80c0cd8646cce4b7", // this ID is from audioSources object
}
const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: audioParams });
const audioIn = audioContext.createMediaStreamSource(mediaStream);
audioIn.connect(dest);
const audioOptions = {
bitsPerSecond: 128000,
mimeType: 'audio/webm; codecs=opus',
};
const recorder = new MediaRecorder(dest.stream, audioOptions);
Because navigator.mediaDevices.enumerateDevices() does not take parameters, such as { audio: true }, enumerateDevices() triggers the camera request.
I use the results of enumerateDevices() to access the device ID, which is then passed into .getUserMedia() to select the specific device. This allows users to select one or more audio inputs for the same recording.
Is there a way of just querying available media for audio devices / excluding video?
Is there another way of identifiying all available audio devices?
How else can I select what device .getUserMedia() returns as a stream?
The only existing information I could find on this was on the shut-down Atom Community forum:
Electron keeps accessing my webcam for NO REASON - two developers discovering the same behaviour in Sept'20 with different Antivirus software. No resolution.
Originally seen using Electron 8.5.0. Issue remains after updating to 13.1.2
Software versions: Electron 13.1.2, ESET 14.2.10.0
Related
I'm developing a webRTC video chat application with a screen-sharing feature, but after getting the video and audio stream, the tab becomes muted.
const stream = await navigator.mediaDevices.getDisplayMedia({
video: true,
audio: true,
preferCurrentTab: true, //used to locate the current tab easily
});
let audio = new Audio("https://www.soundhelix.com/examples/mp3/SoundHelix-Song-3.mp3");
audio.play(); // playing audio why this audio is muted
This is a known bug from Chromiums side: https://bugs.chromium.org/p/chromium/issues/detail?id=1317964&q=preferCurrentTab&can=2
This is the latest update from a Chrome author:
It is on my schedule to implement suppressLocalAudioPlayback, which would then allow me to make the change you're requesting here. Currently, I plan to implement it in August.
So hopefully in one of the next updates, a fix is shipped.
The suppressLocalAudioPlayback audio constraint available in Chrome 109 is what you're looking for. When set to true, it indicates that the browser should stop relaying audio to the local speakers when capture starts. The default value for this constraint is false.
// Prompt the user to share a tab, a window or a screen with audio.
// If successful, stop the captured audio from being played out over
// the local device’s speakers.
const stream = await navigator.mediaDevices.getDisplayMedia({
audio: { suppressLocalAudioPlayback: true },
});
const [audioTrack] = stream.getAudioTracks();
const settings = audioTrack.getSettings();
console.log(settings.suppressLocalAudioPlayback); // true
As of the time of writing, suppressLocalAudioPlayback does not yet work with applyConstraints(). See Chromium Bug 1381959.
I currently played around with the Web Audio API a little bit. I managed to "read" a microphone and play it to my speakers which worked quite seamlessly.
Using the Web Audio API, I now would like to resample an incoming audio stream (aka. microphone) from 44.1kHz to 16kHz. 16kHz, because I am using some tools which require 16kHz. Since 44.1kHz divided by 16kHz is not an integer, I believe I cannot just simply use a low-pass filter and "skip samples", right?
I also saw that some people suggested to use the .createScriptProcessor(), but since it is deprecated I feel kind of bad to use it, so I'm searching a different approach now. Also, I don't necessarily need the audioContext.Destination to hear it! It is still fine if I get the "raw" data of the resampled output.
My approaches so far
Creating an AudioContext({sampleRate: 16000}) --> throws an error: "Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported."
Using an OfflineAudioContext --> but it seems to have no option for streams (only for buffers)
Using an AudioWorkletProcessor to resample. In this case, I think, that I could use the processor to actually resample the input and output the "resampled" source. But I couldn't really figure how to resample it.
main.js
...
microphoneGranted: async function(stream){
audioContext = new AudioContext();
var microphone = audioContext.createMediaStreamSource(stream);
await audioContext.audioWorklet.addModule('resample_proc.js');
const resampleNode = new AudioWorkletNode(audioContext, 'resample_proc');
microphone.connect(resampleNode).connect(audioContext.destination);
}
...
resample_proc.js (assuming only one input and output channel)
class ResampleProcesscor extends AudioWorkletProcessor {
...
process(inputs, outputs, parameters) {
const input = inputs[0];
const output = outputs[0];
if(input.length > 0){
const inputChannel0 = input[0];
const outputChannel0 = output[0];
for (let i = 0; i < inputChannel0.length; ++i) {
//do something with resample here?
}
return true;
}
}
}
registerProcessor('resample_proc', ResampleProcesscor);
Thank you!
Your general idea looks good. While I can't provide the code to do the resampling, I can point out that you might want to start with Sample-rate conversion. Method 1 would work here with L/M = 160/441. Designing the filters takes a bit of work but only needs to be done once. You can also search for polyphase filtering for hints on how to do this effectively.
What chrome does in various parts is to use a windowed-sinc function to resample between any set of rates. This is method 2 in the wikipedia link.
The WebAudio API now allows to resample by passing the sample rate in the constructor. This code works in Chrome and Safari:
const audioStream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false })
const audioContext = new AudioContext({ sampleRate: 16000 })
const audioStreamSource = audioContext.createMediaStreamSource(audioStream);
audioStreamSource.connect(audioContext.destination)
But fails in Firefox that throws a NotSupportedError exception with AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported.
In the example below, I've downsampled the audio coming from the microphone to 8kHz and added a one second delay so we can clearly hear the effect of downsampling:
https://codesandbox.io/s/magical-rain-xr4g80
I am trying to analyse the audio output from the browser, but I don't want the getUserMedia prompt to appear (which asks for microphone permission).
The sound sources are SpeechSynthesis and an Mp3 file.
Here's my code:
return navigator.mediaDevices.getUserMedia({
audio: true
})
.then(stream => new Promise(resolve => {
const track = stream.getAudioTracks()[0];
this.mediaStream_.addTrack(track);
this._source = this.audioContext.createMediaStreamSource(this.mediaStream_);
this._source.connect(this.analyser);
this.draw(this);
}));
This code is working fine, but it's asking for permission to use the microphone! I a not interested at all in the microphone I only need to gauge the audio output. If I check all available devices:
navigator.mediaDevices.enumerateDevices()
.then(function(devices) {
devices.forEach(function(device) {
console.log(device.kind + ": " + device.label +
" id = " + device.deviceId);
});
})
I get a list of available devices in the browser, including 'audiooutput'.
So, is there a way to route the audio output in a media stream that can be then used inside 'createMediaStreamSource' function?
I have checked all the documentation for the audio API but could not find it.
Thanks for anyone that can help!
There are various ways to get a MediaStream which is originating from gUM, but you won't be able to catch all possible audio output...
But, for your mp3 file, if you read it through an MediaElement (<audio> or <video>), and if this file is served without breaking CORS, then you can use MediaElement.captureStream.
If you read it from WebAudioAPI, or if you target browsers that don't support captureStream, then you can use AudioContext.createMediaStreamDestination.
For SpeechSynthesis, unfortunately you will need gUM... and a Virtual Audio Device: first you would have to set your default output to the VAB_out, then route your VAB_out to VAB_in and finally grab VAB_in from gUM...
Not an easy nor universally doable task, moreover when IIRC SpeechSynthesis doesn't have any setSinkId method.
I want to use the SpeechRecognition api with an audio file (mp3, wave, etc.)
Is that possible?
The short answer is No.
The Web Speech Api Specification does not prohibit this (the browser could allow the end-user to choose a file to use as input), but the audio input stream is never provided to the calling javascript code (in the current draft version), so you don't have any way to read or change the audio that is input to the speech recognition service.
This specification was designed so that the javascript code will only have access to the result text coming from the speech recognition service.
Basicly you may use it only with default audioinput device which is choosen on OS level...
Therefore you just need to play you file into your default audioinput
2 options possible:
1
Install https://www.vb-audio.com/Cable/
Update system settings to use VCable device as default audiooutput and audioinput
Play your file with any audio player you have
Recognize it... e.g. using even standard demo UI https://www.google.com/intl/fr/chrome/demos/speech.html
Tested this today, and it works perfectly :-)
2
THIS IS NOT TESTED BY ME, so I cannot confirm that this is working, but you may feed audio file into chrome using Selenium... just like
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
ChromeOptions options = new ChromeOptions();
options.addArguments("--allow-file-access-from-files",
"--use-fake-ui-for-media-stream",
"--allow-file-access",
"--use-file-for-fake-audio-capture=D:\\PATH\\TO\\WAV\\xxx.wav",
"--use-fake-device-for-media-stream");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
ChromeDriver driver = new ChromeDriver(capabilities);
But I'm not sure if this stream will replace default audioinput
Andri deleted this post but I will repost it as I believe it to be the most accurate answer, besides the hackish answers above:
According to MDN you CAN'T do that. You can't feed any stream into recognition service
That's a big problem... You even cannot select microphone used by SpeechRecognition
That is done by purpose, Google want's to sell their CLOUD SPEECH API
You need to use services like CLOUD SPEECH API
You could probably just start the SpeechRecognition engine using the mic and playback the audio file via speakers to have feed back into the mic. It worked for me when I tested it.
Yes, it is possible to get the text transcript of the playback of an audio file using webkitSpeechRecognition. The quality of the transcript depends upon the quality of the audio playback.
const recognition = new webkitSpeechRecognition();
const audio = new Audio();
recognition.continuous = true;
recognition.interimResults = true;
recognition.onresult = function(event) {
if (event.results[0].isFinal) {
// do stuff with `event.results[0][0].transcript`
console.log(event.results[0][0].transcript);
recognition.stop();
}
}
recognition.onaudiostart = e => {
console.log("audio capture started");
}
recognition.onaudioend = e => {
console.log("audio capture ended");
}
audio.oncanplay = () => {
recognition.start();
audio.play();
}
audio.src = "/path/to/audio";
jsfiddle https://jsfiddle.net/guest271314/guvn1yq6/
On Samsung Galaxy S2 Android 6.0.1 + Chrome v55, when I getUserMedia on page load, the video track acquired appears live.
When I select the back camera from my camera select, I trigger another time my gUM with constraints to use that exact facing back cameraId, the video track is ended, I have a black rectangle instead of a stream.
var constraints = {
video: {
deviceId: {
exact: defaultVideoDeviceId
}
},
audio: true
};
gUM wrapper
function gUM(constraints, callback) {
console.debug("WebRTC constraints", constraints);
// Stopping streaming before starting the new one
if (window.streams.local) {
window.streams.local.getTracks().forEach(function(track) {
track.stop();
});
}
navigator.mediaDevices.getUserMedia(constraints)
.then(stream => {
console.debug("New MediaStream > Tracks", stream.getTracks());
window.streams.local = stream;
callback && callback(stream);
})
.catch(err => {
console.log("Raised error when capturing:", err);
});
}
If I switch back to front, it acquires a new MediaStream and it plays the video.
I'm facing a similar problem too.
My test is a bit different since I'm trying to make a GUM of the back camera while I have a GUM a media stream active that is using the front camera.
I tested in several android devices and only the Xiaomi MI Mix 2 with the MIUI beta 875 with android 8.1 works. This can be because that rom uses the new camera2 android api, or because the Mi Mix 2 camera hardware allows the usage of both the back and the front camera at the same time.
The annoying thing is that sometimes, on certain devices, the GUM doesn't fail, but hangs indefinitely.
Maybe listening to the MediaStreamTrack.onended event after the track.stop() method call, can help to understand when resources are completely free so as you can try a new GUM with different constraints.
Please let me know if you discovered something.
Simone