Develop Reference

Take desktop screenshot with Electron

I am using Electron to create a Windows application that creates a fullscreen transparent overlay window. The purpose of this overlay is to:
take a screenshot of the entire screen (not the overlay itself which is transparent, but the screen 'underneath'),
process this image by sending the image as a byte stream to my python server, and
draw some things on the overlay
I am getting stuck on the first step, which is the screenshot capturing process.
I tried option 1, which is to use capturePage():
this.electronService.remote.getCurrentWindow().webContents.capturePage()
.then((img: Electron.NativeImage) => { ... }
but this captures my overlay window only (and not the desktop screen). This will be a blank image which is useless to me.
Option 2 is to use desktopCapturer:
this.electronService.remote.desktopCapturer.getSources({types: ['screen']}).then(sources => {
for (const source of sources) {
if (source.name === 'Screen 1') {
try {
const mediaDevices = navigator.mediaDevices as any;
mediaDevices.getUserMedia({
audio: false,
video: { // this specification is only available for Chrome -> Electron runs on Chromium browser
mandatory: {
chromeMediaSource: 'desktop',
chromeMediaSourceId: source.id,
minWidth: 1280,
maxWidth: 1280,
minHeight: 720,
maxHeight: 720
}
}
}).then((stream: MediaStream) => { // stream.getVideoTracks()[0] contains the video track I need
this.handleStream(stream);
});
} catch (e) {
}
}
}
});
The next step is where it becomes fuzzy for me. What do I do with the acquired MediaStream to get a bytestream from the screenshot out of it? I see plenty of examples how to display this stream on a webpage, but I wish to send it to my backend. This StackOverflow post mentions how to do it, but I am not getting it to work properly. This is how I implemented handleStream():
import * as MediaStreamRecorder from 'msr';
private handleStream(stream: MediaStream): void {
recorder.stop()
const recorder = new MediaStreamRecorder(stream);
recorder.ondataavailable = (blob: Blob) => { // I immediately get a blob, while the linked SO page got an event and had to get a blob through event.data
this.http.post<Result>('http://localhost:5050', blob);
};
// make data available event fire every one second
recorder.start(1000);
}
The blob is not being accepted by the Python server. Upon inspecting the contents of Blob, it's a video as I suspected. I verified this with the following code:
let url = URL.createObjectURL(blob);
window.open(url, '_blank')
which opens the blob in a new window. It displays a video of maybe half a second, but I want to have a static image. So how do I get a specific snapshot out of it? I'm also not sure if simply sending the Javascript blob format in the POST body will do for Python to be correctly interpret it. In Java it works by simply sending a byte[] of the image so I verified that the Python server implementation works as expected.
Any suggestions other than using the desktopCapturer are also fine. This implementation is capturing my mouse as well, which I rather not have. I must admit that I did not expect this feature to be so difficult to implement.

Here's how you take a desktop screenshot:
const { desktopCapturer } = require('electron')
document.getElementById('screenshot-button').addEventListener('click', () => { // The button which takes the screenshot
desktopCapturer.getSources({ types: ['screen'] })
.then( sources => {
document.getElementById('screenshot-image').src = sources[0].thumbnail.toDataURL() // The image to display the screenshot
})
})
Using 'screen' will take a screenshot of the entire desktop.
Using 'windows' will take a screenshot of only the application.
Also refer to these docs: https://www.electronjs.org/docs/api/desktop-capturer

desktopCapturer only takes videos. So you need to get a single frame from it. You can use html5 canvas for that. Here is an example:
https://ourcodeworld.com/articles/read/280/creating-screenshots-of-your-app-or-the-screen-in-electron-framework
Or, use some third party screenshot library available on npm. The one I found needs to have ImageMagick installed on linux, but maybe there are more, or you don't need to support linux. You'll need to do that in the main electron process in which you can do anything that you can do in node.

You can get each frame from taken video like this:
desktopCapturer.getSources({
types: ['window'], thumbnailSize: {
height: 768,
width: 1366
}
}).then(sources => {
for (let s in sources) {
const content = sources[s].thumbnail.toPNG()
console.log(content)
}
})

os x getUserMedia gain control is reducing a constant input signal over time

I'm trying to build a simple fourier transform program using HTML and javascript for a class. I have OS X Catalina, and I'm using Chrome, and grabbing the microphone (at least I think I am).
The javascript for getting the stream setup is:
let constraints = {
audio: {
echoCancellation: false,
autoGainControl: false,
channelCount: 2,
noiseSuppression: false,
sampleRate: my_sampleRate,
sampleSize: 16,
volume: 1.0
},
video: false
};
navigator.mediaDevices.getUserMedia(constraints).then(spectrum).catch(console.log);
let AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext({sampleRate: my_sampleRate});
let analyser = audioCtx.createAnalyser();
analyser.fftSize = my_samples;
analyser.smoothingTimeConstant = 0;
audioCtx.createMediaStreamSource(stream).connect(analyser);
let data_raw = new Uint8Array(analyser.fftSize);
Then I set an interval and in the function that's called, I grab the raw data stream:
analyser.getByteTimeDomainData(data_raw);
The code to make the FFT (I know, I could use the built-in analyzer and getByteFrequencyData but I don't want to use that) is straight forward. When I test it with a 1kHz signal, it shows up in the right place but then fades away. It's as if something is doing an AC filtering over a relatively long period.
Any ideas on how to get rid of this? Note that OS X Catalina still has that pesky "feature" where the noise suppression option that used to be in the Preferences/Sound for the microphone is no longer there.

Upon more investigation...I think part of my problem was normalization to make an FFT histogram. The normalization was increasing, so the peak was decreasing. But I noticed that when I inject a pure sine wave at 1000 Hz, I see a clear peak in the FFT where it should show up however it starts out at some value and quickly falls within a few seconds to something stable. So the hardware (or the analyzer?) is doing something I don't understand. Anyone have a suggestion?

Focus the camera on an object using focusMode on WebRTC

I'm trying to get the camera to focus on an object, by using focusMode on the stream track received from getUserMedia. But after changing the property of focus mode to manual mode, I don't see it reflect on the stream also I could see that after applying focusMode constraint it stays unchanged(focusMode: continuous). The issue is when I capture an image, using the stream, the image is blurry because of lack of focus.
Here is a code snippet.
navigator.mediaDevices.getUserMedia({ video: true })
.then((stream) => {
video.srcObject = stream;
track = stream.getVideoTracks()[0];
track.applyConstraints({
advanced: [
{ focusMode: 'manual', focusDistance: 0.33 }
]
})
});
Does anybody have experience with getting the camera to focus using the manual or single-shot mode? Any help is appreciated.

Checking canvas dimension limitations

Discovering the Web Audio Api I wanted to draw a waveform for sound files. Now, I am aware of the image dimension limitations in some browsers, and I tried to look them up, but they seem to be ever changing (or at least memory differences like Chrome Desktop vs Chrome Mobile).
I tried to look up how to test if an image, or a Canvas / 2D Context can be of a certain size with almost no success. However, when testing this thing in Firefox I did get an error in the console so I tried the following method:
wfWidth = source.buffer.duration*100;
document.getElementById("waveform").width = wfWidth;
wfCtx = document.getElementById("waveform").getContext("2d");
var success = false;
while(success == false)
{
try {
var temp = wfCtx.getImageData(0,0,10,10); // this produced an error further down the code, so I use this as my test case.
success = true;
} catch(err) {
wfWidth = wfWidth / 2;
document.getElementById("waveform").width = wfWidth;
wfCtx = document.getElementById("waveform").getContext("2d");
}
console.log(success);
}
This does seem to work in Firefox as the console first outputs false showing that the canvas is too big and then true after halving the width of the canvas and the waveform is shown.
However, in Google Chrome on Desktop the canvas seems to be of a certain size (as indicated by a scroll bar) but it is totally blank. When I right-click to save image, it is a 0 byte txt file. On Chrome Mobile (android) I get this little square sad face. Guess that method of checking doesn't work in Chrome.
What would be the best way to test if the canvas is, well, valid, and resize if it is not?

WebRTC switch back to front camera with gUM not streaming on Android/Chrome

On Samsung Galaxy S2 Android 6.0.1 + Chrome v55, when I getUserMedia on page load, the video track acquired appears live.
When I select the back camera from my camera select, I trigger another time my gUM with constraints to use that exact facing back cameraId, the video track is ended, I have a black rectangle instead of a stream.
var constraints = {
video: {
deviceId: {
exact: defaultVideoDeviceId
}
},
audio: true
};
gUM wrapper
function gUM(constraints, callback) {
console.debug("WebRTC constraints", constraints);
// Stopping streaming before starting the new one
if (window.streams.local) {
window.streams.local.getTracks().forEach(function(track) {
track.stop();
});
}
navigator.mediaDevices.getUserMedia(constraints)
.then(stream => {
console.debug("New MediaStream > Tracks", stream.getTracks());
window.streams.local = stream;
callback && callback(stream);
})
.catch(err => {
console.log("Raised error when capturing:", err);
});
}
If I switch back to front, it acquires a new MediaStream and it plays the video.

I'm facing a similar problem too.
My test is a bit different since I'm trying to make a GUM of the back camera while I have a GUM a media stream active that is using the front camera.
I tested in several android devices and only the Xiaomi MI Mix 2 with the MIUI beta 875 with android 8.1 works. This can be because that rom uses the new camera2 android api, or because the Mi Mix 2 camera hardware allows the usage of both the back and the front camera at the same time.
The annoying thing is that sometimes, on certain devices, the GUM doesn't fail, but hangs indefinitely.
Maybe listening to the MediaStreamTrack.onended event after the track.stop() method call, can help to understand when resources are completely free so as you can try a new GUM with different constraints.
Please let me know if you discovered something.
Simone