Develop Reference

Take desktop screenshot with Electron

I am using Electron to create a Windows application that creates a fullscreen transparent overlay window. The purpose of this overlay is to:
take a screenshot of the entire screen (not the overlay itself which is transparent, but the screen 'underneath'),
process this image by sending the image as a byte stream to my python server, and
draw some things on the overlay
I am getting stuck on the first step, which is the screenshot capturing process.
I tried option 1, which is to use capturePage():
this.electronService.remote.getCurrentWindow().webContents.capturePage()
.then((img: Electron.NativeImage) => { ... }
but this captures my overlay window only (and not the desktop screen). This will be a blank image which is useless to me.
Option 2 is to use desktopCapturer:
this.electronService.remote.desktopCapturer.getSources({types: ['screen']}).then(sources => {
for (const source of sources) {
if (source.name === 'Screen 1') {
try {
const mediaDevices = navigator.mediaDevices as any;
mediaDevices.getUserMedia({
audio: false,
video: { // this specification is only available for Chrome -> Electron runs on Chromium browser
mandatory: {
chromeMediaSource: 'desktop',
chromeMediaSourceId: source.id,
minWidth: 1280,
maxWidth: 1280,
minHeight: 720,
maxHeight: 720
}
}
}).then((stream: MediaStream) => { // stream.getVideoTracks()[0] contains the video track I need
this.handleStream(stream);
});
} catch (e) {
}
}
}
});
The next step is where it becomes fuzzy for me. What do I do with the acquired MediaStream to get a bytestream from the screenshot out of it? I see plenty of examples how to display this stream on a webpage, but I wish to send it to my backend. This StackOverflow post mentions how to do it, but I am not getting it to work properly. This is how I implemented handleStream():
import * as MediaStreamRecorder from 'msr';
private handleStream(stream: MediaStream): void {
recorder.stop()
const recorder = new MediaStreamRecorder(stream);
recorder.ondataavailable = (blob: Blob) => { // I immediately get a blob, while the linked SO page got an event and had to get a blob through event.data
this.http.post<Result>('http://localhost:5050', blob);
};
// make data available event fire every one second
recorder.start(1000);
}
The blob is not being accepted by the Python server. Upon inspecting the contents of Blob, it's a video as I suspected. I verified this with the following code:
let url = URL.createObjectURL(blob);
window.open(url, '_blank')
which opens the blob in a new window. It displays a video of maybe half a second, but I want to have a static image. So how do I get a specific snapshot out of it? I'm also not sure if simply sending the Javascript blob format in the POST body will do for Python to be correctly interpret it. In Java it works by simply sending a byte[] of the image so I verified that the Python server implementation works as expected.
Any suggestions other than using the desktopCapturer are also fine. This implementation is capturing my mouse as well, which I rather not have. I must admit that I did not expect this feature to be so difficult to implement.

Here's how you take a desktop screenshot:
const { desktopCapturer } = require('electron')
document.getElementById('screenshot-button').addEventListener('click', () => { // The button which takes the screenshot
desktopCapturer.getSources({ types: ['screen'] })
.then( sources => {
document.getElementById('screenshot-image').src = sources[0].thumbnail.toDataURL() // The image to display the screenshot
})
})
Using 'screen' will take a screenshot of the entire desktop.
Using 'windows' will take a screenshot of only the application.
Also refer to these docs: https://www.electronjs.org/docs/api/desktop-capturer

desktopCapturer only takes videos. So you need to get a single frame from it. You can use html5 canvas for that. Here is an example:
https://ourcodeworld.com/articles/read/280/creating-screenshots-of-your-app-or-the-screen-in-electron-framework
Or, use some third party screenshot library available on npm. The one I found needs to have ImageMagick installed on linux, but maybe there are more, or you don't need to support linux. You'll need to do that in the main electron process in which you can do anything that you can do in node.

You can get each frame from taken video like this:
desktopCapturer.getSources({
types: ['window'], thumbnailSize: {
height: 768,
width: 1366
}
}).then(sources => {
for (let s in sources) {
const content = sources[s].thumbnail.toPNG()
console.log(content)
}
})

How to fix images not loading in phaser.js

Whenever I try to load images into phaser they always appear as a green box on my screen?
I've not only tried the root file path like normal, but also every file path imaginable, but it still never works?

I'm assuming you've already set up a web server.. Did you look at developer tools and check if there are any errors? On Google Chrome, right click your web page and hit inspect. Navigate to the Consoles Tab and look for errors. Be sure to also Log XMLHttpRequests, as these will indicate when the request for said image has been fulfilled (just click the Log XMLHttpRequests checkbox).
Did any of the above solutions work for you? For me, I've used a Simple HTTP Server with Python. I was referencing my images correctly and had my web server hosted correctly. When using developer tools, there were no error messages.. I was banging my head against the wall for hours. The Phaser API was not able to tell what the local directory (http://192.168.0.2:8080/) was from the webpage from some reason.
I fixed my issue by using the following path for my images:
'http://192.168.0.2:8080/assets/sky.png'
where 192.168.0.2 was my local IP address hosting the server & 8080 was the port I was using for communications
Alternative to adding that lengthy file path for each of your images, you can call the this.load.setBaseURL('http://192.168.0.2:8080') function in combination with what you already have.
For example, see the following code:
var config = {
type: Phaser.AUTO,
width: 800,
height: 600,
physics: {
default: 'arcade',
arcade: {
gravity: { y: 200 }
}
},
scene: {
preload: preload,
create: create
}
};
var game = new Phaser.Game(config);
function preload ()
{
console.log('preload');
this.load.setBaseURL('http://192.168.0.2:8080');
this.load.image('bombP','bowmb.png');
this.load.image('einMary','/bowmb.png');
}
function create ()
{
console.log('create');
this.add.image(126, 119, 'bombP');
this.add.image(222,269,'einMary');
//var s = game.add.sprite(80,0,'einMary');
//s.rotation=0.219;
}

I should start by checking those two points below :
Check if the image has the right path to the file
Check if the 'key' you are using is the right one

Cordova camera saves to gallery even on false

I'm in a really nasty situation...
My client wants a Cordova application in Ionic Framework v1, and it's imperative that the camera does not save images to gallery. However, when I set the parameter for saving to gallery to false, it is still saving to gallery.
The problem occurs on Android when you take a photo and cancel it. It then saves that picture to gallery and sometimes even saves all other pictures after that.
I would really welcome any kind of help; All I've found so far are some solutions that I find really hard to understand since my knowledge of Java is zero.
Here is my JS code
function capturePhoto() {
var maxDimension = 1280;
var options = {
quality: 80,
destinationType: Camera.DestinationType.DATA_URL,
sourceType: Camera.PictureSourceType.CAMERA,
correctOrientation: true,
targetWidth: maxDimension,
targetHeight: maxDimension,
saveToPhotoAlbum: false
};
This is for camera options.
$cordovaCamera.getPicture(options).then(function (imageData) {
var src = "data:image/jpeg;base64," + imageData;
$scope.photoPreviewSrc = src;
}).catch(function (err) {
});
}

I have checked with your code using cordova. It works fine as expected.
Verify your app in other device once.
I haven't checked it on ionic platform.

You may want to try running something like the following after you receive the image data:
navigator.camera.cleanup(onSuccess, onFail);
function onSuccess() {
console.log("Camera cleanup success.")
}
function onFail(message) {
alert('Failed because: ' + message);
}
From the docs: "camera.cleanup() Removes intermediate image files that are kept in temporary storage after calling camera.getPicture. Applies only when the value of Camera.sourceType equals Camera.PictureSourceType.CAMERA and the Camera.destinationType equals Camera.DestinationType.FILE_URI."
The above relates directly to your use case.

Is it really going to the photo gallery, or just showing up in Android's photo app? The Android default photo browser will show all photos, screenshots, etc. It will also even show just random images - that other Apps may have on the file system, but that aren't photos.
Since in cordova, you don't have great control of the OS, you can use a work around: You can place the images in a hidden directory (starting with a . such .appdata) and this will prevent Android from automatically seeing the images from the "Gallery" app. I had this problem in an Ionic App and solved it that way.

`Preload.js` failed to load some files on Android devices

PreloadJS stuck at a specific step when loading files on Android devices while everything works fine on a desktop browser and iPhone.
The loading process stopped at the final GIF file (as shown in the code). Why this GIF file could not be loaded?
This happened before with desktop browser, but with no error, But at that time it was caused by some non-standard mp3 files. How to deal with this kind of exception when failed to load/init a file?
Here is the code I used to load files.
var preload = new createjs.LoadQueue();
createjs.Sound.alternateExtensions = ["ogg"];
preload.installPlugin(createjs.Sound)
preload.installPlugin(createjs.SpriteSheet)
preload.addEventListener("fileload", updateLoadingProcess); // show loading process
showWelcomeText();
var resources = {
change: "assets/sound/change.mp3",
click: "assets/sound/click.mp3",
collide: "assets/sound/collide.mp3",
game_over: "assets/sound/gameover.mp3",
reset: "assets/sound/reset.mp3",
win: "assets/sound/win.mp3",
interface_assets: "assets/interface.png",
raining_serial: "assets/raining-serial.gif",
background: "assets/background.jpg",
text: "assets/text-" + LANG + ".gif",
};
var loadedResource = 0;
var manifest = [];
for(var i in resources){
manifest.push({id: i, src: resources[i] + _VER_ }); //add version to update cache
}
preload.loadManifest(manifest);
update
I found that some Android devices stopped at another point, the first image(interface.png), and I really don't know why because these browsers don't have a developer tool.
update
The problem is solved by not using XHR for these image files, although I still don't know why. var preload = new createjs.LoadQueue(false); could make it work.

there was a WebAudioPlugin bug with handling errors and a few issues with SoundJS and PreloadJS integration that have been fixed in the latest version SoundJS-Next and PreloadJS-Next. I would suggest trying those and seeing if you still have issues.
Hope that helps.

Getting CODE:4 MEDIA_ERR_SRC_NOT_SUPPORTED when opening video file with VIDEO.js -

The error only happens when i try to view the website(video) over the internet.
Everything works perfectly when i am running locally in visual studio and it also works perfectly when i remote desktop into the production webserver where i published the site to and browse to the site locally there.
If i had to guess it seems like something is timing out when it goes over the internet or maybe there is something on our firewall preventing the movie from streaming to me from the production webserver. I doubt the firewall is the issue because i can view other videos streamed with video.js from other sources.
My developertools console window shows this (I tried to post a screenshot but i didnt have enough rep points):
GET http://166.62.34.149/Videos/Walgreens_8700SKedzieAve.m4v net::ERR_CONNECTION_ABORTEDvideo.js:118 s.loadvideo.js:65 Tvideo.js:75 s.loadVideo.aspx?VideoName=Walgreens_8700SKedzieAve.m4v:59 (anonymous function)video.js:36 s.Hvideo.js:28 t.a.t.ua.extend.ivideo.js:6 dvideo.js:57 t.Player.t.a.extend.ivideo.js:6 dvideo.js:2 tVideo.aspx?VideoName=Walgreens_8700SKedzieAve.m4v:57 (anonymous function)
here is my code:
<script type="text/javascript">
var videoName;
//videojs.autoSetup();
videoName = document.getElementById('lblVideoName').innerHTML;
videojs('my_video_1', {}, function(){
this.src({ type: "video/mp4", src: "/Videos/" + videoName });
this.load();
this.play();
});
videojs('my_video_1').ready(function () {
// Store the video object
var myPlayer = this, id = myPlayer.id();
var aspectRatio = 478 / 850;
function resizeVideoJS() {
var width = document.getElementById(id).parentElement.offsetWidth;
myPlayer.width(width).height(width * aspectRatio);
}
resizeVideoJS();
window.onresize = resizeVideoJS;
});
function PausePlayer() {
var myPlayer = videojs('my_video_1');
myPlayer.pause();
};
</script>
I have set my IIS mime type, so that isnt the issue. Any help you have provide would be greatly appreciated.
Thanks everyone!

I know this is an old oner but just to let you know that I came across this with our website and after digging deep into mime types, changing sources, removing cache and so on, a simple options --> advanced --> "Reset" made everything work fine.
This will obviously only be the case if you can test it working on other PC's somewhere else to confirm all the mime types and video work in the first place.
Hope this is of help to someone in the future as it was the top Google result for me.
Regards
Liam