Develop Reference

Video and audio permissions in JavaScript

I need to tell if a person dismisses the media popup or blocks the media popup
const [permissions, setPermissions] = useState(false)
const handleClick = () => {
setPermissions('pending');
navigator.getUserMedia({ video: true, audio: true }, stream => {
stream.getTracks().forEach(track => {
return setPermissions(track.enabled);
});
}, (error) => {
if(error ==='DOMException: Permission denied'){
setPermissions('denied')
}
if(error === 'DOMException: Permission dismissed'){
setPermissions('dismissed')
}
});
};
I tried using the error string that I got back but it is not working. I need to be able to tell the difference between the errors. Does anyone have a good way to do this? Thanks!

You can't tell the difference in any exact way. Browsers are very uncooperative with Javascript code that tries to access media devices but doesn't get permission from the user. Because cybercreeps.
I've had a bit of success with this to see if the user denied access:
Try again. If you get an immediate (within a couple of seconds) error they probably have already denied access to the devices. If it takes longer, they probably are looking at a repeat of the permission dialog.
Not a great situation. Especially if you want to explain to the user how to go back and un-deny permission. But it's a necessary privacy feature in the surveillance era.

Take desktop screenshot with Electron

I am using Electron to create a Windows application that creates a fullscreen transparent overlay window. The purpose of this overlay is to:
take a screenshot of the entire screen (not the overlay itself which is transparent, but the screen 'underneath'),
process this image by sending the image as a byte stream to my python server, and
draw some things on the overlay
I am getting stuck on the first step, which is the screenshot capturing process.
I tried option 1, which is to use capturePage():
this.electronService.remote.getCurrentWindow().webContents.capturePage()
.then((img: Electron.NativeImage) => { ... }
but this captures my overlay window only (and not the desktop screen). This will be a blank image which is useless to me.
Option 2 is to use desktopCapturer:
this.electronService.remote.desktopCapturer.getSources({types: ['screen']}).then(sources => {
for (const source of sources) {
if (source.name === 'Screen 1') {
try {
const mediaDevices = navigator.mediaDevices as any;
mediaDevices.getUserMedia({
audio: false,
video: { // this specification is only available for Chrome -> Electron runs on Chromium browser
mandatory: {
chromeMediaSource: 'desktop',
chromeMediaSourceId: source.id,
minWidth: 1280,
maxWidth: 1280,
minHeight: 720,
maxHeight: 720
}
}
}).then((stream: MediaStream) => { // stream.getVideoTracks()[0] contains the video track I need
this.handleStream(stream);
});
} catch (e) {
}
}
}
});
The next step is where it becomes fuzzy for me. What do I do with the acquired MediaStream to get a bytestream from the screenshot out of it? I see plenty of examples how to display this stream on a webpage, but I wish to send it to my backend. This StackOverflow post mentions how to do it, but I am not getting it to work properly. This is how I implemented handleStream():
import * as MediaStreamRecorder from 'msr';
private handleStream(stream: MediaStream): void {
recorder.stop()
const recorder = new MediaStreamRecorder(stream);
recorder.ondataavailable = (blob: Blob) => { // I immediately get a blob, while the linked SO page got an event and had to get a blob through event.data
this.http.post<Result>('http://localhost:5050', blob);
};
// make data available event fire every one second
recorder.start(1000);
}
The blob is not being accepted by the Python server. Upon inspecting the contents of Blob, it's a video as I suspected. I verified this with the following code:
let url = URL.createObjectURL(blob);
window.open(url, '_blank')
which opens the blob in a new window. It displays a video of maybe half a second, but I want to have a static image. So how do I get a specific snapshot out of it? I'm also not sure if simply sending the Javascript blob format in the POST body will do for Python to be correctly interpret it. In Java it works by simply sending a byte[] of the image so I verified that the Python server implementation works as expected.
Any suggestions other than using the desktopCapturer are also fine. This implementation is capturing my mouse as well, which I rather not have. I must admit that I did not expect this feature to be so difficult to implement.

Here's how you take a desktop screenshot:
const { desktopCapturer } = require('electron')
document.getElementById('screenshot-button').addEventListener('click', () => { // The button which takes the screenshot
desktopCapturer.getSources({ types: ['screen'] })
.then( sources => {
document.getElementById('screenshot-image').src = sources[0].thumbnail.toDataURL() // The image to display the screenshot
})
})
Using 'screen' will take a screenshot of the entire desktop.
Using 'windows' will take a screenshot of only the application.
Also refer to these docs: https://www.electronjs.org/docs/api/desktop-capturer

desktopCapturer only takes videos. So you need to get a single frame from it. You can use html5 canvas for that. Here is an example:
https://ourcodeworld.com/articles/read/280/creating-screenshots-of-your-app-or-the-screen-in-electron-framework
Or, use some third party screenshot library available on npm. The one I found needs to have ImageMagick installed on linux, but maybe there are more, or you don't need to support linux. You'll need to do that in the main electron process in which you can do anything that you can do in node.

You can get each frame from taken video like this:
desktopCapturer.getSources({
types: ['window'], thumbnailSize: {
height: 768,
width: 1366
}
}).then(sources => {
for (let s in sources) {
const content = sources[s].thumbnail.toPNG()
console.log(content)
}
})

Record the desktop using Electron

I am building an electron in which I need the desktopCapturer api, but I don't fully understand how to use it.
From the api official page (and this example app: https://github.com/hokein/electron-sample-apps/tree/master/desktop-capture) I see that the desktopCapturer only gives me the id's of the sources, not the video streams themselves. For that, I should use navigator.mediaDevices.getUserMedia(). But the constraints object no longer has the mandatory property and because I am using typescript I am getting an error if I try to use it.
I've tried to use the deviceId property instead, but I am getting this error:
Uncaught (in promise) DOMException: Requested device not found (on a device with a webcam I would get the webcam stream instead of that error). Here is my code:
import { desktopCapturer, DesktopCapturerSource } from "electron";
function onLoad(){
desktopCapturer.getSources({
thumbnailSize: {
width: 256,
height: 256,
},
types: ["screen", "window"]
}, (error: Error, srcs: DesktopCapturerSource[]) => {
if (error)
throw error;
let video: HTMLVideoElement | null = document.querySelector("video");
for (let src of srcs)
navigator.mediaDevices.getUserMedia({
video:{
deviceId : src.id
}
}).then((stream:MediaStream)=>{
if(video){
video.srcObject = stream;
video.play();
}
})
})
}
document.addEventListener("DOMContentLoaded", onLoad);
I also tried using navigator.getDisplayMedia(), but I wouldn't get the pop-up prompting to select a source as I would get in Chrome. What should I do to get this working? Thanks in advance!

I found the solution, at least for new since WebRTC is not yet standardized. Copying the navigator object into a variable and casted to any allows the use of the mandatory property on the constraints object since typescript no longer checks for type compatibility

Read the phone number of device using javascript [duplicate]

Is there any way to fetch user’s phone number in Firefox OS?
If so, any help would be appreciated.

According to Mozilla's app permissions page, there is an permission called "phonenumberservice" but there is no information about it. Anyway, the permision is listed under the "Internal (Certified) app permissions", which means that, when available, it can only be used by "system-level apps and default apps created by Mozilla/operators/OEMs".

With Firefox 2.0 you should be able to use Mobile Identity API:
https://wiki.mozilla.org/WebAPI/MobileIdentity
https://bugzilla.mozilla.org/show_bug.cgi?id=1021594
I believe the permission is:
"permissions": {
"mobileid": {} }
And it is privileged.

So, as #Jason said, the Mobile Identity API provides this capability, and not just for certified, but for privileged applications. So it is no longer just for OEMs.
The Mozilla Wiki site shows the API:
dictionary MobileIdOptions {
boolean forceSelection = false;
};
partial interface Navigator {
Promise getMobileIdAssertion(optional MobileIdOptions options);
};
The site also provides a sample code skeleton for this:
function verifyAssertion(aAssertion) {
// Make use of the remote verification API
// and return the verified msisdn.
// NB: This is necessary to make sure that the user *really* controls this phone number!
}
// Request a mobile identity assertion and force the chrome UI to
// allow the user to change a possible previous selection.
navigator.getMobileIdAssertion({ forceSelection: true })
.then(
(assertion) => {
verifyAssertion(assertion)
.then(
(msisdn) => {
// Do stuff with the msisdn.
}
);
},
(error) {
// Process error.
};
);
For this to work, you need to add the mobileid permission in the manifest file, for example like this (I made up the description):
"permissions": {
"mobileid": {
"description": "Required for sending SMS for two factor authentication",
"access": "readonly"
}
}
PS: I made this answer, because most answers are outdated, and the one that isn't, does not contain all useful information.
References:
App Manifest Documentation
Firefox Remote Verification

WebRTC never fires onIceCandidate

I started developing with WebRTC, but that thing never gives me ICE candidates. I set up everything, I'm exchanging the descriptions and stuff, I also made a super-ugly function narrow down there to make sure everything runs correctly, one after another. Signaling state is stable for both, onError is never fired (as expected), but onIceCandidate also (not as expected), and when I want to send a random, empty MediaStream object pc1.addStream(new webkitMediaStream());, it always fires onNegotiationNeeded.
Does anyone have an idea what the heck is wrong with my code? I spent hours of browsing Stack Overflow, HTML5 Rocks, and W3C docs, but I don't understand that. Here is my entire code:
var config={
'iceServers':[{
'url':'stun:stun.l.google.com:19302'
},{
'url':'stun:stun1.l.google.com:19302'
},{
'url':'stun:stun2.l.google.com:19302'
},{
'url':'stun:stun3.l.google.com:19302'
},{
'url':'stun:stun4.l.google.com:19302'
}]
};
var pc1=new webkitRTCPeerConnection(config);
var pc2=new webkitRTCPeerConnection(config);
var onError=function(error)
{
console.error(error);
}
pc1.onicecandidate=function()
{
console.log('PC1 onIceCandidate (finally) fired!');
}
pc2.onicecandidate=function()
{
console.log('PC2 onIceCandidate (finally) fired!');
}
pc1.oniceconnectionstatechange=function()
{
console.log('PC1 oniceconnectionstatechange fired!');
}
pc2.oniceconnectionstatechange=function()
{
console.log('PC2 oniceconnectionstatechange fired!');
}
pc1.onnegotiationneeded=function()
{
console.log('PC1 onnegotiationneeded fired!');
}
pc2.onnegotiationneeded=function()
{
console.log('PC2 onnegotiationneeded fired!');
}
pc1.createOffer(function(offer){
pc1.setLocalDescription(offer,function(){
pc2.setRemoteDescription(new RTCSessionDescription(offer),function(){
pc2.createAnswer(function(answer){
pc2.setLocalDescription(answer,function(){
pc1.setRemoteDescription(new RTCSessionDescription(answer),new Function()/*I don't need you, bro*/,onError);
},onError);
},onError);
},onError);
},onError);
},onError);
BTW I'm developing with Google Chrome. I will make sure it also runs in Firefox, but right now the problem should be cross-browser. I want to make it to data channels before... (but I have nothing against a working solution with Firefox or cross-browser code)

In Chrome 38 and earlier, OfferToReceiveAudio defaulted to true. Starting from Chrome 39, OfferToReceiveAudio defaults to false, as announced by a WebRTC engineer at PSA: Behavior change to PeerConnection.createOffer constraint OfferToReceiveAudio (quoted below).
Because of this change, the SDP returned by createOffer does not contain any media, and therefore the ICE gathering process never starts. You can notice the consequences of this change by observing that the ICE events are never triggered, and the PeerConnection's iceGatheringState and iceConnectionState stay "new".
To make sure that the ICE gathering starts and complete, you have to add media to your offer, e.g. by setting OfferToReceiveAudio:true in the following constraints to your offer (either as a parameter of the PeerConnection constructor, or as a parameter to the peerConnection.createOffer method):
{
mandatory: {
OfferToReceiveAudio: true
}
}
(other ways to get media in the SDP include setting OfferToReceiveVideo:true, or calling peerConnection.addStream with a media stream that you got from getUserMedia)
webrtc-discuss: PSA: Behavior change to PeerConnection.createOffer constraint OfferToReceiveAudio:
I'm going to submit a change (https://webrtc-codereview.appspot.com/16309004/) to change the behavior of RTCPeerConnection.createOffer.
The change is expected to be included in Chrome M39.
What's changed:
Currently if the OfferToReceiveAudio constraint is not specified in PeerConnection.createOffer, the resulted offer SDP will have an "m=audio" line even if there is no audio track attached to the PeerConnection. In other words, OfferToReceiveAudio defaults to true.
After my change, OfferToReceiveAudio no longer defaults to true. Whether the offer SDP has an "m=audio" line depends on if any audio track has been attached to the PeerConnection.
What's not changed:
The behavior of setting an explicit value for OfferToReceiveAudio remains the same, i.e. OfferToReceiveAudio:true will result in an "m=audio" line regardless of the existence of audio tracks; OfferToReceiveAudio:false will result in no "m=audio" line regardless of the existence of audio tracks, unless setLocalDescription has been called with an SDP containing an "m=audio" line, in which case the new offer SDP will mark the audio content inactive instead of removing the audio content.

The above solution of Rob W from Jan 3 '15 worked for me at first but led to another problem.
I included {'offerToReceiveAudio':true,'offerToReceiveVideo':true} in the createOffer and createAnswer calls and onIceCandidate was called as described.
However, if I got it right, offerToReceive... means to only receive but not to send a stream. I figured that out because of the a=recvonly in the sdp offer and the a=sendonly in the sdp answer. As a result, only the caller could see the callees video but not vice versa.
As Rob states correctly:
other ways to get media in the SDP include [...] calling peerConnection.addStream with a
media stream that you got from getUserMedia
Adding the stream was what I had done already in the first place. However, my sending of the sdp happened before that adding because my logical flow was mixed up. Bringing that into the right order (addStream -> sdp = getLocalDescription -> send(sdp)) and removing the offerOptions did the trick for me.
Hope this helps anyone.

Solution in 2020
You have to do 2 things:
include offerToReceiveAudio and offerToReceiveVideo when create RTCPeerConnection
add addTrack
Sample code:
const peerConnection= new RTCPeerConnection({
configuration: {
offerToReceiveAudio: true,
offerToReceiveVideo: true
},
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
})
localStream.getTracks().forEach(track => {
peerConnection.addTrack(track, localStream)
})
const offer = await peerConnection.createOffer()
await peerConnection.setLocalDescription(offer)
peerConnection.onicecandidate = event => {
if (event.candidate) {
console.log('Ice candidate: ', event.candidate)
}
}
... other codes

I found a solution. If you add this code after the config declaration:
var mediaConstraints = {
optional: [],
mandatory: {
OfferToReceiveAudio: true,
OfferToReceiveVideo: true
}
};
and modify the last line like this
},onError,mediaConstraints);
it just works. Don't ask me why.

The doc page should be used as the reference: https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/createOffer
const offerOptions = {
offerToReceiveAudio: true,
offerToReceiveVideo: true
}
myPeerConnection.createOffer ( offerOptions )