WebRTC sequence of events - javascript

Is it mandatory for the WebRTC sequence of events be allowed to execute in the order depicted in this diagram?
WebRTC Flow
The reason I ask is that my one on one WebRTC chat performs flawlessly, however when a third participant enters the video conference, frequently I find one of the RTCPeerConnection shows iceConnectionState: "failed".
I do find that the remote stream is received before I process the Answer and before ICE Candicates are received.
How the following sequence be strictly enforced:
1) Send offer from caller.
2) Receive offer on callee.
3) Process offer on callee, generate answer and send to caller.
4) Receive and process offer on caller.
5) Send ICE candidates from caller to callee.
6) Process ICE candidates on callee.
7) Send ICE candidates from callee to caller.
The debug console shows that ICE candidates start getting generated as soon as the localmedia stream is added to the RTCPeerConnection. Should they be sent over immediately or wait till the sequence allows for sending?

May be too late but for the records:
Yes, the RTCPeerConnection is based on sequence. You can refer here for good explanation. I too struggled to control the flow and errors didn't mean much.
Based on my successful research/implementation (Chrome v74), to enforce sequence do (you can also see any sample online or here):
Send CreateOffer from Caller (signalling part; custom transport to the other users); enable PeerConnection.onnegotiationneeded => CreateOffer() only for Caller
Callee can be ready with video/audio streams added it to RTCPeerConnection on its side, then accepts offer and sends answer to Caller (signalling part)
Caller gets the answer and adds the video/audio streams to RTCPeerConnection conditionally when PeerConnection.connectionState == "new", the tricky parts: condition connectionState == "new" ensures it does not add again as offer / answer might be exchanged many times with state being connecting etc. Another is, if you add video/audio streams before this step, it will be hard to control the ice sdp flowing and raising errors at wrong states (relates to comment from Benjamin Trent, here) (signalling part)
since offer, answer is exchanged, now many ice events flows are exchanged (signalling part)
Connection should be established (assuming STUN configured and if peer to peer not possible, TURN required)
I hope I have highlighted few points missing online when you get hands on into this.

Related

what triggers the webRTC API to connect to stun server

Don't know if this example is correct
The process, call new RTCPeerConnection() then createOffer() then setLocalDescription()
Then I wait for onicecandidate take what it gives and first send the offer and second the icecandidates through the signal server to the other peer
Then the other peer takes the received offer into setRemoteDescription(offer) then the received icecandidates into addIceCandidate(icecandidates) then calls createAnswer() this gives an answer to put in setLocalDescription(answer) this triggers onicecandidate take these icecandidates with the answer=offer and send them back to the other peer
The other peer takes the answer into setRemoteDescription(answer) then the received icecandidates into addIceCandidate(icecandidates)
I think in this example the connection will work when testing inside local network but what if it doesn't because its not a local network, at what step in this example will the API call the STUN server and what other functions do I need to call if it does call the STUN server?
I've found that one way to generate BIND requests to be sent to the STUN server right away is to set the iceCandidatePoolSize option in the configuration to be > 0.
config = {iceServers: [{urls:stun:stunserver.stunprotocol.org}], iceCandidatePoolSize: 1};
peerConnection = new RTCPeerConnection(config); // pretty much starts to resolve the DNS name and sends BIND requests right away.
Hope this helps.
Also: this link is chock-full of great suggestions to troubleshoot webrtc connections.
You need to specify a STUN server in the peer connection's configuration. E.g.:
pc = new RTCPeerConnection({iceServers: [{urls: "stun:stun.1.google.com:19302"}]});
There are no other methods to call, provided it works on a LAN already. You should see additional calls to onicecandidate from this, compared to before. That's it.
Note that a couple of the things you describe happen in parallel, but in short, what triggers the browser to connect to the STUN server is setLocalDescription. It causes the browser's built-in ICE agent to kick off its candidate gathering process for this connection, and STUN is part of that.

Is it possible to convert a WebRTC SDP offer to answer?

I have two peers that want to connect to each other via WebRTC. Typically the first peer would create an offer and send it to the second via a signalling channel/server, the second peer would respond with an answer. This scenario works fine.
However, is it possible to support the case where both peers happen to try to connect to each other simultaneously both sending SDP offers to one another concurrently via the signalling server.
// Both peers do this simultaneously:
const conn = new RTCPeerConnection(null);
const sdpOffer = await conn.createOffer();
await conn.setLocalDescription(sdpOffer);
signalingService.send(peerId, sdpOffer);
// At some point in the future both peers also receive an SDP offer
// (rather than answer) from the other peer whom they sent an offer to
// via the signaling service. If this was an answer we'd call
// RTCPeerConnection.setRemoteDescription, however this doesn't work for an
// offer:
conn.setRemoteDescription(peerSDPOffer);
// In Chrome results in "DOMException: Failed to execute 'setRemoteDescription' on 'RTCPeerConnection': Failed to set remote offer sdp: Called in wrong state: kHaveLocalOffer"
I even tried to "convert" the received peer offers into answers by rewriting the SDP type from offer to answer and setup:actpass to setup:active but that doesn't seem to work, instead I just get a new exception.
So the question is, is this simultaneous connect/offer use case supported in some fashion - or should I close one side/peer RTCPeerConnection & instantiate a new one using RTCPeerConnection.createAnswer this time?
This situation is known as "signaling glare". The WebRTC API does not really define how to handle this (except for something called "rollback" but it is not implemented in any browser yet and nobody has missed it so far) so you have to avoid this situation yourself.
Simply replacing the a=setup won't work since the underlying DTLS mechanism still needs the concept of a client and a server.
The answer for how to avoid glare these days is to use the Perfect Negotiation Pattern: https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Perfect_negotiation
However, what OP described does work with the slight modification of setting setup:active on one peer and setup:passive on the other:
https://codepen.io/evan-brass/pen/mdpgWGG?editors=0010
It might not work for audio / video connections (because those may require negotiating codecs), but I've tested it on Chrome / Firefox / Safari for DataChannel only connections.
You could choose which peer is active and which is passive using whatever system you use to determine 'politeness' in perfect negotiation. One possibility would be to compare the DTLS fingerprints and make whichever one is larger the active peer.

Webrtc Call gets connected even if I'm not adding ice candidate in peer candidate

const pc1 = new RTCPeerConnection(null);
const pc2 = new RTCPeerConnection(null);
async function call(){
const offer = await pc1.createOffer();
pc1.setLocalDescription(offer);
pc2.setRemoteDescription(offer);
const answer = await pc2.createAnswer();
pc2.setLocalDescription(answer);
pc1.setRemoteDescription(pc2.localDescription);
}
async function showVideo(){
const config={audio:true,video:true};
stream = await navigator.mediaDevices.getUserMedia(config);
pc1.addStream(stream);
}
This is simplified version of my code.
Now in this code I'm not adding icecandidate to the peer neither listening to onicecandidate. But if I invoke call() twice my connection gets established.
I used event handler for iceconnectionstate change and found that when I invoke call for first time it iceconnection state is in checking condition and when I invoke it for second time it gets called and state became completed.
So I'm wondering how does even without adding icecandidate to another peer how checking gets initiated and gets connected for the second time?
Four reasons: A bug in your code, a behavior of Chrome, no firewall, and something about how trickle ICE works.
1) A bug in your code
First things first: The following sets pc1.setRemoteDescription(null):
pc2.setLocalDescription(answer);
pc1.setRemoteDescription(pc2.localDescription); // pc2.localDescription == null here
...because setLocalDescription is an asynchronous method that doesn't complete immediately.
Now pc2.localDescription does eventually get set, so the second time you invoke call() it's there, and negotiation works.
To fix this you must wait on the promise using await or then:
await pc2.setLocalDescription(answer);
pc1.setRemoteDescription(pc2.localDescription); // pc2.localDescription is set!
2) No ICE server needed if there's no NAT.
The browser can communicate with other machines on the same LAN, or itself, using "host" candidates (your machine's IP). No ICE servers needed to discover those.
3) Trickle ICE is an optimization.
The signaling (trickling) of individual ice candidates using onicecandidate, is an optimization meant to speed up negotiation. Once setLocalDescription succeeds, the browser's internal ICE Agent starts, inserting ICE candidates, as they're discovered, into the localDescription itself. Wait a few seconds to negotiate, and trickling isn't necessary at all: all ICE candidates will be in the offer and answer transmitted.
4) An interesting behavior in Chrome.
I suspect a race where the second time you invoke call() Chrome's ICE agent remembers the host candidates it has collected from last time, and inserts them into the offer and localDescription immediately, before setLocalDescription's success callback has run to completion. This might be a bug, or it may be how the spec says it should work. In any case, the behavior seems to differ depending on browser atm, so I wouldn't rely on it today.
Steps to Reproduce / Proof of ICE in SDP
Run this fiddle in Chrome and Firefox.
Click The Call! button once, then once more.
Observe Firefox: you'll see 0 candidates (output twice); both times; no connection.
Observe Chrome: it connects with 3 candidates the second time, with candidates.
Now uncomment the line // await wait(2000);
Both browsers now connect, with 4 or 8 candidates after 2 seconds.
The actual number of candidates may vary on your system, but the behavior shouldn't.
When I run this in Firefox, It doesn't connect either time, unless I modify it to wait or trickle candidates).
Browser update: Both browsers have bugs: Firefox is too restrictive and Chrome is too lenient in recovering from ICE failure.

What is the proper way to reject a WebRTC offer?

In WebRTC, there seems to be a very well-defined order in which things happen.
Locally I use getUserMedia to get my local stream, and save the stream to a variable. I create an RTCPeerConnection object, which I name pc, and I add the local stream to it. I add an onaddstream event handler to pc, so that I can save the remote user's stream to a variable, and eventually set it as the src attribute of an HTML element like audio. I also set onicecandidate event handler on my pc to handle ice candidates.
At this point, there is an RTCPeerConnection, but no remote user "connected yet". This is where the "offer/answer" starts. Let's say I'm using websockets for signaling and I receive an offer, which is a message called 'offer' and containing an SDP object. How do I reject it and how should this be dealt with on both endpoints?
For instance, I could send a message 'reject' that would be relayed to the other user. My RTCPeerConnection still exists, and maybe I want to be able to receive other calls. As is, I don't have to do anything to my RTCPeerConnection, correct? Does the other user, who sent the offer, have to do anything? Does he have to close that particular RTCPeerConnection? I would think not, since all he did was create an SDP object, and then outside of WebRTC, through websockets, sent the object over to the other user. He did add the offer using setLocalDescription though. When the offer is rejected, does he need to do anything about this?
When I create an offer, and send it to the other user, if I never get an answer back, can I just send an offer to a third user and then if he sends an answer I'm connected with him?
I haven't found anything about the lifecycle of an RTCPeerConnection.
Proper (spec) way to reject media
The "proper" way to "reject" offered media in an answer hasn't been implemented in any browser yet:
pc.ontrack = e => e.transceiver.stop();
Basically, the WebRTC 1.0 spec has changed quite significantly in this area. In short, a transceiver is an object combining one sender and one receiver, each sending or receiving a single track.
transceiver.stop() lets you reject a single bidirectional m-line (negotiated media) in the signaled SDP media description. E.g. you can reject parts of an offer in your answer, without rejecting the whole thing.
Today
Today, the only way to reject individual m-lines is through mangling the SDP offer/answer manually.
But it sounds like you're not actually asking about that at all. Instead, it sounds like you're asking how to bail out of incomplete signaling and roll a peer connection back to "stable" state.
Rollback to stable state
The offer/answer negotiation cycle is a state machine. The state is pc.signalingState:
You asked if one side walks away from the negotiation, does either side need to do anything before they can re-purpose their connection object for a new attempt with the same or different peer. Well, it depends.
If you've only called createOffer then no rollback of state is necessary, since createOffer is not in the above diagram.
If you've called setLocalDescription however, then you're now in "have-local-offer" state, which means you do need to somehow get back to "stable" state before you can reuse the connection.
Your options are to either finish the negotiation, delete the connection, or rollback to stable state (currently only supported in Firefox, though it is in the spec):
let pc = new RTCPeerConnection();
pc.onnegotiationneeded = async e => {
try {
await pc.setLocalDescription(await pc.createOffer());
console.log(pc.signalingState); // have-local-offer
await pc.setLocalDescription({type: "rollback"});
console.log(pc.signalingState); // stable
} catch(e) {
console.log(e);
}
}
pc.createDataChannel("dummy");
<script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
Of course, you should also let the peer know, usually through out-of-band signaling.
Why this is typically not a problem.
In typical cases, you own the JavaScript on both ends, so this doesn't come up. In other words, the desire to have a connection usually precedes one being made.

WebRTC: How to add stream after offer and answer?

I am working on webRTC video calling. I got datachannel successfully implemented. Now I would like to add video stream to the same peer connection.
I have read that stream should be added before answer and offer. Is there a way to add stream after answer or offer?
In case I have added stream before offer or answer, how could I stop streaming and start it again when needed?
Could there be any issues in maintaining so many streams?
To add stream after creating complete signalling, the Peer connection should renegotiate with stream.
pc1.addstream(stream)
Then once again create offer and send it to other Peer.
Remote peer will add stream and send answer SDP.
To stop streams:
stream.stop();
pc1.removeStream(stream);
In my experience, what Konga Raju advised didn't work. I couldn't send an "updated offer" and have the video streaming actually happen.
I found that this sequence of events works for my case, in which I wish to stream video from peer 1 to peer 2.
set up some way for the peers to exchange messages. (The variance in how people accomplish this is what makes different WebRTC code samples so incommensurable, sadly.)
On each side, set up handlers for the important signalling events. (Some folks have reported that you need to create these handlers at special times, but I haven't found that to be the case.
) There are 3 basic events:
an ice candidate sent from the other side ==> call addIceCandidate with it
an offer message ==> SetRemoteDescription & make an answer & send it
an answer message ===> SetRemoteDescription
On each side, create the peerconnection object with the event handlers we care about: onicecandidate, onremovestream, onaddstream, etc.
ice candidate pops out of the peerconnection object ===> send it to other side
When both peers are present and all the handlers are in place, peer 1 gets a trigger message of some kind to start video capture (the getUserMedia call)
Once getUserMedia succeeds, we have a stream. Call addStream on the peer connection object.
Then peer 1 makes an offer
Due to the handlers we set up earlier, peer 2 sends an answer
Concurrently with this (and rather opaquely), the peer connection object starts producing ice candidates. They get sent back and forth between the two peers and handled (steps 2 & 3 above)
Streaming starts by itself, opaquely, as a result of 2 conditions:
offer/answer exchange
ice candidates received, exchanged, and handled
I haven't found a way to add video after step 9. When I want to change something, I go back to step 3.
MediaStream should be added to peerconnection first only then exchange of offer, answer ,candidates should be done. If the onAddStream() is called ,that mean you are receiving the remote video.

Categories

Resources