Develop Reference

AWS Rekognition timeout error happening randomly

I have an application in Electron that does facial recognition of people to then decide whether or not they can enter the place and for that I'm using Amazon Rekognition.
Everything was working fine (for a few months) until, two days ago, a customer reported to me that the app was behaving strangely, like it wasn't responding to requests for facial recognition.
After several tests, I discovered that what is happening with it is a timeout error, which occurs in all API calls, whether they are looking for faces (SearchFacesByImage) or registering new faces (IndexFaces).
The error says:
{
"message": "connect ETIMEDOUT 3.226.60.54:443",
"errno": -4039,
"code": "TimeoutError",
"syscall": "connect",
"address": "3.226.60.54",
"port": 443,
"time": "2022-12-14T13:50:10.909Z",
"region": "us-east-1",
"hostname": "rekognition.us-east-1.amazonaws.com",
"retryable": true
}
What intrigued me was the fact that everything was working fine, until this behavior just started happening (and I didn't make any code changes/updates to the app running on my client's computer).
And what makes me even more intrigued is that this behavior occurs completely randomly and only on the machine of that client in question. Sometimes the API calls work correctly (returning whether the person was recognized or not), but most of the time, the calls take about 90 seconds to return the timeout error. When executing the same code on my machine (same methods and same CollectionId) everything runs normally and there was no timeout error at any time - while at the exact same moment on my client's machine the behavior continues.
I was using aws-sdk and then switched to #aws-sdk/client-rekognition (thinking that could solve the problem) but the code only worked on a few of the first calls to the API and a few minutes later it got the timeout errors again.
The code I'm using to configure and make calls to Rekognition is basically this:
const { RekognitionClient, IndexFacesCommand, SearchFacesByImageCommand } = require('#aws-sdk/client-rekognition')
const rekognitionClient = new RekognitionClient({
credentials: {
accessKeyId: 'accessKeyId',
secretAccessKey: 'secretAccessKey'
},
region: 'us-east-1'
})
const registerFaceOnRekognition = async (bytes, userId) => {
const params = {
CollectionId: 'collectionId',
Image: { Bytes: bytes },
ExternalImageId: userId,
MaxFaces: 1,
QualityFilter: 'HIGH'
}
const command = new IndexFacesCommand(params)
try {
const { FaceRecords } = await rekognitionClient.send(command)
if (!FaceRecords.length) {
console.log('No faces detected.')
return
}
console.log('Face created:')
console.log(FaceRecords[0].Face.FaceId)
} catch (error) {
console.error(error) // timeout error
}
}
const searchFaceByImageOnRekognition = async (bytes) => {
const params = {
CollectionId: 'collectionId',
Image: { Bytes: bytes },
MaxFaces: 1,
FaceMatchThreshold: 99,
QualityFilter: 'HIGH'
}
const command = new SearchFacesByImageCommand(params)
try {
const { FaceMatches } = await rekognitionClient.send(command)
if (!FaceMatches.length) {
console.log('This face has not been registered yet')
return
}
console.log('Face found:')
console.log(FaceMatches[0].Face.ExternalImageId)
} catch (error) {
console.error(error) // timeout error
}
}
// Method called through the renderer process that has a canvas where the webcam view is reproduced
const onTakePicture = (event, data) => {
const bytes = Buffer.from(data.dataURL.replace('data:image/jpeg;base64,', ''), 'base64')
// If there is a userId, register the face in the image
if (data.userId) {
registerFaceOnRekognition(bytes, data.userId)
return
}
// Else, search for the face in the image
searchFaceByImageOnRekognition(bytes)
}
Just remembering that: during all tests on my client's computer the internet connection was stable and working properly.
What is the best way to investigate and resolve this issue?
UPDATE:
I enabled Rekognition debug logs and they can be found at: https://gist.github.com/IgorSamer/4e58e09f3fa615401f85ca325b794245
In it, the first three requests (2022-12-16T13:48:45.932Z, 2022-12-16T13:53:20.325Z and 2022-12-16T14:19:12.479Z) occur normally. However, all other consecutive requests start to give the timeout error, where, in fact, no data is returned after the [DEBUG] App: endpoints Resolved endpoint: step.
As previously mentioned the internet connection is working fine. I could also managing to reproduce the error via remote access, that is, the machine internet was ok at the time of error.
Is there a possibility that there is a block made by my client's firewall/network that prevents requests from being sent by the SDK after a few successful requests? If yes, what is the best way to investigate this?

Exploration
This is what I would do initially to gather some info:
Verify if this is happening ALL the time with that specific client.
Verify if this is happening ONLY with one client, or more.
Verify if this is happening in one or multiple regions (i.e us-east-1).
Verify if Amazon Recognition has had/or has issues in the affected region during the time window of interest.
Check Recognition's status in the Health dashboard in your AWS console: link
Use AWS Recognition Guidelines and Quotas as a reference to determine if your app/service usage of Recognition is under the set limits.
Note there's a limit on TPS per resource (i.e SearchFacesByImage, IndexFaces) per account.
Possible approaches
Verify if there was a change in the client network/firewall. Just ask.
Replicate your app's API call with AWS CLI and study logs.
Access remotely to your client's device.
Setup temporal AWS credentials (remember to remove access after the test)
Send an API call to the Recognition endpoint. Note that even a 4XX error will be good news, as you got at least some response.
Set up proper logging for your app (as CloudWatch logs may not be enough to troubleshoot).
Check Splunk's APM and NewRelic's APM
I hope this may be of help to at least create a troubleshooting strategy

Reconnect to Laravel Echo server after session disconnection

I am attempting to write an web application with a persistent echo connection to a laravel-echo-server instance, which needs to detect disconnections and attempt to reconnect gracefully. The scenario I am attempting to overcome now is a user's machine has gone to sleep / reawoke and their session key has been invalidated (echo server requires an active session in our app). Detecting this situation from an HTTP perspective is solved - I setup a regular keepAlive, and if that keepAlive detects a 400-level error, it reconnects and updates the session auth_token.
When my Laravel session dies, I cannot tell that has happened from an echo perspective. The best I've found is I can attach to the 'disconnect' event, but that only gets triggered if the server-side laravel-echo-server process dies, rather than the session is invalid:
this.echoConnection.connector.socket.on('connect', function() {
logger.log('info', `Echo server running`);
})
this.echoConnection.connector.socket.on('disconnect', function() {
logger.log('warn', `Echo server disconnected`);
});
On the laravel-echo-server side, I can tell that the connection is dead - it will show this error:
⚠ [7:03:30 PM] - 5TwHN2qUys5VEFP5AAAG could not be authenticated to private.1
I cannot figure out how to catch this failure event programmatically from the client. Is there a way to capture it? Again, I can tell the session is dead eventually because I poll the server regularly via a http keepAlive function, but I would definitely also like to tell directly from the echo connection if possible, as it polls at a much higher natural rate.
As a second (more important) question, if I detect that my session has died, what should I do to recycle the echo connection (after I have logged in again via HTTP and gotten a new auth_token)? Is there anything specific I should call / etc? I've had some success calling disconnect() then setting up the connection again from scratch, but I do see errors such as:
websocket.js:201 WebSocket is already in CLOSING or CLOSED state.
Here is my current (naive) reconnection code, which is my initial connection code with an attempt to disconnect first stapled onto it:
async attemptEchoReconnect() {
if (this.echoConnection !== null) {
this.echoConnection.disconnect();
this.echoConnection = null;
}
const thisConnectionParams = this.props.connections[this.connectionName];
const curThis = this;
this.echoConnection = new Echo({
broadcaster: 'socket.io',
host: thisConnectionParams.echoHost,
authEndpoint: 'api/broadcasting/auth',
auth: {
headers: {
Authorization: `Bearer ` + thisConnectionParams.authToken
}
}
});
this.echoConnection.connector.socket.on('connect', function() {
logger.log('info', `Echo server running`);
})
this.echoConnection.connector.socket.on('disconnect', function() {
logger.log('warn', `Echo server disconnected`);
});
this.echoConnection.join('everywhere')
.here(users => {
logger.log('info', `Rejoined presence channel`);
});
this.echoConnection.private(`private.${this.props.id}`)
.listen(...);
setTimeout(() => { this.keepAlive() }, 120 * 1000);
}
Any help would be so great - these APIs are not well documented to the end that I really want, and I am hoping I can get some stability with this connection rather than having to do something ugly like force restart.

For anyone who needs help with this problem, my above echo reconnection code seems to be pretty stable, along with a keepAlive function to determine the state of the HTTP connection. I am still a bit uncertain of the origin of the console errors I am seeing, but I suspect they have to do with connection loss during a sleep cycle, which is not something I am particularly worried about.
I'd still be interested in hearing other thoughts if anyone has any. I am somewhat inclined to believe long-term stability of an echo connection is possible, though it does appear you have to proactively monitor it with what tools you have available.

WebRTC stuck in connecting state

I have successfully communicated the offer, answer and ice candidates for a WebRTC connection from A to B. At this point, the connection is stuck in the "connecting" state. The initiator (A) seems to timeout or something after a while and switch to the "failed" state, whereas its remote (B) is staying in the "connecting" state permanently.
Any help would be very appreciated.
Creation of peer (A and B):
let peer = new RTCPeerConnection({
iceServers: [
{
urls: [
"stun:stun1.l.google.com:19302",
"stun:stun2.l.google.com:19302",
],
},
{
urls: [
"stun:global.stun.twilio.com:3478?transport=udp",
],
},
],
iceCandidatePoolSize: 10,
});
Creating offer (A):
peer.onnegotiationneeded = async () => {
offer = await peer.createOffer();
await peer.setLocalDescription(offer);
};
Collecting ice candidates (A):
peer.onicecandidate = (evt) => {
if (evt.candidate) {
iceCandidates.push(evt.candidate);
} else {
// send offer and iceCandidates to B through signaling server
// this part is working perfectly
}
};
Creating answer and populating ice candidates (B):
await peer.setRemoteDescription(offer);
let answer = await this._peer.createAnswer();
await peer.setLocalDescription(answer);
// send answer back to A through signaling server
for (let candidate of sigData.iceCandidates) {
await peer.addIceCandidate(candidate);
}
On answer from B through signaling server (A):
await peer.setRemoteDescription(answer);
Detect connection state change (A and B):
peer.onconnectionstatechange = () => {
console.log("state changed")
console.log(peer.connectionState);
}
Also note that there were two occasions where it connected successfully, but I am yet to see it work again.
EDIT: I forgot to mention I am also creating a data channel (the onicecandidate event doesn't seem to call without this). This is called immediately after the RTCPeerConnection is constructed and any event handlers have been attached.
let channel = peer.createDataChannel("...", {
id: ...,
ordered: true,
});
EDIT 2: As #jib suggested, I am now also gathering ice candidates in B and sending them back to A to add. However, the exact same problem persists.
EDIT 3: It seems to connect the first time I hard reload the webpage for A and the webpage for B. Connection stops working again until I do another hard reload. Does anyone have any ideas why this is the case? At least I should be able to continue development for the time being until I can figure out this issue.
EDIT 4: I removed the iceServers I was using and left the RTCPeerConnection constructor blank. Somehow it is much more reliable now. But I am yet to get a successful connection on iOS Safari!

Open this in two browser windows and hit the Connect button in one of them. This is the code:
const pc = new RTCPeerConnection();
call.onclick = async () => {
const stream = await navigator.mediaDevices.getUserMedia({video:true,audio:true})
video.srcObject = stream;
for (const track of stream.getTracks()) {
pc.addTrack(track, stream);
}
};
pc.ontrack = ({streams}) => video.srcObject = streams[0];
pc.oniceconnectionstatechange = () => console.log(pc.iceConnectionState);
pc.onicecandidate = ({candidate}) => sc.send({candidate});
pc.onnegotiationneeded = async () => {
await pc.setLocalDescription(await pc.createOffer());
sc.send({sdp: pc.localDescription});
}
const sc = new localSocket(); // localStorage signaling hack
sc.onmessage = async ({data: {sdp, candidate}}) => {
if (sdp) {
await pc.setRemoteDescription(sdp);
if (sdp.type == "offer") {
await pc.setLocalDescription(await pc.createAnswer());
sc.send({sdp: pc.localDescription});
}
} else if (candidate) await pc.addIceCandidate(candidate);
}
This is the same source for A and B. Replace the localSocket hack with your preferred signaling channel (e.g. websocket).
Don't cache ICE candidates since that defeats the purpose of Trickle ICE. It may appear fast locally, but in real networks ICE may take time.
In fact, sending candidates is meaningless if you delay sending the offer/answer until all local candidates have been gathered, since candidates are already embedded in the offer/answer (pc.localDescription) at that point.

Finally! After a few weeks I have figured out the issue, which wasn't apparent in the code I included in my question, but may still be useful for anyone who is having similar problems.
I assumed that the ice gathering was being completed after the onnegotiationneeded event fired and the offer/answer was created.
Because of this incorrect assumption, I was signaling the offer/answer along with the ice candidates at this stage, but very frequently (always in iOS Safari from my experience) the offer/answer was not yet created at this point.
I solved this by creating two promises for a) the completion of ice candidate gathering, and b) the creation of the offer/answer. I used Promise.all on the two promises, and when they both completed, I sent the ice candidates and offer/answer down through the signaling server all at once.
This works, but of course in the future I should "trickle" this information, by sending bits and pieces as they come, instead of waiting for everything to fully complete. But I'll worry about that in the future, since at the moment I am using HTTP requests, and it is too much hassle.
EDIT: My connection is still always stuck when iceServers are included, so I've created a new question. But local connections when no iceServers are included are now 100% fully reliable :)

Passing extra arguments to callback function

websocketServer.on('connection', function(socket, req) {
socket.on('message', onMessage);
sub.subscribe('chat'); // sub: Redis subscription connection
sub.on('message', onSubMessage);
});
function onMessage(message) {
pub.publish('chat', message); // pub: Redis publishing connection
}
function onSubMessage(channel, message) {
// how to access 'socket' from here?
if (channel === 'chat') socket.send(message);
}
I'm trying to get away with as few state & bindings as possible, to make WS server efficient & to have the ability to just add more machines if I need to scale - state would only make this harder. Im still not understanding everything about Node memory management & garbage collection.
What would be the recommended solution in this case? Move onSubMessage into connection callback to access socket? But function would be then initialized on every connection?
What other choices do I have?
Little background about this:
The user opens a WebSocket connection with the server. If the user sends a message, it gets sent to Redis channel (might also know it as Pub/Sub topic) which broadcasts it to every subscribed client (onSubMessage). Redis Pub/Sub acts as a centralized broadcaster: I don't have to worry about different servers or state, Redis sends a message to everybody who is interested. Carefree scaling.

You can use bind() to pre-define an extra argument to the callback function:
...
sub.on('message', onSubMessage.bind(sub, socket));
...
function onSubMessage(socket, channel, message) {
if (channel === 'chat') socket.send(message);
}
This does create a new function instance for every new connection, so there is probably not a real upside to using a wrapping function in terms of memory usage.

Replacing Socket io with Pubnub (WebRTC)

I am looking into using Pubnub's service to set up WebRTC connections between peers for video.
With this I am hoping to avoid using socket io which is what I am currently using, although I just cannot find any good examples that demonstrate how to do this.
Right now socket io is handling the events emitted from the client and the server. From what I understand, the current node js server would no longer need to handle any of the emitted events since socket io would not be used but this is what I am having problems with. I am not sure how to set up the clients to signal each other the information that they require (who to connect to, etc)
Are there any simple examples or implementations of pubnub being used instead of socket io for a project or perhaps someone can shed some light on something I may not be seeing, thanks!
edit: Also with anyone experienced in Pubnub, is what I am trying to do even possible haha

WebRTC Signaling Exchanging ICE Candidates via PubNub
The goal is to exchange ICE candidate packets between two peers. ICE candidate packets are structured payloads which contain possible path recommendations between two peers. You can use a lib which will take care of the nitty gritty such as http://www.sinch.com/ and below is the general direction you want to take:
Signaling Example Code Follows
<script src="http://cdn.pubnub.com/pubnub-3.6.3.min.js"></script>
<script>(function(){
// INIT P2P Packet Exchanger
var pubnub = PUBNUB({
publish_key : 'demo',
subscribe_key : 'demo'
})
// You need to specify the exchange channel for the peers to
// exchange ICE Candidates.
var exchange_channel = "p2p-exchange";
// LISTEN FOR ICE CANDIDATES
pubnub.subscribe({
channel : exchange_channel,
message : receive_ice_candidates
})
// ICE CANDIDATES RECEIVER PROCESSOR FUNCTION
function receive_ice_candidates(ice_candidate) {
// Attempt peer connection or upgrade route if better route...
console.log(ice_candidate);
// ... RTC Peer Connection upgrade/attempt ...
}
// SEND ICE CANDIDATE
function send_ice_candidate(ice) {
pubnub.publish({
channel : exchange_channel,
message : ice
})
}
Generate ICE Candidates Example Code Follows:
// CREATE ICE CANDIDATES
var pc = new RTCPeerConnection();
navigator.getUserMedia( {video: true}, function(stream) {
pc.onaddstream({stream:stream});
pc.addStream(stream);
pc.createOffer( function(offer) {
pc.setLocalDescription(
new RTCSessionDescription(offer),
send_ice_candidate, // - SEND ICE CANDIDATE via PUBNUB
error
);
}, error );
} );
// ERROR CALLBACK
function error(e) {
console.log(e);
}
})();</script>
More fun details await - https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection