For some reason on all my Bluemix services, I intermittently get the error "Cannot GET /pathname" on my node.js express services. This is intermittent (it works about 1/3 of the time). There is no error or logging shown in the application when this happens (however that response is coming from express I assume).
Any ideas? I have no idea how to progress here. The server has ample resources (memory + CPU).
I've seen this happen before when the user accidently has 2 different applications mapped to the same route/URL. The load balancer is hitting a different application on each time.
Try changing the route to something else and try to recreate the problem....myappname2.mybluemix.net
If that seems to fix it, log in to the UI and confirm that you do not have duplicate applications and that all applications have a unique route.
Related
I am running an Apollo Server with express to create an http server:
const express = require("express");
const cors = require('cors');
const app = express();
const server = new ApolloServer({ ... });
server.applyMiddleware({ app });
// enable pre-flight for cors requests
app.options('*', cors());
// Create the HTTP server
let httpServer = http.createServer(app);
httpServer.listen({ port: config.port });
Locally I can run the server and query it on Apollo Explorer without any issues.
However, when I deploy this server on dev environment, and try to access the Explorer page with the dev endpoint, I get a few errors.
The app.options() line with cors argument somehow seems to have solved part of them but not all.
Errors I am getting (on Dev Tools console):
Failed to load resource: net::ERR_CERT_AUTHORITY_INVALID
POST https://dev.endpoint.service/graphql net::ERR_CERT_AUTHORITY_INVALID
Errors I am getting (as popups on the Explorer page):
Unable to reach server
To diagnose the problem, please run:
npx diagnose-endpoint#1.0.12 --endpoint=https://dev.endpoint.service/graphql
I've tried running the command as instructed in the error and got this result:
Diagnosing https://dev.endpoint.service/graphql
Could not find any problems with the endpoint. Would you please to let us know about this > at explorer-feedback#apollographql.com
Frankly, I'm not even sure I understand the problem.
Am I getting these errors because, even though I launch an http server of Apollo without certificates, I am trying to access it via an https endpoint (which requires certificates)? I have to do this, service is stored in AKS cluster, which is only accessible through the endpoint I am calling. But every service that is already there is also an http service, not https, and is accessible through this same endpoint.
Also, even though these errors are showing up frequently, I am also able to query the server successfully most of the time on Explorer, and the data returned is exactly what I expected, which makes even less sense.
I am using edge browser but also tried chrome, and have the same issues.
How can an error like this be intermittent?
Without any intervention on my part, sometimes it's like this:
Any help, hints, ideas, please.
Thank you so much.
As much as it pains me to admit, it seems the issue is related to the VPN my company is using.
There were a few tells that pointed in this direction, once I started paying attention:
We can't access the endpoint I mentioned without the VPN turned on.
Other services in the AKS behave with the same error, if being called constantly through the same endpoint. I did not think to do that test at first, but when I realized that on Apollo server, the server is constantly doing the introspection thing to check the schema, it means it is being called more often than the other services that do not have this functionality.
We have some monitoring tools, to check the pod statuses and so on, and nothing indicated any problems in this service, or that it needed any kind of pod escalation (due to excessive number of requests).
I actually performed kubectl portforward test linking my localhost directly to the AKS cluster. Calling the service this way bypasses that endpoint which I am, under normal circumstances, forced to use before the request actually reaches the cluster. And I was simultaneously seeing on one window where I was calling the service the normal way showing that error on Apollo Studio, and at the same time on another Apollo Studio window performing the same request with this portforward bypass mechanic, and the latter was working just fine. If it really was a problem with the service, it would be down for both windows.
Other colleagues were testing the service at the same time as me and they were saying the service was working fine for them, until it wasn't. So every developer on my team could be accessing the service at the same time, and the error would just randomly show up for some, but not for others.
There are long periods where the error doesn't occur at all, like during lunch hours, or after work hours, and I assume the VPN traffic will be much lower during those hours.
I am working on a video/audio chat application in React and Django using Mesibo Javascript SDK.
It is working locally on the same network but when trying to connect through different networks, Firefox browser gives me this error
WebRTC: ICE failed, add a TURN server and see about:webrtc for more
details.
In Chrome, this error doesn't show up but still the video/audio chat is not working and gives me
Mesibo_OnCallStatus: 50.
I noticed that Mesibo uses stun:stun.l.google.com:19302 as the STUN server and tried changing it to stun:stun.l.google.com:19305 but of no use. I even created a numb.viagenie.ca account to add a TURN server which worked for a few minutes and then gave this error
webrtc: ice failed, your turn server appears to be broken.
When checked on https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/, chrome is giving error code:701 for all the above turn/stun servers whereas firefox doesn't.
As suggested by Mesibo Documentation, my website is fully secure with https and the Rest APIs are called from Django backend. I'm invoking https://api.mesibo.com/mesibo.js as a script tag in the index.html file in react. I'm also able to successfully get
Mesibo_OnConnectionStatus=1
and the incoming call notification is also working. Upon answering the call, the call status changes from 3 to 5 and gives this webrtc error after sometime without any video/audio rendered.
Please suggest any hint/solution.
To Enable webrtc for remote calling you need your own trun server.
Google don't provide free trun server.
Free Turn Servers are will not work well in production. Well Fell Free to try Your Luck on Them. Not All Are borken
You Need to create Your Own on Cloud like DigitalOcean,AWS.
Or For Tetsing purpose I have a Trick
Soloution.
If Your Doing the testing
Probably you can try to use some Google TURN servers that they use here
https://test.webrtc.org/, I just checked for browser console and found this:
enter image description here
Not Sure About LifeTime. But you will get that.
For Production I will suggest to used DigitalOcean 5$ plan which is very chip.
And you install your own TURN server using coturn on it
If you want You can also use
Node-turn
which is great for node backend.
There is also heroku option of setup
I don't know is there any django or python option.
(tip: try your to build turn server on hosting server. Make sure the port on which server is running is open)
I have an Express Node API on Heroku. It has a few open routes that makes upstream requests to another web server that then executes some queries and returns data.
So client->Express->Apache->DB2.
This has been running just fine for several months on a Heroku Hobby dyno. Now, however, we exposed another route and more client requests are coming in. Heroku is then throwing H12 errors (bc the Express app isn't giving back a response within 30S).
I'm using axios in the Express app to make the requests to Apache and get data. However, I'm not seeing anything fail in the logs. No errors are getting caught to give me some more details about why things could be timing out. Investigated the Apache->DB2 side of things and the bottleneck doesn't seem there and is almost certainly on the Express side of things.
Per Heroku's advice, I connected the app to NewRelic but haven't gained any insights yet. Sounds like this could be a scalability issue with Express and a high number of new requests coming in at a short period of time? There's not particularly that many. i.e ~50/min at highest. Would beefing up the Heroku dyno do anything? Any other ways to actually see what's going on with Express?
It seems like 10-15% of the client requests are receiving the timeout (and seems to happen at a time when there's lots of incoming requests)
Thanks!
I started learning meteor recently and I am developing an application that is similar to a bus timetable app; I have several screens (about 30 devices with browsers) and I connect them all to one localhost that runs my application on cmd (the app is offline using windows 10 with local MongoDB, no online server). Each set of screens display a relevant bus and its timetable, extracted from a MongoDB collection.
The app used to work just fine, but when I added 10 more screens, the clients seem to be disconnected from the server after a couple of minutes; I have meteor calls that display server time on screen, this time shows up as undefined when the server drops and I can't see any of my collections' documents via MeteorToys, I see the collections, but the documents are 0. Also, I can't log in to my admin page (which is a basic user interface that I made, simple MongoDB Accounts collection)
It's worth mentioning that the whole app does not crash; I can still navigate to my pages as my layout, HTML and CSS, still show up, just my server-related functionalities that stop.
I realize it's a traffic issue, as when I disconnect all of the screens and run the app, it works just fine. Also, when I reconnect one by one, it also seems to work fine.
I get no error on my console on the client, and on the server CMD, the application does not crash, it stays on with no error what's so ever.
Also, I added Meteor.status() ping to my console every second, and I get this
{status: "connected", connected: true, retryCount: 0}
which means technically the server is not offline?
I'm very lost, what can I do to rectify this?
Update:
I noticed that I had several ServerSessions that run on every second that get my time from the server. I changed them to normal Sessions and I'm now facing a different issue; I think there is a memory leak somewhere; when it freezes I noticed my RAM usage skyrocketed to 8GB (I have my max_old_space set to 8912 so it shouldn't be a problem) The normal usage was about when it runs is around 600-900MB
Then I get FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
=> Exited with code: 3
and Server Unreachable on the browser with 503 (Service Unavailable)
The server then restarts, and it does the same thing. Help :(
I experienced the same behavior with an app loading lots of data and doing computations. When the nodejs server CPU is on load, the app looses connexion to its node back end, and when the database server CPU is on load, the app looses connexion to the database.
Here are some advices:
Start by this reading if you haven't : https://bulletproofmeteor.com/ It has been written some years ago but still a lot of good practices inside
Monitor your app on your machine, try to understand what is the bottleneck: data transfers, disk reads, cpu intensive tasks ?
Look into your publication/subscription model. You may send too much data to the client, maybe you can limit the amount of data needed ?
if your setup is critical, don't run it on a single windows machine using the meteor command ! : start with building a proper nodejs app using meteor build https://guide.meteor.com/build-tool.html
use nginx to handle incoming connexion, it is robust and scalable
use a separate host for the database, and maybe a mongo replica set of 3 machines (this may solve your issue by giving more database availability, but I haven't tested yet)
Edit regarding the build: After building the app, you will obtain a folder. Here are the steps to launch the app in production mode:
cd into your meteor app folder
meteor build targetFolder --directory
cd targetFolder/bundle
(cd programs/server && npm install)
meteor node targetFolder/main.js #I use meteor node to ensure version compatibility
I'm quite new to Express.js and one of the things that surprised me more at first, compare to other servers such as Apache or IIS, is that Express.js server crashes every time it encounters an uncatched exception or some error turning down the site and making it accessible for users. A terrible thing!
For example, my application is crashing with a Javascript error because a variable is not defined due to a name change in the database table.
TypeError: Cannot call method 'replace' of undefined
This is not such a good example, because I should solve it before moving the site to production, but sometimes similar errors can take part which shouldn't be causing a server crash.
I would expect to see an error page or just an error in that specific page, but turning down the whole server for these kind of things sounds terrifying.
Express error handlers doesn't seem to be enough for this purposes.
I've been reading about how to solve this kind of things in Node.js by using domains, but I found nothing specifically for Express.js.
Another option I found, which doesn't seem to be recommended in all cases, is using tools to keep running a process forever, so after a crash, it would restart itself. Tools like Forever, Upstart or Monit.
How do you guys deal with this kind of problems in Express.js?
The main difference between Apache and nodejs in general is that Apache forks a process per request while nodejs is single threaded, hence if an error occurs in Apache then the process handling that request will crash while the others will continue to work, in nodejs instead the only thread goes down.
In my projects I use monit to check memory/cpu (if nodejs takes to much resources of my vps then monit will restart nodejs) and daemontools to be sure nodejs is always up and running.
I would recommend using Domains along with clusters. There is example in doc itself at http://nodejs.org/api/domain.html. There are also some modules for expressjs https://www.npmjs.org/package/express-domain-middleware.
So when such errors occur use of domain along with cluster will help us separate context of where error occur and will effect only single worker in cluster, we should be logging them and disconnect that worker in cluster and refork it. We can then read logs to fix such errors that need to be fixed in code.
I was facing the same issue and I fixed using try/cache like this. So I created different route files and included each route files in try/cache block like this.
try{
app.use('/api', require('./routes/user'))
}
catch(e)
{
console.log(e);
}
try{
app.use('/api', require('./routes/customer'))
}
catch(e)
{
console.log(e);
}