I am accessing an API that streams (RFC2616 for HTTP/1.1) the response. The stream never ends, but from time to time it throughs an error requiring the program to restart the request. I can't figure out how in NodeJS to restart the request without restarting the program. I've tried loops to check if an error occurred, but I end up creating a blocking code that stops the request methods. The code below works until an error occurs.
const request = require('request');
const options = {
method: 'GET',
url: 'https://api.tradestation.com/v3/marketdata/stream/quotes/IWM',
headers: {Authorization: 'Bearer ' + api.token,},
};
request(options)
.on('data', function (data) {
console.log(data.toString());
//Process data here
})
.on('error', function (err) {
console.error(err);
});
If it helps, the documentation to the API for streaming is here, and the API call is here.
On a side note, I tried using Axios but it doesn't seem to like endless streams (tried onDownloadProgress).
I would stick your request inside a function, you can then call it again on error.. I would also put a slight delay on the retry..
eg.
function callAPI() {
request(options)
.on('data', function (data) {
console.log(data.toString());
//Process data here
})
.on('error', function (err) {
console.error(err);
setTimeout(callAPI, 1000); //wait a second and retry.
});
}
Related
I am using a simple JS script to query a Postgres DB. I simply want to write a simple query to the webpage, but every time I think I'm calling the function, it calls twice
// query the db
rate = () => pool
.query(rateQuery)
.then(res => {return res.rows[0]})
.catch(err => console.log('error: ', err.stack))
const app = http.createServer((request, response) => {
// set response header
response.writeHead(200, { 'Content-Type': 'text/html' });
// get result of promise
r = rate()
.then(res => response.write(JSON.stringify(res), () => {console.log("DONE"); response.end()}))
.catch(err => console.log('error: ', err.stack))
});
app.listen(3000);
When the page is refreshed, it prints DONE twice but I only want it once, any help is much appreciated - thanks.
If you add console.log(request.url) in your request handler, you will likely see that that the second request is when the browser asks your server for the /favicon.ico that represents the site. Browsers do this. When a user types in a URL for a site and hits enter, the browser requests that URL from the target site and then it also asks for /favicon.ico if there wasn't a previously cached icon already for that site.
In general, you should not have an http request handler like this that pays no attention to the request URL path because then you will process anything that the browser or a search crawler or anything sends you. Instead, you should look for a specific URL path and only do your work when it's the desired path and return a 404 response for all other paths.
I'd suggest you change your request handler to this:
const app = http.createServer((request, response) => {
// set response header
if (request.url === "/") {
// get result of promise
rate().then(res => {
response.writeHead(200, { 'Content-Type': 'text/plain' });
response.write(JSON.stringify(res), () => {
response.end();
console.log("DONE");
});
}).catch(err => {
console.log('error: ', err.stack);
response.writeHead(500);
response.end();
});
} else {
response.writeHead(404, "unknown URL");
response.end();
}
});
Note: I change the content-type to text/plain. You had it as text/html, but you weren't sending html at all. You could perhaps make the content-type application/json, but in any case when you're sending JSON, it's not HTML.
This also incorporates cleaner error handling where all error paths send an error response.
P.S. Writing out this simple request handler using the plain http.createServer() request handler reminds me of how much simpler it is to use Express for request handler implementations. It handles so much of this for you while still giving you full control. It's lightweight and simpler.
FYI, here's the program written using Express:
const express = require('express');
const app = express();
app.get("/", (req, res) => {
rate().then(res => {
res.json(res);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
// if we get here, then no other request handler handled the incoming request
// so we send back a 404
app.use((req, res) => {
res.sendStatus(404);
});
app.listen(3000);
There is one possibility. rate already is a function which you defined as
// query the db
rate = () => pool
.query(..........
Therefore, I don't think you would need parenthesis when you call it. Change from this
// get result of promise
r = rate()
.then .........
to this
// get result of promise
r = rate
.then .........
I am making a call from my react app to a node server that needs to run a command on some physical hardware. The hardware takes about three minutes to complete its routine however the request always times out in my react app after only two. The hardware is connected over serial and will print "done" to the command line when finished so I need some way to see when that happens in my node server.
Ive tried increasing the setTimeout on my node server but to my understanding that only impacts the request it makes to other servers and not the requests made to it. I've also tried spawning a child process rather than running exec however I'm not sure how to access it in a later api call if it is scoped into the initial calls function.
I have a function run in my server that looks like this:
function run(command) {
return new Promise((resolve, reject) => {
exec(command, (err, stdout, stderr) => {
if(err){
reject(stderr);
}
resolve(stdout);
});
});
}
My api endpoint exec uses run like so:
app.post('/api/exec', (req, res) => {
req.setTimeout(0); //some commands take a long time, so we shouldnt time out the response
if(req.body.command){
run(req.body.command)
.then(data => {
return res.status(201).send({
message: data
});
}).catch(err => {
return res.status(500).send({
message: err
});
});
} else {
return res.status(400).send({
message: "exec requires a command member"
})
}
});
Finally, I am calling the api from my react app with the following:
execute = (c) => {
fetch(`/api/exec`, {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
command: c
})
}).then (resp => {
... do stuff with my app ...
I expect that the server would run my command until completion and then respond with the stdout from running the command, however my client application times out after ~two minutes of calling the execute function.
Several questions already have good answers regarding how to decrease the timeout time for an api call by wrapping in a timeout promise, however this method does not allow me to exceed the two minute default timeout time.
I am using Puppeteer api (https://github.com/GoogleChrome/puppeteer)
For automation testing.
I want to listen to all http response, need the url and the response data for each one.
I try to so use the page.on('response') function :
page.on('response', response => {
response.text().then( textBody=> {
const req = response.request();;
console.log(req.url())
console.log(textBody)
});
})
Should warp in 'waitForSelector' function , to have a flag that the data is ready?
I try to do so.
The problem is some time i do not see any console.log and some time i do.
I will glad to know what do i do wrong?
There is no need to call response.request(), unless you are trying to obtain the URL of the matching request object.
The following solution will work just fine:
page.on('response', response => {
console.log('Response URL:', response.url());
response.text().then(data => {
console.log('Response Text:');
console.log(data);
});
});
If you are still having issues, it could be because the associated request has failed.
You can check for this error by listening for the requestfailed event and logging the result:
page.on('requestfailed', request => {
console.log('Failed Request URL:', request.url());
console.log('Failed Request Error Message:', request.failure().errorText);
});
I've followed this tutorial's code (https://dialogflow.com/docs/getting-started/basic-fulfillment-conversation) to return results of an API to dialog flow. However my webhook keeps failing. Can someone help me figure out why?
Here's one of the failed conversations:
Here's my code:
'use strict';
const http = require('http');
exports.Hadoop = (req, res) => {
// Get name node server from the request
let nameNodeServer = req.body.queryResult.parameters['nameNodeServer']; // nameNodeServer is a required param
// Call the Hadoop API
getNameNodeInfo(nameNodeServer).then(function(output) {
res.json({ 'fulfillmentText': output }); // Return the results to Dialogflow
}).catch(() => {
res.json({ 'fulfillmentText': 'getNameNodeInfo() Error'- });
});
};
function getNameNodeInfo (nameNodeServer) {
return new Promise((resolve, reject) => {
// Create url for the HTTP request to get the name node info
let url = 'http://' + nameNodeServer + '[rest of url]';
// Make the HTTP request to get the name node info
http.get(url, (res) => {
let body = ''; // var to store the response chunks
res.on('data', (chunk) => {body += chunk; });
res.on('end', () => {
// After all the data has been received, parse the JSON for desired data
let response = JSON.parse(body);
let beans = response['beans'][0];
// Create response
let output = `Percent Used: ${beans['PercentUsed']}`;
// Resolve the promise with the output text
console.log(output);
resolve(output);
});
res.on('error', (error) => {
console.log(`Error calling the Hadoop API: ${error}`);
reject();
});
});
});
}
I believe the getNameNodeInfo function and the retrieval of the name node server are correct, as they logged the correct output in debugging.
Diagnostic Info:
I contacted someone at Dialogflow and this was their response.
Thank you for providing all the information. I have observed in your
code that you have used http requests instead of https. The service
must use HTTPS and the URL must be publicly accessible in order for
the fulfillment to function. Dialogflow does not support self-signed
SSL certs. For information on SSL setup, please refer to this :
https://developers.google.com/web/fundamentals/security/encrypt-in-transit/enable-https
We've had a somewhat different, but related, issue:
Internal Server Error when running an agent.
“status”: {
“code”: 500,
“errorType”: “internal_server_error”,
“errorDetails”: “Internal Server Error”
},
This error was not caused by any changes we introduced. We are using that agent in a dev version of an app and one morning it stopped working.
We tested by creating a .zip and restoring into a new agent. The new agent would work properly, but we would continue to get the 500 error on the agent hooked into our dev app. We submitted a help request and overnight the error got resolved. We suspect that DialogFlow team had to manually reboot the server or something similar.
I have a requirement to access multiple devices over IP from an HTML5 web app.
My approach to working around the cross-domain impossibility of doing this all on the client side has been to "cook" requests from the client inside of express' middleware. A route receives a get or post from the client, and then performs a get or post to the 3rd party device identified by the payload from the client.
I'm using the code to get info from a device. It works just fine when I run it directly from file inside of a client I made for testing purposes. Running directly from file avoids the CORS difficulty because the client is also the server I guess.
When I run the same code from within an express route, I get a 500 error.
Am I trying to do something impossible? I'm only about a week into node, express etc so hopefully it's something dumb and easy to solve. I'm taking fact that I haven't been able to find any other questions quite like this as an indication that there's a proper way to achieve what I need.
// post to find a camera
router.post('/find', function(req, res) {
var url = 'http://' + req.body.addr + '/cgi-bin/aw_cam?cmd=QID&res=1';
console.log(url);
$.ajax({
type: 'GET',
url: url,
dataType: 'html',
success: function (result) {
console.log('success: ' + result);
res.send(result);
},
error: function (xhr, textStatus, err) {
console.log('error: ' + textStatus);
}
});
});
Here's what logged to the server console:
http://192.168.0.10/cgi-bin/aw_cam?cmd=QID&res=1
POST /cameras/find 500 126.593 ms - 1656
Thanks in advance!
ok I found how to do this. The trick is to use Node's built-in http messaging capabilities. I found a good article on how to do this here
The code below does exactly what I wanted from within my custom route middleware. I guess I just learned that I can only use AJAX how I wanted on the client side.
This lets me abstract the hairier details of the device control protocol into the server, leaving my client apps to use JSON/AJAX model to interact with them. Success!
var http = require('http');
// post to find a camera
router.post('/find', function(req, res) {
var url = 'http://' + req.body.addr + '/cgi-bin/aw_cam?cmd=QID&res=1';
console.log(url);
http.get(url, (response) => {
console.log(`Got response: ${response.statusCode}`);
var body = '';
response.on ('data', function(d) {
body += d;
});
response.on ('end', function () {
console.log('received: ' + body);
var reply = {};
if (body.indexOf('OID:') == 0) {
reply.msg = body.slice(4);
reply.ok = true;
} else {
reply.msg = body;
reply.ok = false;
}
res.send(reply);
});
// consume response body
response.resume();
}).on('error', (e) => {
console.log(`Got error: ${e.message}`);
});
});