I am using the node module 'request' to send up some JSON to my REST API.
I have the following call
request({
uri: urlToUse, // Using the url constructed above.
method: "POST", // POSTing to the URI
body: JSON.stringify(jsonItems),
headers: // Allows us to authenticate.
{
'Content-Type' : 'application/json',
'Authorization' : auth
}},
//What to do after the request...
function(err, response, body)
{
// Was there an error?
if (err)
{
// If so, log it.
console.log(err);
}
// Did the server respond?
if (response)
{
// Log it.
console.log(response.statusCode);
// Did the response have a body?
if(body)
{
// Log it.
console.log(body);
}
}
});
I want to add to this - I would like to be able to act on a 429 status code - in order to make it retry the request until complete.
I know how to detect the 429 (using an if statement to check response.statusCode, etc), But I don't know how to make it retry, or if that is even the way to do it best.
It seems to me what you want to do is just wrap all this code in a function and then have it call itself if the response code is 429. perhaps even include a "try attempt" number as the last optional parameter to this function, you can then keep a count of how many times you've called it, and act accordingly to your preference
Related
I have been stuck with a weird issue with CORS for quite some time now. To explain the setup I have a frontend JS snippet (including fetch API call to my own server) which I want to use as an embed code snippet other web applications can use. (ex: kind of like google analytics)
// JS snippet to copy to an unknown website
<script>
// extract data and add to the body
const extracted_data = {}
fetch("https://api.xxxx.com/xxxxxx/create", {
method: "POST",
mode: 'cors',
body: JSON.stringify(extracted_data),
referrer: "origin",
headers: {
'Content-Type': 'application/json',
},
})
.then(function (response) {
// The API call was successful!
if (response.ok) {
return response.json();
} else {
return Promise.reject(response);
}
})
.then(function (data) {
// This is the JSON from our response
console.warn("Successfull!", data);
alert("Success:" + JSON.stringify(data));
})
.catch(function (err) {
// There was an error
console.warn("Something went wrong ->", err);
alert("error:" + err.message);
});
</script>
The problem is even if I have set my fetch API as below and the correct CORS headers are in my preflight response from my API it works only when the API call resolves immediately. If the API takes more time Fetch throws this common error even if the preflight is successful.
TypeError: Failed to fetch
I verified it using adding the below code to my API. Then it stops working and throws the above error. When I don't have any time taking functions inside my API call it works without errors.
// 10 sec delay to check async behavior of the API
await new Promise(resolve => setTimeout(resolve, 10000));
Any recommendations on how I should proceed to resolve this?
I've been stuck on this issue for some time now, I am trying to subscribe to Linkedin's webhook using ngrok for testing on localhost, and have been trying for some time now, i have tried using encode uri's as well but still running into error, I have verified that the APP_ID, profileId and organizationId i'm using are correct, but still i get the same error. I have also tried using the Restli protocol that linkedin suggests in their documentation but to no avail.
let url = `https://api.linkedin.com/v2/eventSubscriptions/(developerApplication:urn:li:developerApplication:${config.IN.APP_ID},user:urn:li:person:${profileId},entity:urn:li:organization:${organizationId},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS)`;
return new Promise((resolve, reject) => {
request(
{
url,
method: 'PUT',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${accessToken}`,
// 'X-Restli-Protocol-Version': '2.0.0',
},
json: {
webhook: "https://url.ngrok.io/api/v1/webhook/linkedin/callback"
},
},
(err, response, body) => {
if (err) {
reject(err);
} else {
resolve(body);
}
},
);
});
I have been receiving this error constantly no matter what I try, I have tried sending the url like this:
https://api.linkedin.com/v2/eventSubscriptions/(developerApplication:urn:li:developerApplication:{${config.IN.APP_ID}},user:urn:li:person:{${profileId}},entity:urn:li:organization:{${organizationId}},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS)
https://api.linkedin.com/v2/eventSubscriptions/${encodeURIComponent((developerApplication:urn:li:developerApplication:${config.IN.APP_ID},user:urn:li:person:${profileId},entity:urn:li:organization:${organizationId},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS))
All I receive is this error
'{"serviceErrorCode":100,"message":"Unpermitted fields present in RESOURCE_KEY: Data Processing Exception while processing fields [/key]","status":403}'
Any help would be appreciated, I have been stuck on this for a while now.
The request seems to be OK , but the method should be GET (not PUT)
One thing is to check which ID are you using for application_id. The application ID is the one in the url - https://www.linkedin.com/developers/apps/<id from here>/settings
. You need to use and uncomment the header for Restli.
I'd say that your url needs to look like this, as this is from their original POSTMAN collection.
https://api.linkedin.com/v2/eventSubscriptions/(developerApplication:urn%3Ali%3AdeveloperApplication%3A{{application_id}},user:urn%3Ali%3Aperson%3A{{person_id}},entity:urn%3Ali%3Aorganization%3A{{organization_id}},eventType:ORGANIZATION_SOCIAL_ACTION_NOTIFICATIONS)
You can validate here, their full collection - https://www.postman.com/linkedin-developer-apis/workspace/e781b3ac-4101-4d60-8981-afcb4812623d/request/16069442-9d0bf046-ea81-4af0-9515-d07246a0ab39
LinkedIn webhooks does not support ngrok URIs
Not able to send response of a REST api callout to browser page using NodeJs server with Express and Request module.
I understand that due to asynchronous nature of callback method, response of api callout cannot be returned in typical style.
I have tried res.send method, but it gives error res.send is not a function.
Below is sample piece of that I have tried.
const options = {
url: endPoint,
method: 'POST',
headers: {
'Authorization': 'Basic',
'Accept': 'application/json',
'Content-Type': 'application/json',
'Accept-Charset': 'UTF-8'
}
};
request(options, (err, res, body) => {
if (err) { return console.log(err); }
//want to send this body to the page.
console.log(JSON.stringify(body));
res.send(body);
});
It gives this error message,
res.send(body);
TypeError: res.send is not a function
at Request.request [as _callback]
Figured the issue.
First problem was as #vipul pointed above. res was response of callout and not instance of global HttpResponse object, so send method was not available on that object. I changed the method,
request(options, (err, response, body) => {
if (err) { return console.log(err); }
//want to send this body to the page.
console.log(JSON.stringify(response.body));
// using the HttpResponse in global context.
this.res.send(JSON.parse(body));
});
Then I faced below error,
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client]
Problem was while making the callout, I was returning the response right after callout method.
.get('/authorize', (req, res) => {
console.log('Authorize Hit! '+req.query.code);
try {
//TODO: Need to utilize state to the current flow in action.
let flowDetails = flow.flowMap.get('wsf');
if(flowDetails) {
//this method makes REST API call
flowDetails.refreshToken(req, res);
//this line was the problem!! Removed it to fix the error.
res.send('Refresh Token Completed!');
}
} catch (error) {
(Object.keys(error).length === 0)?res.send(error.message):res.send(error);
}
})
But due to asynchronous nature of callback function, by the time callout response was received, actual request/response was already returned and request was completed. So removed the res.send right after callout to fix the issue.
Hopefully it will be helpful for others as well :)
I am using firebase for hosting cloud functions, since many functions (about every) I need to make the http request and get the json body to get the data from it. However, the callback doesn't work quite well for me, I've searched some existing answers but still get stuck on this. Here is the code snippet, options are declared before and if I do not put the request within get_request_handler it works fine.:
function get_request_handler(assistant, input_url, callback) {
req(options, function (error, response, body) {
if (!error && response.statusCode == 200) {
var cookie = req.cookie('BPMSTS=' + body );
var headers = {
'Content-Type': 'application/json',
'Cookie': cookie
};
var option = {
url: input_url,
method: 'GET',
headers: headers
}
req(option, function(error, res, body) {
assistant.ask(input_url);
if (!error && res.statusCode == 200) {
callback(JSON.parse(body));
} else {
assistant.ask('inner request with error code: ' + (res.statusCode).toString());
}
});
} else {
assistant.ask('outer request with error code: ' + (response.statusCode).toString());
}
});
}
I call the function as follows:
get_request_handler(assistant, workflow_url, function(cur_json){assistant.ask(cur_json);});
The problem right now is the first request can't be made in the get_request_handler function. In other words, it only goes in to get_request_handler but not go into that request body. If I do not create get_request_handler and left req(options, function (error, response, body) { ... } it works without any problem. Any ideas on this?
Note: I just checked firebase log and it says for this line: req(options, function (error, response, body) it got TypeError: Assignment to constant variable.
at get_request_handler (/user_code/index.js:116:13)
You have a lot of problems here, but the basic one is that you're trying to call assistant.ask() more than once in your response to the user. Once you call assistant.ask(), no further responses can be sent, and none of the other calls to ask() will be handled.
It looks like you're using it for debugging. That seems a really poor choice when you should be using console.log().
You also indicated that you're using Firebase Functions. Note that calls out from a Firebase function are restricted if you're on the free plan. If you're on one of the paid plans there is no restriction, and there is a free tier which should be more than sufficient for testing.
In my node.js webpage I'm making a page preview similar to the Facebook link preview. I'm making a call to get the html of the page, and use it to create the preview.
$.ajax({
type: 'GET',
data: { "html": url },
url: "/htmlTest",
success: function (data) {
imgArray = [];
$('img', data).each(function () {
imgArray.push(this.src);
});
...
This is the server-side code that handles the request.
app.get('/htmlTest', function (req, res) {
res.writeHead(200, { 'content-type': 'text/html' });
request(req.query.html, function (error, response, body) {
if (error) {
res.write(error.toString());
res.end('\n');
}
else if (response.statusCode == 200) {
res.write(body);
res.end('\n');
}
})
});
Now what I've been noticing, is that it will just insert any css the other page uses into my page, which can really screw everything up. Why is this happening?
Also, while I'm at it, does anyone have any better ideas for a facebook-style page preview?
No. writeHead writes HTTP headers to the underlying TCP stream. It has absolutely nothing to do with HTML.
You're running into an issue because your server returns the wholesale HTML content of the requested URL. You then pass this string into jQuery, which is apparently adding contained CSS styles into your document.
Generally, it is a terrible idea to take random code from a user-supplied URL and run in the context of your page. It opens you to gaping security holes – the CSS artifacts you're seeing are one example.
To be blunt, your code has numerous problems, so bear with me as I point out some issues.
app.get('/htmlTest', function (req, res) {
res.writeHead(200, { 'content-type': 'text/html' });
Here, you respond to the browser with a success status (200) beore your server actually does anything. This is incorrect: you should only respond with either a success or error code after you know if the request succeeded or failed.
request(req.query.html, function (error, response, body) {
if (error) {
res.write(error.toString());
res.end('\n');
}
Here would be a good place to respond with an error code, since we know that the request did actually fail. res.send(500, error) would do the trick.
else if (response.statusCode == 200) {
res.write(body);
res.end('\n');
}
And here's where we could respond with a success code. Rather than use writeHead, use Express's set and send methods – things like Content-Length will be correctly set:
res.set('Content-Type', 'text/html');
res.send(body);
Now what happens if response.statusCode != 200? You don't handle that case. error is only set in the case of network errors (such as inability to connect to the target server). The target server can still respond with a non-200 status, and your node server would never respond to the browser. In fact, the connection would hang open until the user kills it. This could be fixed with a simple else res.end().
Even with these issues resolved, we still haven't addressed the fact that it's not a good idea to try to parse arbitrary HTML in the browser.
If I were you, I'd use something that parses HTML into a DOM on the server, and then I'd return only the necessary information back to the browser as JSON. cheerio is the module you probably want to use – it looks just like jQuery, only it runs on the server.
I'd do this:
var cheerio = require('cheerio'), url = require('url'), request = require('request');
app.get('/htmlTest', function(req, res) {
request(req.query.url, function(err, response, body) {
if (err) res.send(500, err); // network error, send a 500
else if (response.status != 200) res.send(500, { httpStatus: response.status }); // server returned a non-200, send a 500
else {
// WARNING! We should probably check that the response content-type is html
var $ = cheerio.load(body); // load the returned HTML into cheerio
var images = [];
$('img').each(function() {
// Image srcs can be relative.
// You probably need the absolute URL of the image, so we should resolve the src.
images.push(url.resolve(req.query.url, this.src));
});
res.send({ title: $('title').text(), images: images }); // send back JSON with the image URLs
}
});
});
Then from the browser:
$.ajax({
url: '/htmlTest',
data: { url: url },
dataType: 'json',
success: function(data) {
// data.images has your image URLs
},
error: function() {
// something went wrong
}
});