I'm trying to retrieve some information from the response that npm Request generated. I'm able to retrieve information like "statusCode" by typing "response.statusCode". However if I want to retrieve other information like "redirectUri", it would show undefined. How am I able to retrieve "redirectUri"?
Below is the code to get the response from the URL that I'm testing;
var request = require('request');
var getRequest = function (url, index) {
request(url, function (error, response, body) {
console.log(response.redirectUri);
});
}
getRequest('https://www.exampleUrl.com', 1);
Below are some of the information from the response;
redirects: [
{ statusCode: 302,
redirectUri:'https://www.exampleurl'.....etc
}],
Please see the response in attached image
Note: I have blurred out the url that I'm testing.
I found my answer from How do I get the redirected url from the nodejs request module?. Set "followRedirect: false" and use "response.headers.location".
var url = 'http://www.google.com';
request({ url: url, followRedirect: false }, function (err, res, body) {
console.log(res.headers.location);
});
Related
I am consuming below API using post. I have another API which I have to consume using get. instead of request.post I used request.get it is giving 401 error. I have to consume another API using GET with basic authentication. anybody can suggest changes for it.
var request = require('request');
var url = 'http://localhost:3000/api/v1/login/'
var user = 'test35';
var pass = 'mypassword';
request.post(
{
uri: url,
form: { username: user, password: pass }
},
function(err, httpResponse, body) {
if (err) {
return console.error('post failed:', err);
}
var json = JSON.parse(body);
console.log('Post successful! Server responded with:', body);
}
);
Does the response of that post request an authentication token (for example an jwt)?
APIs usually return some key that you can use in further requests, for example, in your headers in an get request
I'm tring to get a body response from a RESTFUL API using Node.js and it's "request" lib to send a GET request. Here's the code:
const request = require('request');
const url = 'https://myurl.com';
const headers = {
'x-functions-key': 'mysecretkey'
};
request.get(`${url}${headers}`, (err, response, body) =>{
if (err){
console.log(err)
}
console.log(body)
})
But when I run "node myfile" I get no response from the body, the console return is blank. I think I'm missing something. I've tested the URL and key with Postman and it's working fine, the JSON response appears to me there.
Obs.: I'm new to Node, tried with this tutorial: https://www.twilio.com/blog/2017/08/http-requests-in-node-js.html. The "url" and "key are masked here for security reasons. Any help is fine to me, I'm grateful.
The issue is that you can't just put headers inside the template string. You need to specify them with an options object.
request.get({ url: url, headers: headers }, (err, response, body) => { ... });
Or, with ES6 shorthand:
request.get({ url, headers }, (err, response, body) => { ... });
The problem is that on request.get line you have your URL and headers templates
`${url}${headers}`
Here's the documentation regarding custom headers
So the solution would be creating an options variable like this:
const options = {
url: 'https://myurl.com',
headers: {
'x-functions-key': 'mysecretkey'
}
};
And then passing it to request
request.get(options, (err, response, body) =>{
if (err){
console.log(err)
}
console.log(body)
})
Hope this helps
I want to make GET request to scrape some data thru a proxy server that is randomly generated using the gimmeproxy.com free API.
I am able to get the proxy ip/port and am using
'https-proxy-agent' to setup the agent with the proxy data.
Whenever I try to call any website I always get
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>405 Method Not Allowed</title>
</head><body>
<h1>Method Not Allowed</h1>
<p>The requested method CONNECT is not allowed for the URL
/index.html.en.backup.</p>
</body></html>
Here is my node script:
const request = require('request'), HttpsProxyAgent = require('https-proxy-agent');
generateRandomProxy(function(proxy){
var agent = new HttpsProxyAgent({
proxyHost: proxy.proxyHost,
proxyPort: proxy.proxyPort
});
request({
uri: "http://example.com",
method: "GET",
agent: agent,
timeout: 5000,
}, function(error, response, body) {
console.log(body);
});
})
function generateRandomProxy(cb){
request.get(' https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US',{json:true},function(err,res){
if(!err){cb({
proxyHost: res.body.ip,
proxyPort: res.body.port
})}
else{console.log('problem obtaining proxy')}
})
}
So my question: How can I route my request thru the proxy and then get a returned body that is valid?
As you see now I keep getting the 405 Method Not Allowed
Thank you for any assistance.
Edit: Just found some GimmeProxy wrapper for Node.js: gimmeproxy-request.
It claims to automatically re-route requests through another proxy when one fails.
With this module code would look like this:
const setup = require('gimmeproxy-request').setup;
const request = require('gimmeproxy-request').request;
setup({
api_key: 'your api key',
query: 'get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10', // additional gimmeproxy query parameters
retries: 5, // max retries before fail
test: (body, response) => body.indexOf('captcha') === -1 && response.statusCode === 200 // test function
});
request('https://example.com', {
timeout: 10000 // additional request parameters, see https://github.com/request/request
},
function(err, res, body) {
console.log('err', err)
console.log('res', res)
console.log('body', body)
process.exit()
});
I guess the issue is that you sometimes get not an https proxy from Gimmeproxy, while 'https-proxy-agent' expects https proxy only.
To fix it, use the proxy-agent package of the same author and pass curl field of GimmeProxy response. It will select correct proxy agent implementation.
The following code works for me:
const request = require('request'), ProxyAgent = require('proxy-agent');
generateRandomProxy(function(proxy){
console.log(proxy);
var agent = new ProxyAgent(proxy.curl);
request({
uri: "https://example.com",
method: "GET",
agent: agent,
timeout: 5000,
}, function(error, response, body) {
console.log(error);
console.log(body);
});
})
function generateRandomProxy(cb){
request.get('https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10',{json:true},function(err,res){
if(!err){cb(res.body)}
else{console.log('problem obtaining proxy')}
})
}
Note: If you want to call https websites, you should query for proxies with https support using supportsHttps=true parameter. Also it makes sense to query for fresh proxies with maxCheckPeriod=1800 parameter. Setting minSpeed=10 also helps:
https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10
I want to get all public github repositories of given user from github.
I tried to make it with GET request I read from here. When i try it with curl or in the browser everything is fine, but when I run this code is gives me http 403 status code
var request = require('request');
request.get("https://api.github.com/users/:user")
.on('response', function (response) {
console.log(response.statusCode);
console.log(JSON.stringify(response));
});
I tried using this github api library, but couldn't work around the authetication
var GithubApi = require("github");
var github = new GithubApi({});
github.authenticate({
type: "oauth",
token: "GIT_TOKEN"
});
var getUsersRepos = function (user) {
github.repos.getAll({
username: user
}, function (err, res) {
res.forEach(function (element) {
console.log(`${element.name} - language: ${element.language} - issues: ${element.open_issues} - url: ${element.url}`);
}, this);
});
}
module.exports = {
getRepos: getUsersRepos
};
But when I enter my token I can get only my user information.
Any ideas what I am doing wrong or some better idea will be appreciated
The Github API requires requests to have a valid User-Agent header:
If you provide an invalid User-Agent header, you will receive a 403
Forbidden response.
Github requests that you use your GitHub username, or the name of your application, for the User-Agent header value:
var request = require('request');
options = {
url: "https://api.github.com/users/:user",
headers: {
"User-Agent": "tabula" // Your Github ID or application name
}
}
request.get(options)
.on('response', function (response) {
console.log(response.statusCode);
console.log(JSON.stringify(response));
});
I have tried to get all facebook pages list. But I got an error.
Error is : request is not defined
Code :
var url = 'https://graph.facebook.com/me/accounts';
var accessToken = req.user.facebookAccessToken;
var params = {
access_token: accessToken,
};
request.get({ url: url, qs: params}, function(err, resp, pages) {
// console.log(resp);
pages = JSON.parse(pages);
})
The error you're getting isn't related to Facebook.
"request is not defined" means you're trying to do something with the variable request (call its .get() function in this case) but the variable request hasn't been set anywhere.
Your tags indicate this is in Node. Do you have this line already?
var request = require('request');
If not, add that before the code in your question. That loads the 'request' module you're trying to use.
You got the issue on request ,,
install npm request
require('request').get({
uri: url,
headers: {'content-type': 'application/x-www-form-urlencoded'},
body: require('querystring').stringify(params)
}, function (err, resp, body) {
})