I currently use request to make http requests in node.js. I had at some point encountered an issue where I was getting errors that indicated UNABLE_TO_GET_ISSUER_CERT_LOCALLY. To get around that it set rejectUnauthorized. My working code with request looks like this:
var url = 'someurl';
var options = {
url: url,
port: 443,
// proxy: process.env.HTTPS_PROXY, -- no need to do this as request honors env vars
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
'Accept-Language': 'en-us',
'Content-Language': 'en-us'
},
timeout: 0,
encoding: null,
rejectUnauthorized: false // added this to prevent the UNABLE_TO_GET_ISSUER_CERT_LOCALLY error
};
request(options, function (err, resp, body) {
if (err) reject(err);
else resolve(body.toString());
});
I thought I would try switching to the fetch api using async/await and am now trying to use node-fetch to do the same thing. However, when I do the same thing I am back to the UNABLE_TO_GET_ISSUER_CERT_LOCALLY errors. I read that I needed to use a proxy agent and tried using the proxy-agent module but I am still not having any luck.
Based off of the post https://github.com/TooTallNate/node-https-proxy-agent/issues/11 I thought the following would work:
var options = {
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
'Accept-Language': 'en-us',
'Content-Language': 'en-us'
},
timeout: 0,
encoding: null
};
var proxyOptions = nodeurl.parse(process.env.HTTPS_PROXY);
proxyOptions.rejectUnauthorized = false;
options.agent = new ProxyAgent(proxyOptions);
const resp = await fetch('someurl', options);
return await resp.text();
but I still get the same error. So far the only way I've been able to get around this using node-fetch is to set NODE_TLS_REJECT_UNAUTHORIZED=0 in my environment which I don't really want to do. Can someone help show me how to get rejectUnauthorized to work with node-fetch (presumably using an agent but I don't honestly care how as long as it's specified as part of the request).
This is how I got this to work using rejectUnauthorized and the Fetch API in a Node.js app.
Keep in mind that using rejectUnauthorized is dangerous as it opens you up to potential security risks, as it circumvents a problematic certificate.
const fetch = require("node-fetch");
const https = require('https');
const httpsAgent = new https.Agent({
rejectUnauthorized: false,
});
async function getData() {
const resp = await fetch(
"https://myexampleapi.com/endpoint",
{
agent: httpsAgent,
},
)
const data = await resp.json()
return data
}
Use proxy
You should know that node-https-proxy-agent latest version have a problem and doesn't work with Fetch! You can use older version 3.x and down! And it will work! Otherwise Better you can use the node-tunnel module https://www.npmjs.com/package/tunnel! You can too use the wrapping module proxy-http-agent that is based on node-tunnel https://www.npmjs.com/package/proxy-http-agent! That provide automatic detection of protocol for the proxy! One method for all! And more options and affinity! And both of them support both http and https !
You can see the usage and see a good example of proxy building and setup in this module and repo (check the tests):
https://www.npmjs.com/package/net-proxy
https://github.com/Glitnirian/node-net-proxy#readme
ex:
import { ProxyServer } from 'net-proxy';
import { getProxyHttpAgent } from 'proxy-http-agent';
// ...
// __________ setting the proxy
const proxy = new ProxyServer({
port: proxyPort
});
proxy.server.on('data', (data: any) => { // accessing the server instance
console.log(data);
});
await proxy.awaitStartedListening(); // await server to start
// After server started
// ______________ making the call through the proxy to a server through http:
let proxyUrl = `http://localhost:${proxyPort}`; // Protocol from the proxy is automatically detected
let agent = getProxyHttpAgent({
proxy: proxyUrl,
endServerProtocol: 'http:' // the end server protocol (http://localhost:${localApiServerPort} for example)
});
const response = await fetch(`http://localhost:${localApiServerPort}`, {
method: 'GET',
agent
});
// ___________________ making a call through the proxy to a server through https:
agent = getProxyHttpAgent({
proxy: proxyUrl, // proxy as url string! We can use an object (as tunnel module require too)
rejectUnauthorized: false // <==== here it go
});
const response2 = await fetch(`https://localhost:${localApiHttpsServerPort}`, {
method: 'GET',
agent
});
You can see more examples and details in the doc here:
https://www.npmjs.com/package/proxy-http-agent
And you can too use directly node-tunnel! But the package is just a simple wrapper! That make it more simpler!
Add rejectUnauthorized
For the one that doesn't know well!
As per this thread
https://github.com/node-fetch/node-fetch/issues/15
We use the https.Agent to pass the rejectUnauthorized parameter!
const agent = new https.Agent({
key: fs.readFileSync(`${CERT_PATH}.key`),
cert: fs.readFileSync(`${CERT_PATH}.crt`),
rejectUnauthorized: false
})
A complete example
import https from "https";
const agent = new https.Agent({
rejectUnauthorized: false
});
fetch(myUrl, { agent });
For fetch you can too use an environment variable as follow
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
This way it gonna be set globaly and not per each call! Which may be more appropriate if you are using a constant proxy! For all calls! As when sitting behind the company proxy!
why
By default node fetch! And most of the http requests clients! All use the security and insure a valid ssl Certificate when using https!
To disable this behavior we need to disable that check somehow!
Depending on the libs it may be different!
For fetch that's how it's done!
With http.request! (underlying)
const https = require('https');
const options = {
hostname: 'encrypted.google.com',
port: 443,
path: '/',
method: 'GET',
rejectUnauthorized: false /// <<<== here
};
const req = https.request(options, (res) => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
res.on('data', (d) => {
process.stdout.write(d);
});
});
req.on('error', (e) => {
console.error(e);
});
req.end();
check this:
https://nodejs.org/api/https.html#https_https_request_url_options_callback
Also it's part of tls.connect Options
Which you can check here
https://nodejs.org/api/tls.html#tls_tls_connect_options_callback
Related
Is there any way to write file using buffer array and content-type with put method?
requestify.request('some url', {
method: 'PUT',
body: buffArray, //need modifications here
headers: {
'Content-Type': res_file.headers['content-type']
}
}).then(function (res) {
console.log(res);
})
I could send the data but file not storing in proper way.
working Java code
httpcon.setRequestMethod("PUT");
httpcon.setReadTimeout(100000);
httpcon.setDoOutput(true);
httpcon.setRequestProperty("Content-Type", conenttype);
httpcon.connect();
OutputStream os = httpcon.getOutputStream();
os.write(in.toByteArray(), 0, in.size());
responceCode = httpcon.getResponseCode();
httpcon.disconnect();
My personal advice here is to use the builtin http or https package from Node.JS.
Why? Because you want to write and read binary files that might be large enough to give you problems, and as for what I've tested with requestify, it will give you problems when using binary responses (It stringifies them!).
You can simply use streams which will save you lots of headaches.
You can test it using this, for example:
const fs = require('fs');
const http = require('https');
const req = http.request({
host: 'raw.githubusercontent.com',
path: '/smooth-code/svgr/master/resources/svgr-logo.png',
method: 'GET'
}, res => {
res.pipe(fs.createWriteStream('test.png'));
});
req.end();
and adapted to your provided code:
const fs = require('fs');
const http = require('https');
const req = http.request({
host: 'some-host',
path: '/some/path',
method: 'PUT',
headers: {
'Content-Type': res_file.headers['content-type']
}
}, res => {
res.pipe(fs.createWriteStream('your-output-file.blob'));
});
// This part: If comes from HDD or from another request, I would recommend using .pipe also
req.write(buffArray);
req.end();
Further info:
http package https://nodejs.org/api/http.html
fs package https://nodejs.org/api/fs.html
I want to make GET request to scrape some data thru a proxy server that is randomly generated using the gimmeproxy.com free API.
I am able to get the proxy ip/port and am using
'https-proxy-agent' to setup the agent with the proxy data.
Whenever I try to call any website I always get
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>405 Method Not Allowed</title>
</head><body>
<h1>Method Not Allowed</h1>
<p>The requested method CONNECT is not allowed for the URL
/index.html.en.backup.</p>
</body></html>
Here is my node script:
const request = require('request'), HttpsProxyAgent = require('https-proxy-agent');
generateRandomProxy(function(proxy){
var agent = new HttpsProxyAgent({
proxyHost: proxy.proxyHost,
proxyPort: proxy.proxyPort
});
request({
uri: "http://example.com",
method: "GET",
agent: agent,
timeout: 5000,
}, function(error, response, body) {
console.log(body);
});
})
function generateRandomProxy(cb){
request.get(' https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US',{json:true},function(err,res){
if(!err){cb({
proxyHost: res.body.ip,
proxyPort: res.body.port
})}
else{console.log('problem obtaining proxy')}
})
}
So my question: How can I route my request thru the proxy and then get a returned body that is valid?
As you see now I keep getting the 405 Method Not Allowed
Thank you for any assistance.
Edit: Just found some GimmeProxy wrapper for Node.js: gimmeproxy-request.
It claims to automatically re-route requests through another proxy when one fails.
With this module code would look like this:
const setup = require('gimmeproxy-request').setup;
const request = require('gimmeproxy-request').request;
setup({
api_key: 'your api key',
query: 'get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10', // additional gimmeproxy query parameters
retries: 5, // max retries before fail
test: (body, response) => body.indexOf('captcha') === -1 && response.statusCode === 200 // test function
});
request('https://example.com', {
timeout: 10000 // additional request parameters, see https://github.com/request/request
},
function(err, res, body) {
console.log('err', err)
console.log('res', res)
console.log('body', body)
process.exit()
});
I guess the issue is that you sometimes get not an https proxy from Gimmeproxy, while 'https-proxy-agent' expects https proxy only.
To fix it, use the proxy-agent package of the same author and pass curl field of GimmeProxy response. It will select correct proxy agent implementation.
The following code works for me:
const request = require('request'), ProxyAgent = require('proxy-agent');
generateRandomProxy(function(proxy){
console.log(proxy);
var agent = new ProxyAgent(proxy.curl);
request({
uri: "https://example.com",
method: "GET",
agent: agent,
timeout: 5000,
}, function(error, response, body) {
console.log(error);
console.log(body);
});
})
function generateRandomProxy(cb){
request.get('https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10',{json:true},function(err,res){
if(!err){cb(res.body)}
else{console.log('problem obtaining proxy')}
})
}
Note: If you want to call https websites, you should query for proxies with https support using supportsHttps=true parameter. Also it makes sense to query for fresh proxies with maxCheckPeriod=1800 parameter. Setting minSpeed=10 also helps:
https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10
So, I've made a Node.js API using the express framework. The API supports a POST-request to /login, where the client should include email and password formatted as json in the body. The API will then return a session cookie via the setCookie-header.
I DO see the cookie coming back from the API as a response-cookie, however, the browser isn't storing it, and therefore it is not sent with further requests from the client. I've tried using {credentials: include} since this is a CORS-request. I've also added the cors-module in my node-server (API) to handle the OPTIONS (pre-flight) requests. I've used so many hours trying to figure this out, so any help would be much appreciated.
Side-note: This works completely fine in both Postman and a prototype iOS-app I've developed using the same API, so there shouldn't be any issues on the server itself.
I've included relevant code from the server and the front-end below.
Code from server:
app.use(cors({credentials: true, origin: ['http://expivider.dk', 'http://expivider.herokuapp.com', 'https://expivider.herokuapp.com', 'http://api.expivider.dk']}));
app.use(session({
cookieName: 'session',
secret: SECRET_HERE,
duration: 30 * 60 * 1000,
activeDuration: 5 * 60 * 1000,
cookie: {
// path: '/api', cookie will only be sent to requests under '/api'
maxAge: 60000, // duration of the cookie in milliseconds, defaults to duration above
ephemeral: false, // when true, cookie expires when the browser closes
httpOnly: false, // when true, cookie is not accessible from javascript
secure: false, // when true, cookie will only be sent over SSL. use key 'secureProxy' instead if you handle SSL not in your node process
path: "/"
//domain: "expivider.herokuapp.com"
}
}));
Code from front-end:
const handleRequestWithBodyWithCredentials = function (method, url, body, callback) {
fetch(url, {
method: method,
credentials: 'include',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify(body),
mode: 'cors'
}).then((resp) => (resp.json())).then(function (data) {
callback(data);
});
};
const validate = function () {
let em = document.login.username.value;
let pw = document.login.password.value;
let body = {
'email': em,
'password': pw
};
handleRequestWithBodyWithCredentials('post', LOGIN_NEW, body, showCompanyStats);
console.log();
};
Note: Right now, the front-end is hosted on 'http://expivider.dk', and it makes calls to the api at 'http://api.expivider.dk' (which is actually hosted at 'expivider.herokuapp.com' but I'm using a custom-domain).
Please let me know if you need any more info to help me out!
Hi I am trying to start learning NodeJS now and am in the middle of creating an application. The goal currently is to call a website through node, get an authentication token, then call that website again now with a POST payload which includes my login info and the auth token.
I have created the same program using python and i get a 200 response where in nodeJS i am getting a 302.
I believe thats a quick solution, the main meat of the problem I guess is my lack of understanding in NodeJS where:
1. If I am supposed to nest these requests calls into one another because they are supposed to be a part of the same 'session' and
2. If so how do I go to the last url which is, example.com/poll and be able to store/modify that information (which is just a json) because/if i go to example.com/poll url using a browser, the browser automatically downloads a file which it contains is a JSON format and doesnt just display it, which is what i need. so that i can either save that data in a string or etc. and not download it
In python I do this (Create a session than make the two calls)
url = "https://example.com/"
session = requests.session()
first_req = session.get(url)
auth_token_str = re.search(XXX, first_req.text)
login_url = 'https://example.com/sessions'
payload = { 'session[username_or_email]' : 'username', 'session[password]' : 'password', 'redirect_after_login':'/', 'authenticity_token': authenticity_token }
login_req = session.post(login_url, data=payload, headers=user_agent)
print "login_req response: ", login_req.status_code //gets me 200
then in Node JS:
var initLoad = {
method: 'GET',
url: 'https://example.com/',
headers: {
'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'
}
};
request(initLoad, function(error, response, body) {
if (error) throw new Error(error);
var $ = cheerio.load(body, {xmlMode: false});
var authenticityToken = $("input[name=authenticity_token]").val();
console.log(authenticityToken);
var options = {
method: 'POST',
url: 'https://example.com/sessions',
headers: response.headers,
form: {
'session[username_or_email]': 'someUsername',
'session[password]': 'somePassword',
redirect_after_login: '/',
authenticity_token: authenticityToken
}
};
request(options, function(error, response2, body2) {
if (error) throw new Error(error);
console.log(response2.statusCode); //gets me 302 not 200
var analytics_url = 'https://example.com/poll';
var tripleload = {
method: 'GET',
url: analytics_url,
headers: response2.headers
};
request(tripleload, function(error, response3, body3) {
if (error) throw new Error(error);
res.end(body3);
});
});
});
302 means temporarily moved redirection which you get due error page being displayed to you (or served to your server in this case). There is something with this call that you are doing wrong, maybe url is wrong if generated like this.
Your code is messy due you being newbie in node and due the fact you use request which is barebone and offers little to no comfort in writing this stuff.
Use something like Axios: https://github.com/mzabriskie/axios to make it easier to write requests like this.
I am creating a system that passes http request to a child process in Node js. I cant pass the child process the active Socket using child.send( 'socket', req.socket ) but inside the child I want to recreate the http request and response objects so that they have the headers, parameters, cookies etc.
I am using Express, so if I can recreate the Express req and res object it's even better.
I have been fiddling a bit, but no success.
If I do the following it creates the IncomingMessage object but the headers etc are empty.
var http = require('http');
/* Child Process recieves the Socket */
var incomingMessage = new http.IncomingMessage( socket );
Any ideas if there is any way to accomplish what I want?
You can also this trick for tunneling:
let agent;
if (protocol == 'https:')
agent = new https.Agent();
else
agent = new http.Agent();
agent.createConnection = (opts, callback) => callback(false, socket);
const req = http.request({
method: method,
host: host,
port: port,
protocol: protocol,
path: path,
headers: headers,
agent: agent
}, function (res)
{
console.log(res.headers);
console.log(res.socket.remoteAddress);
console.log(res.socket == socket); // true
});
req.end();