Develop Reference

Using rejectUnauthorized with node-fetch in node.js

I currently use request to make http requests in node.js. I had at some point encountered an issue where I was getting errors that indicated UNABLE_TO_GET_ISSUER_CERT_LOCALLY. To get around that it set rejectUnauthorized. My working code with request looks like this:
var url = 'someurl';
var options = {
url: url,
port: 443,
// proxy: process.env.HTTPS_PROXY, -- no need to do this as request honors env vars
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
'Accept-Language': 'en-us',
'Content-Language': 'en-us'
},
timeout: 0,
encoding: null,
rejectUnauthorized: false // added this to prevent the UNABLE_TO_GET_ISSUER_CERT_LOCALLY error
};
request(options, function (err, resp, body) {
if (err) reject(err);
else resolve(body.toString());
});
I thought I would try switching to the fetch api using async/await and am now trying to use node-fetch to do the same thing. However, when I do the same thing I am back to the UNABLE_TO_GET_ISSUER_CERT_LOCALLY errors. I read that I needed to use a proxy agent and tried using the proxy-agent module but I am still not having any luck.
Based off of the post https://github.com/TooTallNate/node-https-proxy-agent/issues/11 I thought the following would work:
var options = {
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
'Accept-Language': 'en-us',
'Content-Language': 'en-us'
},
timeout: 0,
encoding: null
};
var proxyOptions = nodeurl.parse(process.env.HTTPS_PROXY);
proxyOptions.rejectUnauthorized = false;
options.agent = new ProxyAgent(proxyOptions);
const resp = await fetch('someurl', options);
return await resp.text();
but I still get the same error. So far the only way I've been able to get around this using node-fetch is to set NODE_TLS_REJECT_UNAUTHORIZED=0 in my environment which I don't really want to do. Can someone help show me how to get rejectUnauthorized to work with node-fetch (presumably using an agent but I don't honestly care how as long as it's specified as part of the request).

This is how I got this to work using rejectUnauthorized and the Fetch API in a Node.js app.
Keep in mind that using rejectUnauthorized is dangerous as it opens you up to potential security risks, as it circumvents a problematic certificate.
const fetch = require("node-fetch");
const https = require('https');
const httpsAgent = new https.Agent({
rejectUnauthorized: false,
});
async function getData() {
const resp = await fetch(
"https://myexampleapi.com/endpoint",
{
agent: httpsAgent,
},
)
const data = await resp.json()
return data
}

Use proxy
You should know that node-https-proxy-agent latest version have a problem and doesn't work with Fetch! You can use older version 3.x and down! And it will work! Otherwise Better you can use the node-tunnel module https://www.npmjs.com/package/tunnel! You can too use the wrapping module proxy-http-agent that is based on node-tunnel https://www.npmjs.com/package/proxy-http-agent! That provide automatic detection of protocol for the proxy! One method for all! And more options and affinity! And both of them support both http and https !
You can see the usage and see a good example of proxy building and setup in this module and repo (check the tests):
https://www.npmjs.com/package/net-proxy
https://github.com/Glitnirian/node-net-proxy#readme
ex:
import { ProxyServer } from 'net-proxy';
import { getProxyHttpAgent } from 'proxy-http-agent';
// ...
// __________ setting the proxy
const proxy = new ProxyServer({
port: proxyPort
});
proxy.server.on('data', (data: any) => { // accessing the server instance
console.log(data);
});
await proxy.awaitStartedListening(); // await server to start
// After server started
// ______________ making the call through the proxy to a server through http:
let proxyUrl = `http://localhost:${proxyPort}`; // Protocol from the proxy is automatically detected
let agent = getProxyHttpAgent({
proxy: proxyUrl,
endServerProtocol: 'http:' // the end server protocol (http://localhost:${localApiServerPort} for example)
});
const response = await fetch(`http://localhost:${localApiServerPort}`, {
method: 'GET',
agent
});
// ___________________ making a call through the proxy to a server through https:
agent = getProxyHttpAgent({
proxy: proxyUrl, // proxy as url string! We can use an object (as tunnel module require too)
rejectUnauthorized: false // <==== here it go
});
const response2 = await fetch(`https://localhost:${localApiHttpsServerPort}`, {
method: 'GET',
agent
});
You can see more examples and details in the doc here:
https://www.npmjs.com/package/proxy-http-agent
And you can too use directly node-tunnel! But the package is just a simple wrapper! That make it more simpler!
Add rejectUnauthorized
For the one that doesn't know well!
As per this thread
https://github.com/node-fetch/node-fetch/issues/15
We use the https.Agent to pass the rejectUnauthorized parameter!
const agent = new https.Agent({
key: fs.readFileSync(`${CERT_PATH}.key`),
cert: fs.readFileSync(`${CERT_PATH}.crt`),
rejectUnauthorized: false
})
A complete example
import https from "https";
const agent = new https.Agent({
rejectUnauthorized: false
});
fetch(myUrl, { agent });
For fetch you can too use an environment variable as follow
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
This way it gonna be set globaly and not per each call! Which may be more appropriate if you are using a constant proxy! For all calls! As when sitting behind the company proxy!
why
By default node fetch! And most of the http requests clients! All use the security and insure a valid ssl Certificate when using https!
To disable this behavior we need to disable that check somehow!
Depending on the libs it may be different!
For fetch that's how it's done!
With http.request! (underlying)
const https = require('https');
const options = {
hostname: 'encrypted.google.com',
port: 443,
path: '/',
method: 'GET',
rejectUnauthorized: false /// <<<== here
};
const req = https.request(options, (res) => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
res.on('data', (d) => {
process.stdout.write(d);
});
});
req.on('error', (e) => {
console.error(e);
});
req.end();
check this:
https://nodejs.org/api/https.html#https_https_request_url_options_callback
Also it's part of tls.connect Options
Which you can check here
https://nodejs.org/api/tls.html#tls_tls_connect_options_callback

nodejs request using agent proxy via gimmeproxy.com

I want to make GET request to scrape some data thru a proxy server that is randomly generated using the gimmeproxy.com free API.
I am able to get the proxy ip/port and am using
'https-proxy-agent' to setup the agent with the proxy data.
Whenever I try to call any website I always get
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>405 Method Not Allowed</title>
</head><body>
<h1>Method Not Allowed</h1>
<p>The requested method CONNECT is not allowed for the URL
/index.html.en.backup.</p>
</body></html>
Here is my node script:
const request = require('request'), HttpsProxyAgent = require('https-proxy-agent');
generateRandomProxy(function(proxy){
var agent = new HttpsProxyAgent({
proxyHost: proxy.proxyHost,
proxyPort: proxy.proxyPort
});
request({
uri: "http://example.com",
method: "GET",
agent: agent,
timeout: 5000,
}, function(error, response, body) {
console.log(body);
});
})
function generateRandomProxy(cb){
request.get(' https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US',{json:true},function(err,res){
if(!err){cb({
proxyHost: res.body.ip,
proxyPort: res.body.port
})}
else{console.log('problem obtaining proxy')}
})
}
So my question: How can I route my request thru the proxy and then get a returned body that is valid?
As you see now I keep getting the 405 Method Not Allowed
Thank you for any assistance.

Edit: Just found some GimmeProxy wrapper for Node.js: gimmeproxy-request.
It claims to automatically re-route requests through another proxy when one fails.
With this module code would look like this:
const setup = require('gimmeproxy-request').setup;
const request = require('gimmeproxy-request').request;
setup({
api_key: 'your api key',
query: 'get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10', // additional gimmeproxy query parameters
retries: 5, // max retries before fail
test: (body, response) => body.indexOf('captcha') === -1 && response.statusCode === 200 // test function
});
request('https://example.com', {
timeout: 10000 // additional request parameters, see https://github.com/request/request
},
function(err, res, body) {
console.log('err', err)
console.log('res', res)
console.log('body', body)
process.exit()
});
I guess the issue is that you sometimes get not an https proxy from Gimmeproxy, while 'https-proxy-agent' expects https proxy only.
To fix it, use the proxy-agent package of the same author and pass curl field of GimmeProxy response. It will select correct proxy agent implementation.
The following code works for me:
const request = require('request'), ProxyAgent = require('proxy-agent');
generateRandomProxy(function(proxy){
console.log(proxy);
var agent = new ProxyAgent(proxy.curl);
request({
uri: "https://example.com",
method: "GET",
agent: agent,
timeout: 5000,
}, function(error, response, body) {
console.log(error);
console.log(body);
});
})
function generateRandomProxy(cb){
request.get('https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10',{json:true},function(err,res){
if(!err){cb(res.body)}
else{console.log('problem obtaining proxy')}
})
}
Note: If you want to call https websites, you should query for proxies with https support using supportsHttps=true parameter. Also it makes sense to query for fresh proxies with maxCheckPeriod=1800 parameter. Setting minSpeed=10 also helps:
https://gimmeproxy.com/api/getProxy?get=true&cookies=true&country=US&supportsHttps=true&maxCheckPeriod=1800&minSpeed=10

How to make a POST request using CustomVision api with NodeJS

I'm trying attach an image using the bot emulator tool and sending this image off to the microsofts customvision api, the issue I'm having is that I get
{ Code: 'BadRequestImageFormat', Message: '' }
back from custom the custom vision api call.
I'm using the the request module from npm to handle the calls
// Receive messages from the user and respond by echoing each message back (prefixed with 'You said:')
var bot = new builder.UniversalBot(connector, function (session) {
session.send("Hello"); //session.message.text
// If there is an attachment
if (session.message.attachments.length > 0){
console.log(session.message.attachments[0])
request.post({
url: 'xxx',
encoding: null,
json: true,
headers: {
'Content-Type': 'application/octet-stream',
'Prediction-Key': 'xxx'
},
body: session.message.attachments[0]
}, function(error, response, body){
console.log(body);
});
}
});
I believe that I may be sending the wrong format through to custom vision however I have been unable to figure it out as of yet.

I replicated your issue and it looks like the problem is your 'Content-Type'. You're attempting to pass JSON in your request, but setting the content-type as octet-stream. See my modified code below:
var bot = new builder.UniversalBot(connector, function (session) {
session.send("Hello"); //session.message.text
// If there is an attachment
if (session.message.attachments.length > 0){
console.log(session.message.attachments[0])
request.post({
url: 'https://northeurope.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures',
encoding: null,
json: true,
headers: {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': 'Your API Key...'
},
body: session.message.attachments[0]
},
function (err, response, body) {
if (err) return console.log(err)
console.log(body);
});
}
});
When I run this, I get the error InvalidImageUrl which is to be expected as it's looking for a content on localhost. You could get round this by exposing your localhost using Ngrok.

Verifiy iOS Receipt with Node.js

After struggling a few days trying to get something to work and getting no where, I was wondering if someone has gotten iOS Receipt Validation working on Node.js. I have tried the node module iap_verifier found here but I could not get it to work properly for me. the only response I received back form Apples servers is 21002, data was malformed.
One thing that has worked for me was a client side validation request to apples servers that I got directly from the tutorials provided by Apple here, with the code shown below.
// The transaction looks ok, so start the verify process.
// Encode the receiptData for the itms receipt verification POST request.
NSString *jsonObjectString = [self encodeBase64:(uint8_t *)transaction.transactionReceipt.bytes
length:transaction.transactionReceipt.length];
// Create the POST request payload.
NSString *payload = [NSString stringWithFormat:#"{\"receipt-data\" : \"%#\", \"password\" : \"%#\"}",
jsonObjectString, ITC_CONTENT_PROVIDER_SHARED_SECRET];
NSData *payloadData = [payload dataUsingEncoding:NSUTF8StringEncoding];
// Use ITMS_SANDBOX_VERIFY_RECEIPT_URL while testing against the sandbox.
NSString *serverURL = ITMS_SANDBOX_VERIFY_RECEIPT_URL;
// Create the POST request to the server.
NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:serverURL]];
[request setHTTPMethod:#"POST"];
[request setHTTPBody:payloadData];
NSURLConnection *conn = [[NSURLConnection alloc] initWithRequest:request delegate:self];
[conn start];
I have a bunch of different code I have been using to send a wide array of things to my node server. and all of my different attempts have failed. I have even tried just funneling the "payloadData" I constructed in the client side validation example above to my server and sending that to Apples servers with the following code:
function verifyReceipt(receiptData, responder)
{
var options = {
host: 'sandbox.itunes.apple.com',
port: 443,
path: '/verifyReceipt',
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': Buffer.byteLength(receiptData)
}
};
var req = https.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log("body: " + chunk);
});
});
req.write(receiptData);
req.end();
}
Where the function is passed the payloadData. The response received from Apple is always 21002. I'm still basically a node novice,so I can't figure out what exactly is going wrong. I think there might be some data corruption happening when I am sending the data from ObjC to my Node server, so perhaps I am not transmitting right.
If anyone can point me in the right direction, or provide some example of how they got receipt validation to work in node for them, it would be a great help. It would be great if anyone has had any experience with the iap_verifier module, and exactly what data it requires. I'll provide any code example I need to, as I have been fighting this process for a few days now.
Thanks!

For anyone using the npm library "request", here's how to avoid that bothersome 21002 error.
formFields = {
'receipt-data': receiptData_64
'password': yourAppleSecret
}
verifyURL = 'https://buy.itunes.apple.com/verifyReceipt' // or 'https://sandbox.itunes.apple.com/verifyReceipt'
req = request.post({url: verifyURL, json: formFields}, function(err, res, body) {
console.log('Response:', body);
})

This is my working solution for auto-renewable subscriptions, using the npm request-promise library.
Without JSON stringify-ing the body form, I was receiving 21002 error (The data in the receipt-data property was malformed or missing)
const rp = require('request-promise');
var verifyURL = 'https://sandbox.itunes.apple.com/verifyReceipt';
// use 'https://buy.itunes.apple.com/verifyReceipt' for production
var options = {
uri: verifyURL,
method: 'POST',
headers: {
'User-Agent': 'Request-Promise',
'Content-Type': 'application/x-www-form-urlencoded',
},
json: true
};
options.form = JSON.stringify({
'receipt-data': receiptData,
'password': password
});
rp(options).then(function (resData) {
devLog.log(resData); // 0
}).catch(function (err) {
devLog.log(err);
});

Do you have composed correctly receiptData? Accordlying with Apple specification it should have the format
{"receipt-data": "your base64 receipt"}
Modifying your code wrapping the base64 receipt string with receipt-data object the validation should works
function (receiptData_base64, production, cb)
{
var url = production ? 'buy.itunes.apple.com' : 'sandbox.itunes.apple.com'
var receiptEnvelope = {
"receipt-data": receiptData_base64
};
var receiptEnvelopeStr = JSON.stringify(receiptEnvelope);
var options = {
host: url,
port: 443,
path: '/verifyReceipt',
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': Buffer.byteLength(receiptEnvelopeStr)
}
};
var req = https.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log("body: " + chunk);
cb(true, chunk);
});
res.on('error', function (error) {
console.log("error: " + error);
cb(false, error);
});
});
req.write(receiptEnvelopeStr);
req.end();
}

Node.js: Remotely Submitting Forms

I'm currently working on a sort of Web Proxy for Node.js, but I am having trouble with submitting forms, on most sites I am able to successfully submit a form but on some other sites I am not so fortunate. I can't pinpoint if there is anything I'm doing wrong.
Is there a possible better way of doing this?
Also, how would I be able to handle multipart forms using the Express.js bodyparser?
At the moment this is what I have in the way of form processing is this:
function proxy(req, res,request)
{
var sess = req.session;
var onUrl_Parse = function(url){
var Uri= new URI.URI(url);//Parses incoming url
var options = {
uri: url,
method: req.method
}
options.headers={"User-Agent": "Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0", "Cookie":req.session.cook};
if(req.body) //If x-www-form-urlencoded is posted.
{
var options = {
uri: url,
method: req.method,
body: req.rawBody
}
options.headers={"User-Agent": "Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0", "Cookie":req.session.cook, "Content-Type":"application/x-www-form-urlencoded"};
}
onRequestOptions(options, url);
}
,onRequestOptions = function(options, url)
{
request(options, function(error, response, body)
{
if(!error){
if(response.headers['set-cookie'])
req.session.cook=response.headers['set-cookie'];
Proxy_Parser.Parser(body, url, async, onParse);// Parses returned html return displayable content
}
});
}
,onParse = function(HTML_BODY)
{
if(HTML_BODY=="")
res.end();
res.write(HTML_BODY);
res.end();
console.log("DONEEEEE");
}
Url_Parser.Url(req, URI, onUrl_Parse);
}

I am not sure what exactly you are trying to accomplish, but https://github.com/felixge/node-formidable is a anyway recommended !!

I would start with something like node-http-proxy. All the hard work is done for you and you can just define the routes you want to proxy and put in some handlers for the custom response info.