I am integrating with an external webhook (via Wix.com's API), and I am having an issue where the presence of an Axios (or raw Node HTTP / HTTPS) request is preventing the response from being sent.
Here is my controller code:
const https = require('https')
const request = config =>
new Promise((resolve, reject) => {
const req = https.request(config, res => {
let data = ''
res.on('data', d => {
console.log('GOT DATA:', d)
data += d
})
res.on('close', () => {
resolve(data.toString())
})
})
req.on('error', err => {
reject(err)
})
req.end()
})
const handler = async (req, res) => {
console.log('BEFORE REQUEST')
const result = await request({
hostname: 'example.com',
port: 443,
path: '/',
method: 'GET',
})
console.log('GOT RESULT')
return res.send({ success: true })
}
And I'm seeing the following logs:
BEFORE REQUEST
GOT RESULT
But I'm not getting a response (inspecting via Ngrok).
Note: when I replay the request via Ngrok, or manually send it via curl it does send a response, so it's only producing this error when receiving the request from the Webhook provider. I haven't been able to figure out what is different about the request from the webhook provider vs. the replayed request, which should theoretically be exactly the same.
When I comment out the await request(...) lines, then I do see a resonse.
So, my question is, are there any known interactions between Node HTTP / HTTPS and Express's res.send() function? It seems that after calling the HTTP / HTTPS functions, res.send just stops working.
If you pass your handler function to route:
app.post('/webhook', handler)
Then you should not return the res.send but you have to invoke it.
So your last line should look like this:
res.send({ success: true })
=========================================
Previous answer totally bad:
Your request function is not async. Correct is:
const request = async (config) => ...
I use same pattern without any issues.
I am using Jest to test my Node REST API.
The problem I have currently is that when I try testing my POST routes, I always receive a status code of 302 due to res.redirect("/").
Example of my POST route:
app.post("/login", async (req, res) => {
try {
let username = 'example'
...
return res.redirect("/");
} catch (error) {
return res.redirect("/");
}
});
jest test file:
'use strict';
const request = require('supertest');
const app = require('./index');
...
describe('Test', () => {
test('POST /login', () => {
return request(app)
.post('/login')
.set('username','example')
.expect(?)
});
});
How can I test that the page has redirected successfully?
As per the Express docs, you can specify a response code as such:
res.redirect(301, 'http://example.com')
The docs state "If not specified, status defaults to “302 “Found”."
Edit: HTTP codes 301 and 302 indicate successful redirection; 301 is permanent and 302 is temporary. Both are "successful" as far as a computer is concerned.
I assert the response.headers.location for redirection location. This way I can write test cases by mocking a single class function that causes different redirections.
test('Should handle "/redirectUri"', async () => {
const exchangeForAuthTokenSpy = jest.spyOn(
OAuth.prototype,
'exchangeForAuthToken',
)
exchangeForAuthTokenSpy.mockResolvedValue({
success: true,
access_token: 'access_token',
})
const app = server('', AuthRoutes)
const res = await request(app).get('/redirectUri?code=code&state=state')
expect(exchangeForAuthTokenSpy).toHaveBeenCalledTimes(1)
expect(exchangeForAuthTokenSpy).toHaveBeenCalledWith('code', 'state')
expect(res.status).toEqual(301)
expect(res.headers.location).toContain(
'/callback?code=200&token=access_token',
)
})
It is late, but could help someone. You can test like below
it('redirection test', function (redirect) {
request(app)
.get('/url')
.expect(302, redirect)
});
At present I'm performing the trick of piping a request req to a destination url, and piping the response back to res, like so:
const request = require('request');
const url = 'http://some.url.com' + req.originalUrl;
const destination = request(url);
// pipe req to destination...
const readableA = req.pipe(destination);
readableA.on('end', function () {
// do optional stuff on end
});
// pipe response to res...
const readableB = readableA.pipe(res);
readableB.on('end', function () {
// do optional stuff on end
});
Since request is now officially deprecated (boo hoo), is this trick at all possible using the gaxios library? I thought that setting responseType: 'stream' on the request would do something similar as above, but it doesn't seem to work.
SImilarly, can gaxios be used in the following context:
request
.get('https://some.data.com')
.on('error', function(err) {
console.log(err);
})
.pipe(unzipper.Parse())
.on('entry', myEntryHandlerFunction);
Install gaxios:
npm install gaxios
And then use request with the Readable type specified and with responseType set to 'stream'.
// script.ts
import { request } from 'gaxios';
(await(request<Readable>({
url: 'https://some.data.com',
responseType: 'stream'
}))
.data)
.on('error', function(err) {
console.log(err);
})
.pipe(unzipper.Parse())
.on('entry', myEntryHandlerFunction);
// first-example.ts
import { request } from 'gaxios';
// pipe req to destination...
const readableA = (await request<Readable>({
url: 'http://some.url.com',
method: 'POST',
data: req, // suppose `req` is a readable stream
responseType: 'stream'
})).data;
readableA.on('end', function () {
// do optional stuff on end
});
// pipe response to res...
const readableB = readableA.pipe(res);
readableB.on('end', function () {
// do optional stuff on end
});
Gaxios is a stable tool and is used in official Google API client libraries. It's based on the stable node-fetch. And it goes with TypeScript definitions out of the box. I switched to it from the deprecated request and from the plain node-fetch library.
I guess if you provide responseType as stream and use res.data, you will get a stream which you could pipe like this
const {request} = require("gaxios");
const fs = require("fs");
const {createGzip} = require("zlib");
const gzip = createGzip();
(async () => {
const res = await request({
"url": "https://www.googleapis.com/discovery/v1/apis/",
"responseType": "stream"
});
const fileStream = fs.createWriteStream("./input.json.gz");
res.data.pipe(gzip).pipe(fileStream);
})();
Looks like you are trying to basically forward requests from your express server to the clients. This worked for me.
import { request } from "gaxios";
const express = require("express");
const app = express();
const port = 3000;
app.get("/", async (req: any, res: any) => {
const readableA = (await request({
url: "https://google.com",
responseType: "stream",
})) as any;
console.log(req, readableA.data);
const readableB = await readableA.data.pipe(res);
console.log(res, readableB);
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
I imagine more complicated responses from A will require more nuiance in how to pipe it. But then you can probably just interact with express's response object directly and set the appropriate fields.
I have an image upload endpoint in my API that accepts application/octet-stream requests and handles these streams. I'd like to write test coverage for this endpoint but cannot figure out how to use supertest to stream an image.
Here's my code so far:
import request from 'supertest'
const testImage = `${__dirname}/../../../assets/test_image.jpg`
describe('Upload endpoint', () => {
test('Successfully uploads jpg image', async () =>
request(app)
.post(`${ROOT_URL}${endpoints.add_image.route}`)
.set('Authorization', `Bearer ${process.env.testUserJWT}`)
.set('content-type', 'application/octet-stream')
.pipe(fs.createReadStream(testImage))
.on('finish', (something) => {
console.log(something)
}))
})
This code produces nothing, the finish event is never called, nothing is console logged, and this test suite actually passes as nothing is expected. I cannot chain a .expect onto this request, otherwise I get this runtime error:
TypeError: (0 , _supertest2.default)(...).post(...).set(...).set(...).pipe(...).expect is not a function
How is such a thing accomplished?
This should work. To pipe data to a request you have to tell the readable stream to pipe to the request. The other way is for receiving data from the server. This also uses done instead of async as pipes do not work with async/await.
Also worth nothing is that by default the pipe will call end and then superagent will call end, resulting in an error about end being called twice. To solve this you have to tell the pipe call not to do that and then call end in the on end event of the stream.
import request from 'supertest'
const testImage = `${__dirname}/../../../assets/test_image.jpg`
describe('Upload endpoint', () => {
test('Successfully uploads jpg image', (done) => {
const req = request(app)
.post(`${ROOT_URL}${endpoints.add_image.route}`)
.set('Authorization', `Bearer ${process.env.testUserJWT}`)
.set('content-type', 'application/octet-stream')
const imgStream = fs.createReadStream(testImage);
imgStream.on('end', () => req.end(done));
imgStream.pipe(req, {end: false})
})
})
Edited to add: this has worked for me with small files. If I try testing it with a large test_image.jpg the request times out.
const testImage = `${__dirname}/../../../assets/test_image.jpg`
describe('Upload endpoint', () => {
test('Successfully uploads jpg image', async () =>
request(app)
.post(`${ROOT_URL}${endpoints.add_image.route}`)
.set('Authorization', `Bearer ${process.env.testUserJWT}`)
.attach("name",testImage,{ contentType: 'application/octet-stream' })
.expect(200)
.then(response => {
console.log("response",response);
})
);
});
I had to make assumptions about your upload method taking the body as input instead of multipart form-data. So below is an example where the raw body is passed for upload
const request = require('supertest');
const express = require('express');
const fs = require('fs')
const app = express();
var bodyParser = require('body-parser')
app.use(bodyParser.raw({type: 'application/octet-stream'}))
app.post('/user', function(req, res) {
res.status(200).json({ name: 'tobi' });
});
testImage = './package.json'
resp = request(app)
.post('/user')
resp.set('Authorization', `Bearer Test`).set('Content-Type', 'application/octet-stream')
resp.send(fs.readFileSync(testImage, 'utf-8'))
resp.expect(200)
.then(response => {
console.log("response",response);
}).catch((err) => {
console.log(err)
})
If you use multipart/form-data then below code shows an example
const request = require('supertest');
const express = require('express');
const fs = require('fs')
const app = express();
app.post('/user', function(req, res) {
// capture the encoded form data
req.on('data', (data) => {
console.log(data.toString());
});
// send a response when finished reading
// the encoded form data
req.on('end', () => {
res.status(200).json({ name: 'tobi' });
});
});
testImage = './package.json'
resp = request(app)
.post('/user')
resp.set('Authorization', `Bearer Test`)
resp.attach("file", testImage)
resp.expect(200)
.then(response => {
console.log("response",response);
}).catch((err) => {
console.log(err)
})
I think you actually want to use fs.createReadStream(testImage) to read that image into your request, since fs.createWriteStream(testImage) would be writing data into the file descriptor provided (in this case testImage). Feel free to checkout Node Streams to see how they work in more detail.
I'm not quite sure where you're getting the finish event from for supertest, but you can see how to use the .pipe() method here.
You might also want to consider using supertest multipart attachments, if that better suits your test.
How can I make an HTTP request from within Node.js or Express.js? I need to connect to another service. I am hoping the call is asynchronous and that the callback contains the remote server's response.
Here is a snippet of some code from a sample of mine. It's asynchronous and returns a JSON object. It can do any form of GET request.
Note that there are more optimal ways (just a sample) - for example, instead of concatenating the chunks you put into an array and join it etc... Hopefully, it gets you started in the right direction:
const http = require('http');
const https = require('https');
/**
* getJSON: RESTful GET request returning JSON object(s)
* #param options: http options object
* #param callback: callback to pass the results JSON object(s) back
*/
module.exports.getJSON = (options, onResult) => {
console.log('rest::getJSON');
const port = options.port == 443 ? https : http;
let output = '';
const req = port.request(options, (res) => {
console.log(`${options.host} : ${res.statusCode}`);
res.setEncoding('utf8');
res.on('data', (chunk) => {
output += chunk;
});
res.on('end', () => {
let obj = JSON.parse(output);
onResult(res.statusCode, obj);
});
});
req.on('error', (err) => {
// res.send('error: ' + err.message);
});
req.end();
};
It's called by creating an options object like:
const options = {
host: 'somesite.com',
port: 443,
path: '/some/path',
method: 'GET',
headers: {
'Content-Type': 'application/json'
}
};
And providing a callback function.
For example, in a service, I require the REST module above and then do this:
rest.getJSON(options, (statusCode, result) => {
// I could work with the resulting HTML/JSON here. I could also just return it
console.log(`onResult: (${statusCode})\n\n${JSON.stringify(result)}`);
res.statusCode = statusCode;
res.send(result);
});
UPDATE
If you're looking for async/await (linear, no callback), promises, compile time support and intellisense, we created a lightweight HTTP and REST client that fits that bill:
Microsoft typed-rest-client
Try using the simple http.get(options, callback) function in node.js:
var http = require('http');
var options = {
host: 'www.google.com',
path: '/index.html'
};
var req = http.get(options, function(res) {
console.log('STATUS: ' + res.statusCode);
console.log('HEADERS: ' + JSON.stringify(res.headers));
// Buffer the body entirely for processing as a whole.
var bodyChunks = [];
res.on('data', function(chunk) {
// You can process streamed parts here...
bodyChunks.push(chunk);
}).on('end', function() {
var body = Buffer.concat(bodyChunks);
console.log('BODY: ' + body);
// ...and/or process the entire body here.
})
});
req.on('error', function(e) {
console.log('ERROR: ' + e.message);
});
There is also a general http.request(options, callback) function which allows you to specify the request method and other request details.
Request and Superagent are pretty good libraries to use.
note: request is deprecated, use at your risk!
Using request:
var request=require('request');
request.get('https://someplace',options,function(err,res,body){
if(err) //TODO: handle err
if(res.statusCode === 200 ) //etc
//TODO Do something with response
});
You can also use Requestify, a really cool and very simple HTTP client I wrote for nodeJS + it supports caching.
Just do the following for GET method request:
var requestify = require('requestify');
requestify.get('http://example.com/api/resource')
.then(function(response) {
// Get the response body (JSON parsed or jQuery object for XMLs)
response.getBody();
}
);
This version is based on the initially proposed by bryanmac function which uses promises, better error handling, and is rewritten in ES6.
let http = require("http"),
https = require("https");
/**
* getJSON: REST get request returning JSON object(s)
* #param options: http options object
*/
exports.getJSON = function (options) {
console.log('rest::getJSON');
let reqHandler = +options.port === 443 ? https : http;
return new Promise((resolve, reject) => {
let req = reqHandler.request(options, (res) => {
let output = '';
console.log('rest::', options.host + ':' + res.statusCode);
res.setEncoding('utf8');
res.on('data', function (chunk) {
output += chunk;
});
res.on('end', () => {
try {
let obj = JSON.parse(output);
// console.log('rest::', obj);
resolve({
statusCode: res.statusCode,
data: obj
});
}
catch (err) {
console.error('rest::end', err);
reject(err);
}
});
});
req.on('error', (err) => {
console.error('rest::request', err);
reject(err);
});
req.end();
});
};
As a result you don't have to pass in a callback function, instead getJSON() returns a promise. In the following example the function is used inside of an ExpressJS route handler
router.get('/:id', (req, res, next) => {
rest.getJSON({
host: host,
path: `/posts/${req.params.id}`,
method: 'GET'
}).then(({ statusCode, data }) => {
res.json(data);
}, (error) => {
next(error);
});
});
On error it delegates the error to the server error handling middleware.
Unirest is the best library I've come across for making HTTP requests from Node. It's aiming at being a multiplatform framework, so learning how it works on Node will serve you well if you need to use an HTTP client on Ruby, PHP, Java, Python, Objective C, .Net or Windows 8 as well. As far as I can tell the unirest libraries are mostly backed by existing HTTP clients (e.g. on Java, the Apache HTTP client, on Node, Mikeal's Request libary) - Unirest just puts a nicer API on top.
Here are a couple of code examples for Node.js:
var unirest = require('unirest')
// GET a resource
unirest.get('http://httpbin.org/get')
.query({'foo': 'bar'})
.query({'stack': 'overflow'})
.end(function(res) {
if (res.error) {
console.log('GET error', res.error)
} else {
console.log('GET response', res.body)
}
})
// POST a form with an attached file
unirest.post('http://httpbin.org/post')
.field('foo', 'bar')
.field('stack', 'overflow')
.attach('myfile', 'examples.js')
.end(function(res) {
if (res.error) {
console.log('POST error', res.error)
} else {
console.log('POST response', res.body)
}
})
You can jump straight to the Node docs here
Check out shred. It's a node HTTP client created and maintained by spire.io that handles redirects, sessions, and JSON responses. It's great for interacting with rest APIs. See this blog post for more details.
Check out httpreq: it's a node library I created because I was frustrated there was no simple http GET or POST module out there ;-)
For anyone who looking for a library to send HTTP requests in NodeJS, axios is also a good choice. It supports Promises :)
Install (npm): npm install axios
Example GET request:
const axios = require('axios');
axios.get('https://google.com')
.then(function (response) {
// handle success
console.log(response);
})
.catch(function (error) {
// handle error
console.log(error);
})
Github page
Update 10/02/2022
Node.js integrates fetch in v17.5.0 in experimental mode. Now, you can use fetch to send requests just like you do on the client-side. For now, it is an experimental feature so be careful.
If you just need to make simple get requests and don't need support for any other HTTP methods take a look at: simple-get:
var get = require('simple-get');
get('http://example.com', function (err, res) {
if (err) throw err;
console.log(res.statusCode); // 200
res.pipe(process.stdout); // `res` is a stream
});
Use reqclient: not designed for scripting purpose
like request or many other libraries. Reqclient allows in the constructor
specify many configurations useful when you need to reuse the same
configuration again and again: base URL, headers, auth options,
logging options, caching, etc. Also has useful features like
query and URL parsing, automatic query encoding and JSON parsing, etc.
The best way to use the library is create a module to export the object
pointing to the API and the necessary configurations to connect with:
Module client.js:
let RequestClient = require("reqclient").RequestClient
let client = new RequestClient({
baseUrl: "https://myapp.com/api/v1",
cache: true,
auth: {user: "admin", pass: "secret"}
})
module.exports = client
And in the controllers where you need to consume the API use like this:
let client = require('client')
//let router = ...
router.get('/dashboard', (req, res) => {
// Simple GET with Promise handling to https://myapp.com/api/v1/reports/clients
client.get("reports/clients")
.then(response => {
console.log("Report for client", response.userId) // REST responses are parsed as JSON objects
res.render('clients/dashboard', {title: 'Customer Report', report: response})
})
.catch(err => {
console.error("Ups!", err)
res.status(400).render('error', {error: err})
})
})
router.get('/orders', (req, res, next) => {
// GET with query (https://myapp.com/api/v1/orders?state=open&limit=10)
client.get({"uri": "orders", "query": {"state": "open", "limit": 10}})
.then(orders => {
res.render('clients/orders', {title: 'Customer Orders', orders: orders})
})
.catch(err => someErrorHandler(req, res, next))
})
router.delete('/orders', (req, res, next) => {
// DELETE with params (https://myapp.com/api/v1/orders/1234/A987)
client.delete({
"uri": "orders/{client}/{id}",
"params": {"client": "A987", "id": 1234}
})
.then(resp => res.status(204))
.catch(err => someErrorHandler(req, res, next))
})
reqclient supports many features, but it has some that are not supported by other
libraries: OAuth2 integration and logger integration
with cURL syntax, and always returns native Promise objects.
If you ever need to send GET request to an IP as well as a Domain (Other answers did not mention you can specify a port variable), you can make use of this function:
function getCode(host, port, path, queryString) {
console.log("(" + host + ":" + port + path + ")" + "Running httpHelper.getCode()")
// Construct url and query string
const requestUrl = url.parse(url.format({
protocol: 'http',
hostname: host,
pathname: path,
port: port,
query: queryString
}));
console.log("(" + host + path + ")" + "Sending GET request")
// Send request
console.log(url.format(requestUrl))
http.get(url.format(requestUrl), (resp) => {
let data = '';
// A chunk of data has been received.
resp.on('data', (chunk) => {
console.log("GET chunk: " + chunk);
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
console.log("GET end of response: " + data);
});
}).on("error", (err) => {
console.log("GET Error: " + err);
});
}
Don't miss requiring modules at the top of your file:
http = require("http");
url = require('url')
Also bare in mind that you may use https module for communicating over secured network. so these two lines would change:
https = require("https");
...
https.get(url.format(requestUrl), (resp) => { ......
## you can use request module and promise in express to make any request ##
const promise = require('promise');
const requestModule = require('request');
const curlRequest =(requestOption) =>{
return new Promise((resolve, reject)=> {
requestModule(requestOption, (error, response, body) => {
try {
if (error) {
throw error;
}
if (body) {
try {
body = (body) ? JSON.parse(body) : body;
resolve(body);
}catch(error){
resolve(body);
}
} else {
throw new Error('something wrong');
}
} catch (error) {
reject(error);
}
})
})
};
const option = {
url : uri,
method : "GET",
headers : {
}
};
curlRequest(option).then((data)=>{
}).catch((err)=>{
})