At present I'm performing the trick of piping a request req to a destination url, and piping the response back to res, like so:
const request = require('request');
const url = 'http://some.url.com' + req.originalUrl;
const destination = request(url);
// pipe req to destination...
const readableA = req.pipe(destination);
readableA.on('end', function () {
// do optional stuff on end
});
// pipe response to res...
const readableB = readableA.pipe(res);
readableB.on('end', function () {
// do optional stuff on end
});
Since request is now officially deprecated (boo hoo), is this trick at all possible using the gaxios library? I thought that setting responseType: 'stream' on the request would do something similar as above, but it doesn't seem to work.
SImilarly, can gaxios be used in the following context:
request
.get('https://some.data.com')
.on('error', function(err) {
console.log(err);
})
.pipe(unzipper.Parse())
.on('entry', myEntryHandlerFunction);
Install gaxios:
npm install gaxios
And then use request with the Readable type specified and with responseType set to 'stream'.
// script.ts
import { request } from 'gaxios';
(await(request<Readable>({
url: 'https://some.data.com',
responseType: 'stream'
}))
.data)
.on('error', function(err) {
console.log(err);
})
.pipe(unzipper.Parse())
.on('entry', myEntryHandlerFunction);
// first-example.ts
import { request } from 'gaxios';
// pipe req to destination...
const readableA = (await request<Readable>({
url: 'http://some.url.com',
method: 'POST',
data: req, // suppose `req` is a readable stream
responseType: 'stream'
})).data;
readableA.on('end', function () {
// do optional stuff on end
});
// pipe response to res...
const readableB = readableA.pipe(res);
readableB.on('end', function () {
// do optional stuff on end
});
Gaxios is a stable tool and is used in official Google API client libraries. It's based on the stable node-fetch. And it goes with TypeScript definitions out of the box. I switched to it from the deprecated request and from the plain node-fetch library.
I guess if you provide responseType as stream and use res.data, you will get a stream which you could pipe like this
const {request} = require("gaxios");
const fs = require("fs");
const {createGzip} = require("zlib");
const gzip = createGzip();
(async () => {
const res = await request({
"url": "https://www.googleapis.com/discovery/v1/apis/",
"responseType": "stream"
});
const fileStream = fs.createWriteStream("./input.json.gz");
res.data.pipe(gzip).pipe(fileStream);
})();
Looks like you are trying to basically forward requests from your express server to the clients. This worked for me.
import { request } from "gaxios";
const express = require("express");
const app = express();
const port = 3000;
app.get("/", async (req: any, res: any) => {
const readableA = (await request({
url: "https://google.com",
responseType: "stream",
})) as any;
console.log(req, readableA.data);
const readableB = await readableA.data.pipe(res);
console.log(res, readableB);
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
I imagine more complicated responses from A will require more nuiance in how to pipe it. But then you can probably just interact with express's response object directly and set the appropriate fields.
Related
So, let's say i have /api/cat/fact.js directory.
I wanna to get JSON Data from catfact.ninja
The thing is, i can't use require() or request() package, because if i used require, it would saya Couldnt Found Module..., and if i used request one, instead of returning the JSON Data that you beable to sees in catfact.ninja, it return JSON about the api, like hostname, port, which is i don't need
/API/api/cat/fact.js:
const express = require('express');
const app = express.Router();
const request = require('request')
app.use('', (req, res) => {
const src = 'https://catfact.ninja/fact';
const facts = request({
uri: src,
hostname: 'catfact.ninja',
port: 443,
path: '/fact',
method: 'POST',
json: 'fact'
}, (error, response, body) => {
if (error) console.log(error)
console.log(body, '\n\n' + response.fact)
})
console.log(facts);
return res.jsonp(facts)
})
module.exports = app;
You are returning JSON in the wrong place. It should be returned inside of the callback function.
Here's the solution:
const express = require('express');
const request = require('request-promise')
const app = express();
app.use('', async (req, res) => {
const src = 'https://catfact.ninja/fact';
try {
const response = await request({
uri: src,
port: 443,
method: 'GET',
json: true
})
return res.jsonp(response)
} catch (err) {
return res.jsonp(err)
}
})
function startServer() {
const port = 3000
app.listen(port, () => {
console.info('Server is up on port ' + port)
})
app.on('error', (err) => {
console.error(err)
process.exit(1)
})
}
startServer()
TIP: I suggest using request-promise npm package instead of request package as it provides async-await approach, which is cleaner. Else, you can continue using callback function as second request() function parameter.
Here is how my API works:
You can find SeaweedFS here on GitHub.
And the code here:
// /drivers/seaweedfs.js Defines how API interacts with SeaweedFS
const { error } = require("console");
const http = require("http");
module.exports = class Weed {
constructor(mserver) {
this.mserver = new URL(mserver);
}
get(fileId) {
return new Promise((resolve, reject) => {
let options = {
hostname: this.mserver.hostname,
port: this.mserver.port,
method: "GET",
path: `/${fileId}`,
timeout: 6000,
};
let data;
const fileReq = http.request(options, (res) => {
console.log(`Statuscode ${res.statusCode}`);
res.on("data", (response) => {
data += response;
});
res.on("end", () => {
resolve(data);
});
});
fileReq.on("error", () => {
console.error(error);
reject();
});
fileReq.end();
});
}
};
// /routes/file.js An Express router
const express = require("express");
const router = express.Router();
const Weed = require("../drivers/seaweedfs");
let weedClient = new Weed("http://localhost:60002");
router.get("/:fileId", (req, res) => {
weedClient.get(req.params.fileId)
.then(data=>{
res.write(data)
res.end()
})
}
)
module.exports = router;
MongoDB driver not yet implemented.
When I try to GET a file(using Firefox, Hoppscotch says Could not send request: Unable to reach the API endpoint. Check your network connection and try again.), I get something whose MIME type is application/octet-stream for some reason. It's bigger than the original file. I know there must be some problems with my code, but I don't know where and how to fix it.
I'm having hard times with this package of busboy in some app I'm trying to build wanting to upload images from/to my firebase...
Already tried to look at some of your possible solutions with the same problems but perhaps different code and it didn't work.
May I share mine, so you might give me an idea about what is going wrong?
Here the lines:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp()
const path = require('path');
const os = require('os');
const cors = require('cors')({ origin: true });
const BusBoy = require('busboy')
const fs = require('fs');
const gcConfig = {
projectId: 'meeting-scheduler-9cee1',
keyFilename: "meeting-scheduler-9cee1-firebase-adminsdk-3unr4-09df5fe69a.json"
}
const { Storage } = require("#google-cloud/storage");
exports.upLoadFile = functions.https.onRequest((request, response) => {
cors(request, response,() => {
const busBoy = new BusBoy({ headers: request.headers});
let uploadData = null;
if (request.method !== "POST") {
return response.status(500).json({
message: 'Not Allowed!!'
})
}
// if (!request.headers['Content-Type']){
// return next(new Error('Bad request'));
// }
// else
// next();
busBoy.on('file', (fieldname, file, filename, encoding, mimetype) => {
const filepath = path.join(os.tmpdir(), filename);
uploadData = { file: filepath, type: mimetype };
file.pipe(fs.createWriteStream(filepath));
});
busBoy.on('finish', () => {
const bucket = gcs.bucket('meeting-scheduler-9cee1.appspot.com');
bucket
.upload(uploadData.file, {
uploadType: 'media',
metadata: {
metadata: {
contentType: uploadData.type
}
}
})
.then(() => {
response.status(200).json({
message: 'It Works!'
});
})
.catch(err => {
response.status(500).json({
error: err
})
})
});
busBoy.end(request.rawBody);
});
});
Any time I fire the function in my firebase de error is the same,I'm sharing it too:
Error: Missing Content-Type
at new Busboy (/srv/node_modules/busboy/lib/main.js:23:11)
at cors (/srv/index.js:58:24)
at cors (/srv/node_modules/cors/lib/index.js:188:7)
at /srv/node_modules/cors/lib/index.js:224:17
at originCallback (/srv/node_modules/cors/lib/index.js:214:15)
at /srv/node_modules/cors/lib/index.js:219:13
at optionsCallback (/srv/node_modules/cors/lib/index.js:199:9)
at corsMiddleware (/srv/node_modules/cors/lib/index.js:204:7)
at exports.upLoadFile.functions.https.onRequest (/srv/index.js:57:5)
at cloudFunction (/srv/node_modules/firebase-functions/lib/providers/https.js:49:9)
Ran into this myself. Turns out, main.js in the busboy module checks for content-type instead of Content-Type, and thus does not find the header. Confirmed this was the issue by fixing the casing in main.js (which isn't a feasible solution in most cases), after which the constructor call worked on my request, as expected.
Apparently this was brought up before, but the issue was closed (link).
For anyone using postman and encounters this issue: I didn't realize the file postman was referencing no longer existed, thus content-type didn't exist, but postman never complained. I switched to a valid file and it worked.
I have an image upload endpoint in my API that accepts application/octet-stream requests and handles these streams. I'd like to write test coverage for this endpoint but cannot figure out how to use supertest to stream an image.
Here's my code so far:
import request from 'supertest'
const testImage = `${__dirname}/../../../assets/test_image.jpg`
describe('Upload endpoint', () => {
test('Successfully uploads jpg image', async () =>
request(app)
.post(`${ROOT_URL}${endpoints.add_image.route}`)
.set('Authorization', `Bearer ${process.env.testUserJWT}`)
.set('content-type', 'application/octet-stream')
.pipe(fs.createReadStream(testImage))
.on('finish', (something) => {
console.log(something)
}))
})
This code produces nothing, the finish event is never called, nothing is console logged, and this test suite actually passes as nothing is expected. I cannot chain a .expect onto this request, otherwise I get this runtime error:
TypeError: (0 , _supertest2.default)(...).post(...).set(...).set(...).pipe(...).expect is not a function
How is such a thing accomplished?
This should work. To pipe data to a request you have to tell the readable stream to pipe to the request. The other way is for receiving data from the server. This also uses done instead of async as pipes do not work with async/await.
Also worth nothing is that by default the pipe will call end and then superagent will call end, resulting in an error about end being called twice. To solve this you have to tell the pipe call not to do that and then call end in the on end event of the stream.
import request from 'supertest'
const testImage = `${__dirname}/../../../assets/test_image.jpg`
describe('Upload endpoint', () => {
test('Successfully uploads jpg image', (done) => {
const req = request(app)
.post(`${ROOT_URL}${endpoints.add_image.route}`)
.set('Authorization', `Bearer ${process.env.testUserJWT}`)
.set('content-type', 'application/octet-stream')
const imgStream = fs.createReadStream(testImage);
imgStream.on('end', () => req.end(done));
imgStream.pipe(req, {end: false})
})
})
Edited to add: this has worked for me with small files. If I try testing it with a large test_image.jpg the request times out.
const testImage = `${__dirname}/../../../assets/test_image.jpg`
describe('Upload endpoint', () => {
test('Successfully uploads jpg image', async () =>
request(app)
.post(`${ROOT_URL}${endpoints.add_image.route}`)
.set('Authorization', `Bearer ${process.env.testUserJWT}`)
.attach("name",testImage,{ contentType: 'application/octet-stream' })
.expect(200)
.then(response => {
console.log("response",response);
})
);
});
I had to make assumptions about your upload method taking the body as input instead of multipart form-data. So below is an example where the raw body is passed for upload
const request = require('supertest');
const express = require('express');
const fs = require('fs')
const app = express();
var bodyParser = require('body-parser')
app.use(bodyParser.raw({type: 'application/octet-stream'}))
app.post('/user', function(req, res) {
res.status(200).json({ name: 'tobi' });
});
testImage = './package.json'
resp = request(app)
.post('/user')
resp.set('Authorization', `Bearer Test`).set('Content-Type', 'application/octet-stream')
resp.send(fs.readFileSync(testImage, 'utf-8'))
resp.expect(200)
.then(response => {
console.log("response",response);
}).catch((err) => {
console.log(err)
})
If you use multipart/form-data then below code shows an example
const request = require('supertest');
const express = require('express');
const fs = require('fs')
const app = express();
app.post('/user', function(req, res) {
// capture the encoded form data
req.on('data', (data) => {
console.log(data.toString());
});
// send a response when finished reading
// the encoded form data
req.on('end', () => {
res.status(200).json({ name: 'tobi' });
});
});
testImage = './package.json'
resp = request(app)
.post('/user')
resp.set('Authorization', `Bearer Test`)
resp.attach("file", testImage)
resp.expect(200)
.then(response => {
console.log("response",response);
}).catch((err) => {
console.log(err)
})
I think you actually want to use fs.createReadStream(testImage) to read that image into your request, since fs.createWriteStream(testImage) would be writing data into the file descriptor provided (in this case testImage). Feel free to checkout Node Streams to see how they work in more detail.
I'm not quite sure where you're getting the finish event from for supertest, but you can see how to use the .pipe() method here.
You might also want to consider using supertest multipart attachments, if that better suits your test.
I am creating a generic instance of an api instance to keep my code DRY. Running into an issue exporting the function:
TypeError: request is not a function
index.js
var express = require('express'),
app = express();
const axios = require("axios");
const request = require("./request");
app.get("/api", (req, res) => {
request({
method: 'get',
url: 'https://jsonplaceholder.typicode.com/posts/1'
}).then((resp) => {
console.log(resp);
})
});
app.listen(3000);
request.js
const axios = require("axios");
/**
* Create an Axios Client with defaults
*/
const client = axios.create({
// baseURL: constants.api.url
});
/**
* Request Wrapper with default success/error actions
*/
module.exports.request = function(options) {
const onSuccess = function(response) {
console.debug('Request Successful!', response);
return response.data;
}
const onError = function(error) {
console.error('Request Failed:', error.config);
if (error.response) {
// Request was made but server responded with something
// other than 2xx
console.error('Status:', error.response.status);
console.error('Data:', error.response.data);
console.error('Headers:', error.response.headers);
} else {
// Something else happened while setting up the request
// triggered the error
console.error('Error Message:', error.message);
}
return Promise.reject(error.response || error.message);
}
return client(options)
.then(onSuccess)
.catch(onError);
}
The original code was written with es6 but I guess node doesnt work well so I would like to convert the above function so that node can run it.
module.exports.request = ...
You just exported an object with a request function. That object is not a function.
To do this ES6 style in node you'll need to define the methods as consts:
const onSuccess(response) => {
...
}
const onError(error) => {
...
}
Then export {onSuccess, onError}
If ES6 doesn't work in your node project install babel.