EventSource gets all chunks at once when streaming via Nextjs API Routes

EventSource gets all chunks at once when streaming via Nextjs API Routes - javascript

Problem
I am trying to stream data to my React Frontend via EventSources, which kind of works. The problem is that I receive all chunks of data at once instead of time after time. Which kind of defeats the purpose of the method as it is basically now a GET request.
Code
I am using Next.js api routes and this is my code on this route:
res.writeHead(200, {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
});
answerStream.data
.on("data", (chunk: string) => res.write(chunk))
.on("error", (error: Error) => {
console.error(error);
res.end();
})
.on("end", () => res.end());
This is the code in the Frontend:
const eventSource = new EventSource(url);
eventSource.addEventListener("message", e => {
try {
if (e.data == "[DONE]") eventSource.close();
else {
const messageObject = JSON.parse(e.data);
setArticle(state => (state += messageObject?.choices[0]?.text));
}
} catch (error) {
console.log(error);
}
});
eventSource.addEventListener("close", e => {
console.log("Connection closed with the server");
setSubmitting(false);
});
eventSource.addEventListener("error", e => {
setError(e?.message || "Leider ist ein Fehler aufgetreten");
setSubmitting(false);
eventSource.close();
});
Info
Is there anything in my implementation wrong? Or does Next.js have a strange handling of EventSource?

Solution
Ok, I have found the solution in this Github discussion:
It seems the issue is that the middleware adds a gzip encoding which the browser has negotiated using the header
In order to fix this, it is needed to overwrite that behavior by adding 'Content-Encoding': 'none' to the headers on the server:
res.writeHead(200, {
Connection: 'keep-alive',
'Content-Encoding': 'none',
'Cache-Control': 'no-cache',
'Content-Type': 'text/event-stream',
});
Alternatively, one could use a custom server.js.

Related

How can I add the Accept-Ranges header to the response in nodejs

I'm getting a file from another server and pass it to the client with the following code
(to download the file I'm using axios):
app.get('/download', (req, res) => {
downloadFile(res)
})
async function downloadFile(res) {
const url = 'http://www.sample-videos.com/video123/mp4/720/big_buck_bunny_720p_5mb.mp4'
console.log('Connecting …')
const { data, headers } = await Axios({
url,
method: 'GET',
responseType: 'stream'
})
const totalLength = headers['content-length']
let offset = 0
res.set({
"Content-Disposition": 'attachment; filename="big_buck_bunny_720p_5mb.mp4"',
"Content-Type": "application/octet-stream",
"Content-Length": totalLength,
// "Range": `bytes=${offset}` // my problem is here ....
});
data.on('data', (chunk) => {
res.write(chunk)
})
data.on('close', function () {
res.end('success')
})
data.on('error', function () {
res.send('something went wrong ....')
})
}
So far everything is working properly.
my problem is to add the Accept-Ranges header, I tried a lot but I do not know How to do it?
any help will be appreciated ...

I think the only thing you need to do is to add it here:
res.set({
"Content-Disposition": 'attachment; filename="big_buck_bunny_720p_5mb.mp4"',
"Content-Type": "application/octet-stream",
"Content-Length": totalLength,
"Accept-Ranges": // add here the value your server accepts
});
Check the Accept-Ranges documentation here:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Ranges
Its syntax is like this:
Accept-Ranges: <range-unit>
Accept-Ranges: none

NodeJS express/request: piping a POST request with body parsing issue

I'm trying to pipe a request handling by a remote server, along with the following line:
Unfortunately pipe doesn't work well with post body, could you suggest how can I solve this issue?
self.downloadPriceLists = function (req, res, next) {
const options = {
url: `http://${env.MAILER_HOST}:${env.MAILER_PORT}/getpricelist/`,
method: 'POST',
json: true, // <--Very important!!!
headers: req.headers,
headers: {
'Content-Type': 'application/json;charset=UTF-8',
"Access-Control-Allow-Origin": "*",
},
body: {
userID: req.user.id,
exportAsOf: req.body.exportAsOf,
activationDate: req.body.activationDate,
},
};
console.log("options:", options);
// remoteResponse :: res
// remoteBody :: body
const myReq = request.post(options, function (error, remoteResponse, remoteBody) {
res.setHeader('Access-Control-Expose-Headers', 'Content-Disposition');
remoteResponse.headers.hasOwnProperty('content-disposition') && res.setHeader('Content-disposition', remoteResponse.headers['content-disposition']);
remoteResponse.headers.hasOwnProperty('content-type') && res.setHeader('Content-type', remoteResponse.headers['content-type']);
if (error) {
console.error('request fail:', error);
return res.status(500).end('Error');
}
console.log('submit successful:', remoteResponse.headers);
res.pipe(remoteBody);
});
// Handle errors
myReq.on('error', function (err) {
console.log("++++++++++++sendReq Handle errors:", err);
res.status(500).end("Error:" + err);
});
};

Should you not be piping streams and not scalar data?
res.pipe(remoteBody); does look right to me, if anything, res.pipe(remoteResponse); seems more right.
Have you considered just writing the response of the inner request to the outer one without piping? Like so res.json(remoteBody); ?

How to prevent "other" domains from accessing URL

I have a Node/React application that pulls data from an API.
On the server side (server.js) I have an “internal” URL that grabs the data from the API and then returns it as JSON. Like this:
server.get('/api/data', (req, res) => {
Promise.resolve(getAPIData()).then(data => {
res.writeHead(200, {'Content-Type': 'application/json'})
return res.end(serialize(data))
}).catch((error) => {
console.log(error)
res.writeHead(200, {'Content-Type': 'application/json'})
return res.end(serialize({}))
})
})
Then on the front-end in React (Next.js) I fetch this JSON with the example.com/api/data URL and display it on the page.
My problem is I want to restrict the example.com/api/data URL to be accessed only from my domain (which in this case let's say is example.com). I don’t want someotherdomain.com to be able to access the same data from the url.
I installed helmet, but that didn’t quite do what I wanted. CORS didn’t seem to be the answer either.
Is there a standard way to do this?
Update:
Here's what I tried to do with CORS. Using this package:
https://github.com/expressjs/cors
const cors = require('cors')
...
const corsOptions = {
origin: (process.env.NODE_ENV === 'development') ? 'http://localhost:3000' : 'https://example.com',
optionsSuccessStatus: 200 // some legacy browsers (IE11, various SmartTVs) choke on 204
}
And then on the route:
server.get('/api/data', cors(corsOptions), (req, res) => {
Promise.resolve(getAPIData()).then(data => {
res.writeHead(200, {'Content-Type': 'application/json'})
return res.end(serialize(data))
}).catch((error) => {
console.log(error)
res.writeHead(200, {'Content-Type': 'application/json'})
return res.end(serialize({}))
})
})

Node.js express force client to skip cache (refresh)

I'm sending a file in node.js / express like this:
res.header("Content-Type", mime.lookup(file));
res.sendFile(file);
On sever side I can detect if that file has changed and I want to force client to skip cache only in that case.
something like:
res.header("Content-Type", mime.lookup(file));
if (fileHasChanged(file))
res.header("some-header-telling-client-to-skip-cache", "some-value");
res.sendFile(file);
How can I do that?

Looks like, you need to add 'Cache-Control': 'no-cache' to you the sendFile() options:
const options = {
headers: {
'Cache-Control': 'no-cache',
}
};
res.sendFile(file, options, (err) => {
if (err) {
next(err);
} else {
// Success
}
});

How to resolve NODE.Js HTTP POST "ECONNRESET" Error

I have this function and the below data which is passed into this function returns a ECONNRESET, socket hang up error. However, when the discountCode array is reduced to like only 10 objects, it can POST without any problem.
What could the cause for this problem? I tried to do multiple req.write() by segmenting the data in Buffer, however that doesn't work out well. Any NodeJs ninja could give some insights to this problem?
createObj: function(data, address, port, callback) {
//console.log('Create Reward: '+JSON.stringify(data));
var post_data = JSON.stringify(data);
var pathName = '/me/api/v1/yyy/'+data.idBusinessClient+'/newObj';
//
var options = {
hostname: address,
port: port,
path: pathName,
method: 'POST',
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Accept': 'application/json',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8'
}
};
// http call to REST API server
var req = restHttp.request(options, function(res) {
console.log('HTTP API server PUT Reward response received.');
var resData = '';
res.on('data', function(replyData) {
// Check reply data for error.
console.log(replyData.toString('utf8'));
if(replyData !== 'undefined')
resData += replyData;
});
res.on('end', function() {
//<TODO>Process the data</TODO>
callback(JSON.parse(resData));
});
});
req.write(post_data);
req.end();
console.log('write end');
req.on('close', function() {
console.log('connection closed!');
});
req.on('error', function(err) {
console.log('http request error : '+err);
callback({'error':err});
throw err;
});
req.on('socket', function(socket) {
console.log('socket size:'+socket.bufferSize);
socket.on('data', function(data) {
console.log('socket data:'+data);
});
});
}
]}`

I had the same problem and was able to resolve it by adding a Content-Length header:
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Content-Length': Buffer.byteLength(post_data),
'Accept': 'application/json',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8'
}
However, I still have no clear idea why a missing Content-Length header causes such a trouble. I assume it's some kind of weirdness in the internal Node.js code. Maybe you can even call it a bug, but I'm not sure about that ;)
PS: I'm absolutely interested more information about the cause of this problem. So please leave a comment if you have any idea...

When you change the content of response for sure you need also to update on header the content length:
headers: {
...
'Content-Length': Buffer.byteLength(post_data),
...
}
But i run on this problem also when i try to make multiple request and seems that this is not well managed on different library so a workaround that i have found if this problem persist is to add on headers:
headers: {
...
connection: 'Close'
...
}
So if you are making request on different servers.. this close the connection after finish the process. This worked for me in net, node-http-proxy.

If Express and http-proxy-middleware is used to make the POST call, and some body parser middleware is used like express.json(), the request interceptor fixRequestBody must be used (more info). Otherwise the POST call will hang with the ECONNRESET error.
const express = require('express');
const { createProxyMiddleware, fixRequestBody } = require('http-proxy-middleware');
const app = express();
app.use(express.json());
app.post(
'/path',
createProxyMiddleware('/path', {
target: API_URL,
changeOrigin: true,
pathRewrite: (path, req) => `/something/${req?.body?.someParameter}`,
onProxyReq: fixRequestBody // <- Add this line
});

Had the same problem. The solution for me was to append it to the proxy for it to work. If you're not using a proxy, you can probably just append it to the post request itself.
With proxy:
import express from 'express';
import { createProxyMiddleware } from 'http-proxy-middleware';
import logger from './logger';
// setup routes
server.get('/isAlive', (req, res) => res.send('Alive'));
server.get('/isReady', (req, res) => res.send('Ready'));
server.use(express.static(path.join(__dirname, '../build')));
const restream = (proxyReq, req, res, options) => {
if (req.body) {
let bodyData = JSON.stringify(req.body);
proxyReq.setHeader('Content-Type', 'application/json');
proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
proxyReq.write(bodyData);
}
};
server.use(
'/api',
createProxyMiddleware({
target: 'http://your-backendUrl-api',
onProxyReq: restream,
changeOrigin: true,
proxyTimeout: 30000,
secure: true,
logLevel: 'info',
onError: (err, req, res) => {
logger.error('error in proxy', err, req, res);
},
})
);
E.g without proxy:
import axios, { AxiosResponse } from 'axios';
const api = axios.create({
baseURL: '/api/....',
timeout: 35000,
withCredentials: true,
headers: { Pragma: 'no-cache', 'Cache-Control': 'no-cache' },
validateStatus: (status) => status < 400,
});
const response = await api.post(
`/somepath/${exampleInjectedId}/somepathToRestAPI`,
{
...payload
},
{
baseURL: '/api/...',
timeout: 35000,
withCredentials: true,
headers: {
Pragma: 'no-cache',
'Cache-Control': 'no-cache',
'Content-Length': Buffer.byteLength(
JSON.stringify({
...payload
})
),
},
validateStatus: (status) => status < 400,
}
);

Develop Reference

JavaScript is the programming language of the Web.

EventSource gets all chunks at once when streaming via Nextjs API Routes - javascript

Related

How can I add the Accept-Ranges header to the response in nodejs

NodeJS express/request: piping a POST request with body parsing issue

How to prevent "other" domains from accessing URL

Node.js express force client to skip cache (refresh)

How to resolve NODE.Js HTTP POST "ECONNRESET" Error

Categories

Resources