Develop Reference

EventSource gets all chunks at once when streaming via Nextjs API Routes

Problem
I am trying to stream data to my React Frontend via EventSources, which kind of works. The problem is that I receive all chunks of data at once instead of time after time. Which kind of defeats the purpose of the method as it is basically now a GET request.
Code
I am using Next.js api routes and this is my code on this route:
res.writeHead(200, {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
});
answerStream.data
.on("data", (chunk: string) => res.write(chunk))
.on("error", (error: Error) => {
console.error(error);
res.end();
})
.on("end", () => res.end());
This is the code in the Frontend:
const eventSource = new EventSource(url);
eventSource.addEventListener("message", e => {
try {
if (e.data == "[DONE]") eventSource.close();
else {
const messageObject = JSON.parse(e.data);
setArticle(state => (state += messageObject?.choices[0]?.text));
}
} catch (error) {
console.log(error);
}
});
eventSource.addEventListener("close", e => {
console.log("Connection closed with the server");
setSubmitting(false);
});
eventSource.addEventListener("error", e => {
setError(e?.message || "Leider ist ein Fehler aufgetreten");
setSubmitting(false);
eventSource.close();
});
Info
Is there anything in my implementation wrong? Or does Next.js have a strange handling of EventSource?

Solution
Ok, I have found the solution in this Github discussion:
It seems the issue is that the middleware adds a gzip encoding which the browser has negotiated using the header
In order to fix this, it is needed to overwrite that behavior by adding 'Content-Encoding': 'none' to the headers on the server:
res.writeHead(200, {
Connection: 'keep-alive',
'Content-Encoding': 'none',
'Cache-Control': 'no-cache',
'Content-Type': 'text/event-stream',
});
Alternatively, one could use a custom server.js.

POST request from Node to another server with windows credentials

I am trying to send a post request to a service from my node server.
Node is running on http://localhost:3000. The method I am trying to reach is reachable through http://localhost:80/some/adress/business/layer/myMethod.
var options = {
host: 'localhost',
path: '/some/adress/business/layer/myMethod',
port: '80',
method: 'POST',
headers: {
'Content-type': 'application/json',
'Content-Length': data.length
}
};
var req = http.request(options, function (resu) {
console.log('statusCode: ' + res.statusCode)
resu.on('data', function (d) {
console.log(d);
});
resu.on('error', function (err) {
console.log(err);
});
resu.on('end', function () {
res.jsonp({ result: true });
res.end();
});
});
req.write("data");
req.end();
The request works fine, well more or less. I am getting a 401 status back. The question is: How can I send windows credentials from node to the named server running on localhost:80... ?

Without knowing the exact details of your setup, I can't be sure, but you probably need to use NTLM authentication. There are several libraries that do this for node. Take a look at this question. Hope this helps!

My webserver sends 404 when i make the request through react app, but works fine when I access it directly through chrome or postman

My web server is working fine when I call it through chrome. However when I am using fetch or axiom in my react-app to call the same url, it returns 404 Not Found. The react-app also sends a options request to same url which returns status 200. I have even set this header to allow the origin.
app.use(async (ctx, next) => {
var origin = ctx.headers.origin;
ctx.set({
'Access-Control-Allow-Headers': 'authorization,Content-Type,refresh',
'Access-Control-Allow-Methods': "GET,HEAD,POST,PUT,DELETE",
'Access-Control-Allow-Origin': '*',
'Connection': 'keep-alive',
'Access-Control-Allow-Credentials': true
});
await next();
console.log('______________')
});
require('./routes')(app); //2
my ./routes.js file contain
//router.post('/signin' , signinController.signin);
router.get('/signin' , signinController.signin);
can you please tell me what I am missing here. My axios call
axios.get('http://localhost:3002/signin')
.then(function (response) {
console.log(response);
})
.catch(function (error) {
console.log(error);
});

Alright I tried your code out and it works for me if I use app.use(router.allowedMethods()) middleware to pass preflights, OR declare separate option-routes for each route; Or instead of using router.get() there is router.all('/signin', ...) which will also catch options-requests.
This is my complete server code:
const app = new Koa();
const router = new Router();
router.get('/signin' , (ctx, next) => {
ctx.body = '{"status":"hello"}';
});
// Adding a options-route works (instead of router.allowedMethods())
router.options('/signin' , (ctx, next) => {
ctx.body = '';
});
app.use(async (ctx, next) => {
var origin = ctx.headers.origin;
ctx.set({
'Access-Control-Allow-Headers': 'authorization,Content-Type,refresh',
'Access-Control-Allow-Methods': "GET,HEAD,POST,PUT,DELETE",
'Access-Control-Allow-Origin': '*',
'Connection': 'keep-alive',
'Access-Control-Allow-Credentials': true
});
await next();
console.log('______________')
});
app.use(router.routes());
// app.use(router.allowedMethods()); // This works instead of separate option-routes.
app.use((ctx, next) => {
console.log('here');
next();
});
app.listen(9787);
And this is my call:
axios.defaults.headers.common['Authorization'] = "sometoken";
axios.get('http://localhost:9787/signin').then(response =>{
console.log(response); // {data: {…}, status: 200, ... }
});
(I had to add the header or the CORS won't trigger.)
I've tried setting various origin and headers and it does obey your middleware.
But I did notice that if you choose to use allowedMethods(), that will override your Access-Control-Allow-Methods (and make it useless).

It has to do with cross origin support
You can use this package #koa/cors#2 to add cors support
const Koa = require('koa');
const cors = require('#koa/cors');
const app = new Koa();
app.use(cors());
Heres the link to the package
https://github.com/koajs/cors

Chunked base64 string, piece by piece using http nodejs

From my API(nodejs), I'm accessing a third-party API (using http) to download files.
The service returns a Base64 string, chopped into smaller pieces, to be able to handle larger files.
Is it possible to do multiple http-requests (loop ?) to the third-party service, send each piece in response, to the browser until there is no longer any response from the third-party service?
The reason i want to do this, is because I don't want to consume to much memory on the node server.
I will put the pieces back together in the browser.
Any suggestions on how to do this?
See my current code below.
var request = require('request');
router.post('/getfiledata', function(req, res) {
var fileid = req.body.fileid;
var token = req.headers.authorization;
getFileData(req, res, dbconfig, fileid, token, function(err, chunkOfFile) {
if (err) {
res.status(500).send({
status: 500,
message: err
});
return;
}
res.send(chunkOfFile);
});
});
function getFileData(req, res, dbconfig, fileid, token, next) {
var url ="http://*ip*/service/rest/getfiledata";
var reqbody = {
fileId: fileid
};
var options = {
url: url,
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': token
},
json: true,
body: reqbody
};
/*SOME LOOP HERE TO GET EACH CHUNK AND SEND TO BROWSER*/
request(options, function(err, resp, body) {
if (err) {
console.log(err);
next(err, undefined);
return;
} else {
next(undefined, body)
};
});
};

I think you need Socket.io to push chunks to the browser.
Server :
socket.send("chunk", chunkOfFile)
Client :
let fullString = ""
socket.on("chunk", chunkOfFile => fullString += chunkOfFile )
Something like that

The request library you are using allows for streaming of data from one source to another. Check out the documentation on github.
Here is an example from that page:
request
.get(source)
.on('response', function(response) {
console.log(response.statusCode) // 200
console.log(response.headers['content-type']) // 'image/png'
})
.pipe(request.put(destination))
You may choose to use the http module from Nodejs, as it implements the EventEmitter class too.

I ended up doing a recursive loop from the client. Sending http-requests to my API(node) until the response no longer returns any base64 data chunks.
Thank you guys!

How to resolve NODE.Js HTTP POST "ECONNRESET" Error

I have this function and the below data which is passed into this function returns a ECONNRESET, socket hang up error. However, when the discountCode array is reduced to like only 10 objects, it can POST without any problem.
What could the cause for this problem? I tried to do multiple req.write() by segmenting the data in Buffer, however that doesn't work out well. Any NodeJs ninja could give some insights to this problem?
createObj: function(data, address, port, callback) {
//console.log('Create Reward: '+JSON.stringify(data));
var post_data = JSON.stringify(data);
var pathName = '/me/api/v1/yyy/'+data.idBusinessClient+'/newObj';
//
var options = {
hostname: address,
port: port,
path: pathName,
method: 'POST',
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Accept': 'application/json',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8'
}
};
// http call to REST API server
var req = restHttp.request(options, function(res) {
console.log('HTTP API server PUT Reward response received.');
var resData = '';
res.on('data', function(replyData) {
// Check reply data for error.
console.log(replyData.toString('utf8'));
if(replyData !== 'undefined')
resData += replyData;
});
res.on('end', function() {
//<TODO>Process the data</TODO>
callback(JSON.parse(resData));
});
});
req.write(post_data);
req.end();
console.log('write end');
req.on('close', function() {
console.log('connection closed!');
});
req.on('error', function(err) {
console.log('http request error : '+err);
callback({'error':err});
throw err;
});
req.on('socket', function(socket) {
console.log('socket size:'+socket.bufferSize);
socket.on('data', function(data) {
console.log('socket data:'+data);
});
});
}
]}`

I had the same problem and was able to resolve it by adding a Content-Length header:
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Content-Length': Buffer.byteLength(post_data),
'Accept': 'application/json',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8'
}
However, I still have no clear idea why a missing Content-Length header causes such a trouble. I assume it's some kind of weirdness in the internal Node.js code. Maybe you can even call it a bug, but I'm not sure about that ;)
PS: I'm absolutely interested more information about the cause of this problem. So please leave a comment if you have any idea...

When you change the content of response for sure you need also to update on header the content length:
headers: {
...
'Content-Length': Buffer.byteLength(post_data),
...
}
But i run on this problem also when i try to make multiple request and seems that this is not well managed on different library so a workaround that i have found if this problem persist is to add on headers:
headers: {
...
connection: 'Close'
...
}
So if you are making request on different servers.. this close the connection after finish the process. This worked for me in net, node-http-proxy.

If Express and http-proxy-middleware is used to make the POST call, and some body parser middleware is used like express.json(), the request interceptor fixRequestBody must be used (more info). Otherwise the POST call will hang with the ECONNRESET error.
const express = require('express');
const { createProxyMiddleware, fixRequestBody } = require('http-proxy-middleware');
const app = express();
app.use(express.json());
app.post(
'/path',
createProxyMiddleware('/path', {
target: API_URL,
changeOrigin: true,
pathRewrite: (path, req) => `/something/${req?.body?.someParameter}`,
onProxyReq: fixRequestBody // <- Add this line
});

Had the same problem. The solution for me was to append it to the proxy for it to work. If you're not using a proxy, you can probably just append it to the post request itself.
With proxy:
import express from 'express';
import { createProxyMiddleware } from 'http-proxy-middleware';
import logger from './logger';
// setup routes
server.get('/isAlive', (req, res) => res.send('Alive'));
server.get('/isReady', (req, res) => res.send('Ready'));
server.use(express.static(path.join(__dirname, '../build')));
const restream = (proxyReq, req, res, options) => {
if (req.body) {
let bodyData = JSON.stringify(req.body);
proxyReq.setHeader('Content-Type', 'application/json');
proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
proxyReq.write(bodyData);
}
};
server.use(
'/api',
createProxyMiddleware({
target: 'http://your-backendUrl-api',
onProxyReq: restream,
changeOrigin: true,
proxyTimeout: 30000,
secure: true,
logLevel: 'info',
onError: (err, req, res) => {
logger.error('error in proxy', err, req, res);
},
})
);
E.g without proxy:
import axios, { AxiosResponse } from 'axios';
const api = axios.create({
baseURL: '/api/....',
timeout: 35000,
withCredentials: true,
headers: { Pragma: 'no-cache', 'Cache-Control': 'no-cache' },
validateStatus: (status) => status < 400,
});
const response = await api.post(
`/somepath/${exampleInjectedId}/somepathToRestAPI`,
{
...payload
},
{
baseURL: '/api/...',
timeout: 35000,
withCredentials: true,
headers: {
Pragma: 'no-cache',
'Cache-Control': 'no-cache',
'Content-Length': Buffer.byteLength(
JSON.stringify({
...payload
})
),
},
validateStatus: (status) => status < 400,
}
);