Develop Reference

JS client doesn't read response when server aborts file upload

We have an API (Spring Boot) for file uploads. If file exists it returns 409.
But js client doesn't read this response and fails with error "net::ERR_CONNECTION_ABORTED".
From wireshark dump I see that the backend sends the response and then closes the connection:
I think the problem is that js client doesn't read the response as soon as it available.
However, Postman is able to read the response correctly.
Is it possible to implement Postman's behavior in javascript?
I've tried to read response stream with Fetch API, but no luck.
Sample client code:
function uploadFile() {
try {
console.log("Start file upload");
const selectedFiles = document.getElementById('input').files;
if (selectedFiles.length == 0) {
alert("No file selected");
return;
}
const file = selectedFiles[0];
return fetch("http://localhost:9091/api/v1/storage/upload?fileId=/upload-test/test.mp4", {
method: 'PUT',
body: file,
headers: {
'Content-Type': 'application/octet-stream'
},
})
.then(response => {
console.log("Processing response");
const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
return pump();
function pump() {
return reader.read().then(({ value, done }) => {
if (done) {
console.log("Finished stream reading");
return;
}
console.log(value);
return pump();
});
}
})
.catch((err) => console.error("error:", err));
} catch (error) {
console.log(error);
}
}

Zip image stream using archiver and send as express response

This is on Node/Express/Typescript. I'm trying get an image on my file system, stream it to a zip file, and then stream this zip file to the client. I have to make sure every step is streamed since this will expand to zipping up multiple files, which need to be streamed to the client as a zip.
I have the following code:
import express, { Application, Request, Response } from "express";
import fs from "fs";
import stream from "stream";
import archiver from "archiver";
app.get("/images", async (req: Request, res: Response) => {
const r = fs.createReadStream("appicon.png");
const ps = new stream.PassThrough();
// stream the image
stream.pipeline(
r,
ps,
(err) => {
if (err) {
console.log(err);
return res.sendStatus(400);
}
}
);
// zip the image and send it
let archive = archiver("zip");
archive.on("end", () => {
console.log(archive.pointer() + " total bytes");
console.log("archiver finalized");
})
archive.on('error', (err) => {
return res.status(500).send({
message: err
});
})
res.attachment('output.zip');
ps.pipe(archive);
archive.pipe(res);
archive.finalize();
});
However, when I access my /images route, I get an output.zip file which is empty.
I feel like I'm messing up the order of my pipes somehow.
What am I missing?

I figured out the issue. Here is the code that works:
app.get("/images", async (req: Request, res: Response) => {
const r = fs.createReadStream("appicon.png");
const ps = new stream.PassThrough();
stream.pipeline(
r,
ps,
(err) => {
if (err) {
console.log(err);
return res.sendStatus(400);
}
}
);
//r.pipe(ps); // alternative way to do it without pipeline
// zip the image and send it
let archive = archiver("zip");
archive.on("end", () => {
console.log(archive.pointer() + " total bytes");
console.log("archiver finalized");
})
archive.on('error', (err) => {
return res.status(500).send({
message: err
});
})
// name the output file
res.attachment("output.zip");
// pipe the zip to response
archive.pipe(res);
// add the image from stream to archive
archive.append(ps, {name: "image.png"});
archive.finalize();
});
I had to use archive.append(ps, {name: "image.png"}); to add my image stream to the zip archive.

NODE.JS How do I save JSON data without filling up my storage [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
The title really explains it.
I made a discord bot, and I added a ranking system that had its memory in my filesystem. However, if too many people join, my storage would just get filled up. Perhaps there is a way for me to access the node.js server? Maybe localhost? I just want anything that saves data for free, is not managed by anyone other than me, and doesn't take up my storage.

config
For this answer we'll establish a simple config object to store any values -
// config.json
{"counter":0}
server
We will create a simple server using http.createServer. We will use the request method and request URL to look up a handler or respond with 404 when no handler is found -
// server.js
import { createServer } from "http"
import { readFile, writeFile } from "fs/promises"
const server = createServer(async (req, res) => {
const handler = routes?.[req.method.toLowerCase()]?.[req.url]
if (handler == null) {
res.writeHead(404, {'Content-Type': 'text/plain'})
res.end(`No route for ${req.method} ${req.url}`)
}
else {
await handler(req, res)
res.end()
}
})
server.listen(8000)
Next we define the routes to /getConfig and /saveConfig -
// server.js (continued)
const routes = {
get: {
"/getConfig": async (req, res) => {
res.writeHead(200, {'content-type': 'application/json'})
res.write(await readFile("./config.json"))
}
},
post: {
"/saveConfig": async (req, res) => {
await writeFile("./config.json", await readBody(req))
res.writeHead(204)
},
"/reset": async (req, res) => {
await writeFile("./config.json", JSON.stringify({ counter: 0 }))
res.writeHead(204)
}
}
}
This depends on a reusable helper, readBody -
// server.js (continued)
function readBody(req) {
return new Promise((resolve, reject) => {
const body = []
req.on('data', chunk => body.push(Buffer.from(chunk)))
req.on('end', _ => resolve(Buffer.concat(body).toString()))
req.on('error', reject)
})
}
client
In this case your bot is the http client. The node docs for http.get include this long-winded example, but don't let it worry you -
// example from node docs
http.get('http://localhost:8000/', (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
// Any 2xx status code signals a successful response but
// here we're only checking for 200.
if (statusCode !== 200) {
error = new Error('Request Failed.\n' +
`Status Code: ${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' +
`Expected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
console.log(parsedData);
} catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
You're not expected to copy this verbatim. Imagine writing that much code each time you wanted to fetch some JSON. You can think of the http module as a low-level API that enables you to design higher-level functions -
// client.js
import * as http from "http"
function request (href, { body = "", ...options } = {}) {
return new Promise((resolve, reject) =>
http.request(href, options, res => {
const data = []
res.on('data', chunk => data.push(chunk))
res.on('end', _ => resolve({
status: res.statusCode,
headers: res.headers,
data: Buffer.concat(data).toString()
}))
})
.on('error', reject)
.end(body)
)
}
Above our request function resolves a { status, headers, data } object, and we can write specialized forms get and getJson that make it even easier to intereact with -
// client.js (continued)
async function get (href) {
const { status, headers, data } = await request(href)
if (status < 200 || status >= 300)
throw Error(status)
return { status, headers, data }
}
async function getJson (href) {
const { headers, data } = await get(href)
if (!headers['content-type'].startsWith("application/json"))
throw Error(`expected application/json but received ${headers['content-type']}`)
return JSON.parse(data)
}
We can do the same for post -
// client.js (continued)
async function post (href, body = "") {
const { status, headers, data } = await request(href, { body, method: "POST" })
if (status < 200 || status >= 300)
throw Error(status)
return { status, headers, data }
}
Finally here is our bot code. It reads the config via get, updates the config via post, and re-reads it via get to return the confirmed result -
// client.js (continued)
async function bot() {
const config = await getJson("http://localhost:8000/getConfig")
await post("http://localhost:8000/saveConfig", JSON.stringify({counter: config.counter + 1}))
return getJson("http://localhost:8000/getConfig")
}
bot().then(console.log, console.error)
run
Start the server in your terminal -
$ node ./server.js
In a separate terminal, run the client a few times -
$ node ./client.js
{ counter: 1 }
$ node ./client.js
{ counter: 2 }
$ node ./client.js
{ counter: 3 }
node modules
Above we took a sort of DIY approach to the problem. But this kind of problem has been solved many ways before. There are popular libraries like express and koajs that would make much of this a lot easier. Now that you know the purpose they serve, give 'em a try!

Just use a database, mongoDB atlas will work well in your case because it is cloud based and very easy to set up. You can follow this tutorial to connect your discord bot with mongoDB atlas.

EISDIR - EISDIR: illegal operation on a directory, read

When I try to upload an image into a bucket on the server side I'm getting the error above. I checked using the debugger that the file parameter contains the file's path and not the folder's path. Here's the code :
function uploadFile(file, directory) {
return new Promise((resolve, reject) => {
try {
const bucket = storage.bucket(BUCKET_NAME);
const bucketFile = bucket.file(directory ? `${directory}/${file.originalname}` : file.originalname);
const blobStream = bucketFile.createWriteStream();
blobStream.on('error', err => {
const status = err.status || 500;
console.log(err, status);
reject(err);
});
blobStream.on('finish', async () => {
// The public URL can be used to directly access the file via HTTP.
await bucketFile.makePublic();
const publicUrl = `https://storage.googleapis.com/${bucket.name}/${bucketFile.name}`;
resolve(publicUrl);
});
blobStream.end(file.buffer);
} catch (err) {
reject(err);
}
});
}
Can you help me?

The path of the file was right. But the path to the credentials was wrong

Get Azure uploaded blob file url

I'm uploading a data stream to Azure Storage,
I would get the link to the blob file.
let insertFile = async function (blobName,stream){
const containerName= 'texttospeechudio';
try{
await blobService.createContainerIfNotExists(containerName, {
publicAccessLevel: 'blob'},(err,result, response) => {
if(!err) {
console.log(result);
}
});
let resultstream = blobService.createWriteStreamToBlockBlob(containerName, blobName,(err,result, response)=>{
console.log(res)
});
stream.pipe(resultstream);
stream.on('error', function (error) {
console.log(error);
});
stream.once('end', function (end) {
console.log(end)
//OK
});
}
catch(err) {
console.log(err);
}
}
I added createWriteStreamToBlockBlob callback , but I'm not getting inside it.
I would find a way to get uploaded file url.

There is no file URL returned in the response according to put-blob's rest spec.
And Azure storage's resource URL can be commonly composed with following pattern:
https://{myaccount}.blob.core.windows.net/{mycontainer}/{myblob}