Writing and reading to a file using streams - javascript

The below code writes to a file whatever I type in the console. It also simultaneously reads from the same file and displays whatever that's in the file.
Everything I type in the console is saved in the file. I manually went and checked it. But whatever I type doesn't get displayed simultaneously.
const fs = require('fs');
const Wstream = fs.createWriteStream('./testfile.txt', {encoding: 'utf8'});
const Rstream = fs.createReadStream('./testfile.txt', {encoding: 'utf8'});
process.stdin.pipe(Wstream);
Rstream.pipe(process.stdout);
Why isn't this the same as the following?
process.stdin.pipe(process.stdout);

The read stream will be closed when the data in ./testfile.txt is fully piped. It will not wait for additional entries or changes.
You can use fs.watch, to listen for file changes or even better use something easier like tail-stream or node-tail.

Related

How to convert Base64 string to PNG (currently using Jimp)?

I'm currently creating a real-time chat application. This is a web application that uses node.js for the backend and uses socket.io to connect back and forth.
Currently, I'm working on creating user profiles with profile pictures. These profile pictures will be stored in a folder called images/profiles/. The file will be named by the user's id. For example: user with the id 1 will have their profile pictures stored in images/profiles/1.png. Very self-explanatory.
When the user submits the form to change their profile picture, the browser JavaScript will get the image, and send it to the server:
form.addEventListener('submit', handleForm)
function handleForm(event) {
event.preventDefault(); // stop page from reloading
let profilePicture; // set variable for profile picture
let profilePictureInput = document.getElementById('profilePictureInput'); // get image input
const files = profilePictureInput.files[0]; // get input's files
if (files) {
const fileReader = new FileReader(); // initialize file reader
fileReader.readAsDataURL(files);
fileReader.onload = function () {
profilePicture = this.result; // put result into variable
socket.emit("request-name", {
profilePicture: profilePicture,
id: userID,
}); // send result, along with user id, to server
}
}
I've commented most of the code so it's easy to follow. The server then gets this information. With this information, the server is supposed to convert the sent image to a png format (I can do whatever format, but it has to be the same format for all images). I am currently using the jimp library to do this task, but it doesn't seem to work.
const jimp = require('jimp'); // initialize Jimp
socket.on('request-name', (data) => { // when request has been received
// read the buffer from image (I'm not 100% sure what Buffer.from() does, but I saw this online)
jimp.read(Buffer.from(data.profilePicture), function (error, image) {
if (error) throw error; // throw error if there is one
image.write(`images/profiles/${data.id}.png`); // write image to designated place
}
});
The error I get:
Error: Could not find MIME for Buffer <null>
I've scoured the internet for answers but was unable to find any. I am available to use another library if this helps. I can also change the file format (.png to .jpg or .jpeg, if needed; it just needs to be consistent with all files). The only things I cannot change are the use of JavaScript/Node.js and socket.io to send the information to the server.
Thank you in advance. Any and all help is appreciated.
If you're just getting the data URI as a string, then you can construct a buffer with it and then use the built in fs to write the file. Make sure the relative path is accurate.
socket.on('request-name', data => {
const imgBuffer = Buffer.from(data.profilePicture, 'base64');
fs.writeFile(`images/profiles/${data.id}.png`, imgBuffer);
}

Is it possible to get an image from a message and add it to a folder in Discord.js?

Like is it possible to do something like !image add [image file] and then add the attachment to a folder? I think i can do that with fs, but i'm not sure how
You can use the fs function fs.writeFile() or fs.writeFileSync(). This function accepts the absolute path to a file to write to, and the data to write. In your case, it should be a buffer or stream.
// const fs = require('fs');
fs.writeFileSync('./some_dir/some_file_name.extension', data);
To get the data in question, you should access Message#attachments(), a collection of all attachments on the message. Assuming you only want the first, you can use Collection#first() to narrow down the results.
const attachment = message.attachments.first();
if (!attachment) {
// maybe place in some error handling
}
Unfortunately, the MessageAttachment class doesn't actually hold a buffer/stream representing the attachment, only the URL leading to it. This means you'll need a third-party library such as axios or node-fetch.
// const fetch = require('node-fetch');
fetch(attachment.url)
.then(res => res.buffer())
.then(buffer => {
fs.writeFileSync(`./images/${attachment.name}`, buffer);
});
Make sure to validate that URL to make sure it's an image!
if(!/\.(png|jpe?g|svg)$/.test(attachment.url)) {
// this attachment isn't an image!
// we don't want to be downloading .exe files now, do we?
}
Finally, you should also be weary that if two files are named the same, such as image.png, trying to write the second one will overwrite the first. One way to overcome that issue is to add numerical suffixes to duplicates, such as image.png, image-1.png, image-2.png, etc. That could work out a little like this:
fetch(attachment.url)
.then(res => res.buffer())
.then(buffer => {
let path = `./images/${attachment.name}`;
// increment the suffix every iteration until a file
// by the same name cannot be found
for (let count = 1; fs.existsSync(path); count++) {
path = `./images/${attachment.name}-${count}`;
}
fs.writeFileSync(path, buffer);
});

Use gunzip on a folder

Right so I have a folder full of other folders, which are compressed into .gz files. Inside these folders is text files.
I want to have a program that loops through these text files to see if they contain a specific string, but to do so I need to uncompress them first. I don't want to start messing about with files (unless I can just make them temporarily and delete after), i just want to perform operations on the contents of the .gz folder. I've tried zlib.Gunzip()._outBuffer.toString() which gives a load of gibberish when used on a compressed folder.
How should I proceed?
Had to to something quite similar recently, here's what worked for me:
Basically you just read in the file into a buffer which you then can pass to the gunzip function. This will return another buffer on which you can invoke toString('utf8') in order the get the contents as a string, which is exactly what you need:
const util = require('util');
let {gunzip} = require('zlib');
const fs = require('fs');
gunzip = util.promisify(gunzip);
async function getStringFromGzipFile(inputFilePath) {
const sourceBuffer = await fs.promises.readFile(inputFilePath);
return await gunzip(sourceBuffer);
}
(async () => {
const stringContent = await getStringFromGzipFile('/path/to/file');
console.log(stringContent);
})()
EDIT:
If you want to gunzip and extract a directory, you can use tar-fs which will extract the contents to a specified directory. Once your done processing the files in it you could just remove the directory. Here's how you would gunzip and extract a .tar.gz:
function gunzipFolder(sourceDir, destination) {
fs.createReadStream(sourceDir)
.pipe(zlib.createGunzip())
.pipe(tar.extract(destination));
}

Nodejs - data download and saving in chunks

What I am trying to do
I am fetching an image from a website with node-fetch, and I'm saving it to disk. My question is if I can save that image in chunks rather than downloading all of the image, and then saving all of the downloaded data at once. Doesn't seem like that node way to do things.
node-fetch documentation on how to save images
fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
.then(res => {
const dest = fs.createWriteStream('./octocat.png');
res.body.pipe(dest);
});
That results in a 0 byte big image, on closer inspection res.body.pipe() is undefined, but doesn't throw an error ?
What I currently have
After a bit of investigation I noticed that the "downloaded" data is in a blob form. MDN says that I should "process"(?) that blob with arrayBuffer and then I can save that buffer to disk with .write.
const result = await (await (await fetch(url)).blob()).arrayBuffer();
const fileStream = fs.createWriteStream('./1.jpg').write(Buffer.from(result));
This setup "works", but the saving nor the downloading of the file is not done in chunks, everything is basically synchronous. I feel like I am missing something or I am doing something wrong.

node.js Transfer and saving files using TCP Server

I have a lot of devices sending messages to a TCP Server written in node. The main task of the TCP server is to route some of that messages to redis in order to be processed by another app.
I've written a simple server that does the job quite well. The structure of the code is basically this (not the actual code, details hidden):
const net = require("net");
net.createServer(socket => {
socket.on("data", buffer => {
const data = buffer.toString();
if (shouldRouteMessage(data)) {
redis.publish(data);
}
});
});
Most of the messages are like: {"text":"message body"}, or {"lng":32.45,"lat":12.32}. But sometimes I need to process a message like {"audio":"...encoded audio..."} that spans several "data" events.
What I need in this case is to save the encoded audio into a file and send to redis {"audio":"path/to/audio-file.mp3"} where the route is the file with the audio data received.
One simple option is to store the buffers until I detect the end of the message and then save all them to a file, but that means, among other things, that I must keep the file on memory before saving to disk.
I hope there are better options using streams and pipes. ¿Any suggestions? (some code examples, would be nice)
Thanks
I finally solved, so I post the solution here for documentation purposes (and with some luck, to help others).
The solution was, indeed, quite simple: just open a write stream to a file and write the data packets as they are received. Something like this:
const net = require("net");
const fs = require("fs");
net.createServer(socket => {
socket.on("data", buffer => {
let file = null;
let filePath = null;
const data = buffer.toString();
if (shouldRouteMessage(data)) {
// just publish the message
redis.publish(data);
} else if (isAudioStart(data)) {
// create a write stream to a file and write the first data packet
filePath = buildFilePath(data);
file = fs.createWriteStream(filePath);
file.write(data);
} else if (isLastFragment(data)) {
// if is the last fragment, write it, close the file and publish the result
file.write(data);
file.close();
redis.publish(filePath);
file = filePath = null;
} else if (isDataFragment(data)) {
// just write (stream) it to file
file.write(data);
}
});
});
Note: shouldRouteMessage, buildFilePath, isDataFragment, and isLastFragment are custom functions that depends on the kind of data.
In this way, the incoming data is streamed to the file directly and no need to save the contents in memory before. node's streams rocks!
As always the devil is in the details. Some checks are necesary to, for example, ensure there's always a file when you want to write it. Remember also to set the proper encoding when converting to string (for example: buffer.toString('binary'); did the trick for me). Depending on your data format, the shouldRouteMessage, isAudioStart... and all this custom functions can be more or less complex.
Hope it helps.

Categories

Resources