I'm writing something that is effectively "tailing" a file - every now and then the browser polls back and says "I read up to position 23123, give me anything more". I've written this function which is supposed to give me anything new:
async function readNew( filename, mark )
{
var fileDescriptor = await fs.open( filename, "r" );
var buffer = new Buffer( 32768 );
var result = await fs.read( fileDescriptor, buffer, 0, buffer.length, mark );
await fs.close( fileDescriptor );
return result;
}
and it works perfectly if there is something new. If there's not, for some reason it goes back to the beginning and reads the entire file! So for instance, if mark is 16427 - and the file is of size 16427 - I get 16427 bytes read - and what I want is 0.
Any ideas how I can fix this reliably, and with some level of performance?
Most of the time, there won't be any changes - so I want that case to be as fast as possible.
found a really horrible answer - which is to go back, read two bytes - and if I only get one - decide that I'm at the end - but it seems there must be a better way?
async function readPart( filename, mark )
{
var fileDescriptor = await fs.open( filename, "r" );
var buffer = new Buffer( 32768 );
var result = await fs.read( fileDescriptor, buffer, 0, 2, Math.max( 0, mark-1));
if (result[0] == 1)
return [0, new Buffer(0)];
result = await fs.read( fileDescriptor, buffer, 0, buffer.length-1, mark );
await fs.close( fileDescriptor );
return result;
}
I can't reproduce this behavior. From what I've tested, your function seems to work fine. Please correct me if I misunderstand but after running this test:
readNew(path.join(__dirname, 'test.txt'), 0).then((result) => {
console.log('Read: ' + result.bytesRead + ' bytes');
readNew(path.join(__dirname, 'test.txt'), result.bytesRead).then((result) => {
console.log('Read: ' + result.bytesRead + ' bytes');
});
});
The file test.txt in this case contains: A few bytes!
And this is my output:
Read: 13 bytes
Read: 0 bytes
Perhaps something goes wrong elsewhere in your code, such as supplying a position that is out of range.
EDIT:
Here's the full test using the mz library. It gives the same output as the other test using fs-extra.
const fs = require('mz/fs');
const path = require('path');
async function readNew(filename, mark) {
var fileDescriptor = await fs.open(filename, "r");
var buffer = new Buffer(32768);
var result = await fs.read(fileDescriptor, buffer, 0, buffer.length, mark);
await fs.close(fileDescriptor);
return result;
}
readNew(path.join(__dirname, 'test.txt'), 0).then(([bytesRead, buffer]) => {
console.log('Read: ' + bytesRead + ' bytes');
readNew(path.join(__dirname, 'test.txt'), bytesRead).then(([bytesRead, buffer]) => {
console.log('Read: ' + bytesRead + ' bytes');
});
});
Related
okay, so this is my requirement: The application must read each line from the text file. A class can be used so that after reading a certain number of lines an object is created (in this way the application is more clearly structured). After reading a data set that corresponds to a student's data, it will add this data set to a string (separate so that it is presented in consecutive rows).
So i have these information of 2 students which are one under the other like in the picture below but without the name address etc.(it doesn't show quite right in here).
Ebonie Rangel
7175 Yukon Street
(507) 833-3567
Geography
Keenan Ellwood
2 Elm Lane
(894) 831-6482
History
which are in that file. and after reading every line, I am supposed to add Name in front of the first line, Address in front of the second.. phone and Course and so on.
The result should look like this:
This is what i have for now (I have to use Fetch to get to the file, async and await. or with Promise)
let button = document.getElementById("text-button");
let textArea = document.getElementById("text-area");
button.addEventListener("click", function () {
getData();
});
//cod fetch
async function getData() {
try {
let response = await fetch('fileName.txt');
if (response.status !== 200) {
throw new Error("Error while reading file");
}
let text = await response.text();
textArea.innerHtml = text;
} catch (err) {
textArea.innerHTML = 'Problem occurred: ' + err.message;
}
}
please help! I am stuck since forever on this.
Since you're pulling from a .txt file I think its important to understand the line breaks being used in the file. Here's a decent link I found that says all you need at the top of the article: End of Line or Newline Characters
I opened up the .txt file in Notepad++ like the article recommended and saw this:
The [CR][LF] being displayed after each line means that the newline characters used are \r\n.
When you understand that you realize you can use those line breaks to separate your string at each line break.
Here's the MDN for String.split() String.prototype.split()
String.split('\r\n') will return an Array of items, specifically the strings that were between but not including the \r\n characters.
Let's add this to the getData function:
let button = document.getElementById("text-button");
let textArea = document.getElementById("text-area");
button.addEventListener("click", function () {
getData();
});
//cod fetch
async function getData() {
try {
let response = await fetch('fileName.txt');
if (response.status !== 200) {
throw new Error("Error while reading file");
}
let text = await response.text();
//New stuff:
let arrayOfText = text.split('\r\n');
//Now we could add what we want before the text.
//We need to do every 4 lines so lets use this as a chance to learn % better
arrayOfText = arrayOfText.map((textItem, index) => {
let remainder = (index) % 4 //This will return 0, 1, 2, 3
//switch but you could use anything
switch (remainder) {
case 0:
textItem = 'Name: ' + textItem + '\r\n';
break;
case 1:
textItem = 'Address: ' + textItem + '\r\n';
break;
case 2:
textItem = 'Phone: ' + textItem + '\r\n';
break;
case 3:
textItem = 'Course: ' + textItem + '\r\n\r\n'; //two here to separate the groups
break;
//we need a default so lets make it just return textItem if something goes wrong
default:
break;
};
//Our new array has all the info so we can use
//Array.prototype.join('') with an empty string to make it a string.
//We need those old line breaks though so lets put them
//in the switch returns above.
text = arrayOfText.join('');
//End of my changes/////////////
textArea.innerHtml = text;
} catch (err) {
textArea.innerHTML = 'Problem occurred: ' + err.message;
}
}
I hope this works out for you. Its not the most glamorous solution but its a good learning solution because it uses only things you learn early on in your studies.
Let me know if I can clarify anything!
async function getData() {
try {
let response = await fetch('https://v-dresevic.github.io/Advanced-JavaScript-Programming/data/students.txt');
if (response.status !== 200) {
throw new Error("Error while reading file");
}
let text = await response.text();
const lines = text.split('\n');
const CHUNK_SIZE = 4;
textArea.innerHTML = new Array(Math.ceil(lines.length / CHUNK_SIZE))
.fill()
.map(_ => lines.splice(0, CHUNK_SIZE))
.map(chunk => {
const [Name, Address, Phone, Course] = chunk;
return {Name, Address, Phone, Course};
})
.reduce((text, record) => {
text += Object.keys(record).map(key => `${key} ${record[key]}`).join('\n') + '\n';
return text;
}, '');
} catch (err) {
textArea.innerHTML = 'Problem occurred: ' + err.message;
}
}
I have a test folder with files
file
file (1)
file (2)
If the file exists I add a suffix to a new filename, to prevent overwriting the file. For example
if file exists new name should be file (1)
if file (1) exists new name should be file (2)
if file (2) exists new name should be file (3)
and so on.
The following function works fine, except the value is not returned so I can assign it later.
async function dest_exists_new_name(file) {
const access = fs.promises.access
try {
await access(file, fs.F_OK)
// file exists - generate new name
const info = path.parse(file)
const dir = info.dir
let name = info.name
const ext = info.ext
// generate suffix
let suffix = ' (1)'
const suffix_regex = / \([0-9]+\)$/
if (suffix_regex.test(name)) { // if suffix exists -> increment it
const num = name.split(' ').reverse()[0].replace(/[()]/g,'')
const next_num = parseInt(num) + 1
suffix = ' (' + next_num + ')'
name = name.replace(suffix_regex, '') // remove old suffix
}
// generate suffix end
const new_name = path.join(dir, name + suffix + ext)
// recurse until dest not exists
await dest_exists_new_name(new_name)
} catch {
// file not exist - return its name
// console.log works OK
console.log('new name ->', file)
// return doesn't work - returns undefined if the previous name exists, but works ok if the name doesn't exists
return file
}
}
await dest_exists_new_name('/path/file')
new name -> /path/file (3) // console.log - works OK
undefined // returns undefined, if file previously exists
The question is how can I correctly return the new file name value?
If there are any culprits in such a solution like
accidental file rewriting
infinite recursion
other issues
I will be grateful for the hints on how to improve the function.
Your function will return file, but being an async function, you need to await its return and you cannot do so outside of an async scope. Thus, if you just console.log() its "istantaneous" value, it will indeed return a pending promise, as the return value has not been resolved yet. You may retrieve the correct return value by including your function in an async scope, like this:
let a = async () => {
console.log(await dest_exists_new_name('/path/file'))
}
a();
This will output:
new name -> /path/file
/path/file //here's your file
Now, by adding return await dest_exists_new_name(new_name) you should be able to achive what you want and both console.log() and return the new, non-existent, file name. Here's a complete, reproducible example:
const fs = require('fs');
const path = require('path');
async function dest_exists_new_name(file) {
const access = fs.promises.access
try {
await access(file, fs.F_OK)
const info = path.parse(file)
const dir = info.dir
let name = info.name
const ext = info.ext
let suffix = ' (1)'
const suffix_regex = / \([0-9]+\)$/
if (suffix_regex.test(name)) {
const num = name.split(' ').reverse()[0].replace(/[()]/g, '')
const next_num = parseInt(num) + 1
suffix = ' (' + next_num + ')'
name = name.replace(suffix_regex, '')
}
const new_name = path.join(dir, name + suffix + ext)
return await dest_exists_new_name(new_name)
} catch {
console.log('new name ->', file)
return file
}
}
//Here, make sure that the path to "file" is correct
let a = async() => console.log(await dest_exists_new_name(path.join(__dirname, './file')));
a();
Output, having up to file (2) in the same folder:
new name -> /path/to/file (3)
/path/to/file (3)
Check you try catch and how you are receiving your variable.
async function dest_exists_new_name(file) {
try {
const res = await dest_exists_new_name(file1); // recursion result
} catch (err) {
return Promise.resolve("file not found");
}
}
// usage
let res = await dest_exists_new_name(fileArg);
First of all, you should use await, since it's async function:
// recurse until dest not exists
await dest_exists_new_name(new_name)
About recursion - IMHO, it's always better to use cycle (if it doesn't make code too complicated).
Mixing async-await & promises is not very good. Ideally you should use one style.
I prefer to use destructuring, lambda functions, and other modern features.
So, my variant for async-await, without recursion:
const fs = require('fs');
const path = require('path');
// better to create outside of func, since it doesn't depend on context
const suffix_regex = / \([0-9]+\)$/
const defaultSuffix = ' (1)'
const access = async (...params) => new Promise((resolve) => fs.access(...params, (err) => (err) ? resolve(false) : resolve(true)))
const generate_new_suffix = ({ dir, name, ext }) => {
if (suffix_regex.test(name)) { // if suffix exists -> increment it
const num = name.split(' ').reverse()[0].replace(/[()]/g, '')
const suffix = `(${+num + 1})`;
const cleanName = name.replace(suffix_regex, '') // remove old suffix
return path.join(dir, cleanName + suffix + ext)
}
return path.join(dir, name + defaultSuffix + ext)
}
const dest_exists_new_name = async (file) => {
let newNameGenerated = false
let newFileName = file
while (await access(newFileName, fs.F_OK)) {
console.log(newFileName);
const info = path.parse(newFileName)
newFileName = generate_new_suffix(info)
};
console.log('new name ->', newFileName)
return newFileName
}
(async () => {
console.log(await dest_exists_new_name(path.join(__dirname, 'file')))
})();
I am trying to write multiple csv files from a set of data that I have loaded using the sheet js library. My first attempt was like:
for (let i = 0; i < dataSetDivided.length; i++) {
let exportSet = dataSetDivided[i]
console.log(exportSet)
let ws = XLSX.utils.json_to_sheet(exportSet, {header: finalHeaders})
let wb = XLSX.utils.book_new()
XLSX.utils.book_append_sheet(wb, ws, "SheetJS")
let todayDate = this.returnFormattedDate()
let originalFileName = this.state.fileName
let exportFileName = 'import_' + originalFileName + '_' + todayDate + '(part_' + (i + 1) + ').csv'
XLSX.writeFile(wb, exportFileName)
}
With this code only some files are written I guess because the for loop doesn't wait for the file to be written before continuing.
So I am trying to write each file within a promise like below:
Promise.all(
dataSetDivided.map((exportSet, i) => {
return new Promise((resolve, reject) => {
console.log(exportSet)
let ws = XLSX.utils.json_to_sheet(exportSet, {header: finalHeaders})
let wb = XLSX.utils.book_new()
XLSX.utils.book_append_sheet(wb, ws, "SheetJS")
let todayDate = this.returnFormattedDate()
let originalFileName = this.state.fileName
let exportFileName = 'import_' + originalFileName + '_' + todayDate + '(part_' + (i + 1) + ').csv'
XLSX.writeFile(wb, exportFileName, (err) => {
if (err) {
console.log(err)
reject(err)
} else {
console.log('Created ' + exportFileName)
resolve()
}
})
})
})
)
.then(() => {
console.log('Created multiple files successfully')
})
.catch((err) => {
console.log('ERROR: ' + err)
})
But... this isn't working, again only some files are written and nothing is logged to the console. Can anyone give me any ideas how to make this work or a better way to achieve the goal of writing multiple files like this? There is a XLSX.writeFileAsync method but I can't find any examples of how it works and I'm not sure if that is what I need.
With thanks,
James
UPDATE:
I am now using setTimeout to delay the next writeFile call... this is working for my test cases but I am aware it isn't a good solution, would be much better to have a callback when the file is successfully written:
writeFileToDisk(dataSetDivided, i) {
if (dataSetDivided.length > 0) {
let exportSet = dataSetDivided[0]
let ws = XLSX.utils.json_to_sheet(exportSet, {header: finalHeaders})
let wb = XLSX.utils.book_new()
XLSX.utils.book_append_sheet(wb, ws, "SheetJS")
let todayDate = this.returnFormattedDate()
let originalFileName = this.state.fileName
let exportFileName = 'import_' + originalFileName + '_' + todayDate + '(part_' + i + ').csv'
XLSX.writeFile(wb, exportFileName)
dataSetDivided.splice(0, 1)
i += 1
setTimeout(() => {this.writeFileToDisk(dataSetDivided, i)}, 2500)
}
}
this.writeFileToDisk(dataSetDivided, 1)
Any suggestions how to get this working without simulating the file write time would be much appreciated.
I just tried this (first time) XLSX code and can confirm that it writes the expected workbooks and runs synchronously...
'use strict'
const XLSX = require('xlsx');
let finalHeaders = ['colA', 'colB', 'colC'];
let data = [
[ { colA: 1, colB: 2, colC: 3 }, { colA: 4, colB: 5, colC: 6 }, { colA: 7, colB: 8, colC: 9 } ],
[ { colA:11, colB:12, colC:13 }, { colA:14, colB:15, colC:16 }, { colA:17, colB:18, colC:19 } ],
[ { colA:21, colB:22, colC:23 }, { colA:24, colB:25, colC:26 }, { colA:27, colB:28, colC:29 } ]
];
data.forEach((array, i) => {
let ws = XLSX.utils.json_to_sheet(array, {header: finalHeaders});
let wb = XLSX.utils.book_new()
XLSX.utils.book_append_sheet(wb, ws, "SheetJS")
let exportFileName = `workbook_${i}.xls`;
XLSX.writeFile(wb, exportFileName)
});
Running this yields workbook_0.xls, workbook_1.xls, and workbook_2.xls, each with a single sheet entitled "SheetJS". They all look good in excel, for example, workbook_0 has...
I think you should do the writing asynchronously, and would suggest the following adaptation of the above ...
function writeFileQ(workbook, filename) {
return new Promise((resolve, reject) => {
// the interface wasn't clearly documented, but this reasonable guess worked...
XLSX.writeFileAsync(filename, workbook, (error, result) => {
(error)? reject(error) : resolve(result);
})
})
}
let promises = data.map((array, i) => {
let ws = XLSX.utils.json_to_sheet(array, {header: finalHeaders});
let wb = XLSX.utils.book_new()
XLSX.utils.book_append_sheet(wb, ws, "SheetJS")
let exportFileName = `workbook_${i}.xls`;
return writeFileQ(wb, exportFileName)
});
Promise.all(promises).then(result => console.log(result)).catch(error => console.log(error));
Running this async code, I found that it produced the same expected results and did so asynchronously.
So your original loop looks right, and should work synchronously. The fact that you aren't getting expected results must be caused by something apart from timing (or maybe some timing issue induced by react?).
In any event, if you do want to use the async approach, which I highly recommend, I've shown how to do that (but I worry that might not fully solve the problem unless you sort out what's happening with your first attempt).
XLSX.writeFileAsync does have a callback with the following syntax.
xlsx.writeFileAsync(workbookName, workbook, (err) => {
// It's a callback
});
But this will handle only writing one file asynchronously.
Your case is typical, if you want to do a series things in which each item is asynchronous, then you should not just use iterative methods like loops/map/forEach.
One best library I would suggest for this is 'async'.
'async.parallel' function which takes a array of functions which execute asynchronously and calls callback after all of them finished.
https://caolan.github.io/async/docs.html#parallel
If the concern is to use the library asynchronously for not blocking the server, you should know that this library implementation seems to be synchronous and you should check out the library's server demos README since it has several proposals to workaround this problem: https://github.com/SheetJS/sheetjs/tree/master/demos/server
I'm trying to upload a file to S3 after creation however the file doesn't seem to be closing with the fs.createWriteStream. Hence I keep getting a 0 byte file being uploaded.
function createManifest(manfile) {
console.log(['a'].toString(), ['a', 'b'].toString());
var arrayLength = files.length;
var lastItem = arrayLength - 1;
console.log( chalk.blue("The last item value is:",lastItem))
console.log( chalk.yellow("The arrayLength value is:",arrayLength))
var logStream = fs.createWriteStream("manny_temp.json", {'flags': 'a'});
// use {'flags': 'a'} to append and {'flags': 'w'} to erase and write a new file
// logStream.on('open', function(fd) {
logStream.write('{'+ "\r\n");
logStream.write("\"entries\": [" + "\r\n");
for (var i = 0; i < arrayLength; i++) {
console.log( chalk.inverse(files[i]))
console.log( chalk.blue("The i value is:",i))
if ( i == lastItem) {
logStream.write("{\"url\":\"s3://mybucket/" +files[i] + "\",\"key\":true}" + "\r\n");
} else {
logStream.write("{\"url\":\"s3://mybucket/" +files[i] + "\",\"key\":true}," + "\r\n");
}
}
logStream.write(' ]' + "\r\n");
logStream.write('}');
// }).on('end', function() {
logStream.end();
//fs.renameSync(logStream.path, manfile.toString());
return callback(filepath);
// logStream.close();
//fs.renameSync(logStream.path, "manny.json");
// });
}
I've tried a multitude of ways to get the file to close so that the next function can upload the file upon creation, and even added a sleep, but it always seems to leave a hanging inode.
Using the fs.write seems to write only one line vs writing all the lines in the array / streaming data.
Does anyone have a suggestion?
You need to listen for the close event on the logStream before calling the callback. That is what signals that the backing file descriptor has been closed (no more writing will take place). I should note, this is different than listening for the finish event since finish merely indicates the stream is closed, but not necessarily that the file descriptor is closed, so it is technically less safe to rely on if you're going to do something with the file soon after).
Replace this:
return callback(filepath);
with this:
logStream.on('close', function() {
callback(filepath);
});
It is possible to improve on this further, by instead using the (err, result)-style callbacks common in the node ecosystem:
logStream.on('close', function() {
callback(null, filepath);
});
logStream.on('error', function(err) {
callback(err);
});
That way you can capture any errors that may result from opening or writing to the file.
I'm also assuming you do have callback defined in a parent scope somewhere.
Your question is asked without the basic research though, Here is an attempt to make it more readable code and executable at glance. Hope you will find an answer with below code.
If your question is still the same, re-write your question in a better way.
var chalk = require('chalk');
var fs = require('fs');
function createManifest(manfile) {
var files = ['/test.json'];
console.log(['a'].toString(), ['a', 'b'].toString());
var arrayLength = files.length;
var lastItem = arrayLength - 1;
console.log( chalk.blue("The last item value is:",lastItem))
console.log( chalk.yellow("The arrayLength value is:",arrayLength))
var logStream = fs.createWriteStream("manny_temp.json", {'flags': 'a'});
logStream.write('{'+ "\r\n");
logStream.write("\"entries\": [" + "\r\n");
for (var i = 0; i < arrayLength; i++) {
console.log( chalk.inverse(files[i]))
console.log( chalk.blue("The i value is:",i))
if ( i == lastItem) {
logStream.write("{\"url\":\"s3://mybucket/" +files[i] + "\",\"key\":true}" + "\r\n");
} else {
logStream.write("{\"url\":\"s3://mybucket/" +files[i] + "\",\"key\":true}," + "\r\n");
}
}
logStream.write(' ]' + "\r\n");
logStream.write('}');
logStream.end();
logStream.on('finish', function(){
fs.renameSync("manny_temp.json", "manny.json");
});
};
createManifest();
mscdex is nearly right. You need to listen for the 'finish' event, not the 'close' event because that is emitted by readable streams but this is a writable stream.
And you can pass your handler to 'end' instead. See https://nodejs.org/dist/latest-v4.x/docs/api/stream.html#stream_event_finish
logStream.end(function() {
fs.renameSync("manny_temp.json", "manny.json");
callback(filepath);
});
I tried to edit mscdex's answer to this effect but this was rejected for some reason.
Edit: You'd need to define or pass in 'callback', as well as 'filepath'.
What's the best way to overwrite a line in a large (2MB+) text file using node.js?
My current method involves
copying the entire file into a buffer.
Spliting the buffer into an array by the new line character (\n).
Overwriting the line by using the buffer index.
Then overwriting the file with the buffer after join with \n.
First, you need to search where the line starts and where it ends. Next you need to use a function for replacing the line. I have the solution for the first part using one of my libraries: Node-BufferedReader.
var lineToReplace = "your_line_to_replace";
var startLineOffset = 0;
var endLineOffset = 0;
new BufferedReader ("your_file", { encoding: "utf8" })
.on ("error", function (error){
console.log (error);
})
.on ("line", function (line, byteOffset){
startLineOffset = endLineOffset;
endLineOffset = byteOffset - 1; //byteOffset is the offset of the NEXT byte. -1 if it's the end of the file, if that's the case, endLineOffset = <the file size>
if (line === lineToReplace ){
console.log ("start: " + startLineOffset + ", end: " + endLineOffset +
", length: " + (endLineOffset - startLineOffset));
this.interrupt (); //interrupts the reading and finishes
}
})
.read ();
Maybe you can try the package replace-in-file
suppose we have a txt file as below
// file.txt
"line1"
"line2"
"line5"
"line6"
"line1"
"line2"
"line5"
"line6"
and we want to replace:
line1 -> line3
line2 -> line4
Then, we can do it like this:
const replace = require('replace-in-file');
const options = {
files: "./file.txt",
from: [/line1/g, /line2/g],
to: ["line3", "line4"]
};
replace(options)
.then(result => {
console.log("Replacement results: ",result);
})
.catch(error => {
console.log(error);
});
the result as below:
// file.txt
"line3"
"line4"
"line5"
"line6"
"line3"
"line4"
"line5"
"line6"
More details please refer to its docs: https://www.npmjs.com/package/replace-in-file
This isn't file size focoused solution, but Overwrites a line in a file using node.js. It may help other people that search engines redirect to this post, like me.
import * as fs from 'fs'
const filename = process.argv[2]
const lineIndexToUpdate = parseInt(process.argv[3]) - 1
const textUpdate = process.argv[4]
function filterLine(indexToUpdate, dataString) {
return dataString
.split('\n')
.map((val, index) => {
if (index === indexToUpdate)
return textUpdate
else
return val
})
.join('\n')
}
fs.readFile(filename, 'utf8', (err, data) => {
if (err) throw err
fs.writeFile(filename, filterLine(lineIndexToUpdate, data), (err, data) => {
if (err) throw err
console.log("Line removed")
})
})
Script use exemple:
node update_line.js file 10 "te voglio benne"