The obvious solution is to fs.writeFile, I assume.
But the answer of this question suggests that I should be using a Stream technique.
I'm currently trying this code in order to remove a line from the text file by converting it to an array:
var index = randomIntFromInterval(1, unfinished_searches.length-1); // remove index
unfinished_searches.splice(index, 1);
fs.truncate('unfinished_searches.txt', 0, function()
{
var file = fs.createWriteStream('unfinished_searches.txt');
file.on('error', function(err) { /* error handling */ });
unfinished_searches.forEach(function(v) { file.write(v.join(', ') + '\n'); });
file.end();
})
which returns the following error:
TypeError: undefined is not a function
at join in this line:
unfinished_searches.forEach(function(v) { file.write(v.join(', ') + '\n'); });
If you want to remove line from a big text file, consider to use pipes. Just create read stream, pipe it to your "checking function", and then pipe it to write stream.
This way you don't have to load whole file into memory. In memory will be only small amount of data at once. Of course you can't write it to the same file. But after complete operation, you can rename this file and delete old one.
EDIT:
This post can be usefull for You: Parsing huge logfiles in Node.js - read in line-by-line
Related
I'm trying to write in a text file, but not at the end like appendFile() do or by replacing the entiere content...
I saw it was possible to chose where you want to start with start parameter of fs.createwritestream() -> https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options
But there is no parameter to say where to stop writting, right ? So it remove all the end of my file after I wrote with this function.
const fs = require('fs');
var logger = fs.createWriteStream('result.csv', {
flags: 'r+',
start: 20 //start to write at the 20th caracter
})
logger.write('5258,525,98951,0,1\n') //example a new line to write
Is there a way to specify where to stop writting in the file to have something like:
....
data from begining
....
5258,525,98951,0,1
...
data till the end
...
I suspect you mean, "Is it possible to insert in the middle of the file." The answer to that is: No, it isn't.
Instead, to insert, you have to:
Determine how big what you're inserting is
Copy the data at your insertion point to that many bytes later in the file
Write your data
Obviously when doing #2 you need to be sure that you're not overwriting data you haven't copied yet (either by reading it all into memory first or by working in blocks, from the end of the file toward the insertion point).
(I've never looked for one, but there may be an npm module out there that does this for you...)
You could read/parse your file at first. Then apply the modifications and save the new file.
Something like:
const fs = require("fs");
const fileData = fs.readFileSync("result.csv", { encoding: "utf8" });
const fileDataArray = fileData.split("\n");
const newData = "5258,525,98951,0,1";
const index = 2; // after each row to insert your data
fileDataArray.splice(index, 0, newData); // insert data into the array
const newFileData = fileDataArray.join("\n"); // create the new file
fs.writeFileSync("result.csv", newFileData, { encoding: "utf8" }); // save it
I'm building a script that reads log files, handles what needs to be handled then writes them to a database
Some caveats :
Some log files have a lot of input, multiple times a second
Some log files have few to no input at all
What I try in simple words:
Reading the first line of a file, then deleting this line to go to the next one, while I handle the first line, other lines could be added..
Issues I'm facing
When I try reading a file then processing it, then deleting the
files, some lines have been added
When the app crashes while
handling multiple lines at once for any reason, I can't know what
lines have been processed.
Tried so far
fs.readdir('logs/', (err, filenames) => {
filenames.forEach((filename) => {
fs.readFile('logs/'+filename, 'utf-8', (err, content) => {
//processing all new lines (can take multiple ms)
//deleting file
fs.unlink('logs/'+filename)
});
});
});
Is there not a (native or not) method to 'take' first line(s), or take all lines, from a file at once?
Something similar to what the Array.shift() method does to arrays..
Why you are reading the file at once. Instead you can use the node.js streams.
https://nodejs.org/api/fs.html#fs_class_fs_readstream
This will read the files and output to console
var fs = require('fs');
var readStream = fs.createReadStream('myfile.txt');
readStream.pipe(process.stdout);
You can also go for the npm package node-tail to read the content of a files while new content written to it.
https://github.com/lucagrulla/node-tail
If your log files has been writen as rotate logs. Example: Each hours has each log file, 9AM.log, 10AM.log....When you process the log files, you can skip current file and process another files. ex: now is 10:30 AM o’clock, skip file 10AM.log, solve another files.
I'm trying to write a function that reads a file and when it finds a specific line it deletes everything below it and then it appends another set of lines. I have managed to read the file and find the string I need:
function read() {
lineReader.on('line', function (line) {
console.log('Line from file: ' + line)
if(line.trim() === 'Examples:'){
console.log('Found what I needed, delete everything below this line');
}
});
}
What I'm unable to see is how to delete everything below this line and then append the text I need. I am new to JS and Node.js.
You could do this by opening a file writing stream at the same time.
In the lineReader event handler, put all the lines before "Example" into a separate file stream. When Example comes along, just append your desired set of lines into the second file stream and close the lineReader.
So add something like this:
// before the code
var writer = fs.createWriteStream('new.txt');
// load custom lines from customLines.txt
var customLines = fs.readFileSync('customLines.txt');
// in the on('line') callback:
writer.write(line + "\n");
// if 'Examples:' was found:
writer.write(customLines);
writer.end();
lineReader.close();
I am trying to read the metadata tags from an mp4 file into my javascript using the node module mp4js. I followed the model provided on that page but with some issue.
I tried the following:
var mp4 = require('mp4js');
mp4({ file: 'small.mp4', type: 'local' }, function (err, tags) {
// tags now contains your MP4 tags
console.log('Tags: ' + tags)
$scope.mp4Tags = tags;
});
and I was met with Uncaught TypeError: Cannot read property 'join' of undefined
Does anyone know what I'm missing here? Could it be that the mp4 doesn't have any tags? I just googled "sample mp4" and downloaded something. It's my first time trying this module and I am pretty new to node.
I'm also open to other suggestions for reading mp4 metadata.
Thank you very much for your time and let me know if you need any additional from me.
EDIT 8.26.2015
Following Brad's suggestion, I tried the following code to read metadata from an mp4:
var child_process = require('child_process');
ls = child_process.spawn('ffmpeg/bin/ffprobe.exe', ['-print_format', 'json', 'out.mp4']);
ls.stdout.on('data', function (data) {
console.log('stdout: ' + data);
$scope.mp4data = data;
});
I created mp4.out by following this example for adding a title meta tag using ffmpeg from the command line. Here is a screenshot of my output. Here is the command I entered:
ffmpeg -i in.avi -metadata title="my title" out.flv
After running my app with the code above, the value stored in $scope.mp4data is a Native Buffer that looks like this. All I see are some numbers, but no metadata tags. How can I view the metadata such as the title that I set using the command above? Am I missing something in the arguments? Or maybe I should be looking at something other than data?
Thanks for the help. I may just post this as a new question since the focus has shifted.
EDIT: Solution
Here's where I landed thanks to Brad's help:
$scope.mp4data = "";
ls.stdout.on('data', function (data) {
$scope.mp4data = $scope.mp4data + data.toString();
});
ls.stdout.on('end', function (data) {
$scope.mp4data = JSON.parse($scope.mp4data);
});
works great for reading metadata from my mp4! Thanks Brad!
I recommend using FFprobe (from the FFmpeg project) to do this instead. It supports many containers, including MP4.
You can use ffprobe -print_format json to have it output JSON data that you can parse inside your Node.js app.
Some examples here: https://stackoverflow.com/a/8191228/362536
I'm writing a GTK javascript program that downloads a file and writes it to disk. Here's what my code looks like:
const Gio = imports.gi.Gio;
const Soup = imports.gi.Soup;
// start an http session to make http requests
let _httpSession = new Soup.SessionAsync();
Soup.Session.prototype.add_feature.call(_httpSession, new Soup.ProxyResolverDefault());
// open the file
let file = Gio.file_new_for_path(path);
let fstream = file.replace(null, false, Gio.FileCreateFlags.NONE, null);
// start the download
let request = Soup.Message.new('GET', url);
request.connect('got_chunk', Lang.bind(this, function(message, chunk){
// write each chunk to file
fstream.write(chunk, chunk.length, null);
}));
this._httpSession.queue_message(request, function(_httpSession, message) {
// close the file
fstream.close(null);
});
I get an error on the fstream.write() line:
JS ERROR: !!! Exception was: Error: Unhandled GType GCancellable unpacking GArgument from Number
JS ERROR: !!! message = '"Unhandled GType GCancellable unpacking GArgument from Number"'
JS ERROR: !!! fileName = '"./torbrowser-launcher"'
JS ERROR: !!! lineNumber = '402'
JS ERROR: !!! stack = '"([object _private_Soup_Message],[object _private_Soup_Buffer])#./torbrowser-launcher:402
("2.3.25-2")#./torbrowser-launcher:122
wrapper("2.3.25-2")#/usr/share/gjs-1.0/lang.js:204
("2.3.25-2")#/usr/share/gjs-1.0/lang.js:145
("2.3.25-2")#/usr/share/gjs-1.0/lang.js:239
#./torbrowser-launcher:489
"'
The only reference to this error that I can find is in this thread: https://mail.gnome.org/archives/gnome-shell-list/2012-July/msg00126.html
That person ended up giving up and porting his code to python.
I'm also confused by what the 'got_chunk' callback passes. The chunk field is a Soup.Buffer (http://www.roojs.com/seed/gir-1.2-gtk-3.0/gjs/Soup.Buffer.html). I can get its length with chunk.length, but when I try printing chunk.data it's undefined. When I just print chunk it prints: [object _private_Soup_Buffer].
fstream is a Gio.FileOutputStream (http://www.roojs.com/seed/gir-1.2-gtk-3.0/gjs/Gio.FileOutputStream.html). The write method is: write(String buffer, guint32 count, Cancellable cancellable), and cancellable is optional. Weirdly enough, if I replace the write line with this I still get the exact same error:
fstream.write('test ', 5, null);
I was hitting exactly the same problem. After a lot of trial and error, it boiled down to two issues with the write() call:
It seems that the documentation of the write function you are using (http://www.roojs.com/seed/gir-1.2-gtk-3.0/gjs/Gio.FileOutputStream.html) is wrong; the write method signature is (as far as I can tell):
write(String buffer, Cancellable cancellable, guint32 count)
Yet if you just use fstream.write(chunk, null, chunk.length); you will write a file full of zeros. I don't know why (something to do with the way GJS binds to the underlying C library) but you should use chunk.get_data() instead of just chunk. I.e. replace the write call in your code with:
fstream.write(chunk.get_data(), null, chunk.length);