Develop Reference

Read N last lines of a large shared file with NodeJs

I have two codes one for reading shared file using smb2:
// load the library
var SMB2 = require('smb2');
// create an SMB2 instance
var smb2Client = new SMB2({
share:'\\\\192.168.0.111\\folder'
, domain:'WORKGROUP'
, username:'username'
, password:'password'
});
// read the file
smb2Client.readFile('path\\to\\the\\file.txt', "utf-8", function(error, data){
if(error) throw error;
console.log(data);
});
And the other one for reading the last 20 lines of a local file using read-last-line:
// load the library
const readLastLine = require('read-last-line');
// read the file
readLastLine.read('path\\to\\the\\file.txt', 20).then(function (lines) {
console.log(lines)
}).catch(function (err) {
console.log(err.message);
});
I didn't know how to combine the two of them.
Do you have any suggestions.
Thanks.

If smb2 and read-last-line both supported streams, it would be easy. According to their documentation at least, neither does, but pv-node-smb2 has a method createReadStream.
Here is a stream transformer that processes its input line by line, but keeps only the most recent n lines and outputs them at the end. This avoids keeping huge files entirely in memory:
class Tail extends stream.Transform {
constructor(n) {
super();
this.input = new stream.PassThrough();
this.tail = [];
readline.createInterface({input: this.input, crlfDelay: Infinity})
.on("line", function(line) {
if (this.tail.length === n) this.tail.shift();
this.tail.push(line);
}.bind(this));
}
_transform(chunk, encoding, callback) {
this.input.write(chunk);
callback();
}
_flush(callback) {
callback(null, this.tail.join("\n"));
}
}
(Likely, there is already an NPM package for that.)
Now you can pipe the SMB2 read stream through that transformer:
new require("pv-node-smb2").PvNodeSmb2(...).createReadStream(...)
.pipe(new Tail(20))
.pipe(process.stdout);

I couldn't install pv-node-smb2 package, it gave me too much errors so I tried a Powershell command that works like the tail command in Linux
Get-Content \\192.168.0.111\path\to\the\file.txt -Tail 20
Which returns the last 20 lines of the file and then run that script in NodeJs
let last20lines = "";
let spawn = require("child_process").spawn,child;
child = spawn("powershell.exe",["c:\\wamp64\\www\\powershell_script.ps1"]);
child.stdout.on("data",function(data){
last20lines += data
});
child.stdin.end();

Nodejs - removing substring from a huge file

I need to remove a substring (that appears only in specific known lines of the file) from a file.
there are simple solutions of reading all file data to a string, removing the substring, and then write the fixed data to the file.
here is a code I found in here:
Node js - Remove string from text file
var data = fs.readFileSync('banlist.txt', 'utf-8');
var newValue = data.replace(new RegEx("STRING_TO_REMOVE"), '');
fs.writeFileSync('banlist.txt', newValue, 'utf-8');
My problem is, that the file is huge - up to billion lines of logs, so I can't read all content to the memory.

Why not a simple transform stream and replace()? replace can take a callback as second parameter i.e. .replace(/bad1|bad2|bad3/g, filterWords) in case you need to replace words rather than remove them completely.
const fs = require("fs")
const { pipeline, Transform } = require("stream")
const { join } = require("path")
const readFile = fs.createReadStream("./words.txt")
const writeFile = fs.createWriteStream(
join(__dirname, "words-filtered.txt"),
"utf8"
)
const transformFile = new Transform({
transform(chunk, enc, next) {
let c = chunk.toString().replace(/bad/g, "replaced")
this.push(c)
next()
},
})
pipeline(readFile, transformFile, writeFile, (err) => {
if (err) {
console.log(err.message)
}
})

https://nodejs.org/api/fs.html#fs_fs_read_fd_buffer_offset_length_position_callback
Dont read the whole file at once... read a small buffered piece of it.. and look for your input with that buffered piece.... then increment your buffer starting position and do it again.... would recommend having each buffer start not at the end of the previous buffer... but overlap by at least the expected size of the data being sought so that you dont run into half of your data being at end of one buffer and other half at beginning of the other

You could use a file read stream. However, you would have to find a way to detect if the read data only contains part of the result.

What you probably want to do is use streams so that you are writing after partial reads. this example could probably work for you. you need to copy over the output text file ".tmp" over the original to get the same behavior in your question. It works by reading a chunk and then looking to see if you've come across a new line. then it processes that line, writes it, then removes it from the buffer. This should help with your memory problem.
var fs = require("fs");
var readStream = fs.createReadStream("./BFFile.txt", { encoding: "utf-8" });
var writeStream = fs.createWriteStream("./BFFile.txt.tmp");
const STRING_TO_REMOVE = "badword";
var buffer = ""
readStream.on("data", (chunk) => {
buffer += chunk;
var indexOfNewLine = buffer.search("\n");
while (indexOfNewLine !== -1) {
var line = buffer.substring(0, indexOfNewLine + 1);
buffer = buffer.substring(indexOfNewLine + 1, buffer.length);
line = line.replace(new RegExp(STRING_TO_REMOVE), "");
writeStream.write(line);
indexOfNewLine = buffer.search("\n");
}
})
readStream.on("end", () => {
buffer = buffer.replace(new RegExp(STRING_TO_REMOVE), "");
writeStream.write(buffer);
writeStream.close();
})
There are a few assumptions with this solution such as the data being UTF-8, there only being 1 bad word potentially per line, every line having some text (I didn't test for that), and that every line ends with new line and not some other line ending.
Heres the docs for streams in Node
another thought I had was to use pipe and a transform stream but that seems like over kill.

You can use this code to do it. I'm using fs stream. it's created for read huge files in small memory by chunks. docs
const fs = require('fs');
const readStream = fs.createReadStream('./XXXXX');
const writeStream = fs.createWriteStream('./XXXXXXX');
readStream.on('data', (chunk) => {
const data = chunk.toString().replace('STRING_TO_REMOVE', 'XXXXXX');
writeStream.write(data);
});
readStream.on('end', () => {
writeStream.close();
});

fs.createWriteStream doesn't use back-pressure when writing data to a file, causing high memory usage

Problem
I'm trying to scan a drive directory (recursively walk all the paths) and write all the paths to a file (as it's finding them) using fs.createWriteStream in order to keep the memory usage low, but it doesn't work, the memory usage reaches 2GB during the scan.
Expected
I was expecting fs.createWriteStream to automatically handle memory/disk usage at all times, keeping memory usage at a minimum with back-pressure.
Code
const fs = require('fs')
const walkdir = require('walkdir')
let dir = 'C:/'
let options = {
"max_depth": 0,
"track_inodes": true,
"return_object": false,
"no_return": true,
}
const wstream = fs.createWriteStream("C:/Users/USERNAME/Desktop/paths.txt")
let walker = walkdir(dir, options)
walker.on('path', (path) => {
wstream.write(path + '\n')
})
walker.on('end', (path) => {
wstream.end()
})
Is it because I'm not using .pipe()? I tried creating a new Stream.Readable({read{}}) and then inside the .on('path' emitter pushing paths into it with readable.push(path) but that didn't really work.
UPDATE:
Method 2:
I tried the proposed in the answers drain method but it doesn't help much, it does reduce memory usage to 500mb (which is still too much for a stream) but it slows down the code significantly (from seconds to minutes)
Method 3:
I also tried using readdirp, it uses even less memory (~400mb) and is faster but I don't know how to pause it and use the drain method there to reduce the memory usage further:
const readdirp = require('readdirp')
let dir = 'C:/'
const wstream = fs.createWriteStream("C:/Users/USERNAME/Desktop/paths.txt")
readdirp(dir, {alwaysStat: false, type: 'files_directories'})
.on('data', (entry) => {
wstream.write(`${entry.fullPath}\n`)
})
Method 4:
I also tried doing this operation with a custom recursive walker, and even though it uses only 30mb of memory, which is what I wanted, but it is like 10 times slower than the readdirp method and it is synchronous which is undesirable:
const fs = require('fs')
const path = require('path')
let dir = 'C:/'
function customRecursiveWalker(dir) {
fs.readdirSync(dir).forEach(file => {
let fullPath = path.join(dir, file)
// Folders
if (fs.lstatSync(fullPath).isDirectory()) {
fs.appendFileSync("C:/Users/USERNAME/Desktop/paths.txt", `${fullPath}\n`)
customRecursiveWalker(fullPath)
}
// Files
else {
fs.appendFileSync("C:/Users/USERNAME/Desktop/paths.txt", `${fullPath}\n`)
}
})
}
customRecursiveWalker(dir)

Preliminary observation: you've attempted to get the results you want using multiple approaches. One complication when comparing the approaches you used is that they do not all do the same work. If you run tests on file tree that contains only regular files, that tree does not contain mount points, you can probably compare the approaches fairly, but when you start adding mount points, symbolic links, etc, you may get different memory and time statistics merely due to the fact that one approach excludes files that another approach includes.
I've initially attempted a solution using readdirp, but unfortunately, but that library appears buggy to me. Running it on my system here, I got inconsistent results. One run would output 10Mb of data, another run with the same input parameters would output 22Mb, then I'd get another number, etc. I looked at the code and found that it does not respect the return value of push:
_push(entry) {
if (this.readable) {
this.push(entry);
}
}
As per the documentation the push method may return a false value, in which case the Readable stream should stop producing data and wait until _read is called again. readdirp entirely ignores that part of the specification. It is crucial to pay attention to the return value of push to get proper handling of back-pressure. There are also other things that seemed questionable in that code.
So I abandoned that and worked on a proof of concept showing how it could be done. The crucial parts are:
When the push method returns false it is imperative to stop adding data to the stream. Instead, we record where we were, and stop.
We start again only when _read is called.
If you uncomment the console.log statements that print START and STOP. You'll see them printed out in succession on the console. We start, produce data until Node tells us to stop, and then we stop, until Node tells us to start again, and so on.
const stream = require("stream");
const fs = require("fs");
const { readdir, lstat } = fs.promises;
const path = require("path");
class Walk extends stream.Readable {
constructor(root, maxDepth = Infinity) {
super();
this._maxDepth = maxDepth;
// These fields allow us to remember where we were when we have to pause our
// work.
// The path of the directory to process when we resume processing, and the
// depth of this directory.
this._curdir = [root, 1];
// The directories still to process.
this._dirs = [this._curdir];
// The list of files to process when we resume processing.
this._files = [];
// The location in `this._files` were to continue processing when we resume.
this._ix = 0;
// A flag recording whether or not the fetching of files is currently going
// on.
this._started = false;
}
async _fetch() {
// Recall where we were by loading the state in local variables.
let files = this._files;
let dirs = this._dirs;
let [dir, depth] = this._curdir;
let ix = this._ix;
while (true) {
// If we've gone past the end of the files we were processing, then
// just forget about them. This simplifies the code that follows a bit.
if (ix >= files.length) {
ix = 0;
files = [];
}
// Read directories until we have files to process.
while (!files.length) {
// We've read everything, end the stream.
if (dirs.length === 0) {
// This is how the stream API requires us to indicate the stream has
// ended.
this.push(null);
// We're no longer running.
this._started = false;
return;
}
// Here, we get the next directory to process and get the list of
// files in it.
[dir, depth] = dirs.pop();
try {
files = await readdir(dir, { withFileTypes: true });
}
catch (ex) {
// This is a proof-of-concept. In a real application, you should
// determine what exceptions you want to ignore (e.g. EPERM).
}
}
// Process each file.
for (; ix < files.length; ++ix) {
const dirent = files[ix];
// Don't include in the results those files that are not directories,
// files or symbolic links.
if (!(dirent.isFile() || dirent.isDirectory() || dirent.isSymbolicLink())) {
continue;
}
const fullPath = path.join(dir, dirent.name);
if (dirent.isDirectory() & depth < this._maxDepth) {
// Keep track that we need to walk this directory.
dirs.push([fullPath, depth + 1]);
}
// Finally, we can put the data into the stream!
if (!this.push(`${fullPath}\n`)) {
// If the push returned false, we have to stop pushing results to the
// stream until _read is called again, so we have to stop.
// Uncomment this if you want to see when the stream stops.
// console.log("STOP");
// Record where we were in our processing.
this._files = files;
// The element at ix *has* been processed, so ix + 1.
this._ix = ix + 1;
this._curdir = [dir, depth];
// We're stopping, so indicate that!
this._started = false;
return;
}
}
}
}
async _read() {
// Do not start the process that puts data on the stream over and over
// again.
if (this._started) {
return;
}
this._started = true; // Yep, we've started.
// Uncomment this if you want to see when the stream starts.
// console.log("START");
await this._fetch();
}
}
// Change the paths to something that makes sense for you.
stream.pipeline(new Walk("/home/", 5),
fs.createWriteStream("/tmp/paths3.txt"),
(err) => console.log("ended with", err));
When I run the first attempt you made with walkdir here, I get the following statistics:
Elapsed time (wall clock): 59 sec
Maximum resident set size: 2.90 GB
When I use the code I've shown above:
Elapsed time (wall clock): 35 sec
Maximum resident set size: 0.1 GB
The file tree I use for the tests produces a file listing of 792 MB

You could exploit the returned value from WritableStream.write(): it essentially states if you should continue to read or not. a WritableStream has an internal property that stores the threshold after which the buffer should be processed by the OS. The drain event will be emitted when the buffer has been flushed, i.e. you can call safely call WritableStream.write() without risking to excessively fill the buffer (which means the RAM). Luckily for you, walkdir let you control the process: you can emit pause(pause the walk. no more events will be emitted until resume) and resume(resume the walk) event from the walkdir object, pausing and resuming the writing process on you stream accordingly. Try with this:
let is_emitter_paused = false;
wstream.on('drain', (evt) => {
if (is_emitter_paused) {
walkdir.resume();
}
});
walkdir.on('path', function(path, stat) {
is_emitter_paused = !wstream.write(path + '\n');
if (is_emitter_paused) {
walkdir.pause();
}
});

Here's an implementation inspired by #Louis's answer. I think it's a bit easier to follow and in my minimal testing it performs about the same.
const fs = require('fs');
const path = require('path');
const stream = require('stream');
class Walker extends stream.Readable {
constructor(root = process.cwd(), maxDepth = Infinity) {
super();
// Dirs to process
this._dirs = [{ path: root, depth: 0 }];
// Max traversal depth
this._maxDepth = maxDepth;
// Files to flush
this._files = [];
}
_drain() {
while (this._files.length > 0) {
const file = this._files.pop();
if (file.isFile() || file.isDirectory() || file.isSymbolicLink()) {
const filePath = path.join(this._dir.path, file.name);
if (file.isDirectory() && this._maxDepth > this._dir.depth) {
// Add directory to be walked at a later time
this._dirs.push({ path: filePath, depth: this._dir.depth + 1 });
}
if (!this.push(`${filePath}\n`)) {
// Hault walking
return false;
}
}
}
if (this._dirs.length === 0) {
// Walking complete
this.push(null);
return false;
}
// Continue walking
return true;
}
async _step() {
try {
this._dir = this._dirs.pop();
this._files = await fs.promises.readdir(this._dir.path, { withFileTypes: true });
} catch (e) {
this.emit('error', e); // Uh oh...
}
}
async _walk() {
this.walking = true;
while (this._drain()) {
await this._step();
}
this.walking = false;
}
_read() {
if (!this.walking) {
this._walk();
}
}
}
stream.pipeline(new Walker('some/dir/path', 5),
fs.createWriteStream('output.txt'),
(err) => console.log('ended with', err));

Nodejs out of memory on pipe for pdfMake

I have a Nodejs server that is being used to create about 1200 pdf forms that can be downloaded by a client later. They are being created using pdfmake and then output to a server folder. When I execute the code as written at about 350 documents, Nodejs runs out of memory. I know there must be a better way to save, but I cannot seem to figure it out.
The below method is being called by a map of an array of data from a Mongoose query. The relevant code for creating and saving the form is as follows:
const whichForm = certList => {
certList.map(cert => {
if (cert.Cert_Details !== null) {
switch (cert.GWMA) {
case 'OA':
case 'PC':
// Don't provide reports for Feedlots
if (cert.Cert_Details.cert_type !== null) {
if (cert.Cert_Details.cert_type === 'Irrigation') {
createOAReport(cert);
}
}
break;
case 'FA':
// Don't provide reports for Feedlots
if (cert.Cert_Details.cert_type === 'Irrigation') {
createFAReport(cert);
}
break;
}
}
}
}
Different File:
const PdfPrinter = require('pdfmake/src/printer');
const fs = require('fs');
const createOAReport = data => {
console.log('PC or OA Cert ', data.Cert_ID);
// console.log(data);
let all_meters_maint = [];
data.Flowmeters.map(flowmeter => {
// Each Flow meter
// console.log(`Inside Flowmeter ${flowmeter}`);
if (flowmeter.Active === true) {
let fm_maint = [];
fm_maint.push({
text: `Meter Serial Number: ${flowmeter.Meter_Details.Serial_num}`
});
fm_maint.push({
text: `Type of Meter: ${flowmeter.Meter_Details.Manufacturer}`
});
fm_maint.push({ text: `Units: ${flowmeter.Meter_Details.units}`});
fm_maint.push({ text: `Factor: ${flowmeter.Meter_Details.factor}`});
all_meters_maint.push(fm_maint);
}
docDefinition.content.push({
style: 'tableExample',
table: {
widths: [200, 200, '*', '*'],
body: all_meters_maint
},
layout: 'noBorders'
});
const fonts = {
Roboto: {
normal: path.join(__dirname, '../', '/fonts/Roboto-
Regular.ttf'),
bold: path.join(__dirname, '../', '/fonts/Roboto-Medium.ttf'),
italics: path.join(__dirname, '../', '/fonts/Roboto-Italic.ttf'),
bolditalics: path.join(__dirname, '../', '/fonts/Roboto-
MediumItalic.ttf')
}
};
const printer = new PdfPrinter(fonts);
const pdfDoc = printer.createPdfKitDocument(docDefinition);
// Build file path
const fullfilePath = path.join(
__dirname,
'../',
'/public/pdffiles/',
`${data.Cert_ID}.pdf`
);
pdfDoc.pipe(fs.createWriteStream(fullfilePath));
pdfDoc.end();
};
Is there a different way to save the files that don't force them to be in a stream and will not be kept in memory?

Before we get to the answer, I'm making one huge assumption based on the information in the question. The question states create about 1200 pdf forms. Which means I'm assuming in the function whichForm the parameter certList is an array of 1200 items. Or should I say 1200 items that will call the createOAReport method. You get the idea. I'm assuming the problem is that we are calling that method to create the PDFs 1200 times within that Array.map method. Which makes sense I believe given the question and context of the code.
On to the answer. The major problem is you aren't just trying to create 1200 pdfs. You are trying to create 1200 pdfs asynchronously, which of course puts a strain on the system trying to do all of that work all at once. Maybe even more so on a single thread system like Node.js.
The easy hacky solution is to just increase the memory of Node.js. By using the --max-old-space-size flag and setting the memory size in MB when running your node command. You can find more information about this at this tutorial. But the short version is a command like node --max-old-space-size=8192 main.js. That would increase the memory size of Node.js to 8192 MB or 8 GB.
Few problems with that method. Mainly it's not super scalable. What if someday you have 5000 pdfs you want to create? You'd have to increase that memory size again. And maybe increase the specs on the machine it's being run on.
The second solution, which you could actually probably do with the first solution, is to make this process not asynchronous. Depending on many factors and how optimized the current system is, chances are this will increase the amount of time it takes to create all of these PDFs.
This process is kinda a two step process to code it in. First is to setup your createOAReport function to return a promise to indicate when it's done. The second step is to change your whichForm function to limit how many items can be running asynchronously at any single point in time.
You will have to of course play around with the system to determine how many items you want to run at one time without overloading the system. Fine-tuning that number is not something I focused on, and of course you could probably increase that number by increasing the memory you give Node.js as well.
And of course, there are TONS of different ways to do this. I have a few ideas of methods that are better than the one I'm going to show here, but are a lot more complicated. The foundational idea of limiting how many items are running at once remains the same tho. You can optimize it to fit your needs.
I've developed systems like this before, but I don't think the way I've done it is the best or cleanest way to do it. But at the end of this question I've attached some sample code for your example trying to illustrate my point.
const _ = require('lodash');
const MAX_RUNNING_PROMISES = 10; // You will have to play with this number to get it right for your needs
const whichForm = async certList => {
// If certList is ["a", "b", "c", "d"]
// And we run the following function with MAX_RUNNING_PROMISES = 2
// array would equal [["a", "b"], ["c", "d"]]
certList = _.chunk(certList, MAX_RUNNING_PROMISES);
// Of course you can use something other than Lodash here, but I chose it because it's the first thing that came to mind
for (let i = 0; i < certList.length; i++) {
const certArray = certList[i];
// The following line will wait until all the promises have been resolved or completed before moving on
await Promise.all(certArray.map(cert => {
if (cert.Cert_Details !== null) {
switch (cert.GWMA) {
case 'OA':
case 'PC':
// Don't provide reports for Feedlots
if (cert.Cert_Details.cert_type !== null) {
if (cert.Cert_Details.cert_type === 'Irrigation') {
return createOAReport(cert);
}
}
break;
case 'FA':
// Don't provide reports for Feedlots
if (cert.Cert_Details.cert_type === 'Irrigation') {
return createFAReport(cert);
}
break;
}
}
}));
}
}
Then for your other file. We just have to convert it to return a promise.
const PdfPrinter = require('pdfmake/src/printer');
const fs = require('fs');
const createOAReport = data => {
return new Promise((resolve, reject) => {
console.log('PC or OA Cert ', data.Cert_ID);
// console.log(data);
let all_meters_maint = [];
const flowmeter = data.Flowmeters[0];
if (flowmeter.Active === true) {
let fm_maint = [];
fm_maint.push({
text: `Meter Serial Number: ${flowmeter.Meter_Details.Serial_num}`
});
fm_maint.push({
text: `Type of Meter: ${flowmeter.Meter_Details.Manufacturer}`
});
fm_maint.push({
text: `Units: ${flowmeter.Meter_Details.units}`
});
fm_maint.push({
text: `Factor: ${flowmeter.Meter_Details.factor}`
});
all_meters_maint.push(fm_maint);
}
docDefinition.content.push({
style: 'tableExample',
table: {
widths: [200, 200, '*', '*'],
body: all_meters_maint
},
layout: 'noBorders'
});
const fonts = {
Roboto: {
normal: path.join(__dirname, '../', '/fonts/Roboto-Regular.ttf'),
bold: path.join(__dirname, '../', '/fonts/Roboto-Medium.ttf'),
italics: path.join(__dirname, '../', '/fonts/Roboto-Italic.ttf'),
bolditalics: path.join(__dirname, '../', '/fonts/Roboto-MediumItalic.ttf')
}
};
const printer = new PdfPrinter(fonts);
const pdfDoc = printer.createPdfKitDocument(docDefinition);
// Build file path
const fullfilePath = path.join(
__dirname,
'../',
'/public/pdffiles/',
`${data.Cert_ID}.pdf`
);
pdfDoc.pipe(fs.createWriteStream(fullfilePath));
pdfDoc.on('finish', resolve); // This is where we tell it to resolve the promise when it's finished
pdfDoc.end();
});
};
I just realized after getting really far into this answer that my original assumption is incorrect. Since some of those pdfs might be created within the second function and the data.Flowmeters.map system. So although I'm not going to demonstrate it, you will have to apply the same ideas I have given throughout this answer to that system as well. For now, I have removed that section and am just using the first item in that array, since it's just an example.
You might want to restructure your code once you have an idea of this and just have one function that handles creating the PDF, and not have as many .map method calls all over the place. Abstract the .map methods out and keep it separate from the PDF creation process. That way it'd be easier to limit how many PDFs are being created at a single time.
It'd also be a good idea to add in some error handling around all of these processes.
NOTE I didn't actually test this code at all, so there might be some bugs with it. But the overall ideas and principals still apply.

Multiple pipes on writeStream open event gives undefined

I am downloading a file from a URL and then I want to write the metadata for that file to the destination stream only after I make sure that the destination file is created(i.e. after fs.createWriteStream(path) is successful).So I have used the "open" event of writable stream to proceed further. However, this code gives me the error exactly on the second pipe:
Cannot call method 'pipe' on undefined
There is more code beyond this which uses the hashes that are calculated here. But somehow I am stuck with this error at the moment. I have been struggling on this for quite a while. Any help/pointers are very much appreciated.
Also I tried to run the example
var fs = require('fs');
var digestStream = require('digest-stream')
callHandle();
function callHandle(){
var stream = request.get(url)
var result = {}
handle(reader, result)
}
function handle(reader,metadata){
const writer = fs.createWriteStream('pathToFile');
writer.on('open', function(){
reader.pipe(
digestStream('sha1', 'hex', function(digest, length) {
result.sha1 = digest;
result.size = length;
}))
.pipe(
digestStream('md5', 'hex', function(digest) {
result.md5 = digest;
})
).pipe(writer)
})
}