In my webpage, a user is supposed to upload a zipped file. Within the zipped file are 2 files: another zip file and a txt file. On my server, after receiving the zip, I want to unzip the zip file to extract the zip & txt file, then move those 2 files to a predefined folder. I have a piece of code that extracts the zip file, but the data doesn't seem correct. Firstly, it unzipped a zip and 2 txt file when there should only be 1 txt file. It created an additional 'undefined' txt file. Also, in my txt file, instead of the original data, it was replaced with the following text: '[undefined] [undefined]'.
Can anyone help me on this? The following is my code:
var JSZip = require('JSZip');
fs.readFile( filePath, function(err, data){
if (!err){
var zip = new JSZip();
JSZip.loadAsync(data).then(function(zip){
object.keys(zip.files).forEach(function(filename){
var content = zip.files[filename];
var dest = path + filename;
fs.writeFileSync(dest, content);
});
});
}
});
This is a working version I am using:
var jsZip = require('jszip')
jsZip.loadAsync(file).then(function (zip) {
Object.keys(zip.files).forEach(function (filename) {
zip.files[filename].async('string').then(function (fileData) {
console.log(fileData) // These are your file contents
})
})
})
You can get most of the information you need from http://stuk.github.io/jszip/documentation/examples.html but it's a little hard to get in one place, you have to look around a bit.
It took a bit of digging in their documentation but they have an example that shows how to read the file contents from a ZIP.
You are getting the object that describes the ZIP contents but not the actual content. Here is an adjusted version:
var JSZip = require('JSZip');
fs.readFile(filePath, function(err, data) {
if (!err) {
var zip = new JSZip();
zip.loadAsync(data).then(function(contents) {
Object.keys(contents.files).forEach(function(filename) {
zip.file(filename).async('nodebuffer').then(function(content) {
var dest = path + filename;
fs.writeFileSync(dest, content);
});
});
});
}
});
Here was my strategy in Angular 10 to write a single file to a zip with a custom extension, then later read that same zip to retrieve the json.
Package the file into a zip (custom file endings supported)
import { saveAs } from 'file-saver';
import * as JSZip from 'jszip';
export async function exportJson(
filename: string,
jsonToExport: string,
fileNameEnding = '.zip'
): Promise<string> {
const jsonFile = new Blob([jsonToExport], {
type: 'application/json',
});
if (!jsonFile) return Promise.reject('Error converting file to JSON');
const zipper = new JSZip();
zipper.file(`${filename}.json`, jsonFile);
const zippedFile = await zipper.generateAsync({ type: 'blob' });
const exportFilename = `${filename}${fileNameEnding}`;
saveAs(zippedFile, exportFilename);
return Promise.resolve(exportFilename);
}
Read the file contents from the zip
// file parameter retrieved from an input type=file
export async function readExportedJson(file: File): Promise<Blob> {
const zipper = new JSZip();
const unzippedFiles = await zipper.loadAsync(file);
return Promise.resolve(unzippedFiles).then(unzipped => {
if (!Object.keys(unzipped.files).length) {
return Promise.reject('No file was found');
}
return unzipped.files[Object.keys(unzipped.files)[0]];
}).then(unzippedFile => zipper.file(unzippedFile.name).async('string'));
}
This answer is cordova-plugin-file specific.
As stated in the docs:
Directory entries have to be created successively. For example, the
call fs.root.getDirectory('dir1/dir2', {create:true}, successCallback,
errorCallback) will fail if dir1 did not exist.
I am almost certain that the currently accepted answer cannot guarantee that file content/folders are always retrieved in the same order. This could result in problems with an API such as cordova-plugin-file. Especially when you invoke another async function to asynchronously create the directory on the filesystem.
You may want to filter the directories of your zip archive first and create them in a sync manner before continuing to extract other files as already answered:
const directoryNames = Object.keys(zip.files).filter(name => zip.files[name].dir);
for (const directoryName of directoryNames) {
await this.createDirectory(directoryName, dirEntry);
}
// ...
private createDirectory = (dirName: string, dirEntry: DirectoryEntry) => {
const promise = new Promise<DirectoryEntry>(resolve, reject) => {
dirEntry.getDirectory(dirName, { create: true }, dirEntry => {
resolve(dirEntry);
}, fileError => reject(fileError));
});
return promise;
}
Related
I am writing a function that downloads and converts a pdf into individual jpg files by page. I am using the imagemagick library to do the conversion. I am having trouble with my processPDF() function as it immediately returns undefined. I put a console.log statement immediately before the function returns and it returns the exact value I expect yet that value doesn't seem to be getting outside of the function for some reason.
import im from 'imagemagick'
import { promises as fs } from 'fs'
import path from 'path'
import _ from 'lodash'
import axios from 'axios'
import { v4 as uuid } from 'uuid'
async function processPDF(pdfPath) {
let basename = path.basename(pdfPath, '.pdf')
let outputPath = "./img/" + basename + ".jpg";
console.log(`Converting ${pdfPath}`)
// Take PDF file and generate individual JPG files
await im.convert(["-density", 300, pdfPath, outputPath],async (err) => {
if (err) {
console.log(err)
throw `Couldn't Process ${pdfPath}`
}
else {
// Get every file in Temporary Image Directory
let files = await fs.readdir(`./img/`)
// Append directory into filenames
files = files.map(file => {
return "./img/" + file
})
// We only want the files that match the source pdf's name
files = files.filter((file) => {
return file.includes(basename)
})
console.log(`Getting ${basename} Buffer Data`)
// For each file, read and return the buffer data along with the path
let images = await Promise.all(files.map(async file => {
const contents = await fs.readFile(file)
return { path: file, buffer: contents }
}))
// Since we read the files asynchonously, Reorder the files
images = _.orderBy(images, (image) => {
let regex = /\d*.jpg/
let res = image.path.match(regex)[0]
res = path.basename(res, '.jpg')
return res
})
let output = { pdf: pdfPath, images }
// Returns a value
console.log(output)
// Returns undefined???
return output
}
})
}
export async function downloadAndProcessPDF(url) {
// Fetch PDF from server
let { data } = await axios.get(url, {
responseType: 'arraybuffer',
headers: {
'Content-Type': 'application/json',
'Accept': 'application/pdf'
}
}).catch(e=>{
console.log(e);
throw `Can't retrieve ${url}`
})
// Generate a Unique ID for the pdf since this is called asynchronously, this will be called many times simultaneously
let id = "./pdf/" + uuid() + ".pdf"
await fs.writeFile(id, data);
// tell processPDF to process the pdf in the ./pdf directory with the given filename
let pdfData = await processPDF(id);
// Returns undefined???
console.log(pdfData)
return pdfData
}
If I had to take a wild guess I'd think that im.convert is the function that is giving me trouble. Throughout my source code i'm using promises to handle asynchronous tasks yet im.convert() uses a callback function. I'm not super familiar with how concurrency works between promises and callback functions so I think that's what's probably the issue.
I would check if a ZIP file is corrupted using NodeJS using less CPU and memory as possible.
How to corrupt a ZIP file:
Download a ZIP file
Open the ZIP file using a text editor optimized like Notepad++
Rewrite the header. Only put random characters.
I am trying to reach this goal using the NPM library "node-stream-zip"
private async assertZipFileIntegrity(path: string) {
try {
const zip = new StreamZip.async({ file: path });
const stm = await zip.stream(path);
stm.pipe(process.stdout);
stm.on('end', () => zip.close());
} catch (error) {
throw new Error();
}
}
However, when I run the unit tests I receive an error inside an array:
Rejected to value: [Error]
import zip from 'yauzl';
import path from 'path';
const invalidZipPath = path.resolve('invalid.zip');
const validZipPath = path.resolve('valid.zip');
const isValidZipFile = (filePath) => {
return zip.open(filePath, { lazyEntries: true }, (err, stream ) => {
if (err) {
console.log('fail to read ', filePath);
return false;
}
console.log('success read ', filePath);
return true;
});
}
isValidZipFile(validZipPath);
isValidZipFile(invalidZipPath);
I'm trying to add some files (identity documents) to stripe to create a connected account, but I'm having trouble with uploading them from client side to stripe. My backend is in Node.js and the Stripe documentation says it should use this format:
const Stripe = require('stripe');
const stripe = Stripe('stripeAPIKEY');
var fp = fs.readFileSync('/path/to/a/file.jpg');
var file = await stripe.files.create({
purpose: 'identity_document',
file: {
data: fp,
name: 'file.jpg',
type: 'image/jpg',
},
});
I need to upload the file data (the variable fp), but I can't seem to get the relevant path for when the user uploads their document in the client side in Javascript. Here is my function call to Stripe:
export const uploadPersonIdFile = async (identityDocument: any) => {
const fp = fs.readFileSync(identityDocument);
const personId = await stripe.files.create({
purpose: 'identity_document',
file: {
data: fp,
name: 'idDocument.jpg',
type: 'image/jpg',
},
});
return personId;
}
My client side looks like this:
const inpFileU = $("#utilityButton");
const previewImage = $("#image-preview__image-U");
const previewDefaultText = $("#image-preview__default-text-U");
inpFileU.change(function(){
const file = this.files[0];
if(file){
const reader = new FileReader();
previewImage.css("display", "block");
reader.addEventListener("load", function(){
previewImage.attr("src", this.result);
});
reader.readAsDataURL(file);
utilityFileName = file.name;
await uploadMerchantUtilityDocument({
utilityDocument: utilityFileName
}).then((result) => {
/** #type {any} */
const data = result.data;
const textData = data.text;
console.log(JSON.stringify(data));
console.log(JSON.stringify(textData));
}).catch((error) => {
console.log("Error message: " + error.message);
console.log("Error details: " + error.details);
});
} else {
console.log('no file');
}
});
I upload the file and then the error message response I keep getting is this:
Error: ENOENT: no such file or directory, open 'insuranceImage.jpeg'
How should I upload my file? I think my fp variable is wrong, but I don't know what to replace it with
As far as I understood you are uploading files from the client to the server and from the server you want to upload to stripe API. In this case, when you read a file with fs default encoding is utf8 but maybe when the file uploaded it was encoded as base64. I do not know too much about jquery. Check how it was encoded, so use the correct encoding.
const image= fs.readFileSync('/path/to/file.jpg', {encoding: 'base64'});
I think somehow the image is broken. If the image path is correct, just manually place an image in that directory, and then read from it as utf8 encoding.
since u got this error base64 string is too large to process in fs.readFileSync which means your path is correct.
reader = fs.createReadStream('imagePath', {
flag: 'a+',
// then try this to base64
encoding: 'UTF-8',
start: 5,
end: 64,
highWaterMark: 16
});
// Read and display the file data on console
reader.on('data', function (chunk) {
console.log(chunk);
});
Based on the error and the comment it looks like the path you're providing to readFileSync isn't pointing to where the file you're trying to read exists. From the error it looks like you're passing insuranceImage.jpeg and that the system can't find that file.
Try confirming whether the relative path you're providing is correct, or construct and provide an absolute path instead.
Ok, so after a while I was helped by a pro online. Here's what he did (because stripe's documentation isn't too great on file uploading)...
I created a base64 variable from FileReader(), which came from creating an array base64String = [] This array took hold of the result from reader.load, like so:
reader.addEventListener("load", function(){
base64String.push(this.result);
});
This base64String array was then used as the identityDocument variable for the backend function.
So my base64 variable in the client-side kept on starting with 'image/jpeg; base64,' and I needed to get rid of that. No point using the fs.readFileSync as we're not uploading a file to stripe, we're uploading raw base64 data. So, the following code in node.js fits in well to solve this:
const parts = utilityDocument.split(",");
const base64 = parts[1];
const buffer = Buffer.from(base64, "base64");
so the full function that calls to stripe is so:
export const uploadPersonIdFile = async (uid: any, identityDocument: any) => {
const parts = identityDocument.split(",");
const base64 = parts[1];
const buffer = Buffer.from(base64, "base64");
const personId = await stripe.files.create({
purpose: 'identity_document',
file: {
data: buffer,
name: 'identity.jpg',
type: 'image/jpg',
},
});
await updateMerchantId(uid, { idDocument: personId.id });
return personId;
}
I need to create a zip file with any PDF what I recieved from Storage AWS, and I am trying do this with ADM-zip in NodeJS, but i cant read the final file.zip.
Here is the code.
var zip = new AdmZip();
// add file directly
var content = data.Body.buffer;
zip.addFile("test.pdf", content, "entry comment goes here");
// console.log(content)
// add local file
zip.addLocalFile(`./tmp/boletos/doc.pdf`);
// // get everything as a buffer
var willSendthis = zip.toBuffer();
console.log(willSendthis)
// // or write everything to disk
zip.writeZip("test.zip", `../../../tmp/boletos/${datastring}.zip`);
As it is this only creates a .zip for each file..zip
I was also facing this issue. I looked through a lot of SO posts. This is how I was able to create a zip with multiple files from download urls. Please keep in mind, I'm unsure this is best practice, or if this is going to blow up memory.
Create a zip folder from a list of id's of requested resources via the client.
const zip = new AdmZip();
await Promise.all(sheetIds.map(async (sheetId) => {
const downloadUrl = await this.downloadSds({ sheetId, userId, memberId });
if (downloadUrl) {
await new Promise((resolve) => https.get(downloadUrl, (res) => {
const data = [];
res.on('data', (chunk) => {
data.push(chunk);
}).on('end', () => {
const buffer = Buffer.concat(data);
resolve(zip.addFile(`${sheetId}.pdf`, buffer));
});
}));
} else {
console.log('could not download');
}
}));
const zipFile = zip.toBuffer();
I then used downloadjs in my React.js client to download.
const result = await sds.bulkDownloadSds(payload);
if (result.status > 399) return store.rejectWithValue({ errorMessage: result?.message || 'Error', redirect: result.redirect });
const filename = 'test1.zip';
const document = await result.blob();
download(document, filename, 'zip');
I have a zip file (actually it's an epub file) I need to loop through the files in it and read them without unzipping them to the disk.
I tried to use a Node.js library called JSZip but the content of each file is stored in memory in Buffer and whenever I try to decode the buffer content to string the content returned is unreadable
Here's the code I tried:
const zip = new JSZip();
// read a zip file
fs.readFile(epubFile, function (err, data) {
if (err) throw err;
zip.loadAsync(data).then(function (zip) {
async.eachOf(zip.files, function (content, fileName, callback) {
if (fileName.match(/json/)) {
var buf = content._data.compressedContent;
console.log(fileName);
console.log((new Buffer(buf)).toString('utf-8'));
}
callback();
}, function (err) {
if (err) {
console.log(err);
}
});
});
});
Since unzip seems to be abandoned, I used node-stream-zip with pretty good success.
npm install node-stream-zip
Reading files be all like:
const StreamZip = require('node-stream-zip');
const zip = new StreamZip({
file: 'archive.zip',
storeEntries: true
});
zip.on('ready', () => {
// Take a look at the files
console.log('Entries read: ' + zip.entriesCount);
for (const entry of Object.values(zip.entries())) {
const desc = entry.isDirectory ? 'directory' : `${entry.size} bytes`;
console.log(`Entry ${entry.name}: ${desc}`);
}
// Read a file in memory
let zipDotTxtContents = zip.entryDataSync('path/inside/zip.txt').toString('utf8');
console.log("The content of path/inside/zip.txt is: " + zipDotTxtContents);
// Do not forget to close the file once you're done
zip.close()
});
npm install unzip
https://www.npmjs.com/package/unzip
fs.createReadStream('path/to/archive.zip')
.pipe(unzip.Parse())
.on('entry', function (entry) {
var fileName = entry.path;
var type = entry.type; // 'Directory' or 'File'
var size = entry.size;
if (fileName === "this IS the file I'm looking for") {
entry.pipe(fs.createWriteStream('output/path'));
} else {
entry.autodrain();
}
});