I am developing a desktop app with Electron and Angular 7.
There is a part where I need to zip a folder, which could weight a lot.
I am using ipcRenderer to send from angular to electron the signal to start the zipping.
This is the chunk of the ipcMain:
const { app, BrowserWindow, ipcMain } = require("electron");
const zip = require('file-zip');
...
ipcMain.on('zip', (event, args) => {
const { from, to } = args;
zip.zipFolder(from, to, (error) => {
event.sender.send('zip-response', error);
});
});
The problem is that when the folder has a huge size, the task takes a lot and blocks the rendered process.
I have already tried with 'electron-remote' and its method requireTaskPool, like this:
const zip = require('file-zip');
function zipDir(from, to) {
zip.zipFolder(from, to, (error) => {
return error;
});
}
module.exports = zipDir;
and:
import { requireTaskPool } from 'electron-remote';
const zip = requireTaskPool(require.resolve('./zip'));
ipcMain.on('zip', (event, args) => {
const { from, to } = args;
zip(from, to).then(error => event.sender.send('zip-response', error));
});
but it did not work, "zip" always resolved immediately, without executing the zip function, probably because zip.zipFolder uses a callback.
Any idea?
Related
I'm using the code from this question to archive files using node-archiver and transfer them to S3. My specific task requires me to download a large number of files from URLs, zip them to one archive, and transfer them to S3.
I'm using the "got" Javascript library to do this.
for (const file of files) {
const { resultFileName, fileUrl} = getFileNameAndUrl(file);
if (!fileUrl)
continue;
const downloadStream = got.stream(fileUrl, {
retry: {
limit: 5
}
});
archive.append(downloadStream, { name: resultFileName });
}
The rest of the code is pretty much the same as in the original question. The issue is that script doesn't work well with a huge amount of files (it just finishes execution at some point).
In the perfect world - I want this script to download files, append them to archive and transfer them to S3 using pipes. And the best is to download them in batches (something like Promise.map with concurrency in bluebird). I just don't get how to do it with Streams, as I do have not much experience with them.
archiver package processes one file at a time, so there is no point in downloading several in parallel with got. Follow the example by that link you provided and it should work.
Also, do not open a lot of streams to all files should be zipped. Do that one by one, since streams and archived package have timeouts on opened streams.
I hope this helps.
NOTE: I could not test this because I don't have access to aws s3.
This snippet should download webpages and saves it in zip file, which should contain fs.html & index.html file.
// file:main.mjs
import got from 'got'
import archiver from 'archiver'
import S3 from 'aws-sdk/clients/s3'
import { basename } from 'path'
try {
const urls = ['https://nodejs.org/api/fs.html', 'https://nodejs.org/api/index.html']
const gotconfig = {}
const archive = archiver('zip', {
zlib: { level: 9 },
})
archive.on('warning', function (err) {
if (err.code === 'ENOENT') {
} else {
throw err
}
})
archive.on('error', function (err) {
throw err
})
for (const url of urls) {
// const _url = new URL(url)
archive.append(got.stream(url, gotconfig), { name: basename(url) })
}
const s3 = new S3()
await s3.upload({ Bucket: 'buck', Key: 'key', Body: archive }).promise()
await archive.finalize()
} catch (error) {
console.error(error)
}
this one I have tested & it works. Similar to above but saves zip file in /tmp/test1.zip.
// file: local.mjs
import got from 'got'
import { createWriteStream } from 'fs'
import archiver from 'archiver'
import { basename } from 'path'
try {
const urls = ['https://nodejs.org/api/fs.html', 'https://nodejs.org/api/index.html']
const gotconfig = { }
const output = createWriteStream('/tmp/test1.zip')
const archive = archiver('zip', {
zlib: { level: 9 },
})
output.on('close', function () {
console.log(archive.pointer() + ' total bytes')
console.log('archiver has been finalized and the output file descriptor has closed.')
})
output.on('end', function () {
console.log('Data has been drained')
})
archive.on('warning', function (err) {
if (err.code === 'ENOENT') {
} else {
throw err
}
})
archive.on('error', function (err) {
throw err
})
archive.pipe(output)
for (const url of urls) {
archive.append(got.stream(url, gotconfig), { name: basename(url) })
}
await archive.finalize()
} catch (error) {
console.error(error)
}
I would check if a ZIP file is corrupted using NodeJS using less CPU and memory as possible.
How to corrupt a ZIP file:
Download a ZIP file
Open the ZIP file using a text editor optimized like Notepad++
Rewrite the header. Only put random characters.
I am trying to reach this goal using the NPM library "node-stream-zip"
private async assertZipFileIntegrity(path: string) {
try {
const zip = new StreamZip.async({ file: path });
const stm = await zip.stream(path);
stm.pipe(process.stdout);
stm.on('end', () => zip.close());
} catch (error) {
throw new Error();
}
}
However, when I run the unit tests I receive an error inside an array:
Rejected to value: [Error]
import zip from 'yauzl';
import path from 'path';
const invalidZipPath = path.resolve('invalid.zip');
const validZipPath = path.resolve('valid.zip');
const isValidZipFile = (filePath) => {
return zip.open(filePath, { lazyEntries: true }, (err, stream ) => {
if (err) {
console.log('fail to read ', filePath);
return false;
}
console.log('success read ', filePath);
return true;
});
}
isValidZipFile(validZipPath);
isValidZipFile(invalidZipPath);
Alright so I have these methods in my index.js main process that I want to access from the render process. I have tried two ways to go about this process.
ipcMain and ipcRender
The first idea was to use ipcMain and ipcRender using an "on" and "sendSync" The error I get back is "an object could not be cloned"
Index.js - Main Process
ipcMain.on( "getData", ( event, callBack ) => {
db = new sqlite3.Database(
dbPath,
sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE,
function(error) {
if(error){
log.error(`exception: ${error}`);
//throw error
if (callBack) callBack(error);
}
data.getModel(db, log, function(rawDataStr) {
if (callBack) callBack(rawDataStr);
})
}
)
return db
} );
App.Js - Render Process
window.require('electron').ipcRenderer.sendSync( "getData",function(rawData){
if (rawData.name && rawData.name == 'Error') {
alert('PRT DB is not present');
} else {
sharedObj.rawData = rawData;
app.advanceReadiness();
}
})
#electron/remote
The other solution I tried was to use #electron/remote. I understand the remote module was depreciated, but I was willing to try it. This works when I run the the app locally, but as soon as I build the app with electron-forge it can no longer find my global variable.
Index.js - Main Process
require('#electron/remote/main').initialize()
global.sharedObj = {
getData:function(callBack){
db = new sqlite3.Database(
dbPath,
sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE,
function(error) {
if(error){
log.error(`exception: ${error}`);
//throw error
if (callBack) callBack(error);
}
data.getModel(db, log, function(rawDataStr) {
if (callBack) callBack(rawDataStr);
})
}
)
}
}
App.js - Render Process
var sharedObj = window.require('#electron/remote').getGlobal('sharedObj');
sharedObj.getData(function (rawData) {
if (rawData.name && rawData.name == 'Error') {
alert('PRT DB is not present');
} else {
sharedObj.rawData = rawData;
app.advanceReadiness();
}
});
I suspect that your DB connection isn't cloneable, because this DB object doesn't fit one of the valid values that can be serialized by IPC (inter-process communication). (See this section to see what we can pass between renderer > main process without issue).
You probably need to do something like this. I'm not familiar with using sqlite3 in JS, but this should hopefully get you started on the right track. The general gist is that you should store a reference to your DB in your main.js file and then set up a listener that listens for requests from your front-end page. Once a message is sent to the main.js file (ie. backend), you will query your DB, and then return the results to the front-end by sending an IPC message back (win.webContents.send("fromMain", data); in the example elow).
main.js
const {
app,
BrowserWindow,
ipcMain
} = require("electron");
const path = require("path");
const fs = require("fs");
// Keep a global reference of the window object, if you don't, the window will
// be closed automatically when the JavaScript object is garbage collected.
let win;
let db = new sqlite3.Database(
dbPath,
sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE,
function (error) {
if (error) {
log.error(`exception: ${error}`);
//throw error
if (callBack) callBack(error);
}
data.getModel(db, log, function (rawDataStr) {
if (callBack) callBack(rawDataStr);
})
}
);
async function createWindow() {
// Create the browser window.
win = new BrowserWindow({
width: 800,
height: 600,
webPreferences: {
nodeIntegration: false, // is default value after Electron v5
contextIsolation: true, // protect against prototype pollution
enableRemoteModule: false, // turn off remote
preload: path.join(__dirname, "preload.js") // use a preload script
}
});
// Load app
win.loadFile(path.join(__dirname, "dist/index.html"));
// rest of code..
}
app.on("ready", createWindow);
ipcMain.on("toMain", (event, args) => {
db.retrieveData().then(data => {
// Send result back to renderer process
win.webContents.send("fromMain", data);
});
});
preload.js
const {
contextBridge,
ipcRenderer
} = require("electron");
// Expose protected methods that allow the renderer process to use
// the ipcRenderer without exposing the entire object
contextBridge.exposeInMainWorld(
"api", {
send: (channel, data) => {
// whitelist channels
let validChannels = ["toMain"];
if (validChannels.includes(channel)) {
ipcRenderer.send(channel, data);
}
},
receive: (channel, func) => {
let validChannels = ["fromMain"];
if (validChannels.includes(channel)) {
// Deliberately strip event as it includes `sender`
ipcRenderer.on(channel, (event, ...args) => func(...args));
}
}
}
);
index.html
<!doctype html>
<html lang="en-US">
<head>
<meta charset="utf-8"/>
<title>Title</title>
</head>
<body>
<script>
window.api.receive("fromMain", (data) => {
console.log(`Received ${data} from main process`);
});
window.api.send("toMain", "some data");
</script>
</body>
</html>
I am creating this application with vue-cli and vue-cli-plugin-electron-builder and I'm having this weird problem, specific to electron, where once the application starts the first ReadStream created will not pipe its contents to the given stream.
Now, refreshing the application through CTRL + F5 will make the streams work again, and from there on there won't be any problems anymore.
My current code is something similar to and is called on the Render Thread once the application starts:
public async run() {
await this.scanFolder("/some/path/to/a/folder");
}
private async scanFolder(path: string){
const entries = readdirSync(path, { withFileTypes: true });
for (const entry of entries){
if (!entry.isDirectory()){
const md5 = await calculateMD5(path + "/" + entry.name);
}
}
}
public static async calculateFileMD5(path: string) {
const md5 = createHash("md5");
md5.setEncoding("hex");
console.log("Creating promise");
const promise = new Promise<string>((resolve, reject) => {
const fileStream = createReadStream(path, {
autoClose: true,
emitClose: true
});
fileStream.on("open", () => console.log("STREAM OPEN"));
fileStream.on("ready", () => console.log("STREAM READY"));
fileStream.on("close", () => {
console.log("Stream closed");
resolve(md5.read());
});
fileStream.on("data", data => {
console.log(data);
});
fileStream.on("end", () => {
console.log("Stream ENDED");
resolve(md5.read());
});
fileStream.on("error", error => {
console.error(error);
reject(error);
});
console.log("Piping to MD5");
fileStream.pipe(md5, { end: true });
fileStream.resume();
console.log("Is paused?: " + fileStream.isPaused());
});
console.log("Returning promise");
return promise;
}
Starting up the application with npm run electron:serve and calling the run function will output this:
Creating promise
Piping to MD5
Is paused?: false
Returning promise
STREAM OPEN
STREAM READY
Now, if the application is reloaded through CRTL + F5 the stream properly pipes its contents to the Hash.
Is there anything I can do to make it not require a refresh once the application starts for the streams to work properly?
So I am building an Electron and React App. I am using ghost-script to create imgs of certain pdf files and I want to know how I would tell Node.js to wait for the imgs to be created before bringing up the window and changing the state in the App. These imgs are being used as src for a component and when the component tries to load the img it somewhat retains a broken state because the img src doesn't exist when the state updates.
// this is where I set the state before sending all the data to the renderer process(the front end of the App)
function getStateReady(theState, event) {
let pdfFiles = scanDirectory(currentDir);
let pdfNames = getPdfName(pdfFiles);
imageUrls = getImgName(pdfFiles);
console.log(imageUrls);
createImg(pdfFiles, pdfNames);
switch (theState) {
case 'initialState':
mainWindow.webContents.on('did-finish-load', () => {
mainWindow.webContents.send('initialState', pdfFiles, imageUrls);
})
break;
case 'secondState-reply':
event.sender.send('secondState-reply', pdfFiles, imageUrls);
break;
default:
console.log('a param was missing');
}
}
//these to functions take a pdf file path and its name to create an img
function createImg(pdfPaths, pdfNames) {
pdfNames.forEach((item, index) => {
if(fs.existsSync(path.join(rootDirectory, 'src', 'imgs', item.replace('.pdf', '.jpg')))) {
console.log('image exists');
}
else {
console.log("creating image");
child(returnProcess(pdfPaths[index], item), (err, stdout) => {
if(err) {
console.log(err)
}
console.log(stdout)
})
}
})
}
function returnProcess(pdfPath, pdfName) {
let newPdf = `"${pdfPath}"`
let output = `"${path.join(rootDirectory, 'src', 'imgs', pdfName.replace('.pdf', '.jpg'))}"`;
let mainProcess = `"C:\\Program Files\\gs\\gs9.23\\bin\\gswin64c.exe" -q -o ${output} -sDEVICE=pngalpha -dLastPage=1 ${newPdf}`
return mainProcess;
}
I am not completely sure with code and logic and I have never worked with ghost-script but what I can suggest here to solve this issue is use a call back function. basically pass a callback function to createImg method and execute the callback function which will tell the NodeJS that process is done and front-end should able to display the created image.
Below is the update code.
// this is where I set the state before sending all the data to the renderer process(the front end of the App)
function getStateReady(theState, event) {
let pdfFiles = scanDirectory(currentDir);
let pdfNames = getPdfName(pdfFiles);
imageUrls = getImgName(pdfFiles);
console.log(imageUrls);
createImg(pdfFiles, pdfNames, function (){
switch (theState) {
case 'initialState':
mainWindow.webContents.on('did-finish-load', () => {
mainWindow.webContents.send('initialState', pdfFiles, imageUrls);
})
break;
case 'secondState-reply':
event.sender.send('secondState-reply', pdfFiles, imageUrls);
break;
default:
console.log('a param was missing');
}
});
}
//these to functions take a pdf file path and its name to create an img
function createImg(pdfPaths, pdfNames, fun) {
pdfNames.forEach((item, index) => {
if(fs.existsSync(path.join(rootDirectory, 'src', 'imgs', item.replace('.pdf', '.jpg')))) {
console.log('image exists');
}
else {
console.log("creating image");
child(returnProcess(pdfPaths[index], item), (err, stdout) => {
if(err) {
console.log(err)
}
console.log(stdout)
})
}
})
// call the callback function
fun.call();
}
function returnProcess(pdfPath, pdfName) {
let newPdf = `"${pdfPath}"`
let output = `"${path.join(rootDirectory, 'src', 'imgs', pdfName.replace('.pdf', '.jpg'))}"`;
let mainProcess = `"C:\\Program Files\\gs\\gs9.23\\bin\\gswin64c.exe" -q -o ${output} -sDEVICE=pngalpha -dLastPage=1 ${newPdf}`
return mainProcess;
}