I am working on a Node Express, which would call REST URL to download a zip and extract it.
Below are the steps that I am following
Using the request module, send a POST request to REST API.
REST API returns a zip file. Pipe it to save it to the filesystem
Extract the zip file
Cleanup the zip file (this code is not there yet)
This is how my code looks like
return new Promise(function(resolve, reject) {
var req = require('request');
var extract = require('extract-zip');
var postData = {
username: "abc",
password: "zyz"
}
var options = {
method: 'post',
body: postData,
json: true,
url: "http://localhost:8080/zipdownload"
}
req(options, function(err, res, body) {
if(err) {
console.log('Error posting json ' + err)
return;
}
}).pipe(fs.createWriteStream(USER_DIR + '/MY.zip')).on('finish', function() {
console.log('Finished downloading ZIP file');
extract(USER_DIR + '/MY.zip', {
dir: USER_DIR + '/MyExtractedFolder'
}, function(err) {
if(err) {
console.log('Error extracting zip ' + err);
}
resolve("Promised resolved donwloading and extracting zip");
})
}).on('error', function(error) {
reject('Error in downloading content', error);
});
});
I am able to download the zip file and save it in the file system using .pipe. However when I try to extract it using extract-zip, it gives me the below error
End of central directory record signature not found.
I suspect that extract code is not at the right place and Promise gets resolved while the extract code is still running. As a result, the calling function, tries to do something with the extracted files which do not currently exist.
Also, I am not able to make out where should I put the code for zip cleanup. It needs to trigger once the extraction has been done and not in parallel to extraction.
Related
I am generating the PDF using html-pdf node library and then encrypting it using node-qpdf library.
pdf.create(htmlString, {format: 'A4', timeout: 600000})
.toFile(filePath, function (err, data) {
if (err) {
logger.error('Error while creating the pdf file ::->', err.stack);
return reject(err);
} else {
logger.info('Successfully able to create the file in tmp');
encrypt(filePath, '/tmp/' + _.get(lead, 'code', '00003') + '_receipt_encrypted.pdf');
return resolve(data);
}
});
var encrypt = function (inputFile, outputFile) {
let exec = require('child_process').exec;
var ownerPass = 'ownerpasswd'
var userPass = 'ownerpasswd'
let cmd = 'qpdf --encrypt ' + ownerPass + ' ' + userPass + ' 256 -- ' + inputFile + ' ' + outputFile
exec(cmd, function (err) {
if (err) {
console.error("The error", err, err.stack)
} else {
console.log("The PDF is encrypted")
}
})
}
I am able to generate the enccrypted pdf file in the /tmp/ folder bit when I am trying to send it via Email, I am getting the error
[Error: ENOENT: no such file or directory, open '/tmp/A6rz2K-p2n_receipt_encrypted.pdf'] {
Here is the code for reading file and sending it via email.
if (err) {
console.log('error in reading encrypted file================= ', err, err.stack);
}
var attachment = {
content: data.toString('base64'),
name: encryptedFileName,
type: "application/pdf"
};
sendReceiptEmail(lead,attachment);
logger.log("end of receipt pdf creation");
})
return Promise.resolve();
})```
I guess you're having an issue with async code execution. The call to encrypt looks good, since that function is called inside the pdf..toFile callback function. Maybe you need to make sure sendReceiptEmail is called after encrypt is done, inside the exec callback function.
As Raya commented, I'd suggest to refactor the code to use promises or async code execution as well.
I have two functions in separate files to split up the workflow.
const download = function(url){
const file = fs.createWriteStream("./test.png");
const request = https.get(url, function(response) {
response.pipe(file);
});
}
This function in my fileHelper.js is supposed to take a URL with an image in it and then save it locally to test.png
function uploadFile(filePath) {
fs.readFile('credentials.json', (err, content) => {
if (err) return console.log('Error loading client secret file:', err);
// Authorize a client with credentials, then call the Google Drive API.
authorize(JSON.parse(content), function (auth) {
const drive = google.drive({version: 'v3', auth});
const fileMetadata = {
'name': 'testphoto.png'
};
const media = {
mimeType: 'image/png',
body: fs.createReadStream(filePath)
};
drive.files.create({
resource: fileMetadata,
media: media,
fields: 'id'
}, (err, file) => {
if (err) {
// Handle error
console.error(err);
} else {
console.log('File Id: ', file.id);
}
});
});
});
}
This function in my googleDriveHelper.js is supposed to take the filePath of call and then upload that stream into my google drive. These two functions work on their own but it seems that the https.get works asynchronously and if I try to call the googleDriveHelper.uploadFile(filePath) function after the download, it doesn't have time to get the full file to upload so instead a blank file will be uploaded to my drive.
I want to find a way so that when the fileHelper.download(url) is called, it automatically uploads into my drive.
I also don't know if there is a way to create a readStream directly from the download function to the upload function, so I can avoid having to save the file locally to upload it.
I believe your goal as follows.
You want to upload a file retrieving from an URL to Google Drive.
When you download the file from the URL, you want to upload it to Google Drive without creating the file.
You want to achieve this using googleapis with Node.js.
You have already been able to upload a file using Drive API.
For this, how about this answer?
Modification points:
At download function, the retrieved buffer is converted to the stream type, and the stream data is returned.
At uploadFile function, the retrieved stream data is used for uploading.
When the file ID is retrieved from the response value of Drive API, please use file.data.id instead of file.id.
By above modification, the file downloaded from the URL can be uploaded to Google Drive without creating a file.
Modified script:
When your script is modified, please modify as follows.
download()
const download = function (url) {
return new Promise(function (resolve, reject) {
request(
{
method: "GET",
url: url,
encoding: null,
},
(err, res, body) => {
if (err && res.statusCode != 200) {
reject(err);
return;
}
const stream = require("stream");
const bs = new stream.PassThrough();
bs.end(body);
resolve(bs);
}
);
});
};
uploadFile()
function uploadFile(data) { // <--- Modified
fs.readFile("drive_credentials.json", (err, content) => {
if (err) return console.log("Error loading client secret file:", err);
authorize(JSON.parse(content), function (auth) {
const drive = google.drive({ version: "v3", auth });
const fileMetadata = {
name: "testphoto.png",
};
const media = {
mimeType: "image/png",
body: data, // <--- Modified
};
drive.files.create(
{
resource: fileMetadata,
media: media,
fields: "id",
},
(err, file) => {
if (err) {
console.error(err);
} else {
console.log("File Id: ", file.data.id); // <--- Modified
}
}
);
});
});
}
For testing
For example, when above scripts are tested, how about the following script?
async function run() {
const url = "###";
const data = await fileHelper.download(url);
googleDriveHelper.uploadFile(data);
}
References:
Class: stream.PassThrough
google-api-nodejs-client
I have spent couple of hours trying to get image data as a buffer, search results lead me to using "request" module, others suggestions lead to using other modules in higher version of node, which I cannot use because we depend on node v 6.11 so far.
Here are my trials:
request(imageURL).pipe(fs.createWriteStream('downloaded-img-
1.jpg')).on('close', function () {
console.log('ok');
});
request(imageURL, function (err, message, response) {
fs.writeFile('downloaded-img-2.jpg', response, 'binary', function (err) {
console.log('File saved.');
});
fs.writeFile('downloaded-img-3.jpg', chunks, 'binary', function (err) {
console.log('File saved.');
})
resolve(response);
})
.on('data', function (chunk) {
chunks.push(chunk);
})
.on('response', function (response) {
});
});
The "downloaded-img-1.jpg" gets downloaded correctly, but I have to avoid saving the file on disk, then read it as a stream, it's a PRD environment constraint. So the next option is to use image data, as demonstrated by "downloaded-img-2.jpg" and "downloaded-img-3.jpg", by waiting for the "response" or the hand-made "chunks", the problem is that these 2 images are always corrupted, and I don't know why?
What is the point behind all of that? I am trying to add the image behind the URL in a zip file, and the zip lib I use (js-zip) accepts buffer as an input. Any ideas why I am not getting the "chunks" or the "response" correctly?
I've tested the code below in Node 6.9.2, it will download an image as a buffer. I also write the buffer to a file (just to test all is OK!), the body object is a buffer containing the image data:
"use strict";
var request = require('request');
var fs = require('fs');
var options = {
url: "https://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Hubble2005-01-barred-spiral-galaxy-NGC1300.jpg/1920px-Hubble2005-01-barred-spiral-galaxy-NGC1300.jpg",
method: "get",
encoding: null
};
console.log('Requesting image..');
request(options, function (error, response, body) {
if (error) {
console.error('error:', error);
} else {
console.log('Response: StatusCode:', response && response.statusCode);
console.log('Response: Body: Length: %d. Is buffer: %s', body.length, (body instanceof Buffer));
fs.writeFileSync('test.jpg', body);
}
});
I'm doing an application with react-native. Now I'm trying to send an image from the mobile to the server (Node Js). For this I'm using react-native-image-picker. And the problem is that when I send the image it save a file but it's empty not contain the photo. I think that the problem probably is that the server can't access to the path of the image because is in a different device. But I don't know how I can do it.
React-Native:
openImagePicker(){
const options = {
title: 'Select Avatar',
storageOptions: {
skipBackup: true,
path: 'images'
}
}
ImagePicker.showImagePicker(options, (imagen) =>{
if (imagen.didCancel) {
console.log('User cancelled image picker');
}
else if (imagen.error) {
console.log('ImagePicker Error: ', imagen.error);
}
else if (imagen.customButton) {
console.log('User tapped custom button: ', imagen.customButton);
}
else {
let formdata = new FormData();
formdata.append("file[name]", imagen.fileName);
formdata.append("file[path]", imagen.path);
formdata.append("file[type]", imagen.type);
fetch('http://X/user/photo/58e137dd5d45090d0b000006', {
method: 'PUT',
headers: {
'Content-Type': 'multipart/form-data'
},
body: formdata
})
.then(response => {
console.log("ok");
})
.catch(function(err) {
console.log(err);
})
}})}
Node Js:
addPhotoUser = function (req, res) {
User.findById(req.params.id, function(err, user) {
fs.readFile(req.body.file.path, function (err, data) {
var pwd = 'home/ubuntu/.../';
var newPath = pwd + req.body.file.name;
fs.writeFile(newPath, data, function (err) {
imageUrl: URL + req.body.file.name;
user.save(function(err) {
if(!err) {
console.log('Updated');
} else {
console.log('ERROR: ' + err);
}
res.send(user);
});
});
});
});
};
Yes, the problem is that the filepath is on the local device and not the server. You want to send the actual data returned to you by react-native-image-picker not the uri. It looks like that library encodes the data with base64 so you're going to want send that to your server, not the uri returned from the library because it won't be accessible on a remote server.
What this means is that you won't be reading any files on your server but instead just decoding a base64 string in the response body and writing that to your filesystem.
For the client side:
let formdata = new FormData();
formdata.append("file[name]", imagen.fileName);
formdata.append("file[data]", imagen.data); // this is base64 encoded!
formdata.append("file[type]", imagen.type);
fetch('http://X/user/photo/58e137dd5d45090d0b000006', {
method: 'PUT',
headers: {
'Content-Type': 'multipart/form-data'
},
body: formdata
})
On the server side atob to decode from base64 before writing to the filesystem:
let decoded = atob(req.body.data)
// now this is binary and can written to the filesystem
From there:
fs.writeFile(newPath, decoded, function (err) {
imageUrl: newPath;
user.save(function(err) {
if(!err) {
console.log('Updated');
} else {
console.log('ERROR: ' + err);
}
res.send(user);
});
});
Note, you don't need the filesystem write that's in your code because you're decoding the image that was sent as a b64 string in your request.
There also seems to be some oddities with how you're using that user object. You seem to be only passing a function that handles errors and not any actual data. I don't know what ORM you're using so it's hard to say how it should work. Maybe something like this?
user.save({imageUrl:uriReturnedByFsWrite}, (err, data)=>{...})
Good luck :)
Make an object then send that object to the server. The object will consist of name,path and type, like this:
var imageData = {name: 'image1', path: uri, type: 'image/jpeg'}
Above is a one way to send the image data. The other way is to convert it into BLOB so that server side programmer doesn't have to do this task on their end. You can make BLOB by use of react-native-fetch-blob.
One more way is to directly upload the images to the amazon server(s3) and send the link to the backend..
Function that returns base64 string:
var RNFetchBlob = require('react-native-fetch-blob').default;
getImageAttachment: function(uri_attachment, mimetype_attachment) {
return new Promise((RESOLVE, REJECT) => {
// Fetch attachment
RNFetchBlob.fetch('GET', config.apiRoot+'/app/'+uri_attachment)
.then((response) => {
let base64Str = response.data;
var imageBase64 = 'data:'+mimetype_attachment+';base64,'+base64Str;
// Return base64 image
RESOLVE(imageBase64)
})
}).catch((error) => {
// error handling
console.log("Error: ", error)
});
},
Cheers :)
I'm using the excellent Request library for downloading files in Node for a small command line tool I'm working on. Request works perfectly for pulling in a single file, no problems at all, but it's not working for ZIPs.
For example, I'm trying to download the Twitter Bootstrap archive, which is at the URL:
http://twitter.github.com/bootstrap/assets/bootstrap.zip
The relevant part of the code is:
var fileUrl = "http://twitter.github.com/bootstrap/assets/bootstrap.zip";
var output = "bootstrap.zip";
request(fileUrl, function(err, resp, body) {
if(err) throw err;
fs.writeFile(output, body, function(err) {
console.log("file written!");
}
}
I've tried setting the encoding to "binary" too but no luck. The actual zip is ~74KB, but when downloaded through the above code it's ~134KB and on double clicking in Finder to extract it, I get the error:
Unable to extract "bootstrap" into "nodetest" (Error 21 - Is a directory)
I get the feeling this is an encoding issue but not sure where to go from here.
Yes, the problem is with encoding. When you wait for the whole transfer to finish body is coerced to a string by default. You can tell request to give you a Buffer instead by setting the encoding option to null:
var fileUrl = "http://twitter.github.com/bootstrap/assets/bootstrap.zip";
var output = "bootstrap.zip";
request({url: fileUrl, encoding: null}, function(err, resp, body) {
if(err) throw err;
fs.writeFile(output, body, function(err) {
console.log("file written!");
});
});
Another more elegant solution is to use pipe() to point the response to a file writable stream:
request('http://twitter.github.com/bootstrap/assets/bootstrap.zip')
.pipe(fs.createWriteStream('bootstrap.zip'))
.on('close', function () {
console.log('File written!');
});
A one liner always wins :)
pipe() returns the destination stream (the WriteStream in this case), so you can listen to its close event to get notified when the file was written.
I was searching about a function which request a zip and extract it without create any file inside my server, here is my TypeScript function, it use JSZIP module and Request:
let bufs : any = [];
let buf : Uint8Array;
request
.get(url)
.on('end', () => {
buf = Buffer.concat(bufs);
JSZip.loadAsync(buf).then((zip) => {
// zip.files contains a list of file
// chheck JSZip documentation
// Example of getting a text file : zip.file("bla.txt").async("text").then....
}).catch((error) => {
console.log(error);
});
})
.on('error', (error) => {
console.log(error);
})
.on('data', (d) => {
bufs.push(d);
})