I want to upload a file larger than 150mb.
In the Dropbox API V2 docs, it says you should start an upload session.
The docs say you can't send a POST with more than 150mb of data, but I'm unsure how to achieve that with the upload_session API.
While any individual request shouldn't be larger than 150 MB (and typically you should use a significantly smaller chunk size), you can upload files larger than that by using multiple requests.
There's an example of using upload sessions below. That example uses the Python SDK, but the JavaScript SDK, but it should serve as a useful reference, as the logic is the same. (They both use the same underlying API.)
This uses the Dropbox Python SDK to upload a file to the Dropbox API from the local file as specified by file_path to the remote path as specified by dest_path. It also chooses whether or not to use an upload session based on the size of the file:
f = open(file_path)
file_size = os.path.getsize(file_path)
CHUNK_SIZE = 4 * 1024 * 1024
if file_size <= CHUNK_SIZE:
print dbx.files_upload(f.read(), dest_path)
else:
upload_session_start_result = dbx.files_upload_session_start(f.read(CHUNK_SIZE))
cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
offset=f.tell())
commit = dropbox.files.CommitInfo(path=dest_path)
while f.tell() < file_size:
if ((file_size - f.tell()) <= CHUNK_SIZE):
print dbx.files_upload_session_finish(f.read(CHUNK_SIZE),
cursor,
commit)
else:
dbx.files_upload_session_append(f.read(CHUNK_SIZE),
cursor.session_id,
cursor.offset)
cursor.offset = f.tell()
f.close()
You can quickly upload file chunks using files/upload_session/start, files/upload_session/append_v2 and files/upload_session/finish API endpoints. Here is an example which uses my tiny dropbox v2 api wrapper (dropbox-v2-api):
const CHUNK_LENGTH = 100;
//create read streams, which generates set of 100 (CHUNK_LENGTH) characters of values: 1 and 2
const firstUploadChunkStream = () => utils.createMockedReadStream('1', CHUNK_LENGTH);
const secondUploadChunkStream = () => utils.createMockedReadStream('2', CHUNK_LENGTH);
sessionStart((sessionId) => {
sessionAppend(sessionId, () => {
sessionFinish(sessionId);
});
});
function sessionStart(cb) {
dropbox({
resource: 'files/upload_session/start',
parameters: {
close: false
},
readStream: firstUploadChunkStream()
}, (err, response) => {
if (err) { return console.log('sessionStart error: ', err) }
console.log('sessionStart response:', response);
cb(response.session_id);
});
}
function sessionAppend(sessionId, cb) {
dropbox({
resource: 'files/upload_session/append_v2',
parameters: {
cursor: {
session_id: sessionId,
offset: CHUNK_LENGTH
},
close: false,
},
readStream: secondUploadChunkStream()
}, (err, response) => {
if(err){ return console.log('sessionAppend error: ', err) }
console.log('sessionAppend response:', response);
cb();
});
}
function sessionFinish(sessionId) {
dropbox({
resource: 'files/upload_session/finish',
parameters: {
cursor: {
session_id: sessionId,
offset: CHUNK_LENGTH * 2
},
commit: {
path: "/result.txt",
mode: "add",
autorename: true,
mute: false
}
}
}, (err, response) => {
if (err) { return console.log('sessionFinish error: ', err) }
console.log('sessionFinish response:', response);
});
}
I have an example!
testFile1Data = "test file data 1";
dbx.filesUploadSessionStart({
contents: testFile1Data,
close: true,
})
.then(function (response) {
file1Start = response;
})
.catch(function (err) {
console.log(err);
});
testFile2Data = "test file data 2";
dbx.filesUploadSessionStart({
contents: testFile2Data,
close: true,
})
.then(function (response) {
file2Start = response;
})
.catch(function (err) {
console.log(err);
});
dbx.filesUploadSessionFinishBatch({entries: [
{cursor: {session_id: file1Start.session_id, offset: testFile1Data.length}, commit: {path: "/testFile1.txt"}},
{cursor: {session_id: file2Start.session_id, offset: testFile2Data.length}, commit: {path: "/testFile2.txt"}},
]})
.then(function (response) {
finishBatch = response;
})
.catch(function (err) {
console.log(err);
});
dbx.filesUploadSessionFinishBatchCheck({async_job_id: finishBatch.async_job_id})
.then(function (response) {
finishBatch = response
})
.catch(function (err) {
console.log(err);
});
I got the example from an issue thread on github - https://github.com/dropbox/dropbox-sdk-js/issues/80#issuecomment-283189888
Related
Axios is not returning data from API because of bigger file.
This is my code in api file:
/* Get the archive config information for the device from API */
const getConfigArchiveFileRawData = (deviceId, versionId, fileId) => {
try {
return axios({
method: apiConstant.getMethod,
url:
apiConstant.urlArchiveConfigDeviceFileID +
"/" +
deviceId +
"/version/" +
versionId +
"/file/" +
fileId +
"?processed=true"
}).then(response => {
if (response.data) return response.data;
});
} catch (err) {
toast({
message: err,
flavor: "error",
options: { isHtml: true }
});
}
};
This is component where I am calling:
/**
* Get startup and running config file data and update state
*/
getConfigArchiveFileIDsData = (deviceId, archiveData) => {
try {
getConfigArchiveFileRawData(
deviceId,
archiveData.currentVersionID,
archiveData.runningFileID
).then(data => {
const getRunningConfigArchiveFileRawData = data;
if (getRunningConfigArchiveFileRawData) {
getConfigArchiveFileRawData(
deviceId,
archiveData.previousVersionID,
archiveData.startUpFileID
).then(data => {
const getStartupConfigArchiveFileRawData = data;
if (getStartupConfigArchiveFileRawData) {
this.setState({
startupConfigData: getStartupConfigArchiveFileRawData,
runningConfigData: getRunningConfigArchiveFileRawData,
showLoader: false
});
}
});
}
});
} catch (err) {
toast({
message: err,
flavor: "error",
options: { isHtml: true }
});
}
};
This issue is only coming when the file size is more, for example 4.5 MB file with 100 000 lines,
We get the processed raw file from api and then we pas these files to some component to evaluate.
Update
Getting this error also in console:
unstable_flushDiscreteUpdates: Cannot flush updates when React is already rendering.
I am trying to upload a profile image of a user from React Native App(Android). I am new to uploading image so I searched and found a package called multer. As I am using Mongodb Atlas, multer has a storage option of MemoryStorage and using MemoryStorage I tried uploading the image to Atlas.But the problem I am facing is that I dont the format in which I have to send the data to Nodejs. I use FormData() to send the image details to Nodejs.I may be wrong in choosing the format or package or anywhere.Please correct me if find me wrong anywhere.Below is code for everything.
avatar.js
const avatarSchema = new Schema(
{
profile_avatar: {
type: Buffer,
contentType: String,
// required: [true, 'fee is required'],
},
},
{
timestamps: true,
},
);
const avatar = mongoose.model('avatar', avatarSchema);
module.exports = avatar;
avatar route
let Avatar = require('../models/avatar');
const multer = require('multer');
var storage = multer.memoryStorage();
var upload = multer({storage: storage});
router.route('/add/image').post(upload.single('avatar'), (req, res, next) => {
console.log(req); // it does not get hit
var newAvatar = new Avatar();
newAvatar.profile_avatar = req.file.buffer;
newAvatar.save(function(err) {
console.log('error', err);
if (err) {
console.log(error);
} else {
res.send({message: 'uploaded successfully'});
}
});
});
React Native Code
//Function to select single image from gallery(its working fine)
imagePick = () => {
let options = {
quality: 0.2,
maxWidth: 500,
maxHeight: 500,
allowsEditing: true,
noData: true,
storageOptions: {
skipBackup: false,
path: 'images',
},
};
ImagePicker.launchImageLibrary(options, response => {
console.log('Response = ', response);
if (response.didCancel) {
console.log('User cancelled image picker');
} else if (response.error) {
console.log('ImagePicker Error: ', response.error);
} else if (response.customButton) {
console.log('User tapped custom button: ', response.customButton);
alert(response.customButton);
} else {
this.setState({
imageData: response,
});
}
});
};
//Function to structure FormData
createFormData = photo => {
const data = new FormData();
data.append('avatar', {
name: photo.fileName,
type: photo.type,
uri:
Platform.OS === 'android'
? photo.uri
: photo.uri.replace('file://', ''),
};);
return data;
};
//Function to send Image data to Nodejs(Not Working)
sendImage = () => {
fetch('http://10.0.2.2:5000/user/add/image', {
method: 'POST',
credentials: 'same-origin',
redirect: 'follow',
referrer: 'no-referrer',
body: this.createFormData(this.state.imageData),
})
.then(response => response.json())
.then(response => {
console.log('upload succes', response);
Alert.alert('Upload success!');
})
.catch(error => {
console.log('upload error', error);
Alert.alert('Upload failed!');
});
};
Any other way I can upload the image? Any help would be appreciated.
The goal is to have users enter in some information into a form and spit that out into a PDF. I'm using JSPDF to parse and create the PDF. I've successfully gotten my code to make a printable PDF, but in an effort to not have paper floating around the office, I made a cloud function to instead email that PDF to the customer.
Here is my code on the front end. maildoc is the pdf that I've made, it hasn't been printed or anything. So it only exists in memory.
mailDoc = mailDoc.output('datauri');
mailFunction += "&?data=" + mailDoc;
//axios request to the cloud function
axios.get(mailFunction).then( function (response) {
console.log(response);
}).catch(function (error) {
console.log(error)
})
And here is my code on the cloud function
exports.sendMail = functions.https.onRequest((req, res) => {
cors(req, res, () => {
// getting dest email by query string
//?dest= DestinationEmail
const dest = req.query.dest;
const data = req.query.data;
const mailOptions = {
from: 'whatever <whatever#hoobashaka.com>',
to: dest,
subject: "You're Equipment Return to HBCI", // email subject
attachments :[
{
filename: 'return.pdf',
contentType: 'application/pdf',
path: data,
}
],
};
return transporter.sendMail(mailOptions, (erro, info) => {
if(erro){
return res.send(erro.toString());
}
return res.send('Sended');
});
});
If I try and send the data via URI, I get a 413 error, probably because that URI is enormous. But I can't think of another way of sending that generated PDF to the function.
On your client, instead of uploading the file as a datauri, I'd instead use POST and send the PDF inside the request body (just as if you had submitted a file using a form).
mailDocBlob = mailDoc.output('blob');
const data = new FormData();
data.set('dest', someEmail);
data.append('file', mailDocBlob, 'return.pdf');
axios({
method: 'post',
url: 'https://your-cloud-function.here/sendMail',
data: data,
headers: {
'Content-Type': `multipart/form-data; boundary=${data._boundary}`,
}
})
.then(function (response) {
console.log(response);
})
.catch(function (error) {
console.log(error);
});
On the server you would handle the multipart form data using the busboy package.
const BusBoy = require('busboy');
exports.sendMail = functions.https.onRequest((req, res) => {
cors(req, res, (err) => {
if (err) {
// CORS failed. Abort.
console.log("CORS failed/rejected");
res.sendStatus(403); // 403 FORBIDDEN
return;
}
if (req.method !== 'POST') {
res.set('Allow', 'POST, OPTIONS').sendStatus(405); // 405 METHOD_NOT_ALLOWED
return;
}
let busboy = new BusBoy({headers: req.headers, limits: {files: 1}}); // limited to only a single file
const mailOptions = {
from: 'whatever <whatever#hoobashaka.com>',
to: dest,
subject: "Your Equipment Return to HBCI", // email subject - fixed typo
attachments: []
};
busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
// add new file attachment
mailOptions.attachments.push({
filename: 'return.pdf',
contentType: 'application/pdf',
content: file, // file is a stream
});
})
.on('finish', () => {
if (mailOptions.attachments.length == 0) {
// not enough attachments
res.status(400).send('Error: not enough attachments');
return;
}
return transporter.sendMail(mailOptions, (erro, info) => {
if (erro) {
return res.status(500).send('Error: ' + erro.toString());
}
return res.send('Sent');
})
})
.on('error', (err) => {
console.error(err);
res.status(500).send('Error: ' + err.code);
});
req.pipe(busboy);
});
I'm handling with an webapplication that is not mine and now I've got to send a hundred e-mails.
Unfortunately, the code is not documented and not so well written, that means i have to go testing it to discover what I am able to do and what I'm not. but I don't know how to access this function that is on the code via node. Is it actually possible to do it? Here's the code:
router.post('/aprovadosemail', miPermiso("3"), (req, res) => {
var templatesDir = path.resolve(__dirname, '..', 'templates');
var emailTemplates = require('email-templates');
// Prepare nodemailer transport object
emailTemplates(templatesDir, function(err, template) {
if (err) {
console.log(err);
} else {
var users = [];
projetoSchema.find({"aprovado":true, "categoria":"Fundamental II (6º ao 9º anos)"}, function (err, docs) {
if (err) throw err;
//console.log(docs);
docs.forEach(function(usr) {
let url = "http://www.movaci.com.b/projetos/confirma/"+usr._id+"/2456";
let url2 = "http://www.movaci.com.br/projetos/confirma/"+usr._id+"/9877";
users.push({'email': usr.email, 'projeto': usr.nomeProjeto, 'url': url, 'url2': url2});
});
for (var i = 0; i < users.length; i++) {
console.log(users[i]);
}
const transporter = nodemailer.createTransport(smtpTransport({
host: 'smtp.zoho.com',
port: 587,
auth: {
user: "generic#mail.com",
pass: "genericpassword"
},
getSocket: true
}));
var Render = function(locals) {
this.locals = locals;
this.send = function(err, html, text) {
if (err) {
console.log(err);
} else {
transporter.sendMail({
from: 'no-reply4#movaci.com.br',
to: locals.email,
subject: 'MOVACI - Projeto aprovado!',
html: html,
text: text
}, function(err, responseStatus) {
if (err) {
console.log(err);
} else {
console.log(responseStatus.message);
}
});
}
};
this.batch = function(batch) {
batch(this.locals, templatesDir, this.send);
};
};
// Load the template and send the emails
template('rateada', true, function(err, batch) {
for(var user in users) {
var render = new Render(users[user]);
render.batch(batch);
};
});
res.send('ok');
});
};
});
});
Seems like previous dev did not knew email-templates package deeply (at least have not read how it works).
So in fact it has send method, You can create an email object from email-templates and pass necessary defaults, then You call .send method of it by passing dynamical parts - it just simply merge additional params passed in send arguments, sends mail using nodemailer inside of promise which it returns.
If it's interesting for You - read source code of it: https://github.com/niftylettuce/email-templates/blob/master/index.js
I tried to simplify it modular parts using promises.
I've not debugged it, but You may check my solution and fix it as You wish.
Have 2 files (to routing from handler separately, it may have variables that may conflict and etc):
1) methods/users/aprovadosEmail.js:
const
Email = require('email-templates'),
const
emailTemplatesDir = path.resolve(__dirname + '/../../templates'),
smtpTransportConfig = {
host: 'smtp.zoho.com',
port: 587,
secure: false,
auth: {
user: "no-reply4#movaci.com.br",
pass: "some-password-here"
}
},
createEmail = (template, subject) => {
return new Email({
views: {
root: emailTemplatesDir,
},
transport: smtpTransportConfig,
template,
message: {
from: 'no-reply4#movaci.com.br',
subject
}
});
},
getApprovedUsers = () => {
return new Promise((resolve, reject) => {
const criteria = {
aprovado: true,
categoria:"Fundamental II (6º ao 9º anos)"
};
projetoSchema.find(
criteria,
(error, docs) => {
if(error) return reject(error);
const users = docs.map(doc => {
return {
email: doc.email,
projeto: doc.nomeProjeto,
url: "http://www.movaci.com.b/projetos/confirma/"+doc._id+"/2456",
url2: "http://www.movaci.com.br/projetos/confirma/"+doc._id+"/9877"
};
});
resolve(users);
});
});
},
sendMailToUser = (mail, user) => {
return mail.send({
message: {
to: user.email
},
locals: user
});
},
broadcastMailToUsers = (mail, users) => {
return Promise
.all(
users
.map(user => sendMailToUser(mail, user))
);
};
module.exports = (req, res) => {
const mail = createEmail('rateada', 'MOVACI - Projeto aprovado!'); // mail object
getApprovedUsers()
.then(users => broadcastMailToUsers(mail, users))
.then(result => {
console.log('Result of broadcast:', result);
res.send('ok');
})
.catch(error => {
res.status(500).send(error);
});
};
2) current routes file where routing part that uses module file:
router.post(
'/aprovadosemail',
miPermiso("3"),
require(__dirname+'/../methods/users/aprovadosEmail')
);
I have some codes from node.js project, it contains a post /uploadAvatar router, and a uploadAvatar function:
uploadAvatar: function(req, res) {
req.file('avatar').upload({
// don't allow the total upload size to exceed ~10MB
maxBytes: 10000000
},function whenDone(err, uploadedFiles) {
if (err) {
console.log(err);
return res.negotiate(err);
}
// If no files were uploaded, respond with an error.
if (uploadedFiles.length === 0){
return res.badRequest('No file was uploaded');
}
var avatarFd = uploadedFiles[0].fd;
var json = {fd: avatarFd};
console.log(json);
res.json(200, json);
});
}
Right now I want to using this api to upload image using AFNetworking. Current the swift code like below:
func uploadAvatar(avatar: NSData, success: JSONBlock, fail: ErrorBlock) -> Void {
sessionManager.POST("/uploadAvatar", parameters: nil, constructingBodyWithBlock: { (formData) in
formData.appendPartWithFormData(avatar, name: "avatar")
}, progress: { (_) in
}, success: { (_, responseObject) in
let json = self.jsonFromResponseObject(responseObject)
if json != nil {
success(json)
}
}) { (_, error) in
if let msgData = error.userInfo[AFNetworkingOperationFailingURLResponseDataErrorKey] as? NSData {
if let message = String(data: msgData, encoding: NSUTF8StringEncoding) {
print(message)
}
}
fail(error.localizedDescription);
}
}
The error response I get is: "No file was uploaded". It seems the node.js can't find my image data. What did I do wrong?