I have a problem with performance on a relatively simple parse.com routine:
We have two classes and we want to make a cross product of them. One class contains a single object of boilerplate for the new objects (description etc.) the other class contains "large" sets (only 1000s of objects) of variable data (name, geopoint etc). Each new object also has some of its own columns not just data from the parents.
To do this, we do a query on the second class and perform an each operation. In each callback, we create and populate our new object and give it pointers to its parents. We call save() on each object (with some then() clauses for retries and error handling) and push the returned promise into an array. Finally we return status inside a when() promise on that array of save promises.
We originally created all the objects and then performed a saveall on them but couldn't get good enough error handling out of it - so we moved to when() with chains of promises with retries.
Trouble is, it's slow. It doesn't feel like the type of thing a nosql database should be slow at so we're blaming our design.
What's the best practice for cloning a bunch of objects from one class to another? or is it possible to get better results from saveAll failures?
My current code looks like this:
var doMakeALoadOfObjects = function(aJob,aCaller) {
Parse.Cloud.useMasterKey();
return aJob.save().then(function(aJob) {
theNumTasksPerLoc = aJob.get("numberOfTasksPerLocation");
if (theNumTasksPerLoc < 1) {
theNumTasksPerLoc = 1;
}
var publicJob = aJob.get("publicJob");
return publicJob.fetch();
}).then(function(publicJob) {
var locationList = aJob.get("locationList");
return locationList.fetch();
}).then(function(locationList) {
publicReadACL = new Parse.ACL();
publicReadACL.setPublicReadAccess(true);
publicReadACL.setRoleReadAccess("Admin",true);
publicReadACL.setRoleWriteAccess("Admin",true);
// Can't create a promise chain inside the loop so use this function.
var taskSaver = function(task) {
return task.save.then(function success(){
numTasksMade++;
},
function errorHandler(theError) {
numTimeOuts++;
numTaskCreateFails++;
logger.log("FAIL: failed to make a task for job " + aJob.get("referenceString") + " Error: " + JSON.stringify(theError));
});
};
var taskSaverWithRetry = function(task) {
return task.save().then(function() {
numTasksMade++;
return Parse.Promise.as();
}, function(error) {
logger.log("makeJobLive: FAIL saving task. Will try again. " + JSON.stringify(error));
numTimeOuts++;
return task.save().then(function() {
numTasksMade++;
return Parse.Promise.as();
}, function(error) {
numTimeOuts++;
numTaskCreateFails++;
logger.log("makeJobLive: FAIL saving task. Give up. " + JSON.stringify(error));
return Parse.Promise.as();
});
})
}
for (var j = 0; j < theNumTasksPerLoc; j++) {
var Task = Parse.Object.extend("Task");
var task = new Task();
task.set("column",stuff);
// Can't create a promise chain in the loop so use the function above.
taskSaverArray.push(taskSaverWithRetry(task));
}
return Parse.Promise.when(taskSaverArray);
}).then(function() {
}).then(function() {
// happy happy
},function(error){
// we never land here.
});
}
..."looks" like because I've deleted a lot of the object creation code and some housekeeping we do at the same time. I may have deleted some variable definitions too so I doubt this would run as is.
Related
currently I am struggeling a little bit with node.js (I am new to it) doing different API requests (Usabilla API), work on the results and then combine them in order to work on the whole set (e.g. export).
Requesting the API is not the problem but I can't get the results out to do some other stuff on it (asynchronous code drives me crazy).
Attached please find a overview how I thought to do this. Maybe I am totally wrong about this or maybe you have other more elegant suggestions.
My code works until I have to request the two different API "adresses" (they are provided) and then extract the results to do some other stuff.
My problem here is that there are nested functions with a promise and I cant figure out how to pass this through the parent function inside waterfall to get handled by the next function.
In the code, of course there is nothing parallel as shown in the diagram.
Thats another point, how to do that ? Simply nest parallel and series/ another waterfall inside waterfall ?
I am a little bit confused because that gets more and more complex for a simple problem when this would be done with synchronous code.
Here I build up all my request querys (at the moment 4):
function buildQuery(IDs,callback){
var i = 0;
var max = Object.keys(IDs).length;
async.whilst(
function(){return i < max},
function(callback){
FeedbackQuery[i] =
{
identifier: IDs[i].identifier,
query:
{id: IDs[i].id,
params: {since:sinceDate,}
}
};
i++;
callback(null,i);
})
console.log(FeedbackQuery);
callback (null,FeedbackQuery);
};
I then have to decide which type of query it is and add it to an object which should contain all the items of this identifier type:
function FeedbackRequest(FeedbackQuery,callback)
{
var i = 0;
var max = Object.keys(FeedbackQuery).length;
async.whilst(
function(){return i < max},
function (callback){
identifier = FeedbackQuery[i].identifier;
APIquery = FeedbackQuery[i].query;
switch(identifier)
{
case 'mobilePortal':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.websites.buttons.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsA[item] = feedback;
callback(null, feedbackResultsA);
})
break;
case 'apps':
console.log(FeedbackQuery[i].identifier, 'aktiviert!');
var result = api.apps.forms.feedback.get(APIquery);
result.then(function(feedback)
{
var item = Object.keys(feedbackResults).length;
feedbackResultsB[item] = feedback;
callback(null, feedbackResultsB);
})
break;
}
i++;
callback(null,i);
})
};
Currently the functions are bundled in an async waterfall:
async.waterfall([
async.apply(buildQuery,IDs2request),
FeedbackRequest,
// a function to do something on the whole feedbackResults array
],function (err, result) {
// result now equals 'done'
if (err) { console.log('Something is wrong!'); }
return console.log('Done!');
})
How it actually should be:
Structure
Thank you very much for any tips or hints!
I'm not proficient with async, and I believe if you'r new to this, it's harder than a simple Promise library like bluebird combine with lodash for helpers.
What I would do based on your schemas :
var firstStepRequests = [];
firstStepRequests.push(buildQuery());// construct your first steps queries, can be a loop, goal is to have firstStepRequests to be an array of promise.
Promise.all(firstStepRequests)
.then((allResults) => {
var type1 = _.filter(allResults, 'request_type_1');
var type2 = _.filter(allResults, 'request_type_2');
return {
type1: type1,
type2: type2
};
})
.then((result) => {
result.type1 = //do some work
result.type2 = //do some work
return result;
})
.then((result) => {
//export or merge or whatever.
});
Goal is to have a simple state machine.
UPDATE
If you want to keep identifier for a request, you can use props to have :
var props = {
id_1:Promise,
id_2:Promise,
id_3:Promise
};
Promise.props(props).then((results) => {
// results is {
id_1:result of promise,
id_2:result of promise,
etc...
}
})
You could do something like :
var type1Promises = getType1Requests(); //array of type 1
var type2Promises = getType2Requests(); // array of type 2
var props = {
type_1: Promise.all(type1Promises),
type_2: Promise.all(type2Promises)
}
Promise.props(props).then((result) => {
//result is : {
type_1: array of result of promises of type 1
type_2: array of result of promises of type 2
}
})
I'm just learning Javascript and still pretty new to Parse's cloud code. I've been reading some articles and questions on promises and closures and still don't quite understand how to accomplish what I want do do yet. All the other questions/answer seem to be slightly different or difficult to understand.
I have a function that starts with a query that gets all the "Gyms". For each of those gyms, I need to run several other queries. All those inner queries (inside the loop) need to all complete before I can generate a final report for the gym. I want to understand the following things:
a.) How to allow the correct gym object from that each iteration of the loop to be accessible through the entire chain of queries in that iteration.
b.) Will all the results from the previously executed queries in my chain be available in the following queries? e.g. Can I access newWorkouts in the last function?
function createReports() {
var gymQuery = new Parse.Query(Parse.Object.extend("Gym"));
gymQuery.find({
success: function(results) {
for (var i = 0; i < results.length; ++i) {
/* jshint loopfunc: true */
var gym = results[i];
var newWorkoutsQuery = new Parse.Query(Parse.Object.extend("Workout"));
newWorkoutsQuery.equals("gym", gym);
newWorkoutsQuery.find().then(function(newWorkouts) {
var newLogsQuery = new Parse.Query(Parse.Object.extend("WorkoutLog"));
newLogsQuery.equals("gym", gym);
return newLogsQuery.find();
}).then(function(logsLastWeek) {
//Generate final report for gym using data from all above queries.
//Need access to gym, newWorkouts, and logsLastWeek
});
}
},
error:function() {
console.log("error");
}
});
}
Promise.all() should be able to help you out with this.
First, let's break out a function that retrieves the data for a single gym:
function getGymData(gym) {
var newWorkoutsQuery = new Parse.Query(Parse.Object.extend("Workout"));
newWorkoutsQuery.equals("gym", gym);
var newLogsQuery = new Parse.Query(Parse.Object.extend("WorkoutLog"));
newLogsQuery.equals("gym", gym);
return Promise.all([newWorkoutsQuery.find(), newLogsQuery.find()])
.then(function (results) {
return {
gym: gym,
workouts: results[0],
logs: results[1]
};
});
}
Then use Promise.all() across all the gyms:
function createReports() {
var gymQuery = new Parse.Query(Parse.Object.extend("Gym"));
return gymQuery.find()
.then(function (gyms) {
return Promise.all(gyms.map(getGymData));
})
.then(function (results) {
// results should be an array of objects, each with
// the properties gym, workouts, and logs
})
.catch(function (error) {
console.error(error);
});
}
So I've recently delved into trying to understand promises and the purpose behind them due to javascripts asynchronous behavior. While I "think" I understand, I still struggle with how to promisify something to return the future value, then execute a new block of code to do something else. Two main node modules I'm using:
pg-promise
exceljs
What I'd like to do is read a file, then once fully read, iterate of each worksheet executing DB commands. Then once all worksheets are processed, go back and delete the original file I read. Here is the code I have. I have it working to the point everything writes into the database just fine, even when there are multiple worksheets. What I don't have working is setting it up to identify when all the worksheets have been fully processed, then to go remove the file
workbook.csv.readFile(fileName)
.then(function () {
// this array I was going to use to somehow populate a true/false array.
// Then when done with each sheet, push a true into the array.
// When all elements were true could signify all the processing is done...
// but have no idea how to utilize this!
// So left it in to take up space because wtf...
var arrWorksheetComplete = [];
workbook.eachSheet(function (worksheet) {
console.log(worksheet.name);
db.tx(function (t) {
var insertStatements = [];
for (var i = 2; i <= worksheet._rows.length; i++) {
// here we create a new array from the worksheet, as we need a 0 index based array.
// the worksheet values actually begins at element 1. We will splice to dump the undefined element at index 0.
// This will allow the batch promises to work correctly... otherwise everything will be offset by 1
var arrValues = Array.from(worksheet.getRow(i).values);
arrValues.splice(0, 1);
// these queries are upsert. Inserts will occur first, however if they error on the constraint, an update will occur instead.
insertStatements.push(t.one('insert into rq_data' +
'(col1, col2, col3) ' +
'values($1, $2, $3) ' +
'ON CONFLICT ON CONSTRAINT key_constraint DO UPDATE SET ' +
'(prodname) = ' +
'($3) RETURNING autokey',
arrValues));
}
return t.batch(insertStatements);
})
.then(function (data) {
console.log('Success:', 'Inserted/Updated ' + data.length + ' records');
})
.catch(function (error) {
console.log('ERROR:', error.message || error);
});
});
});
I would like to be able to say
.then(function(){
// everything processed!
removeFile(fileName)
// this probably also wouldn't work as by now fileName is out of context?
});
But I'm super confused when having a promise inside a promise.. I have the db.tx call which is essentially a promise nested inside the .eachSheet function.
Please help a dumb programmer understand! Been beating head against wall for hours on this one. :)
If i understand correctly, you're trying to chain promises.
I suggest you to read this great article on Promises anti-pattern (see 'The Collection Kerfuffle' section)
If you need to execute promises in series, this article suggests to use reduce.
I'll rewrite your snippet to:
workbook.csv.readFile(fileName).then(function () {
processWorksheets().then(function() {
// all worksheets processed!
});
});
function processWorksheets() {
var worksheets = [];
// first, build an array of worksheet
workbook.eachSheet(function (worksheet) {
worksheets.push(worksheet);
});
// then chain promises using Array.reduce
return worksheets.reduce(function(promise, item) {
// promise is the the value previously returned in the last invocation of the callback.
// item is a worksheet
// when the previous promise will be resolved, call saveWorksheet on the next worksheet
return promise.then(function(result) {
return saveWorksheet(item, result);
});
}, Promise.resolve()); // start chain with a 'fake' promise
}
// this method returns a promise
function saveWorksheet(worksheet, result) {
return db.tx(function (t) {
var insertStatements = [];
for (var i = 2; i <= worksheet._rows.length; i++) {
// here we create a new array from the worksheet, as we need a 0 index based array.
// the worksheet values actually begins at element 1. We will splice to dump the undefined element at index 0.
// This will allow the batch promises to work correctly... otherwise everything will be offset by 1
var arrValues = Array.from(worksheet.getRow(i).values);
arrValues.splice(0, 1);
// these queries are upsert. Inserts will occur first, however if they error on the constraint, an update will occur instead.
insertStatements.push(t.one('insert into rq_data' +
'(col1, col2, col3) ' +
'values($1, $2, $3) ' +
'ON CONFLICT ON CONSTRAINT key_constraint DO UPDATE SET ' +
'(prodname) = ' +
'($3) RETURNING autokey',
arrValues));
}
return t.batch(insertStatements);
})
// this two below can be removed...
.then(function (data) {
return new Promise((resolve, reject) => {
console.log('Success:', 'Inserted/Updated ' + data.length + ' records');
resolve();
});
})
.catch(function (error) {
return new Promise((resolve, reject) => {
console.log('ERROR:', error.message || error);
reject();
});
});
}
Don't forget to include the promise module:
var Promise = require('promise');
I haven't tested my code, could contains some typo errors.
Here is my code, it loops through forEach and prints out '1' but never returns from object.save() & never prints out 2, 3 or anything else. I have tried a bunch of other ways but none seems to work.
Note: response.succes(or error) is not being called anywhere, the code is definitely waiting for object.save() to be completed.
var promise = new Parse.Promise();
var query = new Parse.Query("SomeClass");
query.find().then(function(results) {
var promises = [];
results.forEach(function(object) {
object.set("SomeColumnName", true);
console.log('1');
promises.push(object.save(null, {
success: function(result) {
alert('2');
return ;
},
error: function(result, error) {
alert('3');
return ;
}
}));
});
Parse.Promise.when(promises).then(function() {
console.log('inside resolve');
promise.resolve();
}, function() {
console.log('inside reject');
promise.reject();
});
});
return promise;
You're on the right track, but you should take advantage of the fact that most of the sdk functions create and return promises for you. With those, you can substantially simplify the code:
// very handy utility library that provides _.each among many other things
// www.underscorejs.org
var _ = require('underscore');
// answer a promise to modify all instances of SomeClass
function changeSomeClass() {
var query = new Parse.Query("SomeClass");
// if there are more than 100 rows, set query.limit up to 1k
return query.find().then(function(results) { // find returns a promise
_.each(results, function(result) {
result.set("SomeColumnName", true);
});
return Parse.Object.saveAll(results); // and saveAll returns a promise
});
}
Wrap it in a cloud function and call success/error like this:
Parse.Cloud.define("changeSomeClass", function(request, response) {
changeSomeClass().then(function(result) {
response.success(result);
}, function(error) {
response.error(error);
});
});
You can only have one Parse request happening at a time for each object. If multiple requests are sent, all but the first are ignored. You're probably trying to do this while other threads are making Parse requests for those objects. I know that if you save an object, it also saves it's child objects, so you could be hitting a problem with that. Make sure you do as much as you can in background threads with completion blocks, or use saveEventually / fetchEventually where possible.
I'm trying to to pass a variable number of functions into Q.all()
It works fine if I code the array manually - however I want to build it up in a loop as the system wont know how many times to call the function until runtime - and needs to pass a different ID into it for each AJAX call.
I've tried various methods with no success (e.g. array[i] = function() {func}) - I guess eval() could be a last resort.
Any help would be massively helpful.
// Obviously this array loop wont work as it just executes the functions in the loop
// but the idea is to build up an array of functions to pass into Q
var arrayOfFunctions = [];
for(var i in NumberOfPets) {
arrayOfFunctions[i] = UpdatePets(i);
}
// Execute sequence of Ajax calls
Q.try(CreatePolicy)
.then(updateCustomer)
.then(function() {
// This doesn't work - Q just ignores it
return Q.all(arrayOfFunctions)
// This code below works fine (waits for all pets to be updated) - I am passing in the ID of the pet to be updated
// - But how can I create and pass in a dynamic array of functions to achieve this?
// return Q.all([UpdatePets(1), UpdatePets(2), UpdatePets(3), UpdatePets(4), UpdatePets(5), UpdatePets(5)]);
})
.then(function() {
// do something
})
.catch(function (error) {
// error handling
})
.done();
Thanks in advance.
Q.all doesn't expect an array of functions, but an array of promises. Use
Q.try(CreatePolicy)
.then(updateCustomer)
.then(function() {
var arrayOfPromises = [];
var numberOfPets = pets.length;
for (var i=0; i<numberOfPets; i++)
arrayOfPromises[i] = updatePet(pets[i], i); // or something
return Q.all(arrayOfPromises)
})
.then(function() {
// do something
})
.catch(function (error) {
// error handling
});