I'm writing a job that I want to run every hour in the background on Parse. My database has two tables. The first contains a list of Questions, while the second lists all of the user\question agreement pairs (QuestionAgreements). Originally my plan was just to have the client count the QuestionAgreements itself, but I'm finding that this results in a lot of requests that really could be done away with, so I want this background job to run the count, and then update a field directly on Question with it.
Here's my attempt:
Parse.Cloud.job("updateQuestionAgreementCounts", function(request, status) {
Parse.Cloud.useMasterKey();
var query = new Parse.Query("Question");
query.each(function(question) {
var agreementQuery = new Parse.Query("QuestionAgreement");
agreementQuery.equalTo("question", question);
agreementQuery.count({
success: function(count) {
question.set("agreementCount", count);
question.save(null, null);
}
});
}).then(function() {
status.success("Finished updating Question Agreement Counts.");
}, function(error) {
status.error("Failed to update Question Agreement Counts.")
});
});
The problem is, this only seems to be running on a few of the Questions, and then it stops, appearing in the Job Status section of the Parse Dashboard as "succeeded". I suspect the problem is that it's returning prematurely. Here are my questions:
1 - How can I keep this from returning prematurely? (Assuming this is, in fact, my problem.)
2 - What is the best way of debugging cloud code? Since this isn't client side, I don't have any way to set breakpoints or anything, do I?
status.success is called before the asynchronous success calls of count are finished. To prevent this, you can use promises here. Check the docs for Parse.Query.each.
Iterates over each result of a query, calling a callback for each one. If the callback returns a promise, the iteration will not continue until that promise has been fulfilled.
So, you can chain the count promise:
agreementQuery.count().then(function () {
question.set("agreementCount", count);
question.save(null, null);
});
You can also use parallel promises to make it more efficient.
There are no breakpoints in cloud code, that makes Parse really hard to use. Only way is logging your variables with console.log
I was able to utilize promises, as suggested by knshn, to make it so that my code would complete before running success.
Parse.Cloud.job("updateQuestionAgreementCounts", function(request, status) {
Parse.Cloud.useMasterKey();
var promises = []; // Set up a list that will hold the promises being waited on.
var query = new Parse.Query("Question");
query.each(function(question) {
var agreementQuery = new Parse.Query("QuestionAgreement");
agreementQuery.equalTo("question", question);
agreementQuery.equalTo("agreement", 1);
// Make sure that the count finishes running first!
promises.push(agreementQuery.count().then(function(count) {
question.set("agreementCount", count);
// Make sure that the object is actually saved first!
promises.push(question.save(null, null));
}));
}).then(function() {
// Before exiting, make sure all the promises have been fulfilled!
Parse.Promise.when(promises).then(function() {
status.success("Finished updating Question Agreement Counts.");
});
});
});
Related
I'm developing an online store app, and using Parse as the back-end. The count of each item in my store is limited. Here is a high-level description of what my processOrder function does:
find the items users want to buy from database
check whether the remaining count of each item is enough
if step 2 succeeds, update remaining count
check if remaining count becomes negative, if it is, revert remaining count to the old value
Ideally, the above steps should be executed exclusively. I learned that Javascript is a single-threaded and event-based, so here are my questions:
no way in Javascript to put the above steps in a critical section, right?
assume only 3 items are left, and two users try to order 2 of them respectively. The remaining count will end up as -1 for one of the users, so remaining count needs to be reverted to 1 in this case. Imagine another user tries to order 1 item when the remaining count is -1, he will fail although he should be allowed to order. How do I solve this problem?
Following is my code:
Parse.Cloud.define("processOrder", function(request, response) {
Parse.Cloud.useMasterKey();
var orderDetails = {'apple':2, 'pear':3};
var query = new Parse.Query("Product");
query.containedIn("name", ['apple', 'pear']);
query.find().then(function(results) {
// check if any dish is out of stock or not
_.each(results, function(item) {
var remaining = item.get("remaining");
var required = orderDetails[item.get("name")];
if (remaining < required)
return Parse.Promise.error(name + " is out of stock");
});
return results;
}).then(function(results) {
// make sure the remaining count does not become negative
var promises = [];
_.each(results, function(item) {
item.increment("remaining", -orderDetails[item.get("name")]);
var single_promise = item.save().then(function(savedItem) {
if (savedItem.get("remaining") < 0) {
savedItem.increment("remaining", orderDetails[savedItem.get("name")]);
return savedItem.save().then(function(revertedItem) {
return Parse.Promise.error(savedItem.get("name") + " is out of stock");
}, function(error){
return Parse.Promise.error("Failed to revert order");
});
}
}, function(error) {
return Parse.Promise.error("Failed to update database");
});
promises.push(single_promise);
});
return Parse.Promise.when(promises);
}).then(function() {
// order placed successfully
response.success();
}, function(error) {
response.error(error);
});
});
no way in Javascript to put the above steps in a critical section, right?
See, here is the amazing part. In JavaScript everything runs in a critical section. There is no preemption and multiprocessing is cooperative. If your code started running there is simply no way any other code can run before yours completes.
That is, unless your code is done executing.
The problem is, you're doing IO, and IO in JavaScript yields back to the event loop before actually happening kind of like in blocking code. So when you create and run a query you don't actually continue running right away (that's what your callback/promise code is about).
Ideally, the above steps should be executed exclusively.
Sadly that's not a JavaScript problem, that's a host environment problem in this case Parse. This is because you have to explicitly yield control to the other code when you use their APIs (through callbacks and promises) and it is up to them to solve it.
Lucky for you, parse has atomic counters. From the API docs:
To help with storing counter-type data, Parse provides methods that atomically increment (or decrement) any number field. So, the same update can be rewritten as.
gameScore.increment("score");
gameScore.save();
There are also atomic array operations which you can use here. Since you can do step 3 atomically, you can guarantee that the counter represents the actual inventory.
I'm fetching a collection called logCollection from parse.com in a node JS script on my machine. It has 200 elements. Each log has a link to another table (a pointer) called start. I need to fetch this one too.
Here is my code
Parse.User.logIn("user", "pass").then(function(user) {
// Do stuff after successful login.
console.log('succesfully logged in');
return logCollection.fetch();
}).then(function(content) {
console.log('done fetching logs: ' + logCollection.length);
var promises = [];
_.each(logCollection.models, function(thisLog) {
promises.push(thisLog.attributes.start.fetch());
});
// Return a new promise that is resolved when all of the deletes are finished.
return Parse.Promise.when(promises);
});
The thing is, it will fire at least 200 (start) fetch per second, and it will cause problems with the 30 requests per second limit at parse.com.
Is there a better way to do this? How can I slow down the way js fires the requests?
thanks
In a Parse Query, you can get the fully-fetched objects which are pointed to by that object, by using the include method on the query:
var query = new Parse.Query("SomeClass");
query.include('columnName');
query.find().then(function(results) {
// each result will have 'columnName' as a fully fetched parse object.
});
This also works with sub-sub objects:
query.include('columnName.nestedColumnName');
or as an array:
query.include(['columnName', 'anotherPointerColumn']);
I came out with this solution that works very good. It was all this time on the parse documentation.
https://www.parse.com/docs/js_guide#promises-series
The following code will fire one request only after the last one has been finished. Doing so, I can fire many requests without worrying about getting to the limit.
var query = new Parse.Query("Comments");
query.equalTo("post", 123);
query.find().then(function(results) {
// Create a trivial resolved promise as a base case.
var promise = Parse.Promise.as();
_.each(results, function(result) {
// For each item, extend the promise with a function to delete it.
promise = promise.then(function() {
// Return a promise that will be resolved when the delete is finished.
return result.destroy();
});
});
return promise;
}).then(function() {
// Every comment was deleted.
});
I am facing a strange issue with calling socket.on methods from the Javascript client. Consider below code:
for(var i=0;i<2;i++) {
var socket = io.connect('http://localhost:5000/');
socket.emit('getLoad');
socket.on('cpuUsage',function(data) {
document.write(data);
});
}
Here basically I am calling a cpuUsage event which is emitted by socket server, but for each iteration I am getting the same value. This is the output:
0.03549148310035006
0.03549148310035006
0.03549148310035006
0.03549148310035006
Edit: Server side code, basically I am using node-usage library to calculate CPU usage:
socket.on('getLoad', function (data) {
usage.lookup(pid, function(err, result) {
cpuUsage = result.cpu;
memUsage = result.memory;
console.log("Cpu Usage1: " + cpuUsage);
console.log("Cpu Usage2: " + memUsage);
/*socket.emit('cpuUsage',result.cpu);
socket.emit('memUsage',result.memory);*/
socket.emit('cpuUsage',cpuUsage);
socket.emit('memUsage',memUsage);
});
});
Where as in the server side, I am getting different values for each emit and socket.on. I am very much feeling strange why this is happening. I tried setting data = null after each socket.on call, but still it prints the same value. I don't know what phrase to search, so I posted. Can anyone please guide me?
Please note: I am basically Java developer and have a less experience in Javascript side.
You are making the assumption that when you use .emit(), a subsequent .on() will wait for a reply, but that's not how socket.io works.
Your code basically does this:
it emits two getLoad messages directly after each other (which is probably why the returning value is the same);
it installs two handlers for a returning cpuUsage message being sent by the server;
This also means that each time you run your loop, you're installing more and more handlers for the same message.
Now I'm not sure what exactly it is you want. If you want to periodically request the CPU load, use setInterval or setTimeout. If you want to send a message to the server and want to 'wait' for a response, you may want to use acknowledgement functions (not very well documented, but see this blog post).
But you should assume that for each type of message, you should only call socket.on('MESSAGETYPE', ) once during the runtime of your code.
EDIT: here's an example client-side setup for a periodic poll of the data:
var socket = io.connect(...);
socket.on('connect', function() {
// Handle the server response:
socket.on('cpuUsage', function(data) {
document.write(data);
});
// Start an interval to query the server for the load every 30 seconds:
setInterval(function() {
socket.emit('getLoad');
}, 30 * 1000); // milliseconds
});
Use this line instead:
var socket = io.connect('iptoserver', {'force new connection': true});
Replace iptoserver with the actual ip to the server of course, in this case localhost.
Edit.
That is, if you want to create multiple clients.
Else you have to place your initiation of the socket variable before the for loop.
I suspected the call returns average CPU usage at the time of startup, which seems to be the case here. Checking the node-usage documentation page (average-cpu-usage-vs-current-cpu-usage) I found:
By default CPU Percentage provided is an average from the starting
time of the process. It does not correctly reflect the current CPU
usage. (this is also a problem with linux ps utility)
But If you call usage.lookup() continuously for a given pid, you can
turn on keepHistory flag and you'll get the CPU usage since last time
you track the usage. This reflects the current CPU usage.
Also given the example how to use it.
var pid = process.pid;
var options = { keepHistory: true }
usage.lookup(pid, options, function(err, result) {
});
As an exercise to teach myself more about node js I started making a basic CRUD REST server for SimpleDB (sdb) using the aws-sdk.
Everything was running smoothly until I got to a function for reading the domains. The aws-sdk has two functions for this purpose: listDomains and domainMetadata. listDomains returns an array of sdb domain names. domainMetadata will return additional statistics about a domain, but will only return them for one domain at a time. It does not include the domain name in the results.
My script is running listDomains and returning an array in the JSON response just fine. I would like to make my api readDomains function more ambitious though and have it return the metadata for all of the domains in the same single api call. After all, running a handful of domainMetadata calls at the same time is where node's async io should shine.
The problem is I can't figure out how to run a variable number of calls, use the same callback for all of them, match the results of each domainMetadata call to it's domainName (since it's async and they're not guaranteed to return in the order they were requested) and tell when all of the metadata requests have finished so that I can send my final response. Put into code my problem areas are:
domain.receiveDomainList = function(err, data){
var domainList = [];
for(var i=0; i<data.DomainNames.length; i++){
sdb.domainMetaData({"DomainName":data.DomainNames[i]},domain.receiveMetadata);
// alternatively: domainList.push({"DomainName":data.DomainNames[i]});
}
// alternatively:
// async.map(domainList, sdb.domainMetadata, domain.receiveMetadata)
console.log(domainList);
}
domain.receiveMetadata = function (err, data){
// I figure I can stash the results one at a time in an array in the
// parent scope but...
// How can I tell when all of the results have been received?
// Since the domainname used for the original call is not returned with
// the results how do I tell what result matches what request?
}
Based on my reading of async's readme the map function should at least match the metadata responses with the requests through some black magic, but it causes node to bomb out in the aws sync library with an error of " has no method 'makeRequest'".
Is there any way to have it all: requests run in parallel, requests matched with responses and knowing when I've received everything?
Using .bind() you can set the context or this values as well as provide leading default arguments to the bound function.
The sample code below is purely to show how you might use .bind() to add additional context to your response callbacks.
In the code below, .bind is used to:
set a domainResults object as the context for the receiveMetaData callback
pass the current domain name as an argument to the callback
The domainResults object is used to:
keep track of the number of names received in the first request
keep track of the completedCount (incremented on each callback from the metaData request)
keep track of both error and success responses in list
provide a complete callback
Completely untested code for illustrative purposes only:
domain.receiveDomainList = function(err, data) {
// Assuming err is falsey
var domainResults = {
nameCount: data.DomainNames.length,
completeCount: 0,
list: {},
complete: function() {
console.log(this.list);
}
};
for (var i = 0; i < data.DomainNames.length; i++) {
sdb.domainMetaData({ "DomainName": data.DomainNames[i] },
domain.receiveMetadata.bind(domainResults, data.DomainNames[i]));
}
}
domain.receiveMetadata = function(domainName, err, data) {
// Because of .bind, this === domainResults
this.completeCount++;
this.list[domainName] = data || {error: err};
if(this.completeCount === this.nameCount) {
this.complete();
}
}
I'm new at javascript and I've hit a wall hard here. I don't even think this is a Sequelize question and probably more so about javascript behavior.
I have this code:
sequelize.query(query).success( function(row){
console.log(row);
}
)
The var row returns the value(s) that I want, but I have no idea how to access them other than printing to the console. I've tried returning the value, but it isn't returned to where I expect it and I'm not sure where it goes. I want my row, but I don't know how to obtain it :(
Using Javascript on the server side like that requires that you use callbacks. You cannot "return" them like you want, you can however write a function to perform actions on the results.
sequelize.query(query).success(function(row) {
// Here is where you do your stuff on row
// End the process
process.exit();
}
A more practical example, in an express route handler:
// Create a session
app.post("/login", function(req, res) {
var username = req.body.username,
password = req.body.password;
// Obviously, do not inject this directly into the query in the real
// world ---- VERY BAD.
return sequelize
.query("SELECT * FROM users WHERE username = '" + username + "'")
.success(function(row) {
// Also - never store passwords in plain text
if (row.password === password) {
req.session.user = row;
return res.json({success: true});
}
else {
return res.json({success: false, incorrect: true});
}
});
});
Ignore injection and plain text password example - for brevity.
Functions act as "closures" by storing references to any variable in the scope the function is defined in. In my above example, the correct res value is stored for reference per request by the callback I've supplied to sequelize. The direct benefit of this is that more requests can be handled while the query is running and once it's finished more code will be executed. If this wasn't the case, then your process (assuming Node.js) would wait for that one query to finish block all other requests. This is not desired. The callback style is such that your code can do what it needs and move on, waiting for important or processer heavy pieces to finish up and call a function once complete.
EDIT
The API for handling callbacks has changed since answering this question. Sequelize now returns a Promise from .query so changing .success to .then should be all you need to do.
According to the changelog
Backwards compatibility changes:
Events support have been removed so using .on('success') or .success()
is no longer supported. Try using .then() instead.
According this Raw queries documentation you will use something like this now:
sequelize.query("SELECT * FROM `users`", { type: sequelize.QueryTypes.SELECT})
.then(function(users) {
console.log(users);
});