Develop Reference

Assemble paginated ajax data in a Bacon FRP stream

I'm learning FRP using Bacon.js, and would like to assemble data from a paginated API in a stream.
The module that uses the data has a consumption API like this:
// UI module, displays unicorns as they arrive
beautifulUnicorns.property.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
The module that assembles the data requests sequential pages from an API and pushes onto the stream every time it gets a new data set:
// beautifulUnicorns module
var curPage = 1
var stream = new Bacon.Bus()
var property = stream.toProperty()
var property.onValue(function(){}) # You have to add an empty subscriber, otherwise future onValues will not receive the initial value. https://github.com/baconjs/bacon.js/wiki/FAQ#why-isnt-my-property-updated
var allUnicorns = [] // !!! stateful list of all unicorns ever received. Is this idiomatic for FRP?
var getNextPage = function(){
/* get data for subsequent pages.
Skipping for clarity */
}
var gotNextPage = function (resp) {
Array.prototype.push.apply(allUnicorns, resp) // just adds the responses to the existing array reference
stream.push(allUnicorns)
curPage++
if (curPage <= pageLimit) { getNextPage() }
}
How do I subscribe to the stream in a way that provides me a full list of all unicorns ever received? Is this flatMap or similar? I don't think I need a new stream out of it, but I don't know. I'm sorry, I'm new to the FRP way of thinking. To be clear, assembling the array works, it just feels like I'm not doing the idiomatic thing.
I'm not using jQuery or another ajax library for this, so that's why I'm not using Bacon.fromPromise
You also may wonder why my consuming module wants the whole set instead of just the incremental update. If it were just appending rows that could be ok, but in my case it's an infinite scroll and it should draw data if both: 1. data is available and 2. area is on screen.

This can be done with the .scan() method. And also you will need a stream that emits items of one page, you can create it with .repeat().
Here is a draft code (sorry not tested):
var itemsPerPage = Bacon.repeat(function(index) {
var pageNumber = index + 1;
if (pageNumber < PAGE_LIMIT) {
return Bacon.fromCallback(function(callback) {
// your method that talks to the server
getDataForAPage(pageNumber, callback);
});
} else {
return false;
}
});
var allItems = itemsPerPage.scan([], function(allItems, itemsFromAPage) {
return allItems.concat(itemsFromAPage);
});
// Here you go
allItems.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
As you noticed, you also won't need .onValue(function(){}) hack, and curPage external state.

Here is a solution using flatMap and fold. When dealing with network you have to remember that the data can come back in a different order than you sent the requests - that's why the combination of fold and map.
var pages = Bacon.fromArray([1,2,3,4,5])
var requests = pages.flatMap(function(page) {
return doAjax(page)
.map(function(value) {
return {
page: page,
value: value
}
})
}).log("Data received")
var allData = requests.fold([], function(arr, data) {
return arr.concat([data])
}).map(function(arr) {
// I would normally write this as a oneliner
var sorted = _.sortBy(arr, "page")
var onlyValues = _.pluck(sorted, "value")
var inOneArray = _.flatten(onlyValues)
return inOneArray
})
allData.log("All data")
function doAjax(page) {
// This would actually be Bacon.fromPromise($.ajax...)
// Math random to simulate the fact that requests can return out
// of order
return Bacon.later(Math.random() * 3000, [
"Page"+page+"Item1",
"Page"+page+"Item2"])
}
http://jsbin.com/damevu/4/edit

Backbone - Check if Record Already Exists - Is there a better way than this?

In Backbone, I need to check if a model record already exists. Right now, I am doing it by fetching the model by id, and seeing if its "created_at" attribute is undefined. This feels brittle to me. Does anyone have any better recommendations?
var dealProgram = new WhiteDeals.Models.DealProgram({id: servant_id});
dealProgram.fetch({
success: function() {
var program = dealProgram.toJSON();
var datecheck = program.created_at
if(typeof datecheck === 'undefined'){
dealPrograms.create({
title: "",
servant_id: servant.servant_id,
servant_name: servant.name,
servant_master: servant.master
},
{
success: function () {
self.manageServants(servants);
}
}); // End of dealPrograms.create
} else if (datecheck !== undefined) {
console.log("is defined, success!")
self.manageServants(servants);
}; // End of if statement for non-existant dealPrograms
} // End of success
}); // End of dealProgram.fetch

You'll obviously have to check by using a request (whatever the form of the request, you'll have a low amount of data anyway). Guess you should still wonder if it wouldn't be worth fetching all your models at once in a collection so you can make the check client-side (or only the id if it'd be too big to fetch everything).

Javascript Data Layer Architecture Assistance

I'm making a fairly complex HTML 5 + Javascript game. The client is going to have to download images and data at different points of the game depending on the area they are at. I'm having a huge problem resolving some issues with the Data Layer portion of the Javascript architecture.
The problems I need to solve with the Data Layer:
Data used in the application that becomes outdated needs to be automatically updated whenever calls are made to the server that retrieve fresh data.
Data retrieved from the server should be stored locally to reduce any overhead that would come from requesting the same data twice.
Any portion of the code that needs access to data should be able to retrieve it easily and in a uniform way regardless of whether the data is available locally already.
What I've tried to do to accomplish this is build a data layer that has two main components:
1. The portion of the layer that gives access to the data (through get* methods)
2. The portion of the layer that stores and synchronizes local data with data from the server.
The workflow is as follows:
When the game needs access to some data it calls get* method in the data layer for that data, passing a callback function.
bs.data.getInventory({ teamId: this.refTeam.PartyId, callback: this.inventories.initialize.bind(this.inventories) });
The get* method determines whether the data is already available locally. If so it either returns the data directly (if no callback was specified) or calls the callback function passing it the data.
If the data is not available, it stores the callback method locally (setupListener) and makes a call to the communication object passing the originally requested information along.
getInventory: function (obj) {
if ((obj.teamId && !this.teamInventory[obj.teamId]) || obj.refresh) {
this.setupListener(this.inventoryNotifier, obj);
bs.com.getInventory({ teamId: obj.teamId });
}
else if (typeof (obj.callback) === "function") {
if (obj.teamId) {
obj.callback(this.team[obj.teamId].InventoryList);
}
}
else {
if (obj.teamId) {
return this.team[obj.teamId].InventoryList;
}
}
}
The communication object then makes an ajax call to the server and waits for the data to return.
When the data is returned a call is made to the data layer again asking it to publish the retrieved data.
getInventory: function (obj) {
if (obj.teamId) {
this.doAjaxCall({ orig: obj, url: "/Item/GetTeamEquipment/" + obj.teamId, event: "inventoryRefreshed" });
}
},
doAjaxCall: function (obj) {
var that = this;
if (!this.inprocess[obj.url + obj.data]) {
this.inprocess[obj.url + obj.data] = true;
$.ajax({
type: obj.type || "GET",
contentType: "application/json; charset=utf-8",
dataType: "json",
data: obj.data,
url: obj.url,
async: true,
success: function (data) {
try {
ig.fire(bs.com, obj.event, { data: data, orig: obj.orig });
}
catch (ex) {
// this enables ajaxComplete to fire
ig.log(ex.message + '\n' + ex.stack);
}
finally {
that.inprocess[obj.url + obj.data] = false;
}
},
error: function () { that.inprocess[obj.url + obj.data] = false; }
});
}
}
The data layer then stores all of the data in a local object and finally calls the original callback function, passing it the requested data.
publishInventory: function (data) {
if (!this.inventory) this.inventory = {};
for (var i = 0; i < data.data.length; i++) {
if (this.inventory[data.data[i].Id]) {
this.preservingUpdate(this.inventory[data.data[i].Id], data.data[i]);
}
else {
this.inventory[data.data[i].Id] = data.data[i];
}
}
// if we pulled this inventory for a team, update the team
// with the inventory
if (data.orig.teamId && this.team[data.orig.teamId]) {
this.teamInventory[data.orig.teamId] = true;
this.team[data.orig.teamId].InventoryList = [];
for (var i = 0; i < data.data.length; i++) {
this.team[data.orig.teamId].InventoryList.push(data.data[i]);
}
}
// set up the data we'll notify with
var notifyData = [];
for (var i = 0; i < data.data.length; i++) {
notifyData.push(this.inventory[data.data[i].Id]);
}
ig.fire(this.inventoryNotifier, "refresh", notifyData, null, true);
}
There are several problems with this that bother me constantly. I'll list them in order of most annoying :).
Anytime I have to add a call that goes through this process it takes too much time to do so. (at least an hour)
The amount of jumping and callback passing gets confusing and seems very prone to errors.
The hierarchical way in which I am storing the data is incredibly difficult to synchronize and manage. More on that next.
Regarding issue #3 above, if I have objects in the data layer that are being stored that have a structure that looks like this:
this.Account = {Battles[{ Teams: [{ TeamId: 392, Characters: [{}] }] }]}
this.Teams[392] = {Characters: [{}]}
Because I want to store Teams in a way where I can pass the TeamId to retrieve the data (e.g. return Teams[392];) but I also want to store the teams in relation to the Battles in which they exist (this.Account.Battles[0].Teams[0]); I have a nightmare of a time keeping each instance of the same team fresh and maintaining the same object identity (so I am not actually storing it twice and so that my data will automatically update wherever it is being used which is objective #1 of the data layer).
It just seems so messy and jumbled.
I really appreciate any help.
Thanks

You should consider using jquery's deferred objects.
Example:
var deferredObject = $.Deferred();
$.ajax({
...
success: function(data){
deferredObject.resolve(data);
}
});
return deferredObject;
Now with the deferredObject returned, you can attach callbacks to it like this:
var inventoryDfd = getInventory();
$.when(inventoryDfd).done(function(){
// code that needs data to continue
}
and you're probably less prone to errors. You can even nest deferred objects, or combine them so that a callback isn't called until multiple server calls are downloaded.

+1 for Backbone -- it does some great heavy lifting for you.
Also look at the Memoizer in Douglas Crockford's book Javascript the Good Parts. It's dense, but awesome. I hacked it up to make the memo data store optional, and added more things like the ability to set a value without having to query first -- e.g. to handle data freshness.

Backbone.js - problem when saving a model before previous save issues POST(create) instead of PUT(update) request

I've developed a nice rich application interface using Backbone.js where users can add objects very quickly, and then start updating properties of those objects by simply tabbing to the relevant fields. The problem I am having is that sometimes the user beats the server to its initial save and we end up saving two objects.
An example of how to recreate this problem is as follows:
User clicks the Add person button, we add this to the DOM but don't save anything yet as we don't have any data yet.
person = new Person();
User enters a value into the Name field, and tabs out of it (name field loses focus). This triggers a save so that we update the model on the server. As the model is new, Backbone.js will automatically issue a POST (create) request to the server.
person.set ({ name: 'John' });
person.save(); // create new model
User then very quickly types into the age field they have tabbed into, enters 20 and tabs to the next field (age therefore loses focus). This again triggers a save so that we update the model on the server.
person.set ({ age: 20 });
person.save(); // update the model
So we would expect in this scenario one POST request to create the model, and one PUT requests to update the model.
However, if the first request is still being processed and we have not had a response before the code in point 3 above has run, then what we actually get is two POST requests and thus two objects created instead of one.
So my question is whether there is some best practice way of dealing with this problem and Backbone.js? Or, should Backbone.js have a queuing system for save actions so that one request is not sent until the previous request on that object has succeeded/failed? Or, alternatively should I build something to handle this gracefully by either sending only one create request instead of multiple update requests, perhaps use throttling of some sort, or check if the Backbone model is in the middle of a request and wait until that request is completed.
Your advice on how to deal with this issue would be appreciated.
And I'm happy to take a stab at implementing some sort of queuing system, although you may need to put up with my code which just won't be as well formed as the existing code base!

I have tested and devised a patch solution, inspired by both #Paul and #Julien who posted in this thread. Here is the code:
(function() {
function proxyAjaxEvent(event, options, dit) {
var eventCallback = options[event];
options[event] = function() {
// check if callback for event exists and if so pass on request
if (eventCallback) { eventCallback(arguments) }
dit.processQueue(); // move onto next save request in the queue
}
}
Backbone.Model.prototype._save = Backbone.Model.prototype.save;
Backbone.Model.prototype.save = function( attrs, options ) {
if (!options) { options = {}; }
if (this.saving) {
this.saveQueue = this.saveQueue || new Array();
this.saveQueue.push({ attrs: _.extend({}, this.attributes, attrs), options: options });
} else {
this.saving = true;
proxyAjaxEvent('success', options, this);
proxyAjaxEvent('error', options, this);
Backbone.Model.prototype._save.call( this, attrs, options );
}
}
Backbone.Model.prototype.processQueue = function() {
if (this.saveQueue && this.saveQueue.length) {
var saveArgs = this.saveQueue.shift();
proxyAjaxEvent('success', saveArgs.options, this);
proxyAjaxEvent('error', saveArgs.options, this);
Backbone.Model.prototype._save.call( this, saveArgs.attrs, saveArgs.options );
} else {
this.saving = false;
}
}
})();
The reason this works is as follows:
When an update or create request method on a model is still being executed, the next request is simply put in a queue to be processed when one of the callbacks for error or success are called.
The attributes at the time of the request are stored in an attribute array and passed to the next save request. This therefore means that when the server responds with an updated model for the first request, the updated attributes from the queued request are not lost.
I have uploaded a Gist which can be forked here.

A light-weight solution would be to monkey-patch Backbone.Model.save, so you'll only try to create the model once; any further saves should be deferred until the model has an id. Something like this should work?
Backbone.Model.prototype._save = Backbone.Model.prototype.save;
Backbone.Model.prototype.save = function( attrs, options ) {
if ( this.isNew() && this.request ) {
var dit = this, args = arguments;
$.when( this.request ).always( function() {
Backbone.Model.prototype._save.apply( dit, args );
} );
}
else {
this.request = Backbone.Model.prototype._save.apply( this, arguments );
}
};

I have some code I call EventedModel:
EventedModel = Backbone.Model.extend({
save: function(attrs, options) {
var complete, self, success, value;
self = this;
options || (options = {});
success = options.success;
options.success = function(resp) {
self.trigger("save:success", self);
if (success) {
return success(self, resp);
}
};
complete = options.complete;
options.complete = function(resp) {
self.trigger("save:complete", self);
if (complete) {
return complete(self, resp);
}
};
this.trigger("save", this);
value = Backbone.Model.prototype.save.call(this, attrs, options);
return value;
}
});
You can use it as a backbone model. But it will trigger save and save:complete. You can boost this a little:
EventedSynchroneModel = Backbone.Model.extend({
save: function(attrs, options) {
var complete, self, success, value;
if(this.saving){
if(this.needsUpdate){
this.needsUpdate = {
attrs: _.extend(this.needsUpdate, attrs),
options: _.extend(this.needsUpdate, options)};
}else {
this.needsUpdate = { attrs: attrs, options: options };
}
return;
}
self = this;
options || (options = {});
success = options.success;
options.success = function(resp) {
self.trigger("save:success", self);
if (success) {
return success(self, resp);
}
};
complete = options.complete;
options.complete = function(resp) {
self.trigger("save:complete", self);
//call previous callback if any
if (complete) {
complete(self, resp);
}
this.saving = false;
if(self.needsUpdate){
self.save(self.needsUpdate.attrs, self.needsUpdate.options);
self.needsUpdate = null;
}
};
this.trigger("save", this);
// we are saving
this.saving = true;
value = Backbone.Model.prototype.save.call(this, attrs, options);
return value;
}
});
(untested code)
Upon the first save call it will save the record normally. If you quickly do a new save it will buffer that call (merging the different attributes and options into a single call). Once the first save succeed, you go forward with the second save.

As an alternative to the above answer, you could achieve the same affect by overloading the backbone.sync method to be synchronous for this model. Doing so would force each call to wait for the previous to finish.
Another option would be to just do the sets when the user is filing things out and do one save at the end. That well also reduce the amount of requests the app makes as well

How to implement super fast ASP.NET AJAX callbacks/PageMethods

I am designing an ASP.NET web application (.NET 4.0) which basically has a page that should interact with the code behind every 1-2 seconds (Using Client callbacks or PageMethods via ScriptManager or jQuery.ajax) It'll be hosted on an intranet, so a 1-2 second refresh rate is kind of reasonable.
How can I make the page to access the web service/pagemthod in the code behind in a timeply manner (e.g. every 1 second). Should I use a javascript timer (I'm not familiar with javascrip very much)?
Although the site is hosted on an intranet, but I still need to implement a good approach to reach the desired refresh rate. the amount of data being transfered is about 1KB in each interaction. What are your recommendations on my design regarding this? (using callbacks or ScriptManager or jQuery.ajax,... any pitfalls I should avoid)
Thanks.

A 1kb request every 1-2 seconds is reasonable using either approach. A page method or web service (they're nearly identical under the hood) that does nothing will respond in just a few milliseconds if you're dealing with a fast network/server.
The limiting factor will be how long the meat of your server-side method takes to complete (i.e. if it's involves a database or external service, that's going to slow you down more than the overhead of the service).

I use Webservices, since they are much faster. But if you doing with UpdatePanels, Webservices are useless. Futher I would say, you shouldnt update the page every x seconds, but first ask, if there is an update to do at all. This saves a lot ;-)
This might be a little example, i didn´t try but worked once like this. Is a ms ajax version, needs scriptmanager
Type.registerNamespace("myproject");
myproject.updateControl = function () {
myproject.updateControl.initializeBase(this);
this._xhr = null;
this._updating = false;
this._timer = null;
}
myproject.updateControl.prototype = {
initialize: function () {
myproject.updateControl.callBaseMethod(this, 'initialize');
this.startTimer();
},
startTimer: function () {
if (this._timer) clearTimeout(this._timer);
this._timer = setInterval(Function.createDelegate(this, this._timerWork), 2000);
},
stopTimer: function () {
clearTimeout(this._timer);
this._timer = null;
},
_timerWork: function () {
if (this._updating || !this._checkXhr()) return;
this._xhr = Sys.Net.WebServiceProxy.invoke("myServicePath Or null if PageMethod", "checkForUpdate",
false,
null,
Function.createDelegate(this, this._onCheckedUpdate));
},
_onCheckedUpdate: function (data) {
this._xhr = null;
if (data.needsUpdate) {
this._update();
}
},
_udpate: function () {
if (!this._checkXhr) return;
this._updating = true;
this._xhr = Sys.Net.WebServiceProxy.invoke("servicepath", "updateMe", false, { param: param }, Function.createDelegate(this, this._updateSucces));
},
_updateSuccess: function (data) {
alert("yeah i´m get updated");
this._updating = false
this._xhr = null;
},
_checkXhr: function () {
if (this._xhr()) {
if (confirm("There is an active call to the Server. If you wait to long, it may have been broken. Do you want to Abort the current call?")) {
this._xhr.get_executor().abort();
this._xhr = null;
return true;
} else {
return false;
}
}
return true;
},
dispose: function () {
myproject.updateControl.callBaseMethod(this, 'dispose');
}
}
myproject.updateControl.registerClass('myproject.updateControl', Sys.Component);
usage
$create(myproject.updateControl);
or
var upd = new myproject.updateControl();
upd.initialize();

"Timer" on the client side is a bad idea. You can use setInterval(method, timespan) to force a call every n milliseconds, but if the server ever gets backed up, you can then start stacking requests and you'll start getting responses out of order (even on a non-slow network).
I recommend using setTimeout(method, timespan) in your ajax code in the call processing logic to setup the next call.
Example (using jQuery):
function getStuff()
{
$.get(
'myurl.aspx?r=' + Math.random(), // stop caching issues
function(data) {
$('#myDiv').html(data);
setTimeout(getStuff, 2000); // you might want to set this to 1900 if you need it closer to every 2 seconds
}
);
}
setTimeout(getStuff, 2000); // the initial timer initialization