Develop Reference

Assemble paginated ajax data in a Bacon FRP stream

I'm learning FRP using Bacon.js, and would like to assemble data from a paginated API in a stream.
The module that uses the data has a consumption API like this:
// UI module, displays unicorns as they arrive
beautifulUnicorns.property.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
The module that assembles the data requests sequential pages from an API and pushes onto the stream every time it gets a new data set:
// beautifulUnicorns module
var curPage = 1
var stream = new Bacon.Bus()
var property = stream.toProperty()
var property.onValue(function(){}) # You have to add an empty subscriber, otherwise future onValues will not receive the initial value. https://github.com/baconjs/bacon.js/wiki/FAQ#why-isnt-my-property-updated
var allUnicorns = [] // !!! stateful list of all unicorns ever received. Is this idiomatic for FRP?
var getNextPage = function(){
/* get data for subsequent pages.
Skipping for clarity */
}
var gotNextPage = function (resp) {
Array.prototype.push.apply(allUnicorns, resp) // just adds the responses to the existing array reference
stream.push(allUnicorns)
curPage++
if (curPage <= pageLimit) { getNextPage() }
}
How do I subscribe to the stream in a way that provides me a full list of all unicorns ever received? Is this flatMap or similar? I don't think I need a new stream out of it, but I don't know. I'm sorry, I'm new to the FRP way of thinking. To be clear, assembling the array works, it just feels like I'm not doing the idiomatic thing.
I'm not using jQuery or another ajax library for this, so that's why I'm not using Bacon.fromPromise
You also may wonder why my consuming module wants the whole set instead of just the incremental update. If it were just appending rows that could be ok, but in my case it's an infinite scroll and it should draw data if both: 1. data is available and 2. area is on screen.

This can be done with the .scan() method. And also you will need a stream that emits items of one page, you can create it with .repeat().
Here is a draft code (sorry not tested):
var itemsPerPage = Bacon.repeat(function(index) {
var pageNumber = index + 1;
if (pageNumber < PAGE_LIMIT) {
return Bacon.fromCallback(function(callback) {
// your method that talks to the server
getDataForAPage(pageNumber, callback);
});
} else {
return false;
}
});
var allItems = itemsPerPage.scan([], function(allItems, itemsFromAPage) {
return allItems.concat(itemsFromAPage);
});
// Here you go
allItems.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
As you noticed, you also won't need .onValue(function(){}) hack, and curPage external state.

Here is a solution using flatMap and fold. When dealing with network you have to remember that the data can come back in a different order than you sent the requests - that's why the combination of fold and map.
var pages = Bacon.fromArray([1,2,3,4,5])
var requests = pages.flatMap(function(page) {
return doAjax(page)
.map(function(value) {
return {
page: page,
value: value
}
})
}).log("Data received")
var allData = requests.fold([], function(arr, data) {
return arr.concat([data])
}).map(function(arr) {
// I would normally write this as a oneliner
var sorted = _.sortBy(arr, "page")
var onlyValues = _.pluck(sorted, "value")
var inOneArray = _.flatten(onlyValues)
return inOneArray
})
allData.log("All data")
function doAjax(page) {
// This would actually be Bacon.fromPromise($.ajax...)
// Math random to simulate the fact that requests can return out
// of order
return Bacon.later(Math.random() * 3000, [
"Page"+page+"Item1",
"Page"+page+"Item2"])
}
http://jsbin.com/damevu/4/edit

Validate URLs in jQuery or Javascript before sending

I have a basic HTML text area which will be used to have URLs pasted into it. Once some URLs are passed into the text area, those will be sent to a server-side script for processing via AJAX. I'll be binding those whole process to a keyUp event.
The issue is: How will I know I'm sending valid URLs to the script with a client-side check? I don't want to start sending URLs to the PHP script without having them validated in Javascript/jQuery first.
This would be quite easy to solve of the text area accepted one URL only, but the text area needs to accept multiple URLs separated by line breaks. So for example, I'd need to validate this:
http://someurl.com/something.ex
https://someurl.com/somethingelse.ext
I-M-NOT-AN-URL
So from the above, only the URLs would be sent to the server and I-M-NOT-AN-URL would be ignored.
I've not tried anything in regards to this issue since I'm not very familiar with JS, nor found anything as I couldn't come up with a relevant search term I guess, so I'm asking here for help.
Any kind of help on how to tackle this issue would be appreciated.
Update
Based on the comments and answer below, I've come up with the following Javascript/jQuery. I don't know if it efficient, therefore I'm sharing it with you for feedback and help. I don't seem to know how to prepare logic that well in JS... That's lame from my side.
Anyway here I go:
var char_start = 10;
var index = 0;
var urls = $('textarea.remote-area');
var val_ary = [];
var urls_ary = [];
var single_url = '';
urls.keyup(function(){
if (urls.val().length >= char_start)
{
var has_lbrs = /\r|\n/i.test(urls.val());
if (has_lbrs) {
val_ary = urls.val().split('\n');
for (var i = 0; i < val_ary.length; i++)
{
if (!validate_url(val_ary[i]))
{
continue;
}
urls_ary[i] = val_ary[i];
}
}
else
{
if (validate_url(urls.val()))
{
single_url = urls.val();
}
}
if (urls_ary.length > 0)
{
for (var i = 0; i < urls_ary.length; i++)
{
$.ajax({
// do AJAX here.
});
}
}
else
{
$.ajax({
// do AJAX here.
});
}
}
});
function validate_url(url)
{
if(/^([a-z]([a-z]|\d|\+|-|\.)*):(\/\/(((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:)*#)?((\[(|(v[\da-f]{1,}\.(([a-z]|\d|-|\.|_|~)|[!\$&'\(\)\*\+,;=]|:)+))\])|((\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5]))|(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=])*)(:\d*)?)(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)*)*|(\/((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)+(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)*)*)?)|((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)+(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)*)*)|((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)){0})(\?((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)|[\uE000-\uF8FF]|\/|\?)*)?(\#((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)|\/|\?)*)?$/i.test(url)){
return true;
}
return false;
}

The jQuery validation plugin makes use of a method such as this:
var anyURL = "http://www.yahoo.com/";
if(/^([a-z]([a-z]|\d|\+|-|\.)*):(\/\/(((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:)*#)?((\[(|(v[\da-f]{1,}\.(([a-z]|\d|-|\.|_|~)|[!\$&'\(\)\*\+,;=]|:)+))\])|((\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5]))|(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=])*)(:\d*)?)(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)*)*|(\/((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)+(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)*)*)?)|((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)+(\/(([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)*)*)|((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)){0})(\?((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)|[\uE000-\uF8FF]|\/|\?)*)?(\#((([a-z]|\d|-|\.|_|~|[\u00A0-\uD7FF\uF900-\uFDCF\uFDF0-\uFFEF])|(%[\da-f]{2})|[!\$&'\(\)\*\+,;=]|:|#)|\/|\?)*)?$/i.test(anyURL)) {
/* the URL is valid */
} else {
/* the URL is invalid)
}
You can use that code directly or use the validation plugin itself.
Please note: it may be that the plugin has evolved and the actual code is different now. Nonetheless, the above should help you.

Loading csv files content into dictionary of objects

I'm trying to load the content of the several CSV files into a new array. CSV files have a typical structure, with a label in the first row, and values (both string and real numbers) separated by commas. This part of code is responsible for loading the data for future use with Google Maps Api (not a problem for now, since I'm stuck on just loading the data). I would like to have a structure, in which I could call an element by it's name, that's why the var nodedata = {}; is created.
So the thing I totally don't get is why some part of the code is not being executed at all? console.log(nodedata); is empty, at least not in my Firefox console.
That's my attempt to the problem - links to the csv files are in the code.
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.js"></script>
<script src="http://jquery-csv.googlecode.com/files/jquery.csv-0.71.js"></script>
<script type="text/javascript">
var nodes = {};
var generation = {};
var nodedata = {};
$.get('https://dl.dropboxusercontent.com/u/25575808/energy/nodes.csv', function (response) {
nodes = $.csv.toObjects(response);
console.log(nodes);
});
$.get('https://dl.dropboxusercontent.com/u/25575808/energy/generation.csv', function (response) {
generation = $.csv.toObjects(response);
console.log(generation);
});
function getGeneration (nodename){
gen = 0;
for (var i = 0; i < generation.length; i++) {
if (generation[i].datetime == "2013-01-01 01:00"){
if (generation[i].node == nodename){
gen = gen + Number(generation[i]["output (MW)"])
}
}
}
return gen;
}
for (var i = 0; i < nodes.length; i++) {
nodedata[nodes[i].Node] = {
center: new google.maps.LatLng(nodes[i].Latitude,nodes[i].Longitude),
nodegen : getGeneration(nodes[i].Node)
}
}
console.log(nodedata);

I believe the problem you're having is unrelated to the usage of CSV data, rather it is the fact that the data is being loaded asynchronously.
You are executing 2 $.get() requests to load the files, which will take some time to download the files. The browser does not wait for them to finish before continuing through the rest of the code.
Therefore, it is possible for console.log(nodedate) to be executed before any data exists inside the nodes array.
An easy way to handle this is to stack your callback functions so that the first GET request completes -> run the 2nd GET request -> finally, run the processing code.
Check out this reorganization of the code: http://jsfiddle.net/Vr7sw/
(I removed the Google Maps line since I don't have the library loaded)

the problem is, the $.get requests are asynchronous (see jquery documentation), try to call to a function, into your callback body like this :
function nodesToJson(nodes) {
for (var i = 0; i < nodes.length; i++) {
nodedata[nodes[i].Node] = {
center: new google.maps.LatLng(nodes[node].Latitude,nodes[node].Longitude),
nodegen : getGeneration(nodes[i].Node)
}
}
console.log(nodedata);
}
$.get('https://dl.dropboxusercontent.com/u/25575808/energy/nodes.csv', function (response) {
nodes = $.csv.toObjects(response);
//when the request are ready, process the nodes
nodesToJson(nodes);
});

Why is this javascript object property undefined?

I am using an approach described in detail at Dictionary Lookups in Javascript (see the section"A Client-Side Solution") to create an object that contains a property for each word in the scrabble dictionary.
var dict = {};
//ajax call to read dictionary.txt file
$.get("dictionary.txt", parseResults);
function parseResults(txt) {
var words = txt.split( "\n");
for (var i=0; i < words.length; i++){
dict[ words[i] ] = true;
}
console.log(dict.AAH);
console.log(dict);
if (dict.AAH == true) {
console.log('dict.AAH is true!');
}
}
(updated code to use an earlier answer from Phil)
I can't figure out why dict.AAH is returning undefined, but the dict object looks fine in the console. Screenshots from Firebug below.
Console:
Drilled down into "Object { }"
How can I check a given word ("AAH", in this case) and have it return true if it is a property in the dict object defined as true?
Live example
Code on Github

The problem isn't your code. You have invisible characters in your words, which you fail to clean up.
You can verify this by using this as your results parser
function parseResults(txt) {
// clean the words when we split the txt
var words = txt.split("\n")
.map($.trim)
.splice(0,3); // Keep only 3 first ones
if(btoa(words[2]) !== btoa('AAH')){ // Compare in Base64
console.log('YOU HAVE HIDDEN CHARS!');
}
}
And you can fix it by whitelisting your characters.
function parseResults(txt) {
// clean the words when we split the txt
var words = txt.split("\n").map(function(el){
return el.match(/[a-zA-Z0-9]/g).join('');
});
for (var i=0; i < words.length; i++){
dict[ words[i] ] = true;
}
console.log(dict.AAH);
console.log(dict);
if (dict.AAH == true) {
console.log('dict.AAH is true!');
}
}
I would recommend cleaning it up on the server side since running regex on every element in an array as large as seen in your live site might cause performance issues.

It's probably a race condition. You're loading the dictionary in a GET and then immediately (while the request is being made) those console.log commands are being called (and the one comes back undefined). Then the data is actually loaded by the time you debug. Everything should be done in a callback or deferred. It's an understandable quirk of debuggers that's caught me up before.

Get ajax requests are asynchronous. This means that while the whole operation that occurs in the ajax request is going, javascript keeps reading the next lines.
The problem then is you are logging values that the ajax request did not manage to retrieve early enough.
To get around the issue you can include the log calls inside your ajax request callback as below
var dict = {};
//ajax call to read dictionary.txt file
$.get("dictionary.txt", function( txt ){
var words = txt.split( "\n");
for (var i=0; i < words.length; i++){
dict[ words[i] ] = true;
}
//Now inside these console.log will run once you DO have the data
console.log(dict.AAH);
console.log(dict);
});
//Stuff out here will run whether or not asynchronous request has finished
I WOULD RECOMMEND USING THE WHEN METHOD IN JQUERY FOR THIS TYPE OF SCENARIOS EVEN MORE AS THE BEST SOLUTION
HERE IS HOW WHAT I THINK WOULD BE MOST PROPER FOR COMPLEX PROJECTS
var dict = {};
//ajax call to read dictionary.txt file
function getDictionary(){
return $.ajax("dictionary.txt");
}
/*I recommend this technique because this will allow you to easily extend your
code to maybe way for more than one ajax request in the future. You can stack
as many asynchronous operations as you want inside the when statement*/
$.when(getDictionary()).then(function(txt){//Added txt here...forgot callback param before
var words = txt.split( "\n");
for (var i=0; i < words.length; i++){
dict[ words[i] ] = true;
}
//Now inside these console.log will run once you DO have the data
console.log(dict.AAH);
console.log(dict);
});

You're trying to output dict before it has been populated by the $.get success handler.
Try this:
// If the browser doesn't have String.trim() available, add it...
if (!String.prototype.trim) {
String.prototype.trim=function(){return this.replace(/^\s\s*/, '').replace(/\s\s*$/, '');};
String.prototype.ltrim=function(){return this.replace(/^\s+/,'');};
String.prototype.rtrim=function(){return this.replace(/\s+$/,'');};
String.prototype.fulltrim=function(){return this.replace(/(?:(?:^|\n)\s+|\s+(?:$|\n))/g,'').replace(/\s+/g,' ');};
}
/**
* Parses the response returned by the AJAX call
*
* Response parsing logic must be executed only after the
* response has been received. To do so, we have to encapsulate
* it in a function and use it as a onSuccess callback when we
* place our AJAX call.
**/
function parseResults(txt) {
// clean the words when we split the txt
var words = txt.split("\n").map($.trim);
for (var i=0; i < words.length; i++){
dict[ words[i] ] = true;
}
console.log(dict.AAH);
console.log(dict);
if (dict.AAH == true) {
console.log('dict.AAH is true!');
}
}
// global object containing retrieved words.
var dict = {};
//ajax call to read dictionary.txt file
$.get("dictionary.txt", parseResults);
As another user commented, jQuery's $.when lets you chain such code.
By the way, if all you want to do is know if a word is in the results you can do:
function parseResults(txt) {
// clean the words when we split the txt
var words = txt.split("\n").map($.trim);
if ($.inArray('AAH', words)) {
console.log('AAH is in the result set');
}
}

I think the problem lays in that you have dict defined as an object but use it as an array.
Replace var dict = {} by var dict = new Array() and your code should work (tried with your live example on Google Chrome).

Returning a value from a jQuery Ajax method

I'm trying to use Javascript in an OO style, and one method needs to make a remote call to get some data so a webpage can work with it. I've created a Javascript class to encapsulate the data retrieval so I can re-use the logic elsewhere, like so:
AddressRetriever = function() {
AddressRetriever.prototype.find = function(zip) {
var addressList = [];
$.ajax({
/* setup stuff */
success: function(response) {
var data = $.parseJSON(response.value);
for (var i = 0; i < data.length; i++) {
var city = data[i].City; // "City" column of DataTable
var state = data[i].State; // "State" column of DataTable
var address = new PostalAddress(postalCode, city, state); // This is a custom JavaScript class with simple getters, a DTO basically.
addressList.push(address);
}
}
});
return addressList;
}
}
The webpage itself calls this like follows:
$('#txtZip').blur(function() {
var retriever = new AddressRetriever();
var addresses = retriever.find($(this).val());
if (addresses.length > 0) {
$('#txtCity').val(addresses[0].getCity());
$('#txtState').val(addresses[0].getState());
}
});
The problem is that sometimes addresses is inexplicably empty (i.e. length = 0). In Firebug the XHR tab shows a response coming back with the expected data, and if I set an alert inside of the success method the length is correct, but outside of that method when I try to return the value, it's sometimes (but not always) empty and my textbox doesn't get populated. Sometimes it shows up as empty but the textbox gets populated properly anyways.
I know I could do this by getting rid of the separate class and stuffing the whole ajax call into the event handler, but I'm looking for a way to do this correctly so the function can be reused if necessary. Any thoughts?

In a nutshell, you can't do it the way you're trying to do it with asynchronous ajax calls.
Ajax methods usually run asynchronous. Therefore, when the ajax function call itself returns (where you have return addressList in your code), the actual ajax networking has not yet completed and the results are not yet known.
Instead, you need to rework how the flow of your code works and deal with the results of the ajax call ONLY in the success handler or in functions you call from the success handler. Only when the success handler is called has the ajax networking completed and provided a result.
In a nutshell, you can't do normal procedural programming when using asynchronous ajax calls. You have to change the way your code is structured and flows. It does complicate things, but the user experience benefits to using asynchronous ajax calls are huge (the browser doesn't lock up during a networking operation).
Here's how you could restructure your code while still keeping the AddressRetriever.find() method fairly generic using a callback function:
AddressRetriever = function() {
AddressRetriever.prototype.find = function(zip, callback) {
$.ajax({
/* setup stuff */
success: function(response) {
var addressList = [];
var data = $.parseJSON(response.value);
for (var i = 0; i < data.length; i++) {
var city = data[i].City; // "City" column of DataTable
var state = data[i].State; // "State" column of DataTable
var address = new PostalAddress(postalCode, city, state); // This is a custom JavaScript class with simple getters, a DTO basically.
addressList.push(address);
}
callback(addressList);
}
});
}
}
$('#txtZip').blur(function() {
var retriever = new AddressRetriever();
retriever.find($(this).val(), function(addresses) {
if (addresses.length > 0) {
$('#txtCity').val(addresses[0].getCity());
$('#txtState').val(addresses[0].getState());
}
});
});

AddressRetriever = function() {
AddressRetriever.prototype.find = function(zip) {
var addressList = [];
$.ajax({
/* setup stuff */
success: function(response) {
var data = $.parseJSON(response.value);
for (var i = 0; i < data.length; i++) {
var city = data[i].City; // "City" column of DataTable
var state = data[i].State; // "State" column of DataTable
var address = new PostalAddress(postalCode, city, state); // This is a custom JavaScript class with simple getters, a DTO basically.
addressList.push(address);
processAddresss(addressList);
}
}
});
}
}
function processAddresss(addressList){
if (addresses.length > 0) {
$('#txtCity').val(addresses[0].getCity());
$('#txtState').val(addresses[0].getState());
}
}
or if you want don't want to make another function call, make the ajax call synchronous. Right now, it is returning the array before the data is pushed into the array

Not inexplicable at all, the list won't be filled until an indeterminate amount of time in the future.
The canonical approach is to do the work in your success handler, perhaps by passing in your own callback. You may also use jQuery's .when.

AJAX calls are asynchroneous, which means they don't run with the regular flow of the program. When you execute
if (addresses.length > 0) {
addresses is in fact, empty, as the program did not wait for the AJAX call to complete.