I'm learning FRP using Bacon.js, and would like to assemble data from a paginated API in a stream.
The module that uses the data has a consumption API like this:
// UI module, displays unicorns as they arrive
beautifulUnicorns.property.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
The module that assembles the data requests sequential pages from an API and pushes onto the stream every time it gets a new data set:
// beautifulUnicorns module
var curPage = 1
var stream = new Bacon.Bus()
var property = stream.toProperty()
var property.onValue(function(){}) # You have to add an empty subscriber, otherwise future onValues will not receive the initial value. https://github.com/baconjs/bacon.js/wiki/FAQ#why-isnt-my-property-updated
var allUnicorns = [] // !!! stateful list of all unicorns ever received. Is this idiomatic for FRP?
var getNextPage = function(){
/* get data for subsequent pages.
Skipping for clarity */
}
var gotNextPage = function (resp) {
Array.prototype.push.apply(allUnicorns, resp) // just adds the responses to the existing array reference
stream.push(allUnicorns)
curPage++
if (curPage <= pageLimit) { getNextPage() }
}
How do I subscribe to the stream in a way that provides me a full list of all unicorns ever received? Is this flatMap or similar? I don't think I need a new stream out of it, but I don't know. I'm sorry, I'm new to the FRP way of thinking. To be clear, assembling the array works, it just feels like I'm not doing the idiomatic thing.
I'm not using jQuery or another ajax library for this, so that's why I'm not using Bacon.fromPromise
You also may wonder why my consuming module wants the whole set instead of just the incremental update. If it were just appending rows that could be ok, but in my case it's an infinite scroll and it should draw data if both: 1. data is available and 2. area is on screen.
This can be done with the .scan() method. And also you will need a stream that emits items of one page, you can create it with .repeat().
Here is a draft code (sorry not tested):
var itemsPerPage = Bacon.repeat(function(index) {
var pageNumber = index + 1;
if (pageNumber < PAGE_LIMIT) {
return Bacon.fromCallback(function(callback) {
// your method that talks to the server
getDataForAPage(pageNumber, callback);
});
} else {
return false;
}
});
var allItems = itemsPerPage.scan([], function(allItems, itemsFromAPage) {
return allItems.concat(itemsFromAPage);
});
// Here you go
allItems.onValue(function(allUnicorns){
console.log("Got "+ allUnicorns.length +" Unicorns");
// ... some real display work
});
As you noticed, you also won't need .onValue(function(){}) hack, and curPage external state.
Here is a solution using flatMap and fold. When dealing with network you have to remember that the data can come back in a different order than you sent the requests - that's why the combination of fold and map.
var pages = Bacon.fromArray([1,2,3,4,5])
var requests = pages.flatMap(function(page) {
return doAjax(page)
.map(function(value) {
return {
page: page,
value: value
}
})
}).log("Data received")
var allData = requests.fold([], function(arr, data) {
return arr.concat([data])
}).map(function(arr) {
// I would normally write this as a oneliner
var sorted = _.sortBy(arr, "page")
var onlyValues = _.pluck(sorted, "value")
var inOneArray = _.flatten(onlyValues)
return inOneArray
})
allData.log("All data")
function doAjax(page) {
// This would actually be Bacon.fromPromise($.ajax...)
// Math random to simulate the fact that requests can return out
// of order
return Bacon.later(Math.random() * 3000, [
"Page"+page+"Item1",
"Page"+page+"Item2"])
}
http://jsbin.com/damevu/4/edit
Related
Script must log postback information about call detail from zvonok.com to google spreadsheets. I has write function which only append row to sreadsheet - no update or modify of any cell in code and during few manual test calls rows has been append correct, but when my client began his usual call campaign, calls and postbacks going very often one after other, values in last row began changing few times and in some cases leave strange values
I seen behavior like this first time made short video record:
https://youtu.be/0_H_mVAbp4g
here is one column with strange value
2103052006092385
2,10305E+15
210305412464544
I have found 9 cases from 248 rows.
Client has show me excel from his user cabinet, totally was maded 5649 calls, so in google spreadsheets must be 5649 rows instead 248.
function getJsonFromUrl(url) {
var query = url;
var result = {};
if (query == undefined){
return result;
}
query.split("&").forEach(function(part) {
var item = part.split("=");
result[item[0]] = decodeURIComponent(item[1]);
});
return result;
}
function doGet(e){
const ctCompl = 'ct_completed';
var doc = SpreadsheetApp.openById(SHEET_KEY);
var sheet = doc.getSheetByName(SHEET_NAME);
var row = [];
if(typeof e !== undefined){
mArr = getJsonFromUrl(e.queryString);
for (i in mArr) if( i == ctCompl) {
row.push(convTimeLong(mArr[i]));
} else
row.push(mArr[i]);
sheet.appendRow(row);
} else {
sheet.appendRow(['e undefined!']);
}
SpreadsheetApp.flush();
return handleResponse(e)
}
function convTimeLong(dateTime) {
let d = new Date();
let dt=dateTime.replace('+', 'T');
try {
var res = Utilities.formatDate(d,"GMT+2", "dd.MM.yyyy HH:mm");
return res
} catch(e){
return dateTime; }
}
executions dashboard show status "completed' everywhere, execution time longest - 1.688 s
Client has set delay 5 second between call's, right now I don't now is percent of lost postback's decreased after delay was set or not, but it still very high.
https://youtu.be/0_H_mVAbp4g
In general, using Google Sheets as a database is a bad idea. It's not designed for this so it could fail really bad. Using a proper database will make everything much, much easier. If you are using the spreadsheet to then cook the data, I'd advise to use a function that imports data like IMPORTXML (see reference).
That being said, if you insist on using Sheets, you could try using locks:
function appendRow(sheet, row) {
const lock = LockService.getScriptLock()
while (!lock.tryLock(100000)) /* Spin the lock until it gets aquired */;
try {
sheet.appendRow(row)
SpreadsheetApp.flush()
} finally {
lock.releaseLock()
}
}
To use it, you only need to pass the sheet and the values to add: sheet.appendRow(row) to appendRow(sheet, row).
It will make sure that entries don't get overridden. Note that this will slow down the code a lot and the script can time out if there are a lot of requests.
i have a Meteor Application which is very "slow" as there are a lot of API-Calls.
What i try to do is to break apart the loading/calls.
What i just did is:
i have loading template via iron-router
i waitOn for the first API-Call has finished
then i start the next API-calls in the Template.myTemplate.rendered - function
This was already a big benefit for the speed of my Application, but i want to break it up even more as the second call is in fact more like 5-25 API-calls.
So what i try to do now is inside the rendered function is a self-calling function which calls itself as long as there are no more to do and saves the response inside a session. (Until now it just rewrites, but even to this point i can´t get)
Template.detail.rendered = function(){
//comma separated list of numbers for the API-Call
var cats = $(this.find(".extra")).attr('data-extra').split(',');
var shop = $(this.find(".extra")).attr('data-shop');
var counter = 0;
var callExtras = function(_counter){
var obj = {
categories : [cats[_counter]],
shop : shop
};
if(_counter <= cats.length){
Meteor.subscribe('extra', obj,function(result){
//TODO dickes todo... nochmal nachdenken und recherchieren
//console.log(_counter);
Session.set('extra',Extra.find('extra').fetch()[0].results);
counter++;
callExtras(counter);
});
}
};
callExtras(counter);
Session.set('loading_msg', '' );
};
Now i have again problems with my reactive parts of the app desscribed here - Meteor: iron-router => waitOn without subscribe As i can´t find a proper way to update my client-side per user base collection. Also in the docs it is described the publish method also creates a new collection. (The new document´s ID) here - http://docs.meteor.com/#/full/publish_added
here is the publish from server
Meteor.publish('extra', function(obj){
var that = this;
Meteor.call('extra', obj, function(error, result){
if (result){
//console.log(result);
that.added("extra", "extra", {results: result});
//that.changed('extra','extra',{results: result});
that.ready();
} else {
//that.ready();
}
});
});
So my question is: Is there from scratch a better way to structuring my code means solving the problem somehow different? If not how can i achive it the cleanest way? Because for my understanding this is just strange way to do it.
EDIT:
For example.
Can i do a per-user-collection (maybe only client-side like now) and push data from the server and just subscribe to this collection? But then how can i check when the async API-Call has finshed to start the next round. So the view gets data piece by piece. I am just confused right now.
My fault was simple as i thaught: You don´t need to use subscribe.
I just added "error,result" in the callback of Meteor.call
Only "result" leads to the result is always undefined.
var cats = $(this.find(".extra")).attr('data-extra').split(',');
var shop = $(this.find(".extra")).attr('data-shop');
var counter = 0;
var callExtras = function(_counter){
var obj = {
categories : [cats[_counter]],
shop : shop
};
if(_counter <= cats.length){
Meteor.call('extra', obj,function(error,result){
var actual_session = Session.get('extra');
if(actual_session === false){
actual_session = [];
}
actual_session = actual_session.concat(result);
Session.set('extra',actual_session);
counter++;
callExtras(counter);
});
}
};
callExtras(counter);
Then in the template helper
"extra" : function(){
return Session.get('extra');
},
In my Win 8 app, based on a blank template, I have successfully added search contract and it seems to work despite the fact that I have not linked it to any data yet, so, for now, when I search any term in my app it simply takes me to the searchResults page with the message "No Results Found" this is what I was expecting initially.
Now what I wish to do is link my database into the searchResults.js file so that I can query my database. Now outside of the search contract I have tested and connected my Db and it works; I did this using WinJS.xhr, to connect to my web-service which in turn queries my database and returns a JSON object.
In my test I only hardcoded the url, however I now need to do two things. Move the test WinJS.xr data for connecting my DB into the search contract code, and second - change the hardcoded url to a dynamic url that accepts the users search term.
From what I understand of Win 8 search so far the actual data querying part of the search contract is as follows:
// This function populates a WinJS.Binding.List with search results for the provided query.
_searchData: function (queryText) {
var originalResults;
// TODO: Perform the appropriate search on your data.
if (window.Data) {
originalResults = Data.items.createFiltered(function (item) {
return (item.termName.indexOf(queryText) >= 0 || item.termID.indexOf(queryText) >= 0 || item.definition.indexOf(queryText) >= 0);
});
} else {`enter code here`
originalResults = new WinJS.Binding.List();
}
return originalResults;
}
});
The code that I need to transfer into this section is as below; now I have to admit I do not currently understand the code block above and have not found a good resource for breaking it down line by line. If someone can help though it will be truly awesome! My code below, I basically want to integrate it and then make searchString be equal to the users search term.
var testTerm = document.getElementById("definition");
var testDef = document.getElementById("description");
var searchString = 2;
var searchFormat = 'JSON';
var searchurl = 'http://www.xxx.com/web-service.php?termID=' + searchString +'&format='+searchFormat;
WinJS.xhr({url: searchurl})
.done(function fulfilled(result)
{
//Show Terms
var searchTerm = JSON.parse(result.responseText);
// var terms is the key of the object (terms) on each iteration of the loop the var terms is assigned the name of the object key
// and the if stament is evaluated
for (terms in searchTerm) {
//terms will find key "terms"
var termName = searchTerm.terms[0].term.termName;
var termdefinition = searchTerm.terms[0].term.definition;
//WinJS.Binding.processAll(termDef, termdefinition);
testTerm.innerText = termName;
testDef.innerText = termdefinition;
}
},
function error(result) {
testDef.innerHTML = "Got Error: " + result.statusText;
},
function progress(result) {
testDef.innerText = "Ready state is " + result.readyState;
});
I will try to provide some explanation for the snippet that you didn't quite understand. I believe the code you had above is coming from the default code added by Visual Studio. Please see explanation as comments in line.
/**
* This function populates a WinJS.Binding.List with search results
* for the provided query by applying the a filter on the data source
* #param {String} queryText - the search query acquired from the Search Charm
* #return {WinJS.Binding.List} the filtered result of your search query.
*/
_searchData: function (queryText) {
var originalResults;
// window.Data is the data source of the List View
// window.Data is an object defined in YourProject/js/data.js
// at line 16 WinJS.Namespace.define（"Data" ...
// Data.items is a array that's being grouped by functions in data.js
if (window.Data) {
// apply a filter to filter the data source
// if you have your own search algorithm,
// you should replace below code with your code
originalResults = Data.items.createFiltered(function (item) {
return (item.termName.indexOf(queryText) >= 0 ||
item.termID.indexOf(queryText) >= 0 ||
item.definition.indexOf(queryText) >= 0);
});
} else {
// if there is no data source, then we return an empty WinJS.Binding.List
// such that the view can be populated with 0 result
originalResults = new WinJS.Binding.List();
}
return originalResults;
}
Since you are thinking about doing the search on your own web service, then you can always make your _searchData function async and make your view waiting on the search result being returned from your web service.
_searchData: function(queryText) {
var dfd = new $.Deferred();
// make a xhr call to your service with queryText
WinJS.xhr({
url: your_service_url,
data: queryText.toLowerCase()
}).done(function (response) {
var result = parseResultArrayFromResponse(response);
var resultBindingList = WinJS.Binding.List(result);
dfd.resolve(result)
}).fail(function (response) {
var error = parseErrorFromResponse(response);
var emptyResult = WinJS.Binding.List();
dfd.reject(emptyResult, error);
});
return dfd.promise();
}
...
// whoever calls searchData would need to asynchronously deal with the service response.
_searchData(queryText).done(function (resultBindingList) {
//TODO: Display the result with resultBindingList by binding the data to view
}).fail(function (resultBindingList, error) {
//TODO: proper error handling
});
I'm making a fairly complex HTML 5 + Javascript game. The client is going to have to download images and data at different points of the game depending on the area they are at. I'm having a huge problem resolving some issues with the Data Layer portion of the Javascript architecture.
The problems I need to solve with the Data Layer:
Data used in the application that becomes outdated needs to be automatically updated whenever calls are made to the server that retrieve fresh data.
Data retrieved from the server should be stored locally to reduce any overhead that would come from requesting the same data twice.
Any portion of the code that needs access to data should be able to retrieve it easily and in a uniform way regardless of whether the data is available locally already.
What I've tried to do to accomplish this is build a data layer that has two main components:
1. The portion of the layer that gives access to the data (through get* methods)
2. The portion of the layer that stores and synchronizes local data with data from the server.
The workflow is as follows:
When the game needs access to some data it calls get* method in the data layer for that data, passing a callback function.
bs.data.getInventory({ teamId: this.refTeam.PartyId, callback: this.inventories.initialize.bind(this.inventories) });
The get* method determines whether the data is already available locally. If so it either returns the data directly (if no callback was specified) or calls the callback function passing it the data.
If the data is not available, it stores the callback method locally (setupListener) and makes a call to the communication object passing the originally requested information along.
getInventory: function (obj) {
if ((obj.teamId && !this.teamInventory[obj.teamId]) || obj.refresh) {
this.setupListener(this.inventoryNotifier, obj);
bs.com.getInventory({ teamId: obj.teamId });
}
else if (typeof (obj.callback) === "function") {
if (obj.teamId) {
obj.callback(this.team[obj.teamId].InventoryList);
}
}
else {
if (obj.teamId) {
return this.team[obj.teamId].InventoryList;
}
}
}
The communication object then makes an ajax call to the server and waits for the data to return.
When the data is returned a call is made to the data layer again asking it to publish the retrieved data.
getInventory: function (obj) {
if (obj.teamId) {
this.doAjaxCall({ orig: obj, url: "/Item/GetTeamEquipment/" + obj.teamId, event: "inventoryRefreshed" });
}
},
doAjaxCall: function (obj) {
var that = this;
if (!this.inprocess[obj.url + obj.data]) {
this.inprocess[obj.url + obj.data] = true;
$.ajax({
type: obj.type || "GET",
contentType: "application/json; charset=utf-8",
dataType: "json",
data: obj.data,
url: obj.url,
async: true,
success: function (data) {
try {
ig.fire(bs.com, obj.event, { data: data, orig: obj.orig });
}
catch (ex) {
// this enables ajaxComplete to fire
ig.log(ex.message + '\n' + ex.stack);
}
finally {
that.inprocess[obj.url + obj.data] = false;
}
},
error: function () { that.inprocess[obj.url + obj.data] = false; }
});
}
}
The data layer then stores all of the data in a local object and finally calls the original callback function, passing it the requested data.
publishInventory: function (data) {
if (!this.inventory) this.inventory = {};
for (var i = 0; i < data.data.length; i++) {
if (this.inventory[data.data[i].Id]) {
this.preservingUpdate(this.inventory[data.data[i].Id], data.data[i]);
}
else {
this.inventory[data.data[i].Id] = data.data[i];
}
}
// if we pulled this inventory for a team, update the team
// with the inventory
if (data.orig.teamId && this.team[data.orig.teamId]) {
this.teamInventory[data.orig.teamId] = true;
this.team[data.orig.teamId].InventoryList = [];
for (var i = 0; i < data.data.length; i++) {
this.team[data.orig.teamId].InventoryList.push(data.data[i]);
}
}
// set up the data we'll notify with
var notifyData = [];
for (var i = 0; i < data.data.length; i++) {
notifyData.push(this.inventory[data.data[i].Id]);
}
ig.fire(this.inventoryNotifier, "refresh", notifyData, null, true);
}
There are several problems with this that bother me constantly. I'll list them in order of most annoying :).
Anytime I have to add a call that goes through this process it takes too much time to do so. (at least an hour)
The amount of jumping and callback passing gets confusing and seems very prone to errors.
The hierarchical way in which I am storing the data is incredibly difficult to synchronize and manage. More on that next.
Regarding issue #3 above, if I have objects in the data layer that are being stored that have a structure that looks like this:
this.Account = {Battles[{ Teams: [{ TeamId: 392, Characters: [{}] }] }]}
this.Teams[392] = {Characters: [{}]}
Because I want to store Teams in a way where I can pass the TeamId to retrieve the data (e.g. return Teams[392];) but I also want to store the teams in relation to the Battles in which they exist (this.Account.Battles[0].Teams[0]); I have a nightmare of a time keeping each instance of the same team fresh and maintaining the same object identity (so I am not actually storing it twice and so that my data will automatically update wherever it is being used which is objective #1 of the data layer).
It just seems so messy and jumbled.
I really appreciate any help.
Thanks
You should consider using jquery's deferred objects.
Example:
var deferredObject = $.Deferred();
$.ajax({
...
success: function(data){
deferredObject.resolve(data);
}
});
return deferredObject;
Now with the deferredObject returned, you can attach callbacks to it like this:
var inventoryDfd = getInventory();
$.when(inventoryDfd).done(function(){
// code that needs data to continue
}
and you're probably less prone to errors. You can even nest deferred objects, or combine them so that a callback isn't called until multiple server calls are downloaded.
+1 for Backbone -- it does some great heavy lifting for you.
Also look at the Memoizer in Douglas Crockford's book Javascript the Good Parts. It's dense, but awesome. I hacked it up to make the memo data store optional, and added more things like the ability to set a value without having to query first -- e.g. to handle data freshness.
I'm trying to make an HTTP GET request using the jQuery get() function, but I'm having some trouble.
Here's what my code looks like:
// get the links on the page
var pageLinks = $.find('#pageLinks');
// loop through each of the links
$(pageLinks).find('a').each(function(){
if($(this).attr('title') !== "Next Page"){
// make a GET request to the URL of this link
$.get($(this).attr("href"), function(data) {
console.log("here");
var temp = parse_page(data);
// concatenate the return string with another
bdy = bdy+String(temp);
console.log("done");
});
}
});
There are multiple pages that I need to get data from. Since the get() function is asynchronous, I get the pages in a random order. Secondly, the concatenation does not work. Even though I get each of the pages, they're not put into bdy.
Can anyone suggest how I might deal with this?
Thanks a lot!!
Construct bdy after all pages are retrieved, i.e. store get results in a dictionary or array; wait for all gets to finish; then assemble them in the correct order.
I tried this one and it works:
// get the links on the page
var pageLinks = $('a');
var bdy
// loop through each of the links
$(pageLinks).each(function(){
console.log(this);
// make a GET request to the URL of this link
$.get($(this).attr("href"), function(data) {
// concatenate the return string with another
bdy = bdy + data.toString();
console.log(bdy);
});
});
As an example of what #muratgu has said:
var results = [];
var count = 0;
function allDone() {
var bdy = results.join("");
// do stuff with bdy
}
// get the links on the page
var pageLinks = $.find('#pageLinks');
// filter the links so we're left with the links we want
var wantedLinks = $(pageLinks).find('a').filter(function (idx) {
return $(this).attr('title') !== "Next Page";
});
// remember how many links we're working on
count = wantedLinks.length;
// loop through each of the links
wantedLinks.each(function (idx) {
// make a GET request to the URL of this link
$.get($(this).attr("href"), function (data) {
console.log("here");
var temp = parse_page(data);
results[idx] = temp;
// Decrement the count.
count--;
if (count === 0) {
// All done.
allDone();
}
});
});
You could go further and abstract this into a data type that can perform N async downloads, and then notify you when all are complete.
I just found that there are modules that allow one to manage the control flow in JS. The ones I found are:
Async
Step
For help using the above modules, see my follow up question here.