Develop Reference

Best Server API and Client Side Javascript Interaction Methods?

Currently, I'm using setTimeout() to pause a for loop on a huge list so that I can add some styling to the page. For instance,
Eg: http://imdbnator.com/process?id=wtf&redirect=false
What I use setTimeOut for:
I use setTimeout() to add images,text and css progress bar (Why doesn't Progress Bar dynamically change unlike Text?2).
Clearly, as you can see it is quite painful for a user to just browse through the page and hover over a few images. It gets extremely laggy. Is there any any workaround to this?
My FOR Loop:
Each for loop makes an ajax request on the background to a PHP API. It definitely costs me some efficiency there but how do all other websites pull it off with such elegance? I mean, I've seen websites show a nice loading image with no user interference while it makes an API request. While I try to do something like that, I have set a time-out everytime.
Is that they use better Server-Client side interaction languages like the node.js that I've heard?
Also, I'e thought of a few alternatives but run into other complications. I would greatly appreciate if you can help me on each of these possible alternatives.
Method 1:
Instead of making an AJAX call to my PHP API through jQuery, I could do a complete server side script altogether. But then, the problem I run into is that I cannot make a good Client Side Page (as in my current page) which updates the progress bar and adds dynamic images after each of the item of the list is processed. Or is this possible?
Method 2: (Edited)
Like one the useful answers below, I think the biggest problem is the server API and client interaction. Websockets as suggested by him look promising to me. Will they necessarily be a better fix over a setTimeout? Is there any significant time difference in lets say I replace my current 1000 AJAX requests into a websocket?
Also, I would appreciate if there is anything other than websocket that is better off than an AJAX call.
How do professional websites get around with a fluidic server and client side interactions?
Edit 1: Please explain how professional websites (such as http://www.cleartrip.com when you are requesting for flight details) provide a smooth client side while processing the server side.
Edit 2: As #Syd suggested. That is something that I'm looking for.I think there is a lot of delay in my current client and server interaction. Websockets seem to be a fix for that. What are the other/ best ways for improving server cleint interaction apart from the standard AJAX?

Your first link doesn't work for me but I'll try to explain a couple of things that might help you if I understand your overall problem.
First of all it is bad to have synchronous calls with large amount of data that require processing in your main ui thread because the user experience might suffer a lot. For reference you might want to take a look into "Is it feasible to do an AJAX request from a Web Worker?"
If I understand correctly you want to load some data on demand based on an event.
Here you might want to sit back and think what is the best event for your need, it's quite different to make an ajax request every once in a while especially when you have a lot of traffic. Also you might want to check if your previous request has completed before you initialize the next one (this might not be needed in some cases though). Have a look at async.js if you want to create chained asynchronous code execution without facing the javascript "pyramid of doom" effect and messy code.
Moreover you might want to "validate - halt" the event before making the actual request. For example let's assume a user triggers a "mouseenter" you should not just fire an ajax call. Hold your breath use setTimeout and check if the user didn't fire any other "mouseenter" event for the next 250 ms this will allow your server to breath. Or in implementations that load content based on scroll. You should not fire an event if the user scrolls like a maniac. So validate the events.
Also loops and iterations, we all know that if the damn loop is too long and does heavy lifting you might experience unwanted results. So in order to overcome this you might want to look into timed loops (take a look at the snippet bellow). basically loops that break after x amount of time and continue after a while. Here are some references that helped me with a three.js project. "optimizing-three-dot-js-performance-simulating-tens-of-thousands-of-independent-moving-objects" and "Timed array processing in JavaScript"
//Copyright 2009 Nicholas C. Zakas. All rights reserved.
//MIT Licensed
function timedChunk(items, process, context, callback){
var todo = items.concat(); //create a clone of the original
setTimeout(function(){
var start = +new Date();
do {
process.call(context, todo.shift());
} while (todo.length > 0 && (+new Date() - start < 50));
if (todo.length > 0){
setTimeout(arguments.callee, 25);
} else {
callback(items);
}
}, 25);
}
cleartip.com will probably might use some of these techniques and from what I've seen what it does is get a chunk of data when you visit the page and then upon scroll it fetches other chunks as well. The trick here is to fire the request a little sooner before the user reaches the bottom of the page in order to provide a smooth experience. Regarding the left side filters they only filter out data that are already in the browser, no more requests are being made. So you fetch and you keep something like cache (in other scenarios though caching might be unwanted for live data feeds etc).
Finally If you are interested for further reading and smaller overhead in data transactions you might want to take a look into "WebSockets".

You must use async AJAX calls. Right now, the user interaction is blocked while the HTTP ajax request is being done.
Q: "how professional websites (such as cleartrip.com) provide a smooth client side while processing the server side."
A: By using async AJAX calls

When making multiple ajax requests, is there a "good pattern" for tracking which data the responses line up with?

I build a lot of client-side js apps and have worked mostly with Backbone. Backbone always expects your api to return a copy of the created/changed/deleted model when such ajax actions are completed, allowing it to easily determine which "model" just got updated.
I'm working in a new scenario (new framework, freshly built API) in which all I get back from the API are success/error status codes. The whole "return the entire model" seems like a crutch...but the more I work trying to keep track of these async actions the more it feels like a necessary one.
Is there some kind of convention (outside of the Backbone world, perhaps) for more easily handling this kind of thing?
Edit: I may want to point out I'm using FLUX so the uni-directional flow is preventing me from just cross-wiring everything on either side of the dispatcher.

Your question title is more clear to me than the description, so I'll go with that.
If you are having difficulties in keeping track of multiple ajax requests results, you should use Promises.
For example, with the Q library you can do something like this:
Q.all([AjaxModel1(), AjaxModel2(), AjaxModel3()])
.spread(function(resultModel1, resultModel2, resultModel3) {
});
If what you need to know is how to get a model every time you make a change to them, I would need more details about your framework/server/code.

Isn't it dangerous to have query information in javascript using breezejs?

Just starting to play with breeze.js because of the obvious gains in coding time, i.e. managing to access model data from the server direct within Javascript (I am a newbie here, so obviously bare with!).
In the past I have used the stock ajax calls to get/post data to the server, and I have used a few different client tools in the past to provide some help in querying local data, such as jLinq.
My question is this. Isn't it dangerous to have essentially full model query access in Javascript? I must be missing something because it looks like a really well thought through tool. In the past I have at least controlled what can be sent to the client via the backend query process, and again using something like jLinq etc I could filter the data etc. I can also understand the trade-off perhaps with gaining the direct query/none-duplicating local model problem, so just if anyone could provide some insight to this?
Thanks!
EDIT
Obviously I am not the only one, however I am guessing there is a reasonable response - maybe limiting the data being requested using DTO methods or something? The other question posted is here

It can be dangerous to expose the full business model. It can be dangerous to allow unrestrained querying of even that part of the model that you want to expose to the client. This is true whether you offer an easy-to-query API or one that is difficult to query.
That's why our teams are careful about how we construct our services.
You should only expose types that your client app needs. If you want to limit access to authorized instances of a type, you can write carefully prescribed non-queryable service methods. Breeze can call them just fine. You don't have to use the Breeze query facilities for every request. You'll still benefit from the caching, related-entity-navigation, change-tracking, validation, save-bundling, cache-querying, offline support.
Repeat: your service methods don't have to return IQueryable. Even when they do return IQueryable, you can easily write the service method to constrain the query results to just those entities the user is authorized to see.
Fortunately, you can blend the two approaches in the same service or in collaborating services.
Breeze gives you choices. It's up to you to exercise those choices wisely. Go out there and design your services to fit your requirements.

Breeze isn't meant to be your business logic in that sense. Keeping in mind the rule of thumb that if you do something in Javascript, anyone can do it, you ought to be restricting the visibility of your own service data as needed.
In other words, it's useful for you if you meant to make the data publicly visible anyway. But only expose the entities that you're happy exposing and allowing anyone to query; another way to look at it is that your API becomes a public API for your website (but not one you advertise and tell everyone to use).
I am personally not a fan of doing things this way as there is a dependency created on the schema of the backend implementation. If I want to make changes to my database tables, I now have to take my Javascript into consideration. I also lack in terms of integration and unit testing.
However, it can have its uses if you want to quickly build a website feature on non-sensitive data without having to build the service methods and various layers of implementation of it.

What about when you expose the Metadata? Isn't that considered dangerous. IMHO is not safe to expose metadata from the DbContext. I know you can construct metadata on the client, but the point is to do things as quickly as possible(if safe).

Lift session-valid ajax callback from a static javascript

I am currently implementing a graph visualisation tool using lift on the server side and d3 ( a javascript visualisation framework) for all the visualisation. The problem I have is that in the script I want to get session dependent data from the server.
So basically, my objective is to write lift-valid ajax callbacks in a static js script.
What I have tried so far
If you feel that the best solution is one that I already tried feel free to post a detailed answer telling me how to use it exactly and how it completely solves my problem.
Write the ajax callback in another script using lift and call it from the main script
This solution, which is similar to a hidden text input is probably the more likely to work. However it is not elegant and it would mean that I would have to load a lot of scripts on load, which is not really conveniant.
This seems to be one of the prefered solutions in the lift community as explained in this discussion on the mailing list.
REST interface
Usually what one would do to get data from a javascript function in lift is to create a REST interface. However this interface will not be linked to any session. This is the solution I got from my previous question: Get json data in d3 from lift snippet
Give function as argument of script
Another solution would be to give the ajaxcallback as an argument of the main script called to generate my graph. However I expect to have a lot of callbacks and I don't want to have to mess with the arguments of my script.
Write the whole script in lift and then serve it to the client
This solution can be elegant, however my script is very long and I would really prefer that it remainss static.
What I want
On client side
While reviewing the source code of my webpage I found that the callback for an ajaxSelect is:
<select onchange="liftAjax.lift_ajaxHandler('F966066257023LYKF4=' + encodeURIComponent(this.value), null, null, null)" name="F96606625703QXTSWU" id="node_delete" class="input">
Moreover, there is a variable containing the state of the page in the end of the webpage:
var lift_page = "F96606625700QRXLDO";
So, I am wondering if it is possible to simulate that my ajaxcall is valid using this liftAjax.lift_ajaxHandler function. However I don't know the exact synthax to use.
On server side
Since I "forged" a request on client side, I would now like to get the request on client side and to dispatch it to the correct function. This is where the LiftRules.dispatch object seems the best solution: when it is called, all the session management has been made (the request is authentified and linked to a session), however I don't know how to write the correct piece of code in the append function.
Remark
In lift all names of variables are changed to a random string in order to increase the security, I would like to have the same behavior in my application even if that will probably mean that I will have to "give" the javascript these values. However an array of 15 string values is still a better tradeoff than 15 functions as argument of a javascript function.
Edit
While following my research I found this page : Mapping server functions to client actions which somehow explains the goal of named functions even if it stil didn't lead me to a working solution.

Quick Answer
Rest in Lift does not have to be stateless. If you register your RestHelper with LiftRules.dispatch.append, then it will be handled statefully and Session information will be available through the S object as usual.
Long Answer
Since you seem interested, and it's come up on SO before, here's a more detailed explanation of how server-side functions are registered and called in Lift. If you haven't worked with Lift for some time, look away. What follows should not in any way be used to evaluate Lift or its complexity. This is purely library developer level stuff and a majority of Lift users go about their development blissfully unaware of it.
How it works
When you create stateful callbacks, typically by using the methods within the SHtml object, what you are really doing is registering objects of type S.AFuncHolder within the context of the users session, each with a unique ID. The unique ID that was generated during this process is what you're seeing when you come across a pattern like F96606625700QRXLDO. When data is submitted, via form post, ajax, or whatever, Lift will check the request for these function ids and execute the associated function if they exist. There are several helpers that provide more specific types of AFuncHolder, like S.SFuncHolder (accepts a single string query parameter) and S.BinFuncHolder (parameter is multipart form data) but they all return Any and behind the scenes Lift will collect those return values to create the proper type of response. A JsCmd, for instance, will result in a JavaScriptResponse that executes the command. You can also return a LiftResponse directly.
How to use it
AFuncHolders are registered using the S.fmapFunc method. You'd call it like this
S.fmapFunc(SFuncHolder({ (str: String) =>
doSomethingAwesomeWithAString(str)
}))(id => <input type="text" name={id} value=""/>)
The first parameter is your function, wrapped in the proper *FuncHolder type and the second parameter is a function that takes the generated id and outputs something. The something that gets output is what you will include on the page. It should somehow result in the id being sent to the server as a query parameter so that your function is executed.
Putting it all together
You could use the above to make your own Ajax calls, but when Lift makes an ajax call there are a few other considerations:
1) Most browsers only allow so many simultaneous connections to a given domain. Three seems to be the magic number.
2) AFuncHolders will often close over the scope of the snippet they are contained within and if multiple ajax requests are handled at once, each in its own thread, bad things can happen.
To combat these issues, the liftAjax.lift_ajaxHandler function queues each ajax request, ensuring that only one at a time is sent to the server.
The drawback to this approach is that it can make it difficult to make an Ajax call where the result needs to be passed to a callback. JQuery autocomplete, for instance, provides a callback function when input changes that accepts a list of matches. If you are manually calling LiftAjax.lift_ajaxHandler though, you can provide your own callback functions for success & error and I would recommend that you look at the source of those functions in your browser for more information on how they work.
There's actually more to it, like how Lift restores RequestVars on ajax callbacks (which is where the lift_page comes in, but that's about all I'm prepared to explain over coffee on a Saturday morning :)
Good luck with your app!

XmlHttpRequest return values

I'm looking for (arguably) the correct way to return data from a XmlHttpRequest. Options I see are:
Plain HTML. Let the request format the data and return it in a usable format.
Advantage: easy to consume by the calling page.Disadvantage: Very rigid, stuck with a fixed layout.
XML. Let the request return XML, format it using XSLT on the calling page.Advantage: the requested service is easily consumed by other sources.Disadvantage: Is browser support for XSLT good enough?
JSON. Let the request return JSON, consume it using javascript, render HTML accordingly.Advantage: easier to 'OO-ify' the javascript making the request. Disadvantage: Probably not as easy to use as the previous two options.
I've also thought about going for option one while abstracting the view logic in the called service in such a way that switching in and out different layouts would be trivial. Personally I think this option is the best out of three, for compatibility reasons.
While typing this, I got another insight. Would it be a good idea to allow all three response formats, based on a parameter added to the request?

If you're looking for a quick solution that should work with most available frameworks, I'd go for JSON. It's easy to start with and works.
If you're trying to build a larger application that you're going to extend (in terms of size or maybe your own API for 3rd party extensions) I'd go for XML. You could write a proxy to provide the information in JSON or HTML too, but having XML as the main source is definitly worth the time and effort while building the app.
Like #John Topley said: it depends.

I'd agree with John Topley - it depends on the application. There's a good article on quirksmode that discusses the advantages and disadvantages of each format that you might want to read: http://www.quirksmode.org/blog/archives/2005/12/the_ajax_respon.html

I think this sort of depends on the level of "ajaxyness" your app is going to have. If your front end is a "rich client", al'a gmail, I'd go with the JSON solution, as you'd have to solve the problem of having client side view generation anyway. If you're using ajax sparingly, to provide simple messages to the user, update a few fields now and then, etc, then I'd go with option 1, since most of your view logic is already on the server.

I think trying to use XmlHttpRequest will be a huge headache, unless its the type of headache you don't mind - to do it properly you're almost reinventing the wheel. Then again, people like to reinvent wheels in their spare time, just to say, "Hey, I did it". Not me...
I would get a framework like prototype or Extjs, that has alot of data loading functions built in for XML and JSON, plus you'll get more predictable results, as the frameworks have event handlers to make sure your XmlHttpRequest succeeded or failed. Plus you get support for all the various browsers.

Different response format shouldn't be difficult to produce.
JSON works best for me, i like to keep OO in js, and don't know how to parse XML well :)