I'm trying to get a list of tweets (in this case 25 with a specific hashtag) using $.ajax and twitter's search functionality.
There's 2 things that are a bit unclear to me:
How long is the duration for cache: true (how long does it take untill it sends a new request instead of using the cache)? At the moment it seems to be every few seconds but I'd like to know the precise duration and if/how this can be altered.
This retrieving the information from twitter will be done by about 2000-3000 people at random moments, but probably not by that same person for a little while after. How will twitter respond to this (block the IP address because of too many requests maybe)? This is why I want to cache the information for about 1-5 minuutes, to lessen the amount of requests.
TL;DR: I'm making alot of requests to the twitter search functionality, how do I lessen the load?
The cache time for $.ajax is most likely browser dependant so you should not rely on this to balance your load against the Twitter API. This caching can be turned off by passing { cache: false } into the settings parameter. Instead you should implement your own cacheing system.
Twitter has a page describing this: https://dev.twitter.com/docs/rate-limiting.
It says that it sends some headers to tell you how many requests you have left within a certain timespan. What you will have to do is just cache long enough so that you do no exceed the limits Twitter imposes on you. You should do some testing to see how different load on your service will affect the total number of calls to the Twitter API and design your cache system accordingly. The most optimal solution here will probably be something like from a to b'o-clock i will need to cache for c minutes and from d to e'o-clock i will need to cache for f minutes. Another solution might be to implement a system to keep track of load as it is happening and determine cache times in real time. If both of these solutions seams to daunting you can probably get away with just picking a long enough cache time so that you will never exceed the limit.
Related
I've got an application that I intend to set a lock flag in my database that would exclude others from viewing that same page if set.
However, once set - I have no idea how to "unset" it. I could make it up to the user to unset the flag, but that seems unnecessary.
I'd want to simply look for the browser to leave the page, make a call to the database, and unlock the page.
How does one do this "type" (not looking for the exact way) of thing with JSF/Javascript/jQuery (all options)
There's really not a reliable way to do this, that I've seen anyway.
You can use the browser's onbeforeunload event to tell the server, "Hey I'm leaving the page now.". The issue is you can't actually block the page from unloading. If the user is actually closing the browser, any open sockets are going to be closed immediately. Your web server may or may not get the request in time. I've had very flaky results with this approach.
One approach that might work is to employ some sort of timeout mechanism. The page would ping the server every 30 seconds or what not, saying "I'm still here." If the server did not get this update after a few minutes, it would invalidate the session and free up that document. Perhaps this could be optimized by checking for the last ping when someone new came along. One issue with this is if someone left the page, the next user might have to wait a minute or two before they could go to the page. You'd then have to find a ping frequency that doesn't flood your server with traffic, but also doesn't make the next user have to wait too long.
It's also possible to combine these two methods. When the user leaves the page, trap the onbeforeunload event and immediately invalidate the session. However, if it didn't work, the session would time out after a minute of not being pinged.
Are there better solutions?
If you really need to lock a document in a web app so multiple users can't edit it, you might want to investigate your overall design. Are you afraid of users clobbering data? If so, maybe employ a mechanism that can resolve merge conflicts, or detect if both sets of changes can be combined.
If you wanted to go truly Web 2.0, you could design something similar to Google Docs, where changes appear live as they're made. No need for a Save button anywhere!
Sending a "keep-alive signal" might be an option. Something along these lines on the frontend side, combined with session cookies.
setInterval(function() {
var img = new Image();
var src = "http://examle.com/keepalive.gif?cachebuster=" +
Math.ceil(Math.random() * 10000 );
}, 1000);
I have created a website for a friend. Because he wished to have a music player continue to play music through page loads, I decided to load content into the page via ajax (facilitated by jQuery). It works fine, it falls back nicely when there is no javascript, and the back/forward buttons are working great, but it's dreadfully slow on the server.
A couple points:
The initial page load is fairly quick. The Chrome developer console tells me that "index.php" loads in about 2.5 seconds. I have it set up so that query string params dictate which page is loaded, and this time frame is approximately accurate for them all. For the homepage, there is 8.4KB of data loaded.
When I load the content in via an ajax request, no matter the size of the data downloaded, it takes approximately 20 seconds. The smallest amount of data that is loaded in this way is about 500 bytes. There is obviously a mismatch here.
So Chrome tells me that the vast majority of the time spent is "waiting" which I take to mean that the server is dealing with the request. So, that can only mean, I guess, that either my code is taking a long time, or something screwy is going on with the server. I don't think it's my code, because it's fairly minimal:
$file = "";
if (isset($_GET['page'])) {
$file = $_GET['page'];
} else if (isset($_POST['page'])) {
$file = $_POST['page'];
} else {
$file = "home";
}
$file = 'content/' . $file . '.php';
if (file_exists($file)) {
include_once($file);
} else {
include_once('content/404.php');
}
This is in a content_loader.php file which my javascript (in this scenario) sends a GET request to along with a "page" parameter. HTML markup is returned and put into a DIV on the page.
I'm using the jQuery .get() shorthand function, so I don't imagine I could be messing anything up there, and I'm confident it's not a Javascript problem because the delay is in waiting for the data from the server. And again, even when the data is very small, it takes about 20 seconds.
I currently believe it's a problem with the server, but I don't understand why a request made through javascript would be so much slower than a request made the traditional way through the browser. As an additional note, some of the content pages do connect to a MySQL database, but some do not. It doesn't seem to matter what the page requires for processing or how much data it consists of, it takes right around 20 seconds.
I'm at a loss... does anyone know of anything that might explain this? Also, I apologize if this is not the correct place for such a question, none of the other venues seemed particularly well suited for the question either.
As I mentioned in my comment, a definite possibility could be reverse DNS lookups. I've had this problem before and I bet it's the source of your slow requests. There are certain Apache config directives you need to watch out for in both regular apache and vhost configs as well as .htaccess. Here are some links that should hopefully help:
http://www.tablix.org/~avian/blog/archives/2011/04/reverse_dns_lookups_and_apache/
http://betabug.ch/blogs/ch-athens/933
To find more resources just Google something like "apache slow reverse dns".
A very little explanation
In a reverse DNS lookup an attempt is made to resolve an IP address to a hostname. Most of the time with services like Apache, SSH and MySQL this is unnecessary and it's a bad idea as it only serves to slow down requests/connections. It's good to look for configuration settings for your different services and disable reverse DNS lookups if they aren't needed.
In Apache there are certain configuration settings that cause a reverse lookup to occur. Things like HostnameLookups and allow/deny rules specifying domains instead of IP addresses. See the links above for more info.
As you suggested in your comment, the PHP script is executing quickly once it finally runs. The time is spent waiting on Apache - most likely to do a reverse DNS lookup, and failing. You know the problem isn't with your code, it's with the other services involved in the request.
Hope this helps!
Right now I'm using AJAX to pull in a list of active streams (TwitchTV) and it's viewers and I'm requesting this every second. At time the list of streams to check can get quite lengthy so I plan on splitting the ajax requests into 2 or 3 parts:
1) Get Number of Viewers for Current Stream (Check every 1 Second)
2) Split Stream in Half and Check 1st Half of List for Active Streamers (Check every 5 Seconds)
3) Check 2nd Half of List for Active Streamers (Check every 5 Seconds)
so I would have 3 requests running simultaneously but I'm worried about what the load time will come down to. Since it is constantly pulling in data would it make the page slower? Would the user likely notice? Is it better to keep 1 ajax request for big amounts of data or is it better to use multiple ajax requests for smaller pieces of data? Is ajax really the best thing to pull in constantly changing live data?
The answer to your various questions is probably "It depends":
The ajax requests by themselves shouldn't make anything slower. These are asynchronous requests, so they will only actually cause the user's browser any significant (and probably still not noticeable) load when the request completes.
One thing that could potentially slow your app down (or cause the user to notice in an unpleasant way) is the DOM manipulation when the request completes. Changing your current number of streaming users in-place probably won't hurt, but depending on the number of streams/how you are displaying them in a list, redrawing this could potentially be very expensive/cause lag on things like redraw.
An alternative to using Ajax (depending on what browsers you wish to support) is to use websockets. This way you can keep a connection open and the server can tell the application when the data needs to change, instead of the need to poll for it.
Why do you need to break your list up into a first half and a second half?
One way to cut down on the amount of data you're sending back and forth might be to send some sort of signal indicating the last bit of data you received. For example, when your timeline on twitter.com updates every few seconds, the ajax request sends along the id of the most recent tweet it received, so that the server knows not to waste time sending any data older than that. Depending on your use case this might be effective.
Why do some web-applications use the http-get parameter rnd? What is the purpose of it? What problems are solved by using this parameter?
This could be to make sure the page/image/whatever isn't taken from the user's cache. If the link is different every time then the browser will get it from the server rather than from the cache, ensuring it's the latest version.
It could also be to track people's progress through the site. Best explained with a little story:
A user visits example.com. All the links are given the same random number (let's say 4).
The user opens a link in a new window/tab, and the link is page2.php?rnd=4. All the links in this page are given the random number 7.
The user can click the link to page3.php from the original tab or the new one, and the analytics software on the server can tell which one by whether it has rnd=4 or rnd=7.
All we can do is suggest possibilities though. There's no one standard reason to put rnd= in a URL, and we can't know the website designer's motives without seeing the server software.
Internet Explorer and other browsers will read an image URL, download the image, and store it in a cache.
If your application is going to be updating the image regular, and so you want your users to not see a cached image, the URL needs to be unique each time.
Therefore, adding a random string ensures this will be unique and downloaded into the cache each time.
It's almost always for cache-busting.
As has been suggested by others. This kind of behaviour is usually used to avoid caching issues when you are calling a page that returns dynamic content data.
For example, say you have a page that gets some current user information such as "mysite.com/CurrentUserData". Now on the first call to this page, the user data will be returned as expected, but depending on the timing and caching settings, the second call may return the same data - even though the expected data may have been updated.
The main reason for caching is of course to optimise the speed of frequent request. But in the instance where this is not wanted, adding a random value as a query string parameter is known to be a widely used solution.
There are however other ways to get around this issue. For example if you were doing an Ajax request with javascript/JQuery. You could set the cache to false in your call...
$.ajax({url: 'page.html', cache: false});
you could also change it for all page calls on document load with...
$.ajaxSetup({cache: false}});
If you were to do an MVC application, you can even disable the caching on the control action methods with an attribute like so...
[OutputCache(NoStore = true, Duration = 0, VaryByParam = "*")]
public ActionResult NonCacheableData()
{
return View();
}
(thanks to a quick copy and paste from here)
I dare say there are also settings in IIS you could apply to get the same affect - though I have not been that far with this yet.
I implemented a REST service and i'm using a web page as client.
My page has some javascript functions that performs several times the same http get request to REST server and process the replies.
My problem is that the browser caches the first reply and not actualy sends the following requests..
Is there some way to force the browser execute all the requests without caching?
I'm using internet explorer 8.0
Thanks
Not sure if it can help you, but sometimes, I add a random parameter in the URL of my request in order to avoid being cached.
So instead of having:
http://my-server:8080/myApp/foo?bar=baz
I will use:
http://my-server:8080/myApp/foo?bar=baz&random=123456789
of course, the value of the random is different for every request. You can use the current time in milliseconds for that.
Not really. This is a known issue with IE, the classic solution is to append a random parameter at the end of the query string for every request. Most JS libraries do this natively if you ask them to (jQuery's cache:false AJAX option, for instance)
Well, of course you don't actually want to disable the browser cache entirely; correct caching is a key part of REST and the fact that it can (if properly followed by both client and server) allow for a high degree of caching while also giving fine control over the cache expiry and revalidation is one of the key advantages.
There is though an issue, as you have spotted, with subsequent GETs to the same URI from the same document (as in DOM document lifetime, reload the page and you'll get another go at that XMLHttpRequest request). Pretty much IE seems to treat it as it would a request for more than one copy of the same image or other related resource in a web page; it uses the cached version even if the entity isn't cacheable.
Firefox has the opposite problem, and will send a subsequent request even when caching information says that it shouldn't!
We could add a random or time-stamped bogus parameter at the end of a query string for each request. However, this is a bit like screaming "THIS IS SPARTA!" and kicking our hard-won download into a deep pit that no Health & Safety inspector considered putting a safety rail around. We obviously don't want to repeat a full unconditional request when we don't need to.
However, this behaviour has a time component. If we delay the subsequent request by a second, then IE will re-request when appropriate while Firefox will honour the max-age and expires headers and not re-request when needless.
Hence, if two requests could be within a second of each other (either we know they are called from the same function, or there's the chance of two events triggering it in close succession) using setTimeout to delay the second request by a second after the first has completed will make it use the cache correctly, rather than in the two different sorts of incorrect behaviour.
Of course, a second's delay is a second's delay. This could be a big deal or not, depending primarily on the size of the downloaded entity.
Another possibility is that something that changes so rapidly shouldn't be modelled as GETting the state of a resource at all, but as POSTing a request for a current status to a resource. This does smell heavily of abusing REST and POSTing what should really be a GET though.
Which can mean that on balance the THIS IS SPARTA approach of appending random stuff to query strings is the way to go. It depends, really.