I'm doing some research on how to properly cache AJAX responses, since that speeds up a page with lots of AJAX requests. I found this piece on the Yahoo website:
Let's look at an example. A Web 2.0 email client might use Ajax to download the user's address book for autocompletion. If the user hasn't modified her address book since the last time she used the email web app, the previous address book response could be read from cache if that Ajax response was made cacheable with a future Expires or Cache-Control header. The browser must be informed when to use a previously cached address book response versus requesting a new one. This could be done by adding a timestamp to the address book Ajax URL indicating the last time the user modified her address book, for example, &t=1190241612. If the address book hasn't been modified since the last download, the timestamp will be the same and the address book will be read from the browser's cache eliminating an extra HTTP roundtrip. If the user has modified her address book, the timestamp ensures the new URL doesn't match the cached response, and the browser will request the updated address book entries.
This makes it only less clear. The reason I want to know all this, is that I'm building a simple webpage where users can add shortcuts to websites. They see a grid of icons and can click on or search for the website they need. This is only meant as a project to get to know PHP and most importantly AJAX a lot better; nothing that actual users will ever see.
As you can imagine, the search function slows the website down a lot. Especially since it's performing an AJAX request after every typed letter. Therefore, I think it would greatly improve the website if some parts of this would be cached.
You have to make up your mind first:
Do you want a cached response used, or do you want the server to create a fresh one each time?
It isn't clear to me what you are asking exactly. What do you want to be cached, and how long?
If you want a cached response, simply use an URL you used before. (In combination with an expires-header)
For example:
http://www.example.com/myAjaxHelper.php?q=ab
If server responds with an expires header that says "one week from now", you will get the cached response for a week. Your browser will make that so.
(Remember YOU can set up the expires value in the response headers)
There are problems with browsers NOT respecting the expires header, and sometimes they fetch the cached version instead of the new one, even if it expired in the past. (Notably older IE needed a few of extra headers to fix that, and it is highly confusing)
If you want a fresh response (and this can lead to slowness because on each key-up you make a roundtrip to the server), make a NEW url.
Eg:
http://www.example.com/myAjaxHelper.php?q=ab?time=< ?php echo time(); ?>
This will lead to a new URL each second. And that one will never be cached, because the URL differs (only for the time=.... part, but that is enough to force a new request).
Related
I am new in AJAX and I find this one very confusing. Please help me figure it out.
I pursue the curriculum of w3schools and I got this example from here:
https://www.w3schools.com/xml/ajax_xmlhttprequest_send.asp
I didn't understand what is the difference between cached file and non cached file in these two examples because both examples deliver the same results and why do I have to add a unique Id to the url?
If you request an url, the browser or other proxys on the way to the server might cache the result, so if you request it more than once, it will be delivered faster. But the content might get out of date.
If you change the url, the caching mechanism between, from browser and other proxys on the way, will not recognize the request as the former request.
If you keep sending the same number, it might get cached, so it has to be unique.
But you cannot avoid a cached result from the server itself
This is not a duplicate of questions such as this, but rather the opposite: I have a form that I'm submitting via jQuery
$('<form>', {
action : 'service',
method : 'post',
target : '_blank'
}).append(
$('<input>', {
type : 'hidden',
name : 'payload',
value : JSON.stringify(payload)
})
).appendTo('body').submit().remove();
This is done so that I can open a different page with HTML.
Since I need to submit quite a lot of complex information, what I actually do is serialize them all into a big JSON string, then create a form with only one field ("payload") and submit that.
The receiving end has a filter that goes like this:
if the method is POST,
and there is only one submitted variable,
and the name of that one variable is "payload",
then JSON-decode its value and use it to create fake GET data.
So when the GET data grows too much I can switch methods without modifying the actual script, which notices no changes at all.
It always worked until today.
What should happen
The server should receive a single POST submission, and open the appropriate response in a popup window.
What actually happens instead
The server does receive the correct POST submission...
...apparently ignores it...
...and immediately after that, the browser issues a GET with no parameters, and it is the result of that parameterless GET that gets (pardon the pun) displayed in the popup window.
Quite unsurprisingly, this is always a "You did not submit any parameters" error. Duh.
What I already did
verified that this method works, and has always worked for the last couple of years with different forms and different service endpoints
tried replacing the form with a hardcoded <FORM> in HTML, without any jQuery whatsoever. Same results. So, this is not a jQuery problem.
tried with different browsers (it would not have helped if it only worked on some browsers: I need to support most modern browsers. However, I checked. Luckily, this failure reproduces in all of them, even on iPhones).
tried sending few data (just "{ test: 0 }").
tried halting the endpoint script as soon as it receives anything.
checked Stack Overflow. I found what seems to be the same problem, in various flavours, but it's of little comfort. This one has an interesting gotcha but no, it does not help.
checked firewalls, proxies, adblockers and plugins (I'm now using plain vanilla Firefox).
called the IT guys and asked pointed questions about recent SVN commits. There were none.
What I did not yet do
Check the HTTPS conversation at low level (I don't have sufficient access).
Compared the configuration, step by step, of a server where this works and the new server where it does not.
Quite clearly, put my thinking hat on. There must be something obvious that I'm missing and I'm setting myself up for a sizeable facepalm.
Use a tool like hurl.it or Postman to manually send a request to the server. The tools will nicely display the response from the server including all HTTP headers. I suspect the server responds with a redirect (Status code 30X) which leads to a GET request being issued after the POST completes.
Update: HTTP redirects
HTTP redirects do not necessarily use the same HTTP method or even the same data to issue a request to the redirect target. Especially for non-idempotent requests this could be a security issue (you don't generally want your form submission to be automatically re-submitted to another address). However, HTTP gives you both options:
[...] For this reason, HTTP/1.1 (RFC 2616) added the new status codes 303 and 307 [...], with 303 mandating the change of request type to GET, and 307 preserving the request type as originally sent. Despite the greater clarity provided by this disambiguation, the 302 code is still employed in web frameworks to preserve compatibility with browsers that do not implement the HTTP/1.1 specification.
[from Wikipedia: HTTP 302]
Also for 301s:
If the 301 status code is received in response to a request of any type other than GET or HEAD, the client must ask the user before redirecting.
[from Wikipedia: HTTP 301]
Due to decisions that are completely outside of my control, I am in the following situation:
I have a product listing on catalog.org
Clicking the "Add to Cart" button on a product makes an AJAX JSONP request to secure.com/product/add/[productKey], which saves the cart record to the database, sets a cookie with the cart ID, and returns a true response (or false if it failed)
Back on catalog.org, if the response is true, another AJAX JSONP request is made to secure.com/cart/info, which reads the cart ID cookie, fetches the record, and returns the number of items in the cart
Back on catalog.org once again, the response is read and an element on the page is updated showing the number of items in the cart (if any)
At this point, clicking the "Go to Cart" button on catalog.org displays the cart summary on secure.com
This works beautifully in Firefox 17, Chrome 32 and IE 11. It also works in IE8 - IE10 on our development and test environments, where catalog.org is catalog.development.com and catalog.test.com and secure.com is secure.development.com and secure.test.com respectively.
However, after we deployed to production, this stopped working in IE8 - IE10. After adding a product to the cart, the number of items in the cart is updated successfully on catalog.org. Then, after clicking the "Go to Cart" button on catalog.org, the cart summary on secure.com shows nothing because it can't read the cookie. Going to Cache > "View cookie information" in IE develeoper tools shows no cart ID cookie. It should be there, just like it is there in other browsers and in our development and test environments.
I believe what's happening is IE is blocking third party cookies. We have added a P3P compact policy header to all requests on secure.com, but the cookie is still not being set. The header we are setting is:
P3P: CP="CAO PSA OUR"
Why doesn't adding the compact policy header fix this in IE8 - IE10? How can I fix this to work in all versions of IE?
Solution
There are several good ideas posted below. I accepted #sdecima's because it sounded the most promising. We ended up combining some of these ideas but managed to avoid XDomainRequest:
Clicking the "Add to Cart" button on a product makes an AJAX JSONP
request to secure.com/product/add/[productKey], which saves the cart
record to the database, sets a cookie with the cart ID, and returns a
true response (or false if it failed)
We changed the action at secure.com/product/add to return a JSON object with a boolean indicating success or failure and the cart ID.
Back on catalog.org, if the response is true, another AJAX JSONP
request is made to secure.com/cart/info, which reads the cart ID
cookie, fetches the record, and returns the number of items in the
cart
We changed the callback function to check for both properties in the response object. If success is true and the cart ID is present, we create a hidden iframe on the page. The src attribute of the iframe is set to a new endpoint we added to secure.com. This action accepts a cart ID parameter and saves the cart ID cookie. We no longer need to save the cookie in the secure.com/product/add action.
Next, we changed the action at secure.com/cart/info to accept a cart ID parameter. This action will use the cart ID parameter if present to fetch the cart information, otherwise it will still attempt to read the cookie. This extra check would be unnecessary if we could guarantee that the iframe had finished loading and the cookie had been saved on secure.com, but we have no way of knowing when the iframe has finished loading on catalog.org due to browser security restrictions.
Finally, the P3P header CP="CAO PSA OUR" is still required for this to work in IE7 - IE10. (Yes, this works in IE7 now too :)
We now have a solution (albeit an incredibly complex one) for saving and accessing cross domain cookies that works in all major browser, at least as far back as we can reliably test.
We will probably refactor this some more. For one thing, the second AJAX JSONP request to secure.com/cart/info is redundant at this point since we can return all the information we need in the original request to secure.com/product/add action (a side benefit of changing that action to return a JSON object - plus we can return an error message indicating exactly why it failed if there was an error).
In short
Cookies will NOT go through a cross-origin request on IE 8 and 9. It should work on IE 10 and 11 though.
IE 8 and 9
On IE8/9 XMLHttpRequest partially supports CORS, and cross-origin requests are made with the help of the XDomainRequest object which does NOT send cookies with each request.
You can read more about this on the following official MSDN Blog post:
http://blogs.msdn.com/b/ieinternals/archive/2010/05/13/xdomainrequest-restrictions-limitations-and-workarounds.aspx
Particularly this part:
5 . No authentication or cookies will be sent with the request
In order to prevent misuse of the user’s ambient authority (e.g.
cookies, HTTP credentials, client certificates, etc), the request will
be stripped of cookies and credentials and will ignore any
authentication challenges or Set-Cookie directives in the HTTP
response. XDomainRequests will not be sent on previously-authenticated
connections, because some Windows authentication protocols (e.g.
NTLM/Kerberos) are per-connection-based rather than per-request-based.
IE 10+
Starting with IE10, full CORS support was added to XMLHTTPRequest and it should work fine with a correct Access-Control-Allow-Origin header property on the response from the server (that wishes to set the cookie on the browser).
More about this here:
http://blogs.msdn.com/b/ie/archive/2012/02/09/cors-for-xhr-in-ie10.aspx
And here:
http://www.html5rocks.com/en/tutorials/cors/
Workarounds on IE 8 and 9
The only way to go around this on IE8/9 is, quoting the same MSDN post as above:
Sites that wish to perform authentication of the user for cross-origin
requests can use explicit methods (e.g. tokens in the POST body or
URL) to pass this authentication information without risking the
user’s ambient authority.
Bottom line: third party cookies are commonly blocked by privacy/advertisement blocking extensions and should be considered unreliable. You'll be shooting yourself in the foot leaving it in production.
The syntax suggests that the endpoint has ambitions to one day become RESTful. The only problem with that is using cookies, which throws the whole "stateless" concept out of the window! Ideally, changes should be made to the API. If you are not integrating with a third party (i.e. "secure.com" is operated by your company) this is absolutely the correct way to deal with the issue.
Move the cartId out of the secure.com cookie into its querystring:
secure.com/product/add/9876?cartId=1234 //should be a POST
Where to get a valid cartId value? We can persist it in some secure-com-cart-id cookie set for catalog domain, which will avoid any cross-domain issues. Check that value and, if present, append to every secure.com request as above:
$.post('secure.com/product/add/9876', { //needs jQuery.cookie
cartId: $.cookie('secure-com-cart-id')
});
If you don't have a valid cartId, treat it as a new user and make the request without the parameter. Your API should then assign a new id and return it in the response. The "local" secure-com-cart-id cookie can then be updated. Rinse and repeat.
Voila, you've just persisted an active user cart without polluting API calls with cookies. Go yell at your architect. If you can't do that (changing API syntax or yelling), you'll have to set up a tunnel to secure.com endpoint so that there will be no cross-domain request - basically something sitting at catalog.org/secure-com-endpoint which will channel the requests to secure.com verbatim. It's a workaround specifically to avoid making changes to the API, just don't do it with code and have proper Apache/IIS/F5 rules set up to handle it instead. A quick search comes up with several explanations, this one looks pretty good to me.
P.S.: this is a classic XY problem in my opinion. The solution isn't necessarily about persisting 3rd party cookies but about passing necessary parameters to a 3rd party while persisting the data somewhere.
Although a correct solution would be a change of architecture, if you're looking for a quick, temporary solution:
JSONP files are actually just javascript. You could add a line of code to set cookies to the front of your JSONP.
eg instead of:
callback({"exampleKey": "exampleValue"});
Your JSONP could look like:
document.cookie="cartID=1234";
callback({"exampleKey": "exampleValue"});
If you control the DNS records, create a new entry so both servers are in the same domain.
Is there 1 database serving catalog.org and secure.com or can they communicate?
If so then, you got it.
When catalog.org servers a cookie, save it in the db.
When secure.com servers a cookie, save it in the db.
Then you can determine who's cart belongs to which user.
This is a fun problem to consider......Update 2:
When a user goes to catalog.org:
check if he has a cat_org cookie, if not, then:
in catalog.org:
create a key value pair and save in the db {cat_cookie_id, unique_number}
set cat_cookie_id in browser
instruct the browser to ajax visit secure.com/register/unique_number
in secure.com
read unique_number from url
create a secure_cookie id
save in the db {cat_cookie_id, unique_number, secure_cookie_id}
delete unique_number, as this is a one-time use key
Now the db can map cat_cookie_id to secure_cookie_id and vice versa.
Let's assume that I have created my REST service smoothly and I am returning json results.
I also implemented API key for my users to communicate for my service.
Then Company A started using my service and I gave them an API key.
Then they created an HttpHandler for bridge (I am not sure what is the term here) in order not to expose API key (I am also not sure it is the right way).
For example, lets assume that my service url is as follows :
www.myservice.com/service?apikey={key_comes_here}
Company A is using this service from client side like below :
www.companyA.com/services/service1.ashx
Then they start using it on the client side.
Company A protected the api key here. That's fine.
But there is another problem here. Somebody else can still grab www.companyA.com/services/service1.ashx url and starts using my service.
What is the way of preventing others from doing that?
For the record, I am using WCF Web API in order to create my REST services.
UPDATE :
Company A's HttpHandler (second link) only looks at the host header in order to see if it is coming from www.companyA.com or not. but in can be faked easily I guess.
UPDATE 2 :
Is there any known way of implementing a Token for the url. For example, lets say that www.companyA.com/services/service1.ashx will carry a querystring parameter representing a TOKEN in order for HttpHandler to check if the request is the right one.
But there are many things here to think about I guess.
You could always require the client to authenticate, using HTTP Basic Auth or some custom scheme. If your client requires the user to login, you can at least restrict the general public from obtaining the www.companyA.com/services/service1.ashx URL, since they will need to login to find out about it.
It gets harder if you are also trying to protect the URL from unintended use by people who legitimately have access to the official client. You could try changing the service password at regular intervals, and updating the client along with it. That way a refresh of the client in-browser would pull the new password, but anyone who built custom code would be out of date. Of course, a really determined user could just write code to rip the password from the client JS programmatically when it changes, but you would at least protect against casual infringers.
With regard to the URL token idea you mentioned in update 2, it could work something like this. Imagine every month, the www.companyA.com/services/service1.ashx URL requires a new token to work, e.g. www.companyA.com/services/service1.ashx?token=January. Once it's February, 'January' will stop working. The server will have to know to only accept current month, and client will have to know to send a token (determined at the time the client web page loads from the server in the browser)
(All pseudo-code since I don't know C# and which JS framework you will use)
Server-side code:
if (request.urlVars.token == Date.now.month) then
render "This is the real data: [2,5,3,5,3]"
else
render "401 Unauthorized"
Client code (dynamic version served by your service)
www.companyA.com/client/myajaxcode.js.asp
var dataUrl = 'www.companyA.com/services/service1.ashx?token=' + <%= Date.now.month %>
// below is JS code that does ajax call using dataUrl
...
So now we have service code that will only accept the current month as a token, and client code that when you refresh in the browser gets the latest token (set dynamically as current month). Since this scheme is really predictable and could be hacked, the remaining step is to salted hash the token so no one can guess what it is going to be .
if (request.urlVars.token == mySaltedHashMethod(Date.now.month)) then
and
var dataUrl = 'www.companyA.com/services/service1.ashx?token=' + <%= mySaltedHashMethod(Date.now.month) %>
Which would leave you with a URL like www.companyA.com/services/service1.ashx?token=gy4dc8dgf3f and would change tokens every month.
You would probably want to expire faster than every month as well, which you could do my using epoch hour instead of month.
I'd be interested to see if someone out there has solved this with some kind of encrypted client code!
What you're describing is generally referred to as a "proxy" -- companyA's public page is available to anyone, and behind the scenes, it makes the right calls to your system. It's not uncommon for applications to use proxies to get around security -- for example, the same-origin policy means that your javascript can't make Ajax calls to, say, Amazon -- but if you proxy it on your own system, you can get around this.
I can't really think of a technical way to prevent this; once they've pulled data from your service, they can use that data however they want. You have legal options, of course; you can make it a term of service that proxying isn't allowed, and pull their API key if they don't comply. But most likely, if you haven't already included that in the TOS, you'd have to wait for, say, a renewal of their subscription to your service.
Presumably if they're making server-side HTTP requests to your service, those requests are all coming from the same IP address, so you could block that address. You'd probably want to tell them first, and they could certainly get around that if they wanted to.
With the second link exposed by Company A I don't think you can do much. As I understand it, you can only check whether the incoming request comes from Company A or not.
But each request issued to www.companyA.com/.. can't be distinguished from original request from Company A. Everyone they let in uses their referrer as a disguise.
I implemented a REST service and i'm using a web page as client.
My page has some javascript functions that performs several times the same http get request to REST server and process the replies.
My problem is that the browser caches the first reply and not actualy sends the following requests..
Is there some way to force the browser execute all the requests without caching?
I'm using internet explorer 8.0
Thanks
Not sure if it can help you, but sometimes, I add a random parameter in the URL of my request in order to avoid being cached.
So instead of having:
http://my-server:8080/myApp/foo?bar=baz
I will use:
http://my-server:8080/myApp/foo?bar=baz&random=123456789
of course, the value of the random is different for every request. You can use the current time in milliseconds for that.
Not really. This is a known issue with IE, the classic solution is to append a random parameter at the end of the query string for every request. Most JS libraries do this natively if you ask them to (jQuery's cache:false AJAX option, for instance)
Well, of course you don't actually want to disable the browser cache entirely; correct caching is a key part of REST and the fact that it can (if properly followed by both client and server) allow for a high degree of caching while also giving fine control over the cache expiry and revalidation is one of the key advantages.
There is though an issue, as you have spotted, with subsequent GETs to the same URI from the same document (as in DOM document lifetime, reload the page and you'll get another go at that XMLHttpRequest request). Pretty much IE seems to treat it as it would a request for more than one copy of the same image or other related resource in a web page; it uses the cached version even if the entity isn't cacheable.
Firefox has the opposite problem, and will send a subsequent request even when caching information says that it shouldn't!
We could add a random or time-stamped bogus parameter at the end of a query string for each request. However, this is a bit like screaming "THIS IS SPARTA!" and kicking our hard-won download into a deep pit that no Health & Safety inspector considered putting a safety rail around. We obviously don't want to repeat a full unconditional request when we don't need to.
However, this behaviour has a time component. If we delay the subsequent request by a second, then IE will re-request when appropriate while Firefox will honour the max-age and expires headers and not re-request when needless.
Hence, if two requests could be within a second of each other (either we know they are called from the same function, or there's the chance of two events triggering it in close succession) using setTimeout to delay the second request by a second after the first has completed will make it use the cache correctly, rather than in the two different sorts of incorrect behaviour.
Of course, a second's delay is a second's delay. This could be a big deal or not, depending primarily on the size of the downloaded entity.
Another possibility is that something that changes so rapidly shouldn't be modelled as GETting the state of a resource at all, but as POSTing a request for a current status to a resource. This does smell heavily of abusing REST and POSTing what should really be a GET though.
Which can mean that on balance the THIS IS SPARTA approach of appending random stuff to query strings is the way to go. It depends, really.