Develop Reference

How to persistently cache data fetched from remote server with javascript

I am building a reactjs (with hooks) web app which presents the user some data fetched from remote server pages (using a simple proxy).
Data on remote server changes about once per week, so I would like to persistently cache data (for example using LocalStorage) on the client, until server side pages are not updated, for a better user experience.
I'm using axios for data fetching, but I could also use a basic fetch.
The problem is that I can't understand how to cache data on the client until any update on the server: I see for example axios supports caching mechanisms (also using interceptors), but no way I can specify something like ETag or If-None-Match request header, but I can only specifiy a fixed amount of time before the cache is invalidated.
It is possible I'm missing something obvious, and that I'm asking something unfeasible...

I can think of a few ways solving this:
1) using a DDP protocol https://en.wikipedia.org/wiki/Distributed_Data_Protocol or some polling mechanism (e.g. fetch every x minutes)
2) add some "expireAt" timestamp in the response itself and use that one on the client to figure out when to check again for updates
3) using websockets with some pub/sub mechanism
They all have their ups and downs and you'd need to check if they are feasible for your application.
Best,
Sebo

Is it possible to prevent cookies to be sent in every HTTP request?

I recently found (here: Does every web request send the browser cookies?) that every HTTP request contains the cookies related to a domain every time a request is made to that same domain.
Given this, what happens when the request is not sent through a browser but from Node.js, for example? Is it possible that no information is sent in the request?
Is it also possible to prevent it to be sent in the browser requests?

Browsers
Is not possible to prevent browser to send cookies.
This is why is generally it is recommended (Yahoo developer Best practice, see section Use Cookie-free Domains for Components) to serve static content like css, images, from a different domain that is cookie free.
When the browser makes a request for a static image and sends cookies together with the request, the server doesn't have any use for those cookies. So they only create network traffic for no good reason. You should make sure static components are requested with cookie-free requests. Create a subdomain and host all your static components there.
Programmatically
From any programming language, instead, you can choose if you like to send cookies or not.
Cookie management is done by the programmer, because libraries are written to make single requests.
So if you make a first request that return cookies, you need to explicit read them, hold them locally somewhere, and eventually put them in a second request to the same server if you need.
So from NodeJS if you don't explicitly add cookies in your requests the http call doesn't hold them.

You Can Use Fetch with the credentials option set to omit
see
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

You can strip cookies with a proxy server. For example our product WinGate will allow you to modify requests (and responses), and you could use this to clear the Cookie header in requests.
However, this will prevent a large number of websites from functioning properly, as cookies are used to transport session IDs so that the server can identify each connection / request your browser makes as being from the same "session". HTTP itself does not have any concept of session.
Disclaimer: I work for Qbik who make WinGate.

Differenciate Between User Requests and AJAX/Resource Requests

I'm attempting to create an app with Node.js (using http.createServer()) which will be a single page application with requests for data via XMLHttpRequest. To do this I need to be able to differentiate between a user navigating to my domain, and AJAX requests and requests generated by the browser for linked resources.
If the request is from the user I always want to return the index.html page which will handle requesting content but if the request is browser generated or AJAX and is for CSS, Javascript or other linked files I want to serve those files. Is there any way to detect this?
Looking at the request headers for the different file types I saw the referer header appeared when the request for content was generated by the page. I figured that was the solution I was looking for but that header is also set when a user clicks on a link to the page making it useless.
The only other thing which seems to change is the accept header which could sort of work but might not be a catch all solution. Any user requests always seem to have text/html as the preferred return type regardless of which url was entered. I could detect that but I'm pretty sure AJAX requests for html files would also have that accept header which would cause problems.
Is there anything I'm missing here (any headers or properties I can look for)?
Edit: I do not need the solution to protect files and I don't care about users bypassing it with their own requests. My intention is not to hide files or make them secure, but rather to keep any data that is requested within the scope of the app.
For example, if a user navigates to http://example.com/images/someimage.jpg they are instead shown the index.html file which can then show the image in a richer context and include all of the links and functionality to go with it.
TL/DR: I need to detect when someone is trying to access the app to then serve them the index page and have that send them the content they want. I also need to detect when the browser has requested resources (JS, CSS, HTML, images, etc) needed by the app to be able to actually return the resource not the index file.

In terms of HTTP protocol there are NO difference between a user-generated-query and a browser-generated-query.
Every query is just... a query.
You can make a query with a command line, with a browser, you can click a link, send some ascii text via telnet, request a proxy which will make the query for you, the server goal is never to identify how the query was requested by the user.
See for example a request made by a user on a reverse proxy cache, this query will never reach your server (response comes from the cache), the first query made to build this response could have been made by a real user or by a browser.
In terms of security trying to control that the user is never requesting data by-himself cannot be done by detecting that the query is a real human click (and search google for clickjacking if you want to be afraid). Every query that a browser can make can also be played by the user, every one, you have no way to prevent that.
Some browsers plugins are even doing pre-fetching, detecting links on the page and making the request before you do it yourself (if it's a GET query).
For ajax, some libraries like JQuery will add an X-Requested-With: XMLHttpRequest header, and this is used on most framework to detect ajax mode.
But it is more robust to depend on a location policy for that (like making your ajax queries with a /format/ajax, which could also be used on other ways (like /format/json, /format/html, or /format/csv).
Spending time on a location policy based routing is certainly more usefull.
But one thing can make a difference, POST queries are not indempotent, it means the browser cannot make a POST query without a real user interaction, because a POST query may alter the state of the session or the state of the server data (but js can make POST queries, this is just a default behavior of browsers). The browser will never automatically retrieve a POST query, so you could make a website where all users interactions are POST queries (via forms or via some js altering link clicks to send POST ajax queries instead). But I'm not that's your real goal.

Not technically an answer to the question but I found a simple solution which does what I want: prefix all app based requests with a subdomain eg. http://data.example.com/. It's then really simple to check the host header for that subdomain: if present send the resource else send the index page.

Is browser cache aware of javascript xmlhttp requests?

When I fetch a page using GET request in javascript, does the browser cache it the same way as it does when I click that link or type it in address bar?
If not, since I have already fetched the page, is there a way that I can add it (programmatically) to the browser cache?

When the browser fetches web pages, it is also using a GET request. Chances are that all GET requests go through the same caching mechanism in the browser, though there is no specification that formalizes how that works.
There is no programmatic way to add something to the browser's own cache other than just requesting the resource and letting the browser's cache do its normal thing with it. If you want to know if all common browsers will cache it in this way, then you need to make sure the server-side header settings are set appropriately (to allow it to be cached) and then test each browser to make sure it's cached like you want.
If you are staying within the same page and want to make sure something is not requested more than once from the same page, you can implement your own cache within that page's javascript code. You just store the result in a javascript variable the first time it is requested and then a function you implement to fetch this resource just checks your own local storage object to see if the resource is already here. If not, it requests it via a GET and then saves the result. You could make a simple version of this that was hardcoded to one particular resource or a more general version that saved the URL and result and a timestamp and implemented more typical caching behaviors.
If you want it to be cached across pages and your testing finds that the built-in browser caches are not adequate, then you can use Local Storage to store the data (probably with a timestamp) and then just check the local storage before requesting it with a GET request.

Using non persistent Http Cookies to deliver out of band data to the browser

Imagine that your web application maintains a hit counter for one or multiple pages and that it also aggressively caches those pages for anonymous visitors. This poses the problem that at least the hitcount would be out of date for those visitors because although the hitcounter is accurately maintained on the server even for those visitors, they would see the old cached page for a while.
What if the server would continue to serve them the cached page but would pass the updated counter in a non-persistent http cookie to be read by a piece of javascript in the page that would inject the updated counter into the DOM.
Opinions?

You are never going to keep track of the visitors in this manner. If you are aggressively caching pages, intermediate proxies and browsers are also going to cache your pages. And so the request may not even reach your server for you to track.
The best way to do so would be to use an approach similar to google analytics. When the page is loaded, send an AJAX request to the server. This ajax request would increment the current counter value on the server, and return the latest value. Then the client side could could show the value returned by the server using javascript.
This approach allows you to cache as aggressively as you want without losing the ability to keep track of your visitors.

you can also get the page programmatically via asp or php out the cache yourself and replace the hitcounter.