I'm working with a public API that is poorly configured, i.e. the server doesn't set the Cache-Control header when it should.
Can I use HTTP request headers or alter response headers, so that a response is cached at the browser level for some period of time?
In the context of this API the response will be fresh until a specific data time calculated previously.
I looked quite a bit, but cannot seem to find anything related to setting request headers.
Note, I'm working in javascript and using fetch() to make requests.
Here is the state of the current headers:
Request Headers
:authority: fantasy.premierleague.com
:method: GET
:path: /api/entry/1157414/event/3/picks/
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-GB,en-US;q=0.9,en;q=0.8
sec-ch-ua: "Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87"
sec-ch-ua-mobile: ?0
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Response Headers
accept-ranges: bytes
content-length: 571
content-type: text/html
date: Fri, 25 Dec 2020 16:57:32 GMT
server: nginx/1.18.0
via: 1.1 google, 1.1 varnish, 1.1 varnish
x-cache: MISS, MISS
x-cache-hits: 0, 0
x-served-by: cache-lhr7366-LHR, cache-fra19130-FRA
x-timer: S1608915452.118518,VS0,VE22
Related
I have a web page served by FastAPI that on a button click is initiating a POST request using pure Javascript to a route in my API which then should redirect to an external page.
The Javascript:
function submit(url) {
let xhr = new XMLHttpRequest();
xhr.open("POST", url, false);
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.send(); }
In my app FastAPI app, I added the following:
app.add_middleware(
CORSMiddleware,
allow_credentials=True,
allow_origins=['*', 'http://127.0.0.1:8080', 'http://127.0.0.1:8080/OpryW?', 'http://127.0.0.1:8080/OpryW'],
allow_methods=["*"],
allow_headers=["*"],
)
When clicking the button, however, I am still getting a CORS error.
Here are some details about the request:
POST: http://127.0.0.1:8080/OpryW
Request headers:
POST /OpryW HTTP/1.1
Host: 127.0.0.1:8080
Connection: keep-alive
Content-Length: 19
Pragma: no-cache
Cache-Control: no-cache
sec-ch-ua: "Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36
sec-ch-ua-platform: "macOS"
Content-Type: application/json
Accept: */*
Origin: http://127.0.0.1:8080
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://127.0.0.1:8080/OpryW?
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Response headers:
HTTP/1.1 307 Temporary Redirect
date: Mon, 01 Nov 2021 20:00:10 GMT
server: uvicorn
location: http://www.google.com
access-control-allow-origin: *
access-control-allow-credentials: true
Transfer-Encoding: chunked
Why am I still getting a CORS error?
Thanks
For some reason cookies set by server are not being saved, can't access them from within my js code, in devtools in Application tab in cookies section it's all empty, but somehow when I make a request to the server after the initial one, those cookies that apparently didn't save are being included in the header.
I have set withCredentials to true in axios config, as well as CORS_ALLOW_CREDENTIALS to true on my backend.
Here are the headers from the request that should save cookies:
Response Header:
access-control-allow-credentials: true
access-control-allow-origin: <frontend url>
content-length: 22
content-type: text/html; charset=utf-8
date: Mon, 01 Feb 2021 12:04:09 GMT
server: nginx/1.19.2
set-cookie: csrftoken=<token>; expires=Mon, 31 Jan 2022 12:04:09 GMT; Max-Age=31449600; Path=/; SameSite=Lax
set-cookie: sessionid=<sessionid>; expires=Mon, 01 Feb 2021 13:04:09 GMT; HttpOnly; Max-Age=3600; Path=/
strict-transport-security: max-age=31536000
vary: Cookie, Origin
x-frame-options: SAMEORIGIN
Request Headers:
:authority: <backend url>
:method: POST
:path: /login/
:scheme: https
accept: application/json, text/plain, */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
content-length: 66
content-type: application/json;charset=UTF-8
dnt: 1
origin: <frontend url>
pragma: no-cache
referer: <current frontend url query>
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-site
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36
As you can see in the response I'm getting two cookies to be set, sessionid and csrftoken, those are not being saved.
What's weird those cookies are present in the following request:
Request Headers:
:authority: <backend url>
:method: GET
:path: /numbers/
:scheme: https
accept: application/json, text/plain, */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
cookie: csrftoken=<token>; sessionid=<sessionid>
dnt: 1
origin: <frontend url>
pragma: no-cache
referer: <current frontend url query>
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-site
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36
How is it possible for the cookies to not be set but still included in the request? And what can cause this behavior? I really need them to be saved.
My backend and frontend are located at different urls, my backend is running on Django with a rest-framework.
UPDATE
I've also tried to set samesite on the cookies to none, as well as changing request content-type to application/x-www-form-urlencoded. Still no luck.
After hours spent troubleshooting this problem I found out a solution, somewhat. My frontend app and backend app are hosted from different urls, but under the same domain, like that: <frontend>.<mydomain>.com and <backend>.<mydomain>.com
When frontend application was making a request to the backend, backend was sending set-cookie headers that would save the cookies under <backend>.<mydomain>.com (I was able to confirm this in here: chrome://settings/siteData), which made them inaccessible from within <frontend>.<mydomain>.com, I were able to fix that by adding a domain property to the set-cookie header that would point to .<mydomain>.com. To do that I had to create a middleware in Django that would add my predefined domain name to the cookies before being send back to the client.
I'm not really 100% happy with this because if my applications were served from different domains, I'm not sure if I could make the cookies work at all, since setting set-cookie to a different domain is not allowed by the browser.
I'm working on the React + Express + mongo project.
Everything works locally, but when I deployed project on heroku.com, a problem appeared with GET requests.
It returns 304 and instead of body JSON, i'm getting this error:
To run this application, you need to enable JavaScript.
Instead of json response, I'm getting HTML response with this information about JS. In body I should get normal data JSON that I display on the page.
I will write again, everything WORKS locally ( i get 200 and json response), I don't know what's going on with this heroku.
I thought that problem is 304 code, so I used middleware that disables cache (no-cache), the result is 200, but there is still no body.
REQUEST ON CHROME NETWORK
GENERAL:
Request URL: MYURL.com/...
Request Method: GET
Status Code: 200 OK
Remote Address: 34.255.19.16:443
Referrer Policy: no-referrer-when-downgrade
RESPONSE HEADERS::
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate
Connection: keep-alive
Content-Length: 2366
Content-Type: text/html; charset=UTF-8
Date: Fri, 01 Nov 2019 13:54:09 GMT
Etag: W/"93e-16e271de930"
Expires: 0
Last-Modified: Fri, 01 Nov 2019 13:18:22 GMT
Pragma: no-cache
Server: Cowboy
Surrogate-Control: no-store
Via: 1.1 vegur
X-Powered-By: Express
REQUEST HEADERS:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjVkYTZkNjZhYTExYWI5MTM3MDU1MTY2OCIsIm5hbWUiOiJGaWxpcCIsImlhdCI6MTU3MjYxNjQ0MiwiZXhwIjoxNTcyNjIwMDQyfQ.c4uPda0njISa3VWNX0kK5cPRVW2X6A3wBNnt5hc5N-k
Connection: keep-alive
Content-Type: application/json
Cookie: AUTHORIZATION_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjVkYTZkNjZhYTExYWI5MTM3MDU1MTY2OCIsIm5hbWUiOiJGaWxpcCIsImlhdCI6MTU3MjYxNjQ0MiwiZXhwIjoxNTcyNjIwMDQyfQ.c4uPda0njISa3VWNX0kK5cPRVW2X6A3wBNnt5hc5N-k
Host: dashboard-pwa.herokuapp.com
Referer: https://dashboard-pwa.herokuapp.com/tickets/show/5dbb2310f67d543c84053a79
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
I want to get 200 with JSON body response.
adam tropp thank you! It helped me a lot. It turns out that i had redirect to client index.html in wrong place, it should be placed after routes declaration, not before (just like I had before).
file server.js:
app.use('/api/user', require('./routes/user'));
// For any request that doesn't
// match one above, send back React's index.html file.
app.get('*', (req, res) => {
res.sendFile(path.join(__dirname+'/client/build/index.html'));
});
Thanks, once again.
The setup uses a Backbone Model, Nginx server. The users enter their username and password which is then passed via a post. The server authenticates and returns a session cookie.
When the backend and front-end are on the same server (e.g. connect via localhost) the cookie is stored. However when the connection is remote, it is not stored in Chrome; however, it is stored in Safari and FireFox.
Ajax is setup via
$.ajaxSetup({
crossDomain: true,
xhrFields: {
withCredentials: true
}
});
The request headers are
POST /login HTTP/1.1
Host: localhost:8102
Connection: keep-alive
Content-Length: 59
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://127.0.0.1:9102
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36
Content-Type: application/json
DNT: 1
Referer: http://127.0.0.1:9102/somefolder
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
The response headers are
HTTP/1.1 200 OK
access-control-allow-origin: http://127.0.0.1:9102
access-control-allow-credentials: true
vary: origin,accept-encoding
access-control-expose-headers: WWW-Authenticate,Server-Authorization
content-type: text/html; charset=utf-8
set-cookie: na-auth-token=encrypted-string; Max-Age=86400; Expires=Thu, 27 Apr 2017 13:32:44 GMT; HttpOnly; SameSite=Strict; Path=/
cache-control: no-cache
content-encoding: gzip
Date: Wed, 26 Apr 2017 13:32:44 GMT
Connection: keep-alive
Transfer-Encoding: chunked
In FireFox and Safari the cookie is stored just fine, but in Chrome it gets the response and tosses the cookie without any notification.
Update
The cookie is actually being saved under the localhost domain, however when you navigate back to the page (e.g. via a window.location.reload) the cookie disappears.
So the answer to my question lied in a library we were using
hapi-auth-cookie
The package updated and a flag was introduced isSameSite. Changed this value to false to allow the cors cookie to persist between pages loads.
I'm trying to do a cross-domain POST with jQuery using CORS, but after the OPTIONS request is sent (and responded to), the POST doesn't happen. My request/response filters are shown below. My jQuery post code is also shown. It looks like the response has the proper CORS response headers, so I'm not sure why this isn't working.
var params = {};
params.data = myPostData;
params.headers = {
'x-pingother' : 'pingpong'
};
$.ajax(params);
Request Header
OPTIONS /my/service/url HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Origin: http://localhost:9876
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type,x-pingother
Pragma: no-cache
Cache-Control: no-cache
Response Header:
HTTP/1.1 200 OK
Access-Control-Allow-Origin: http://localhost:9876
Access-Control-Allow-Methods: POST,GET,OPTIONS
Access-Control-Allow-Headers: x-pingother
Content-Length: 703
Content-Type: application/vnd.sun.wadl+xml
Allow: OPTIONS,POST
Last-Modified: Sat, 19 Jan 2013 13:46:51 EST
Server: Jetty(6.1.25)
I'm running automated tests from a local server (JSTestDriver) that connects to a web server running on a different port. The web server runs on 8080 and the page making the request runs on 9876.
This was solved by adding the response headers as specified in
jQuery CORS request not working when sending Range request
Did you try to explicitly tell jQuery to enable cors?
$.support.cors = true;