Develop Reference

Login to Amazon with Cookies and Scrape Kindle Highlights

Amazon Kindle has the following site where you can access highlights from your Kindle: from https://read.amazon.com/notebook.
I want to scrape these highlights but none of my attempts have worked so far. I know that this should be possible because Readwise does this and a note-taking app named Reflect.app has a chrome extension where they also managed to do this.
I found this URL that returns relevant JSON data with a list of books read (this is the first piece of the puzzle): https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency. If I go to the Network tab and copy the request as a cURL and import it on Postman, the request ends up working successfully.
I've tried making the request on the client side with Fetch and Axios and with Python on the backend but neither approach has worked.
When I make the request on the client side, I get the error Access to fetch at 'https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency' from origin 'http://localhost:3000' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'.
Here is the code I am using, which I got by copying the request on the Network tab as a fetch request:
const request = await fetch(
"https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency",
{
headers: {
accept: "*/*",
"accept-language": "en-US,en;q=0.9",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "none",
},
referrerPolicy: "strict-origin-when-cross-origin",
body: null,
method: "GET",
mode: "cors",
credentials: "include",
}
);
I've tried many iterations of this. For example, I have tried setting the mode to "no-cors" but then I get an empty response. I also tried adding the session cookie information in the header of the request but that doesn't make a difference.
So I tried making the request from the backend using Python and Flask. When I do a get request using the Requests library, the url gets redirected to the amazon login page. I tried adding the session cookie as well and that didn't make a difference.
This is the code I have:
#api.route('/data')
def data():
session = requests.session()
request_url = 'https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency'
headers = {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"cookie": "<cookie session info>"
}
params = {
"referrerPolicy": "strict-origin-when-cross-origin",
}
req = session.get(request_url, headers=headers, params=params)
I think what I need to do is find a way to authenticate with Amazon when making the request and I believe the way to do that is with a cookie. Right now I am just trying to hardcode my own cookie information but I would need a way to get the cookie from the user's session as well.
I'm not sure what I'm missing here. Or maybe there is a better way to do what I want to do?

Cookies are being blocked with my fetch call

This is my fetch request:
fetch(`${process.env.REACT_APP_API_DOMAIN}/auth/user/full`, { method: "GET", credentials: "include" })
It should send my session cookies, but im getting this error in the request cookies on network tab in chrome:
This means my backend doesn't recieve the cookie and thinks the user is not authenticated.

So it was my backend cookie settings. I tried to set sameSite: false (what it says in the docs of cookie) but turns out proper way is sameSite: "none". The library i use is cookie-session which refer you to cookie docs for sameSite.

How to correctly assign Headers to fetch request in Javascript

I'm quite new to more advanced APIs, and I'm trying to send a GET request to an external API using fetch, with the appropriate Headers as detailed by the API owner.
However, I'm still receiving a 403 Forbidden error, and it seems that the headers are not actually being sent with the request as the Chrome DevTools shows "Provisional headers are being shown".
I'm using a CORS proxy: https://cors-anywhere.herokuapp.com/, which has worked with other simpler API requests.
const proxy = 'https://cors-anywhere.herokuapp.com/';
const api = `${proxy}https://api-example.com`; // Obfuscated
// Generate the data
fetch(api, data = {}, {
credentials: "include",
method: "GET",
headers: {
Accept: "application/json",
"Content-Type": "application/json",
Authorization: "Bearer eLrw3eXlljyFRjaul5UoYZLNgpUeapbXSFKmLc5SVaBgv8azUtoKn7B062PjbYoS",
"User-Agent": "any-name"
},
body: JSON.stringify(data)
})
.then(response => {
return response.text();
})
The API request works in Postman and using curl, but with my application I receive a 403 Forbidden response. As also mentioned, only provisional headers are shown in the request headers; none of the headers I had set.
Any help would be much appreciated. Thanks!

It looks as though you are passing an empty object as the options. The fetch() function only takes two parameters, the resource (uri) and an object of options (see: https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch). You have an empty object data = {} as the second parameter and your options specified as an unused, third parameter. I believe what you want is to remove the data parameter, especially since you don't need to send a body in a GET request.
fetch(api, {
credentials: "include",
method: "GET",
headers: {
Accept: "application/json",
"Content-Type": "application/json",
Authorization: "Bearer eLrw3eXlljyFRjaul5UoYZLNgpUeapbXSFKmLc5SVaBgv8azUtoKn7B062PjbYoS",
"User-Agent": "any-name"
}
})
.then(response => {
return response.text();
})

The api works in Postman and curl and if you are sure you are sending all request and headers same way then it probably is CORS issue. You have not provided enough information to truly understand if that is the case.
However I am trying to explain what I understand how CORS work for browsers. Browsers before making a request (e.g GET, POST, DELETE etc) makes an OPTIONS request. If the server that handles the request sees that the request is allowed for that host (using the origin and a few other factors), the server responds with a successful response. When browsers see that the OPTIONS request is successful then the browser executes the actual request (GET, POST, DELETE, whatever).
Sometimes for local development you may need to overcome this as localhost will not be supported by the server. In this case you can use browser extensions that intercepts your xhr requests and mocks a successful OPTIONS request for your browser and your browser thinks server responded with successful for OPTIONS call so then it allows the call.
Sending the headers with your request will not work. The api server must allow options request to be returned with status 200 for your app to be able to make that call from browser.
All the above is based on that you sent the request from your browser the same way as from postman or curl. You can verify that if you use a network monitor app like Fiddler if you use windows. If you are on macOS or Linux, I am not aware of a tool like Fiddler, there must be tools but as I don't work on those platform, I cannot suggest another tool to monitor network.

How to config headers in axios on Laravel application

I'm trying to build a small application on Vuejs 2.0 where I'm having axios call and in configuration I'm setting the headers something like this:
const headers = {
'Accept': 'application/json',
'Content-Type': 'application/json',
'origin': 'http://www.conxn.co.in',
'Access-Control-Allow-Origin': '*'
}
And calling the url something like this:
axios.post('http://www.conxn.co.in/CoxnsvcA.svc/Login', postData,{headers: getLoginHeader()})
But I can see the origin is not changed from local host:
Guide me how can I set the Origin to the url as mentioned.
Thanks

That's not how CORS works. You cannot modify the origin header being sent out because that's set by the browser for security purposes. The server needs to allow the request by returning the Access-Control-Allow-Origin header. It doesn't look like the server you're requesting ajax from is setup correctly for CORS requests.

Angular how not to send OPTION when sending auth token in HTTP Request

Having an issue with CORS, trying to do a GET request with Authorization token but having an issue with OPTIONS. Wondering how to not send OPTIONS when requesting GET to another server?
$http({
url: request_data.url,
type: request_data.method,
withCredentials: true,
headers: {
'Authorization': localStorage.getItem('Auth'),
'Content-Type': 'application/json; charset=utf-8'
},
crossDomain: true
}).then(function(response) {
//process your data here
vm.prodData = response.data;
console.log(response.data);
});
I can't send a request because the CORS issue, probably due to the header Authorization part.

You can't.
Authorization headers are not "safe", so you can't include them in a cross origin request without permission.
The preflight OPTIONS request is there to ask for permission.
You are doing something unusual that is a potential risk, so you have to change the server you are making the request to so it will tell the browser that it is OK to make the request.

Is far as i know when youre trying a CORS request it will always send an OPTIONS request first to check if youre allowed to make requests, you have to deal with that on server side, for instance in php i do the following:
if($_SERVER['REQUEST_METHOD'] == 'OPTIONS') {
header( "HTTP/1.1 200 OK" );
exit();
}
that way if the client send an OPTIONS request it will automaticly send an 200 OK response and then it will wait for the correct request (GET, POST, etc)