Amazon Kindle has the following site where you can access highlights from your Kindle: from https://read.amazon.com/notebook.
I want to scrape these highlights but none of my attempts have worked so far. I know that this should be possible because Readwise does this and a note-taking app named Reflect.app has a chrome extension where they also managed to do this.
I found this URL that returns relevant JSON data with a list of books read (this is the first piece of the puzzle): https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency. If I go to the Network tab and copy the request as a cURL and import it on Postman, the request ends up working successfully.
I've tried making the request on the client side with Fetch and Axios and with Python on the backend but neither approach has worked.
When I make the request on the client side, I get the error Access to fetch at 'https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency' from origin 'http://localhost:3000' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'.
Here is the code I am using, which I got by copying the request on the Network tab as a fetch request:
const request = await fetch(
"https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency",
{
headers: {
accept: "*/*",
"accept-language": "en-US,en;q=0.9",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "none",
},
referrerPolicy: "strict-origin-when-cross-origin",
body: null,
method: "GET",
mode: "cors",
credentials: "include",
}
);
I've tried many iterations of this. For example, I have tried setting the mode to "no-cors" but then I get an empty response. I also tried adding the session cookie information in the header of the request but that doesn't make a difference.
So I tried making the request from the backend using Python and Flask. When I do a get request using the Requests library, the url gets redirected to the amazon login page. I tried adding the session cookie as well and that didn't make a difference.
This is the code I have:
#api.route('/data')
def data():
session = requests.session()
request_url = 'https://read.amazon.com/kindle-library/search?query=&libraryType=BOOKS&sortType=recency'
headers = {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"cookie": "<cookie session info>"
}
params = {
"referrerPolicy": "strict-origin-when-cross-origin",
}
req = session.get(request_url, headers=headers, params=params)
I think what I need to do is find a way to authenticate with Amazon when making the request and I believe the way to do that is with a cookie. Right now I am just trying to hardcode my own cookie information but I would need a way to get the cookie from the user's session as well.
I'm not sure what I'm missing here. Or maybe there is a better way to do what I want to do?
Related
I'm quite new to more advanced APIs, and I'm trying to send a GET request to an external API using fetch, with the appropriate Headers as detailed by the API owner.
However, I'm still receiving a 403 Forbidden error, and it seems that the headers are not actually being sent with the request as the Chrome DevTools shows "Provisional headers are being shown".
I'm using a CORS proxy: https://cors-anywhere.herokuapp.com/, which has worked with other simpler API requests.
const proxy = 'https://cors-anywhere.herokuapp.com/';
const api = `${proxy}https://api-example.com`; // Obfuscated
// Generate the data
fetch(api, data = {}, {
credentials: "include",
method: "GET",
headers: {
Accept: "application/json",
"Content-Type": "application/json",
Authorization: "Bearer eLrw3eXlljyFRjaul5UoYZLNgpUeapbXSFKmLc5SVaBgv8azUtoKn7B062PjbYoS",
"User-Agent": "any-name"
},
body: JSON.stringify(data)
})
.then(response => {
return response.text();
})
The API request works in Postman and using curl, but with my application I receive a 403 Forbidden response. As also mentioned, only provisional headers are shown in the request headers; none of the headers I had set.
Any help would be much appreciated. Thanks!
It looks as though you are passing an empty object as the options. The fetch() function only takes two parameters, the resource (uri) and an object of options (see: https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch). You have an empty object data = {} as the second parameter and your options specified as an unused, third parameter. I believe what you want is to remove the data parameter, especially since you don't need to send a body in a GET request.
fetch(api, {
credentials: "include",
method: "GET",
headers: {
Accept: "application/json",
"Content-Type": "application/json",
Authorization: "Bearer eLrw3eXlljyFRjaul5UoYZLNgpUeapbXSFKmLc5SVaBgv8azUtoKn7B062PjbYoS",
"User-Agent": "any-name"
}
})
.then(response => {
return response.text();
})
The api works in Postman and curl and if you are sure you are sending all request and headers same way then it probably is CORS issue. You have not provided enough information to truly understand if that is the case.
However I am trying to explain what I understand how CORS work for browsers. Browsers before making a request (e.g GET, POST, DELETE etc) makes an OPTIONS request. If the server that handles the request sees that the request is allowed for that host (using the origin and a few other factors), the server responds with a successful response. When browsers see that the OPTIONS request is successful then the browser executes the actual request (GET, POST, DELETE, whatever).
Sometimes for local development you may need to overcome this as localhost will not be supported by the server. In this case you can use browser extensions that intercepts your xhr requests and mocks a successful OPTIONS request for your browser and your browser thinks server responded with successful for OPTIONS call so then it allows the call.
Sending the headers with your request will not work. The api server must allow options request to be returned with status 200 for your app to be able to make that call from browser.
All the above is based on that you sent the request from your browser the same way as from postman or curl. You can verify that if you use a network monitor app like Fiddler if you use windows. If you are on macOS or Linux, I am not aware of a tool like Fiddler, there must be tools but as I don't work on those platform, I cannot suggest another tool to monitor network.
I am working on a Vue js project which uses an API build in PHP from another domain. API-calls are done with AXIOS.
When i log in via the API i get get back my PHPSESSID. This cookie is not stored in the browser and i cannot access the value in the response. I need this PHPSESSID value to send with future-requests and get related data, stay logged in etc..
axios.create({
withCredentials: true,
credentials: 'same-origin',
headers: {
'content-type': 'application/x-www-form-urlencoded'
}
})
The response header contains:
Set-Cookie
PHPSESSID=6amji3ps4tdnkhflbd8depjas0; path=/
Can someone, point me in the right way please?
Many thanks in advance!
I am trying to login by calling an API via a POST HTTP request.
post(
postLogin(email),
JSON.stringify({password: passwd}),
{ headers: { "Content-Type":"application/json" },
credentials: 'include' // i also tried with 'same-origin'
}
)
I am using Redux and React. In the API response, I receive the Set-Cookie header (I can see it in browser dev tools), but for some reason I cannot access it in my code and the browser doesn't set the cookie. I'm using Chrome Version 63.0.3239.84. It is a cross-origin request, so I have the following CORS headers set so I think it is not from here.
Access-Control-Allow-Credentials:true
Access-Control-Allow-Headers:*
Access-Control-Allow-Methods:GET, OPTIONS
Access-Control-Allow-Origin:*
Do you have any suggestions or someone has this issue too? Thank you!
I have finally fixed it. The problem was that my application and the API were not in the same domain. I configure a local NGINX in order to have them under the same domain and it works fine.
I have a WEB API which I am consuming from POSTMAN, and it works perfectly fine:
Headers:
Content-Type:application/json
X-Developer-Id:asdasdas
X-Api-Key:asdasdas
Authorization:Bearer sasdasdsa
Time-Zone:Morocco Standard Time
When I do a GET request in POSTMAN it works fine, however from angular 2 (Ionic 2) I get the following error:
Request header field Time-Zone is not allowed by Access-Control-Allow-Headers in preflight response.
let params: URLSearchParams = new URLSearchParams();
params.set('date', date);
//Header
let headers = new Headers({
'Content-Type': AppSettings.ContentType,
'X-Developer-Id': AppSettings.XDeveloperId,
'X-Api-Key': AppSettings.XApiKey,
'Time-Zone': AppSettings.time_zone,
'Authorization': AppSettings.Authorization + localStorage.getItem("AccessToken")
});
var RequestOptions: RequestOptionsArgs = {
url: AppSettings.UrlAvailability + userId,
method: 'GET',
search: params,
headers: headers,
body: null
};
return this.http.get((AppSettings.UrlAvailability + userId), RequestOptions)
.map(res => res.json())
.do(data => { data },
err => console.log('error: ' + err));
First I would think that the API developers have to do something on the server side, like enabling that Time-Zone Header on CORS, however if that would be the case then we would get the same error on POSTMAN, but it works fine there.
What am I missing here?
This is something you need to configure on the server. You first need to make sure you have CORS support. I don't use ASP.NET, so I don't know how to do it. I'm pretty sure a quick google search will find you the answer. Then you need to make sure in that server CORS config, that special headers you want the client to be able to send are added to the CORS allowed headers. That's what the error is saying: that the headers are not included in the response header Access-Control-Allow-Headers. The response header would look like
Access-Control-Allow-Headers: X-Developer-Id, X-Api-Key, Time-Zone, Authorization
To learn more about CORS, see the MDN
First I would think that the API developers have to do something on the server side, like enabling that Time-Zone Header on CORS, however if that would be the case then we would get the same error on POSTMAN, but it works fine there
No, Postman does not have the same restrictions. It is a native desktop app. Fun fact: 99% of people who post questions on SO that hava a CORS problem, have somewhere in their post "...but it work with Postman!". So don't feel bad :-)
I think you should remove some of your headers and check you content-type so your request could be considered as a "simple request" and then won't trigger a CORS preflight as explained in the doc.
source:
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#examples_of_access_control_scenarios
Apart from the headers automatically set by the user agent (for example, Connection, User-Agent, or the other headers defined in the Fetch spec as a forbidden header name), the only headers which are allowed to be manually set are those which the Fetch spec defines as a CORS-safelisted request-header, which are:
Accept
Accept-Language
Content-Language
Content-Type (please note the additional requirements below)
I keeps getting the error when doing PUT request to AWS apigateway with Fetch api in ReactJS. Pretty much sure I've enabled CORS in aws and depoyed it.
the fetch code looks like:
fetch(urlUpdate, {
method: "PUT",
headers: {
"Content-Type": "application/json"
},
mode: 'cors',
body: JSON.stringify({
"Item": value;
})
});
GET and POST requests are all working, have't check DELETE yet.
The documentation clearly states that you need to manually add other methods unless you are limiting your resources to GET, HEAD, or POST:
You must set up an OPTIONS method to handle preflight requests to support CORS. However, OPTIONS methods are optional if 1) an API resource exposes only the GET, HEAD or POST methods and 2) the request payload content type is application/x-www-form-urlencoded, multipart/form-data or text/plain and 3) the request does not contain any custom headers. When possible, we recommend to use OPTIONS method to enable CORS in your API.
Scroll to the bottom of the docs page. You need to add
method.response.header.Access-Control-Allow-Methods : "'*'"