I make a POST request and the request just sits, pending until it eventually fails. I've monitored the nginx logs and the node server logs and the request doesn't even register. This works for anyone else that I've had test it except one other colleague. If I use the edge browser or a different computer it works fine.
I have attempted to make POST requests to other (custom) servers and it hangs on options there as well. I have also made the POST request with jQuery and it fails the same way.
It's maybe worth noting that I am using the withCredentials flag.
Headers:
Provisional headers are shown
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:GET
Origin:http://localhost:8080
Referer:http://localhost:8080/<path>
User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
The request:
public login(user) {
const endpoint = `http://<url>`;
let headers = new Headers();
headers.append('Content-type', 'application/json');
return this.http
.post(endpoint, JSON.stringify(user), {
headers: headers,
});
}
I subscribe to the call in my component:
this._accountService.login(this.user)
.subscribe(res => {
console.log("logged in!");
if (res.json().status === "success") {
window.location.href = `/home/${this.org}/${this.product}`;
}
else {
// What other options are there?
console.log("Do something else maybe?");
}
},
err => {
this.invalidLogin = true;
console.log("Ye shall not pass!");
});
Successful user's headers
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:<url>
Origin:<url>
Referer:<url>
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.33 Safari/537.36
From chrome://net-internals/#events
t=61869793 [st= 0] +REQUEST_ALIVE [dt=60162]
--> has_upload = false
--> is_pending = true
--> load_flags = 34624 (DO_NOT_SAVE_COOKIES | DO_NOT_SEND_AUTH_DATA | DO_NOT_SEND_COOKIES | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
--> load_state = 14 (WAITING_FOR_RESPONSE)
--> method = "OPTIONS"
--> net_error = -1 (ERR_IO_PENDING)
--> status = "IO_PENDING"
--> url = "<url>"
t=61929955 [st=60162] -HTTP_STREAM_PARSER_READ_HEADERS
--> net_error = -324 (ERR_EMPTY_RESPONSE)
t=61929955 [st=60162] -HTTP_TRANSACTION_READ_HEADERS
--> net_error = -324 (ERR_EMPTY_RESPONSE)
t=61929955 [st=60162] -URL_REQUEST_START_JOB
--> net_error = -324 (ERR_EMPTY_RESPONSE)
t=61929955 [st=60162] URL_REQUEST_DELEGATE [dt=0]
t=61929955 [st=60162] -REQUEST_ALIVE
--> net_error = -324 (ERR_EMPTY_RESPONSE)
I'm really guessing this is related to something that is cached in my browser(s) but I really cannot find what. I've cleared all cookies and anything that could be stored. Where else can I check to clear things? This is clearly something local to my computer/browser (and one other unfortunate person).
Please try to subscribe() to the observable.
return this.http
.post(endpoint, JSON.stringify(user), {
headers: headers,
}).subscribe(() => console.log("POST done!"));
Have you tried setting the 'Cache-Control' in your headers? I think in jQuery you can simply set
$.ajax({
cache: false
});
or adding a header with a regular ajax request
request.setRequestHeader("Cache-Control", "no-cache");
Why don't you just prevent getting into OPTIONS request loop . It really drives you crazy at times . Other browsers do not trigger OPTIONS request but chrome and firefox does to ensure CORS . I have successfully used this library named as xdomain from github , and it really works !! Their github introduction page introduce xdomain as a CORS alternative . And most importantly i used it in JQuery , but it also does support Angular's http service . Have a look at it . It may help you for good :) . Here's the link to library Xdomain CORS Alternative
There are issues with CORS and using localhost as the domain (which you have listed in the ORIGIN headers). Typically CORS / OPTIONS requests don't work properly when localhost is involved for certain security reasons, but hanging isn't normally what happens so this might not be the correct answer but its worth a shot!
Try adding a new host to your local machine and removing localhost from the equation. Just throwing this idea out there and hope that it might help you out!
As per comment below
Your server appears to allow the connection, but it does not appear to send a response. Are you able to post the headers from a successful OPTIONS request to prove that the server is actually able to handle these requests.
Related
I want to scrape the contents of LinkedIn using requests and bs4 but I'm facing a problem with the JavaScript that is loading the page after I sign in(I don't get the home page directly), I don't wanna use Selenium
here is my code
import requests
from bs4 import BeautifulSoup
class Linkedin():
def __init__(self, url ):
self.url = url
self.header = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"}
def saveRsulteToHtmlFile(self, nameOfFile=None):
if nameOfFile == None:
nameOfFile ="Linkedin_page"
with open(nameOfFile+".html", "wb") as file:
file.write(self.response.content)
def getSingInPage(self):
self.sess = requests.Session()
self.response = self.sess.get(self.url, headers=self.header)
soup = BeautifulSoup(self.response.content, "html.parser")
self.csrf = soup.find(attrs={"name" : "loginCsrfParam"})["value"]
def connecteToMyLinkdin(self):
self.form_data = {"session_key": "myemail#mail.com",
"loginCsrfParam": self.csrf,
"session_password": "mypassword"}
self.url = "https://www.linkedin.com/uas/login-submit"
self.response = self.sess.post(self.url, headers=self.header, data=self.form_data)
def getAnyPage(self,url):
self.response = self.sess.get(url, headers=self.header)
url = "https://www.linkedin.com/"
likedin_page = Linkedin(url)
likedin_page.getSingInPage()
likedin_page.connecteToMyLinkdin() #I'm connected but java script still loading
likedin_page.getAnyPage("https://www.linkedin.com/jobs/")
likedin_page.saveRsulteToHtmlFile()
I want help to pass the javascript loads without using Selenium...
Although it's technically possible to simulate all the calls from Python, at a dynamic page like LinkedIn, I think it will be quite tedious and brittle.
Anyway, you'd open "developer tools" in your browser before you open LinkedIn and see how the traffic looks like. You can filter for the requests from Javascript (in Firefox, the filter is called XHR).
You would then simulate the necessary/interesting requests in your code. The benefit is the servers usually return structured data to Javascript, such as JSON. Therefore you won't need to do as much HTML parsing.
If you find not progressing very much this way (it really depends on the particular site), then you will probably have to use Selenium or some alternative such as:
https://robotframework.org/
https://miyakogi.github.io/pyppeteer/ (port of Puppeteer to Python)
You should send all the XHR and JS requests manually [in the same session which you created during login]. Also, pass all the fields in request headers (copy from the network tools).
self.header_static = {
'authority': 'static-exp2.licdn.com',
'method': 'GET',
'path': '/sc/h/c356usw7zystbud7v7l42pz0s',
'scheme': 'https',
'accept': '*/*',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-GB,en;q=0.9,en-US;q=0.8,hi;q=0.7,la;q=0.6',
'cache-control': 'no-cache',
'dnt': '1',
'pragma': 'no-cache',
'referer': 'https://www.linkedin.com/jobs/',
'sec-fetch-mode': 'no-cors',
'sec-fetch-site': 'cross-site',
'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Mobile Safari/537.36'
}
def postConnectionRequests(self):
urls = [
"https://static-exp2.licdn.com/sc/h/62mb7ab7wm02esbh500ajmfuz",
"https://static-exp2.licdn.com/sc/h/mpxhij2j03tw91bpplja3u9b",
"https://static-exp2.licdn.com/sc/h/3nq91cp2wacq39jch2hz5p64y",
"https://static-exp2.licdn.com/sc/h/emyc3b18e3q2ntnbncaha2qtp",
"https://static-exp2.licdn.com/sc/h/9b0v30pbbvyf3rt7sbtiasuto",
"https://static-exp2.licdn.com/sc/h/4ntg5zu4sqpdyaz1he02c441c",
"https://static-exp2.licdn.com/sc/h/94cc69wyd1gxdiytujk4d5zm6",
"https://static-exp2.licdn.com/sc/h/ck48xrmh3ctwna0w2y1hos0ln",
"https://static-exp2.licdn.com/sc/h/c356usw7zystbud7v7l42pz0s",
]
for url in urls:
self.sess.get(url,headers=self.header_static)
print("REQUEST SENT TO "+url)
I called the postConnectionRequests() function after before saving the HTML content, and received the complete page.
Hope this helps.
XHR is send by JavaScript and Python will not run JavaScript code when it will get page using requests and beautifulsoup. Tools like Selenium loads page and runs JavaScript. You can also use Headless Browsers.
I have a problem with vue-resource causing a Provisional headers are shown on Chrome, using jQuery on the other hand work without any problem
The problem only happens with Chrome + vue-resource
Reproduction Link
Chrome 57.0.2987
Windows 7
I don't have adblock or origin installed, and it happen even on guest mode on Chrome
A simple call set with setInterval
new Vue({
el: '#main',
data: {
summary: null
},
methods: {
updateSummary: function() {
/*
$.post( "summary.php", function( data ) {
if(typeof response.body.summary != 'undefined'){
this.summary = response.body.summary;
}
});
*/
this.$http.post('summary.php').then(function(response) {
if(typeof response.body.summary != 'undefined'){
this.summary = response.body.summary;
}
});
}
},
mounted: function () {
this.updateSummary();
setInterval(function () {
this.updateSummary();
}.bind(this), 2000);
}
});
https://jsfiddle.net/7vo2s8z3/1/
Steps to reproduce
usually it happens when I leave the page open for a few hours
What is Expected?
A 200 code response with content served
What is actually happening?
I get a request with these headers
Request URL:http://127.0.0.1:8080/monitor/summary.php
Referrer Policy:no-referrer-when-downgrade
Request Headers
Provisional headers are shown
Accept:application/json, text/plain, /
Content-Type:application/json;charset=utf-8
Origin:http://127.0.0.1:8080
Referer:http://127.0.0.1:8080/monitor/
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
X-Requested-With:XMLHttpRequest
And looking at chrome://net-internals/#events the cause of failure is
85487: URL_REQUEST
http://127.0.0.1:8080/monitor/summary.php
Start Time: 2017-04-18 09:38:43.826
t=29028 [st= 0] +REQUEST_ALIVE [dt=24184]
--> priority = "MEDIUM"
--> url = "http://127.0.0.1:8080/monitor/summary.php"
t=29029 [st= 1] +DELEGATE_INFO [dt=24183]
--> delegate_blocked_by = "RedirectToFileResourceHandler"
t=53211 [st=24183] CANCELLED
--> net_error = -2 (ERR_FAILED)
t=53212 [st=24184] -REQUEST_ALIVE
I believe this happens when the actual request is not sent, usually when you are loading a cached resource.
Basically you sent a POST request to port 8080 which caused the "CAUTION: provisional headers are shown" message as seen in the inspector and then it seems the request was blocked all together.
Based on that, one possible solution is to setup nginx to proxy pass the request from the usual SSL port of 443 to the node SSL port of 8080 (node has to be on a higher port as it cannot be ran as root in prod). I guess Chrome doesn't like SSL requests to unconventional SSL ports, though I definitely agree their error message could be more specific.
I'm encountering a 403 status code with an UnrecognizedClientException in the x-amzn-errortype header of the response to my API Gateway GET Request using the generated Javascript SDK. The Resource being called utilizes IAM Auth which differentiates the users role based on their user group.
Here is my API Client Initialize Function
function initializeAPIClient(accessKey, secretKey, sessionToken){
var config = {
region : region,
accessKey : accessKey,
secretKey : secretKey,
sessionToken : sessionToken
}
apigClient = apigClientFactory.newClient(config);
}
Here is my GET request Function
function testCall(){
var params = '';
var body = '';
var additionalParams = '';
apigClient.testCallGet(params, body, additionalParams)
.then(function(result){
alert("Permissions are available to this user.");
})
.catch(function(result){
alert("Permissions are NOT available to this user.");
});
}
Here are my request headers:
:authority:[API_ENDPOINT]
:method:GET
:path:/[STAGE]/[RESOURCE]
:scheme:https
accept:application/json
accept-encoding:gzip, deflate, sdch, br
accept-language:en-US,en;q=0.8
authorization:AWS4-HMAC-SHA256 Credential=[ACCESS_KEY_ID]/20170406/[REGION]/execute-api/aws4_request, SignedHeaders=accept;host;x-amz-date, Signature=[SIGNATURE]
origin:http://localhost:8000
referer:http://localhost:8000/php/[PAGE].php/?username=[USERNAME]&sessionToken=[SESSION_TOKEN]
user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
x-amz-date:20170406T180808Z
x-amz-security-token:[SESSION_TOKEN]
I'm not sure what could be causing this. The solutions recommended when I search UnrecognizedClientException seem to suggest doing what I'm already doing.
I've solved my own issue, so here's the answer for anybody who runs into a similar logic error. Do NOT use the Id token as your session token, which is what I was doing. The id token is used to generate the session token, along with the access key and secret key. Do not confuse the two.
I am trying to use an interceptor to add a custom header to every request in an AngularJS App using the following code:
angular.module('app').factory('httpRequestInterceptor', function () {
return {
request: function (config) {
config.headers['testheader'] = 'testheaderworks';
return config;
}
};
});
angular.module('app').config(function ($httpProvider) {
$httpProvider.interceptors.push('httpRequestInterceptor');
});
This code was copied from the answer to this question
Unfortunately, when I examine the resulting requests, I get the following:
Provisional headers are shown
Access-Control-Request-Headers:accept, testheader
Access-Control-Request-Method:GET
Origin:http://localhost:61577
Referer:http://localhost:61577/
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
I confirmed this in both the network tab in Chrome and on the server side. Why is the custom header key 'testheader' added to Access-Control-Request-Headers rather than the general headers? What happened to the value? Is there another way to add custom headers to every AngularJS request that avoids this issue?
In case anyone reads this and is having the same issue:
The problem was that Angular was making a cross origin request, which the browser was preventing. In order to enable this request I had to enable the header on the server side. In our case (NodeJs) the code to do this was:
app.use(function (req, res, next) {
res.header("Access-Control-Allow-Headers", testheader");
next();
});
Using node.js and the Request package from the browser (via browserify), I am using CORS to do a HTTP GET request on a separate domain.
On the server, when I set 'Access-Control-Allow-Origin' to the wildcard '*', I get the following error on the client:
XMLHttpRequest cannot load .... A wildcard '*' cannot be used in the
'Access-Control-Allow-Origin' header when the credentials flag is
true. Origin '...' is therefore not allowed access.
The HTTP request header looks like this:
Accept:*/*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8,ja;q=0.6
Access-Control-Request-Headers:withcredentials
Access-Control-Request-Method:GET
Cache-Control:no-cache
Connection:keep-alive
Host:localhost:3000
Origin:http://localhost:9966
Pragma:no-cache
Referer:http://localhost:9966/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
So clearly the problem is Access-Control-Request-Headers:withcredentials in the header, right?
To be able to remove this, I need to set the 'withcredentials' property of the 'XMLHttpRequest' object to 'false'. However, I cannot figure out where node.js or the Request package are creating the 'XMLHttpRequest' object, and how I can even access this.
Thanks.
After some investigation, I discovered that the withCredentials setting can be passed in via the options parameter object:
var req = http.request({
withCredentials: false
}, function(res) {
//...
});
req.end();
If undefined, the default setting is true.
Reference from the http-browserify/lib/request.js source:
if (typeof params.withCredentials === 'undefined') {
params.withCredentials = true;
}
try { xhr.withCredentials = params.withCredentials }
catch (e) {}