Develop Reference

If I pull a url with automatic redirects from a .json, how can I make the output the page that I would have been redirected to? - Javascript [duplicate]

I was wondering if anyone knew how to handle redirects with the Request npm from sites such as bitly or tribal or Twitter's t.co URLs. For example, if I have web page that I want to scrape with the Request npm and the link I have to get to that page is a bity or shortened URL that is going to redirect me, how do I handle those redirects?
I found that the Request npm has a "followRedirect" options set to true by default. If I set that to false I can get the next link that the page will redirect me to by scraping that page that is returned, but that isn't the best because I don't know how many redirects I am going to have to go through.
Right now I am getting a 500 error. When I have "followRedirect" set to true. When I have "followRedirect" set to false, I can get each redirect page. Again, I don't know how many redirect pages I will have to go through. Code is below:
var options = {
followRedirect: false
};
request('http://t.co/gJ74UfmH4i', options, function(err, response, body){
// when options are set I get the redirect page
// when options are not set I get a 500
});

At first, you need to get the last redirect url, using followAllRedirects: true parameter
request('http://t.co/gJ74UfmH4i', {
method: 'HEAD',
followAllRedirects: true
}, function(err, response, body) {
var url = response.request.href
})
>
The second part is making request to final url, with some browser-like headers
request(url, {
headers: {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.46 Safari/537.36"
},
}, function(err, response, body) {
//here is your body
})

The Request package follows HTTP 3xx redirects by default but the URL you are using is returning an HTTP 200 with a META REFRESH style of redirect. I'm not sure if Request supports this particular style of redirect so you may need to parse the response and follow it manually.
GET http://t.co/gJ74UfmH4i HTTP/1.1
HTTP/1.1 200 OK
cache-control: private,max-age=300
content-length: 208
content-type: text/html; charset=utf-8
date: Fri, 28 Aug 2015 16:28:59 GMT
expires: Fri, 28 Aug 2015 16:33:59 GMT
server: tsa_b
set-cookie: muc=b0a729d6-9a30-466c-9cd9-57306369613f; Expires=Wed, 09 Aug 2017 16:28:59 GMT; Domain=t.co
x-connection-hash: 28133ba91da8c83d45afa434e12f8a72
x-response-time: 9
x-xss-protection: 1; mode=block
<noscript><META http-equiv="refresh" content="0;URL=http://nyti.ms/1EmZJhP"></noscript><title>http://nyti.ms/1EmZJhP</title><script>window.opener = null; location.replace("http:\/\/nyti.ms\/1EmZJhP")</script>
One possible route to understanding the issue would be to use a function for followRedirect to see if you can find out where it's failing.
From the README:
followRedirect - follow HTTP 3xx responses as redirects (default: true). This property can also be implemented as function which gets response object as a single argument and should return true if redirects should continue or false otherwise.

Make a http request with ajax for basic authorization and cors

The config for my httpd server 111.111.111.111 (supposed).
Config for cors and basic auth in /etc/httpd/conf/httpd.conf.
<Directory "/var/www/html">
Options Indexes FollowSymLinks
AllowOverride AuthConfig
Require all granted
Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Allow-Methods "POST, GET, PUT, DELETE, OPTIONS"
Header always set Access-Control-Allow-Credentials "true"
Header always set Access-Control-Allow-Headers "Authorization,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With"
</Directory>
Make some more configs for basic authorization on my server 111.111.111.111.
cd /var/www/html && vim .htaccess
AuthName "login"
AuthType Basic
AuthUserFile /var/www/html/passwd
require user username
Create password for username.
htpasswd -c /var/www/html/passwd username
Reboot httpd with :
systemctl restart httpd
The /var/www/html/remote.html on the server 111.111.111.111 .
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="Access-Control-Allow-Origin" content="*" />
</head>
<body>
<p>it is a test </p>
</body>
</html>
Test it with username and passwd when to open 111.111.111.111\remote.html?username=xxxx&password=xxxx in browser.
it is a test
Get the response header with curl.
curl -u xxxx:xxxx -I http://111.111.111.111/remote.html
HTTP/1.1 200 OK
Date: Thu, 06 Sep 2018 00:59:56 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization,X-PINGOTHER,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With
Last-Modified: Wed, 05 Sep 2018 15:01:05 GMT
ETag: "f5-575210b30e3cb"
Accept-Ranges: bytes
Content-Length: 245
Content-Type: text/html; charset=UTF-8
Add a parameter OPTIONS in header .
curl -X OPTIONS -i http://111.111.111.111/remote.html
HTTP/1.1 401 Unauthorized
Date: Thu, 06 Sep 2018 06:42:04 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization,X-PINGOTHER,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With
WWW-Authenticate: Basic realm="please login"
Content-Length: 381
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
Add OPTIONS and basic authorization in header.
curl -X OPTIONS -u xxxx:xxxxx -i http://111.111.111.111/remote.html
HTTP/1.1 200 OK
Date: Thu, 06 Sep 2018 06:42:54 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization,X-PINGOTHER,DNT,User-Agent,Keep-Alive,Content-Type,accept,origin,X-Requested-With
Allow: POST,OPTIONS,GET,HEAD,TRACE
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Ok, everything is good status.
Let's try ajax's basic authorization.
The /var/www/html/test.html on my local apache.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title></title>
</head>
<body>
<script src="http://127.0.0.1/jquery-3.3.1.js"></script>
<script>
function Ajax( ) {
var url = 'http://111.111.111.111/remote.html';
$.ajax(url, {
type:"get",
dataType: 'html',
withCredentials: true,
username: "xxxx",
password: "xxxx",
success:function(response){
mytext = $("#remote");
mytext.append(response);
},
error: function (e) {
alert("error");
}
});
};
</script>
<input type="button" value="show content" onclick="Ajax();">
<p id="remote">the content on remote webpage</p>
</body>
</html>
To click show content button when to input 127.0.0.1/test.html,i got the error:
GET http://111.111.111.111/remote.html 401 (Unauthorized)
I have given a detailed description based on httpd setting (centos7) and ajax and others related to the issue, please download my code and save it in your vps and local htdocs directory, replace ip with your real ip, reproduce the process.
I beg you not to make any comments until you reproduce the process.
you may find what happened, maybe it is same as mine here.
Two important elements in the issue.
1.httpd setting in the file /etc/httpd/conf/httpd.conf.
2.ajax code
Which one is wrong?
How to fix it?
I have some clue to solve the issue, thanks to #sideshowbarker.
reason
The problem turns into another one:
How to configure the apache to not require authorization for OPTIONS requests?
I have tried as Disable authentication for HTTP OPTIONS method (preflight request) say.
Disable authentication for HTTP OPTIONS method (preflight request)e
<Directory "/var/www/html">
<LimitExcept OPTIONS>
Require valid-user
</LimitExcept>
</Directory>
systemctl restart httpd,failed.

If you console.log the error you will see a message saying:
Failed to load http://111.111.111.111/remote.html: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. Origin 'http://127.0.0.1/' is therefore not allowed access. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.
This is to protect against CSRF attacks.
E.g. If wildcard could be used in this case: Malicious-SiteA could access privileged resources without authorization on target-SiteB through browser-initiated ajax reqests, just because victim at some point provided crendentials for SiteB, while he was on trusted-SiteC.
So the solution is to just change Access-Control-Allow-Origin from "*" to "http://127.0.0.1", (SiteC address in the example above)
Now in order for curl -X OPTIONS -i http://111.111.111.111/remote.html to work while keeping active authentication for other methods, you need to add the following to .htaccess, or to httpd.conf:
<LimitExcept OPTIONS>
AuthName "login"
AuthType Basic
AuthBasicProvider file
AuthUserFile /var/www/cors-auth.passwd
Require user username
</LimitExcept>
EDIT:
LimitExcept OPTIONS is required if you need to prefill the username/password (as in your case), instead of expecting the browser to prompt a dialog. But to make it work on all browsers (chrome/ff/edge),
I had to replace this:
withCredentials: true,
username: "xxxx",
password: "xxxx",
with this:
beforeSend: function (xhr){
xhr.setRequestHeader('Authorization', "Basic " + btoa("xxxx:xxxx"));
},

try putting localhost instead of 'http://111.111.111.111/remote.html'
'http://localhost/remote.html .It should work .It may be your public ip address issue where port 80 is not open.

Cross-Origin Request Blocked between Node dev server and Spring Boot application

My stack is as follows:
Backend: Spring boot(Java) exposed at :8088
Frontend: Vue hosted on a Node development server exposed at :8080
On the frontend, I am re-configuring axios in a http-common.js to put the baseURL to the Spring boot application, and allow connection from the node development server:
import axios from 'axios'
export const AXIOS = axios.create({
baseURL: `http://localhost:8088`,
headers: {
'Access-Control-Allow-Origin': 'http://localhost:8080'
}
})
However, when attempting to make a post request to log in, I will get the following message in the console:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8088/api/login. (Reason: CORS preflight channel did not succeed).
Which makes me think: Is the issue with the spring boot application?
But no, in the main method, I have enabled CORS globally when reaching the /api/* endpoints from the node application running at :8080:
#Bean
public WebMvcConfigurer corsConfigurer() { // Enables CORS globally
return new WebMvcConfigurerAdapter() {
#Override
public void addCorsMappings(CorsRegistry registry) {
registry.addMapping("/api/*").allowedOrigins("http://localhost:8080");
}
};
}
To me it looks as if it should be configured correctly. However, as of now, the following POST of username + password never even reaches the backend Spring boot application at all. The issue must be with the Node application?
This is the Login method in the frontend:
login ({commit}, authData) {
AXIOS.post('/api/login', {
username: authData.username,
password: authData.password,
withCredentials: true
})
.then(res => {
console.log(res)
commit('authUser', {
token: res.data.idToken,
userId: res.data.localId
})
})
.catch(error => console.log(error))
}
To further solidate my point, i can cURL to the spring boot application and get the correct response(a valid JWT!):
Request:
curl -i -H "Content-Type: application/json" -X POST -d '{
"username": "sysadmin",
"password": "sysadmin"
}' http://localhost:8088/api/login
Response:
HTTP/1.1 200
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
X-Application-Context: application:8088
authentication: <very long JWT string>
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
So, via cUR - I get a HTTP 200 OK, and a valid JWT. But via the same POST method from :8080, I get a 403 and a warning message.
As per other posts, I have attempted to add CORS to my dev server configuration(Node/Express):
var app = express()
app.use(cors())
app.options('*', cors())
app.use(function (req, res, next) {
// Website you wish to allow to connect
res.setHeader('Access-Control-Allow-Origin', 'http://localhost:8088')
// Request methods you wish to allow
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS, PUT, PATCH, DELETE')
// Request headers you wish to allow
res.setHeader('Access-Control-Allow-Headers', 'X-Requested-With,content-type')
// Set to true if you need the website to include cookies in the requests sent
// to the API (e.g. in case you use sessions)
res.setHeader('Access-Control-Allow-Credentials', true)
// Pass to next layer of middleware
next()
})
The result is exactly the same as previously

Adding the 'Access-Control-Allow-Origin' header to your ajax post call is useless since is part of cors specification and must be set by the server as part of the http response.
export const AXIOS = axios.create({
baseURL: `http://localhost:8088`,
headers: {
//you can remove this header
'Access-Control-Allow-Origin': 'http://localhost:8080'
}
})
You can curl the application because the cors exception is caused by the browser disallowing you to access the payload. The browser performs the preflight (OPTION) request before any Cross domain call, and before your actual http request to make sure you have the rights to see the payload, you can see it just inspecting the console under the network tab.
the issue is most likely server side, somehow you did not configure correctly the cors header to your http response.
make sure you're setting not only the Access-Control-Allow-Origin header (that must contain the specific domain, not * since you're in credential mode), but even Access-Control-Allow-Credential since you're sending credentials, and the Access-Control-Allow-Methods (that must contain at least the PUSH and the OPTION methods)
in your chrome dev tools console under the network tab if you inspect your ajax call you can see the header of the http response, should end up with something like this.

Have you tried to add #CrossOrigin to your login REST method?
#CrossOrigin(origins = "http://localhost:8080")
#GetMapping("/greeting")
public Greeting greeting(#RequestParam(required=false, defaultValue="World") String name) {
System.out.println("==== in greeting ====");
return new Greeting(counter.incrementAndGet(), String.format(template, name));
}
Update: I just read this on javadoc:
Exact path mapping URIs (such as "/admin") are supported as well as Ant-style path patterns (such as "/admin/**").
I don't see here a path with one star, but your path is a correct Ant-style path..

Ajax Angular SSL CORS

I actually do not understand this issue. I am not very much into SSL and certificates.
A script on test.kanubox.de (You can try it there and look at the source code) uses ajax to call rest server on sandbox.api.kehrwasser.com/kanubox/v1. Obviously CORS is needed and works well without SSL, thus I assume that CORS is set up correctly. The header data on an OPTIONS-request (preflight) to the API confirms
Access-Control-Allow-Origin: *
Upgrade: h2,h2c
Pragma: no-cache
Keep-Alive: timeout=5, max=100
Content-Type: application/json
Content-Encoding: gzip
Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT
Server: Apache/2.4
Expires: Fri, 19 Aug 2016 12:15:58 GMT
Access-Control-Max-Age: 500
Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, authorization, accept, client-security-token, Access-Control-Allow-Origin, X-Frame-Options
But when I switch to https://test.kanubox.de and call the API at https://sandbox.api.kehrwasser.com/kanubox/v1 I get CORS error from FireFox like "(Cross-Origin blocked)
Reason: CORS-Header 'Access-Control-Allow-Origin' missing
(Translated error message)
The certificate is from my hoster and verified by my hoster itself. I'm not sure but is it "self-signed" then? So maybe FF blocks it because it doesn't trust it?
Here is my code:
var test = angular.module("test", []);
test.constant('apiConfig', {
apiUrl: "https://sandbox.api.kehrwasser.com/kanubox/v1"
});
test.controller("TestController", function($scope, $http, apiConfig) {
var credentials = { mail: "user#mailserver.com", password: "12345" };
// POST REQUEST VIA SSL
$http({
url: apiConfig.apiUrl + "/users/auth/",
method: 'POST',
data: credentials
}).success(function(data, status, headers, config) {
$scope.variable = data;
}).error(function(data, status, headers, config) {
$scope.variable = data;
});
});

If I browse to https://test.kanubox.de/, then the server certificate is not known in my firefox browser. It is indead a self signed certificated, issued by "Hostpoint DV SSL CA - G2" itself!
To make that SSL certificate work, you need the "Hostpoint" root certificate in your browser. That is exactly how you made it work! So, is was a trusted ROOT certificate issue.
When the SSL problem is solved, then you can look at the CORS issue.
The certificate which is used in https://sandbox.api.kehrwasser.com/kanubox/v1/ is a issued by the well known CA "Let's incrypt". That works fine.

Request npm: Handling Redirects

I was wondering if anyone knew how to handle redirects with the Request npm from sites such as bitly or tribal or Twitter's t.co URLs. For example, if I have web page that I want to scrape with the Request npm and the link I have to get to that page is a bity or shortened URL that is going to redirect me, how do I handle those redirects?
I found that the Request npm has a "followRedirect" options set to true by default. If I set that to false I can get the next link that the page will redirect me to by scraping that page that is returned, but that isn't the best because I don't know how many redirects I am going to have to go through.
Right now I am getting a 500 error. When I have "followRedirect" set to true. When I have "followRedirect" set to false, I can get each redirect page. Again, I don't know how many redirect pages I will have to go through. Code is below:
var options = {
followRedirect: false
};
request('http://t.co/gJ74UfmH4i', options, function(err, response, body){
// when options are set I get the redirect page
// when options are not set I get a 500
});

At first, you need to get the last redirect url, using followAllRedirects: true parameter
request('http://t.co/gJ74UfmH4i', {
method: 'HEAD',
followAllRedirects: true
}, function(err, response, body) {
var url = response.request.href
})
>
The second part is making request to final url, with some browser-like headers
request(url, {
headers: {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.46 Safari/537.36"
},
}, function(err, response, body) {
//here is your body
})

The Request package follows HTTP 3xx redirects by default but the URL you are using is returning an HTTP 200 with a META REFRESH style of redirect. I'm not sure if Request supports this particular style of redirect so you may need to parse the response and follow it manually.
GET http://t.co/gJ74UfmH4i HTTP/1.1
HTTP/1.1 200 OK
cache-control: private,max-age=300
content-length: 208
content-type: text/html; charset=utf-8
date: Fri, 28 Aug 2015 16:28:59 GMT
expires: Fri, 28 Aug 2015 16:33:59 GMT
server: tsa_b
set-cookie: muc=b0a729d6-9a30-466c-9cd9-57306369613f; Expires=Wed, 09 Aug 2017 16:28:59 GMT; Domain=t.co
x-connection-hash: 28133ba91da8c83d45afa434e12f8a72
x-response-time: 9
x-xss-protection: 1; mode=block
<noscript><META http-equiv="refresh" content="0;URL=http://nyti.ms/1EmZJhP"></noscript><title>http://nyti.ms/1EmZJhP</title><script>window.opener = null; location.replace("http:\/\/nyti.ms\/1EmZJhP")</script>
One possible route to understanding the issue would be to use a function for followRedirect to see if you can find out where it's failing.
From the README:
followRedirect - follow HTTP 3xx responses as redirects (default: true). This property can also be implemented as function which gets response object as a single argument and should return true if redirects should continue or false otherwise.