I have an account at Vimeo.com and am trying to figure out how to log in using the Python Requests package. Here are my initial observations:
Logging in is done via a post request to https://vimeo.com/log_in?ssl=0&iframe=0&popup=0&player=0&product_id=0 and requires an email, password, action, service, and token to be passed as body data
The token for the post request appears to be different from the xsrft token but I'm not able to get either as the token appears to be generated via javascript and only valid for each session (though it isn't clear)
Cookies are necessary
Here is some initial code:
import requests
from bs4 import BeautifulSoup
url = 'http://vimeo.com/log_in?ssl=0&iframe=0&popup=0&player=0&product_id=0'
payload = {'email': 'some#email.com',
'password': 'privatepassword',
'action': 'login',
'service': 'vimeo',
'token': '?????'
}
response = requests.post(url, data=payload, allow_redirects=True)
soup = BeautifulSoup(response.text)
I can't figure out how to get the correct token for the post request and whether the token is associated with the vuid or xsrft token.
Related
In this test case am sending an axios post request with userId and password to ExpressJS server running with passportjs local. Server respond with status code 200, and send appropriate header with set-cookie.
I need subsequent request to be treated as authorized request, for that tried following options, but none seems to be working. It getting rejected with status code 401.
First call with userid and password, responded with status 200
const userDoc = {
userId: 'test-user-1',
userName: 'Test User 1',
emailId: 'test.user.1#abc.xom',
password: 'test-password'
} ;
let resp
resp = await axios({method : 'post', url : 'http://localhost:4040/auth/local', data : {userId: userDoc.userId, password: userDoc.password },withCredentials: true })
following options are used to send next request
send cookies received as part of 1st request
const headers = { headers : {Cookie: resp.headers['set-cookie'][0] } };
send header as it is received as part of 1st request
const headers = { headers : resp.headers};
send withCredentials: true along with above headers.
Second call is made with either of above option
resp = await axios({method : 'post', url : 'http://localhost:4040/v1/master/account', data : accountDoc , headers, withCredentials: true})
used httpAgent, keepAlive with axios instance
const axios = require('axios')
const http = require("http")
const httpAgent = new http.Agent({keepAlive : true , timeout :1000})
const instance = axios.create({httpAgent})
const resp1 = await instance({method : 'post', url : 'http://localhost:4040/auth/local', data : {userId: userDoc.userId, password: userDoc.password, } , withCredentials: true })
const resp2 = await instance({method : 'post', url : 'http://localhost:4040/v1/master/account', data : accountDoc , withCredentials: true })
Rejected with status code 401
-- Error: Request failed with status code 401
at createError (/home/Projects/FinAccounts2003/node_modules/axios/lib/core/createError.js:16:15)
at settle (/home/Projects/FinAccounts2003/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/home/Projects/FinAccounts2003/node_modules/axios/lib/adapters/http.js:269:11)
at IncomingMessage.emit (events.js:412:35)
at endReadableNT (internal/streams/readable.js:1334:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)
Server code is standard passport-js local code, which working well with browser.
It may be duplicate of some of the questions, solutions given are 1) withCredentials: true, already tried above 2) Authorization: Bearer ${token} - not applicable in this case, in passport js, cookie is directly set, and not getting token.
One solution that worked for me was using the modules tough-cookie and axios-cookiejar-support. I combined them in a persistent-client.js file, and then I was able to maintain the session between requests (commonJS):
const axios = require('axios').default;
const { CookieJar } = require('tough-cookie');
const { wrapper } = require('axios-cookiejar-support');
module.exports = function () {
const jar = new CookieJar();
const client = wrapper(axios.create({ jar }));
return client;
}
There are two different ways to send the session authorization token from the server to the client (web browser)
Via (HttpOnly) response headers.
Via the response body.
And there are two different ways to authorize client requests (send the session token from the web browser to the server.)
A. Automatic: HttpOnly headers
B. Manual: Authorization: Bearer [TOKEN]
Usually method 1 is used with method A, and method 2 is used with method B. I think you are mixing them up.
If the server is using Set-Cookie to send the session token, then I think the browser automatically sends the session token automatically on all future requests (to the same domain).
Can you confirm what the actual contents of the set-cookie header are from the server? Note you will probably not be able to check this via JS if these are HttpOnly cookies; inspect the dev console "Network" tab. You can also check to see if any new cookies were set from the "Application" tab.
If the client does actually need to manually send the token via headers, the header needs to fit a specific Authorization cookie format. (Which you are not doing. You are simply echoing the headers received from the server.)
See my response to a similar question.
I don't believe you should be using any third party packages for this, especially not if they're directly accessing the cookies using javascript (which is an XSS security vulnerability). Cookies should be set using secure and http-only and never be accessed using Document.cookie directly.
Make sure that passport is actually setting your cookie and that you're correctly sending back the cookie on the login. Verify that it's been set in your browser.
Make sure that you have CORS enabled in express, that you've specified the domain you're making requests from and that you've enabled credentials in CORS.
Make sure that you're using withCredentials on your axios requests.
Make sure that you've set the cookie using the correct domain and path.
I have a js client (vuejs) and a backend using DRF both in local.
I use this package to generate the token : https://github.com/davesque/django-rest-framework-simplejwt
I use this package https://www.npmjs.com/package/axios-auth-refresh to handle refresh token logic.
The main goal is to intercept a request when it return a 401 response, perform a refresh token request and then resolve the orginal request with the new token.
It works when the original request is a GET request but not when it is a POST request.
When using a POST request :
The orgin request fall in 401 when the token expire then the interceptor occur but the server respond with 405 method not allowed:
-https://imgur.com/C1tchvb
the method from the request from the interceptor does not match the method in the code shown above (line 3 & 4) : as you can see the server receive the payload from the origin request as method of the request :
-https://imgur.com/nlAknMi
I found this post : App Script sends 405 response when trying to send a POST request
i try to change the headers as advised but it did not work
How is the payload from the orginal resquest becoming the method of the interceptor when the origin request is a Post request with a payload ?
Here the code from the javascript client :
const refreshAuthLogic = failedRequest => axios(
{
method: 'post',
url: 'auth/refresh',
data: { refresh: store.state.token.refresh }
}).then(tokenRefreshResponse => {
store.dispatch('refreshToken', tokenRefreshResponse.data)
return Promise.resolve()
})
const instance = axios.create({
baseURL: '/api/'
})
instance.interceptors.request.use(config => {
config.headers.Authorization = `Bearer ${store.state.token.access}`
return config
})
createAuthRefreshInterceptor(instance, refreshAuthLogic)
EDIT
I manage to get it work but i don't really understand:
the problem is related to DJANGO/ DRF and not axios
it seems that when a POST request is done and fail ( here with 401) the server keeped the data.
Here the part i can't explain :
when the request of the interceptor (to refresh token) hit the server, it messes with the data of previous request.
I had to add a middleware in django to clear the body when the request fails with 401 and it worked for me. But it is not a proper solution i guess.
Unfortunately the lib is loosely mantained and it's flawed in some aspects.
Eg: concurrent requests are not correctly queued when the request is sent with and invalid token but the response arrives when a new token is already issued.
As is, if you look at the lib source, you'll find in the very first lines:
/** #type {Object} */
const defaults = {
/** #type {Number[]} */
statusCodes: [
401 // Unauthorized
]
};
This means that only 401 code is managed and the statusCodes are not exported so them remains private.
If you want to continue to use this library you can fork it in order to change what does not fit with your stack or simply copy the source, edit it and use it as a local service.
I'm developing a Flask server to communicate between some backend Python functionality and Javascript clients over the web. I'm attempting to utilize Flask's session variable to store user specific data over the course of their time interacting with the app. I've removed most of the application specific code below but the core problem I'm experiencing remains.
Here is my the code for my (simplified) Flask app:
import json
import os
from flask import Flask, jsonify, request, session
app = Flask(__name__)
app.secret_key = 'my_secret_key'
#app.route('/', methods=['GET'])
def run():
session['hello'] = 'world'
return jsonify(session['hello'])
#app.route('/update', methods=['POST'])
def update():
return jsonify(session['hello'])
if __name__ == '__main__':
app.run(host='0.0.0.0')
Utilizing Postman, I can make a GET request to my server and receive the expected output of "world". I can then make a POST request with an arbitrary body and receive the same expected output of "world" (again using Postman).
When using Chrome, I can visit my server IP and see the expected output "world" on the page. I can also manually make a GET request using Javascript (in Chrome's console) and receive the same response as expected. However, my problem arises when trying to send a POST request to the server using Javascript; the server shows a KeyError: 'hello' when trying to make this request.
Here is the Javascript I'm using to make the POST request:
var url = 'http://my_server_ip/update';
fetch(url, {
method: 'POST',
body: JSON.stringify('arbitrary_string'),
headers: new Headers({
'Content-Type': 'application/json'
})
})
.then(response => response.json())
.then((data) => {
console.log(data);
})
What's going wrong here? Why can I make the GET/POST requests with Postman just fine but run into errors making the same requests with Javascript?
The caveats section of the fetch documentation says:
By default, fetch won't send or receive any cookies from the server, resulting in unauthenticated requests if the site relies on maintaining a user session.
It is recommended to use AJAX to exchange information with Flask views.
Meanwhile, in your code for the Flask app, the session object is a dictionary. Now, if you access a dictionary with its key session['hello'] and if this key does not exist, a Keyerror is raised. To get around this error, you can use the get() method for dictionaries.
What is happening is: the fetch request does not find the hello key(or GET the session value from the Flask view) in the Flask session.
user = session.get('hello')
return jsonify(session_info=user)
But this will still give you a null value for the session { session_info: null }. Why is that so?
When you send GET/POST requests to the Flask server, the session is initialized and queried from within Flask. However, when you send a Javascript fetch POST request, you must first GET the session value from Flask and then send it as a POST request to your Flask view which returns the session information.
In your code, when the POST request is triggered from fetch, when I send the payload data to Flask, it is received correctly and you check this using request.get_json() in the Flask view:
#app.route('/update', methods=['POST'])
def update():
user = session.get('hello')
payload = request.get_json()
return jsonify(session_info=user, payload=payload)
This will return { payload: 'arbitrary_string', session_info: null }. This also shows that fetch does not receive the session information because we did not call GET first to get the session information from Flask.
Remember: The Flask session lives on the Flask server. To send/receive information through Javascript you must make individual calls unless there is a provision to store session cookies.
const fetch = require('node-fetch');
var url_get = 'http://my_server_ip';
var url_post = 'http://my_server_ip/update';
fetch(url_get, {
method:'GET'
}).then((response)=>response.json()).then((data) =>fetch(url_post, {
method: 'POST',
body: JSON.stringify(data),
dataType:'json',
headers: {
'Content-Type': 'application/json'
}
})
.then(response => response.json())
.then((postdata) => {
console.log(postdata);
}));
The Flask views will change slightly:
#app.route('/', methods=['GET'])
def set_session():
session['hello'] = 'world'
return jsonify(session['hello'])
#app.route('/update', methods=['POST'])
def update():
payload = request.get_json()
return jsonify(session_info=payload)
When you trigger the Javacript request now, the output will be: { session_info: 'world' }
After a few hours of testing, I managed to figure out the issue. Although I think #amanb's answer highlights the problem, I'm going to answer my own question because what I found is ultimately a simpler solution.
In order to make the POST request return the expected value, I simply needed to add a credentials: 'same-origin' line to the fetch body. This looks like the following:
var url = 'http://my_server_ip/update';
fetch(url, {
method: 'POST',
body: JSON.stringify('arbitrary_string'),
credentials: 'same-origin', // this line has been added
headers: {
'Content-Type': 'application/json'
}
})
.then(response => response.json())
.then((data) => {
console.log(data);
})
According to Mozilla's Fetch usage guide,
By default, fetch won't send or receive any cookies from the server,
resulting in unauthenticated requests if the site relies on
maintaining a user session.
So it seems I looked over this. Changing the credentials to allow communication of the cookie/session between client and server resolved the issue.
For logging in, I'm doing something like:
function setHeader(xhr) {
// as per HTTP authentication spec [2], credentials must be
// encoded in base64. Lets use window.btoa [3]
xhr.setRequestHeader("Authorization", "Basic " + btoa(username + ':' + password));
}
$.ajax({type: "POST", url: AUTH_URL, beforeSend: setHeader}).
fail(function(resp){
console.log('bad credentials.')
}).
done(function(resp){
});
after which, I'm storing the Session in local storage.
However, for logging out, I'm unable to figure out how to use this session to send with the request header, so that django's : request.logout() logs out the user having that session id
For login you can add view similar to this one:
import json
import requests
from django.shortcuts import render_to_response
from django.http import HttpResponseRedirect
#csrf_protect
def login(request):
if request.method == "POST":
login = requests.post('http://your_url/api-token-auth/', data={'username': request.POST['username'], 'password': request.POST['password']})
response = json.loads(login.text)
if response.status_code == 200:
token = response['token']
request.session.flush()
request.session['user'] = request.POST['username']
if request.session.test_cookie_worked():
request.session.delete_test_cookie()
return HttpResponseRedirect("/")
else:
error = "Error"
request.session.set_test_cookie()
return render_to_response("login.html", {"error": error}, RequestContext(request))
For logout all you have to do in your view is:
def logout(request):
request.session.flush()
return HttpResponseRedirect('/')
On your API side, you have to define api-token-auth in urls: here is the tutorial for more informations
url(r'^api-token-auth/', 'rest_framework.authtoken.views.obtain_auth_token')
This way you will get your token for communication with the API. Beside TokenAuthentication you can define and SessionAuthentication. More about that you can find in the above tutorial
You are using HTTP Basic Authentication, which does not define a way to log users out. It is not tied to the Django session, so you can't clear that. You could potentially clear out the token from session storage, and send an invalid token, though the browser may opt to send the original credentials (untested).
There are quite a few questions about it on Stack Overflow. Your best bet looks like sending invalid credentials, hoping that the user's browser will invalidate any saved ones.
You may be able to use a form of token-based authentication, such as TokenAuthentication or OAuth, which will not be intercepted by the browser. This way you will not need to worry about logging users out, as the authentication is tied directly to requests made with the token.
I have successfully logged in to the API Server using following code in meteorjs
var request = Npm.require('request');
request('http://api-server-link-here',
{
'auth' : {
'user': 'username',
'pass': 'password',
'sendImmediately': false
}
}
, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(response);
}
But after this login when i go for using search query using code below. I get Http Status 401 error.
var request = Npm.require('request');
request("http://search-query-link-here",
{
//search query parameters are here
},
function(error,response,body){
console.log(response);
});
Can anybody please give me pointers of why this issue is happening. Or if this is possible that i get a working example here?
One more thing that needs to be told here is that I'm doing login with the help of digest authentication.
Obviously, it's kind of hard to provide you with a working example for an unknown API.
But let's just try to sort it out. Logically, when you send your search query, you are supposed to identify yourself. Most of the public API servers use either tokens (most probably) or session cookies in order to authorize access to a resource. So, whenever you do your second (search) request, you have either provide a token or send a cookie you are getting from the first (login) request.
Long story short, verify which mechanism is being used by the API server.
if it's tokens: check the response body from the login request; most probably there will be an access token you'd send then in the Authorization header of your API calls: headers: {'Authorization': 'Bearer ...'} in your request options;
if it's sessions: make sure you have the cookie jar enabled on login and you it for subsequent requests: https://github.com/mikeal/request#requestjar