Exploring the 'request' package to perform some automated tasks at the office.
The requirement is to automatically log into a server (but there are many many), navigate to a web page and then collect information then create a report with it.
Thing is, these servers first ask for a user account, then they generate a challenge number that one would copy and paste into another server to be resolved as the challenge response. This response is pasted back into the web page I want the script to log in.
Trying to figure out how to go about doing this with 'request'. Following code only gets as far as looping through the page where to enter the user name:
"use strict"
var request = require('request');
var getJar = request.jar();
var opts = {
method: 'GET',
uri: 'https://serverIP/path/to/post/action',
agentOptions: {
rejectUnauthorized: false
},
jar: getJar,
followAllRedirects: true
};
var postJar = request.jar();
var postOpts = {
method: 'POST',
uri: 'https://serverIP/path/to/post/action',
form: {
userName: 'init'
},
agentOptions: {
rejectUnauthorized: false
},
postJar: postJar,
followAllRedirects: true,
headers: {
'Referer': 'value I see in dev console'
}
};
request(opts, function (err, res, body) {
console.log('GET error is: ' + JSON.stringify(err));
console.log('GET response is: ' + res);
console.log('GET body is: ' + body);
var cookie_string = getJar.getCookieString(opts.uri);
var cookies = getJar.getCookies(opts.uri);
console.log(JSON.stringify(cookies) + '\n' + cookie_string);
postOpts.postJar.setCookie(cookie_string, postOpts.uri);
request(postOpts, function (err, res, body) {
console.log('POST error is: ' + JSON.stringify(err));
console.log('POST response is: ' + res);
console.log('POST body is: ' + body);
});
});
The form parameters in the snippet are things I see in Firefox's developer console under Network > POST request > Params when I submit the user name through the browser.
The code is only taking me back to the page where the user name is requested. It kind of gets stuck there. I know this by logging the value of the POST body to console, it spits out the HTML code asking for that value, instead of the HTML where you see the challenge string and the input field where to paste the challenge response.
What may I do differently?
I used 'request-debug' package for further insight. Turned out, the first GET request happened successfully, from which I take the cookie=sessionId value, and inject it into the POST message jar as shown in my initial snippet.
With the debug on, I was able to detect a redirection to a URL saying 'cookiesRequired' at the end, request was following this redirection and generating a second GET to it, where I was shown HTML code with the message 'Invalid session you must be logged in'.
It then hit me that, the cookie-session system was not working properly. I saw from the second GET that it inserted a header with the session id taken from the cookie. So I injected this header into postJar like this:
var cookie_string = getJar.getCookieString(opts.uri);
console.log('ATTENTION--- The cookie from getJar is:\t' + cookie_string);
postOpts.headers.cookie = cookie_string;
This line doesnt even affect the behavior so far:
postOpts.jar.setCookie(cookie_string, '/');
After inserting the cookie header I am now getting prompted to resolve the next phase of the authentication.
Related
I'm trying to build a python tool that sends an email to a helpdesk. it does this using flask, javascript, jQueryUI and python. I have set up a dialogue box to capture information needed for a ticket. this information (as well as some information about the user's browser) is then saved as a variable called emailinfo which is passed through a fetch request to flask, and then to a function in python that sends an email. I know that the function I've written will send an email because I've tested it. I also know that flask is receiving the emailInfo variable and that it has information in it, because flask prints the variable it receives and the JS console also prints the information it receives. both of these are fine, but when it tries to send the email, it doesn't send anything.
JavaScript/jQueryUI:
emailInfo = JSON.stringify("User's Name: " + name.val() + " "
+ "User's E-Mail: " + + email.val() + " "
+ "Reported Issue: " + issue.val() + " "
+ "More Details: " + " "
+ start.val(), null, 2)
console.log(emailInfo)
//get IP/browser information
$.getJSON('http://ip-api.com/json', function(data) {
//put together email info
emailInfo = emailInfo + " " + JSON.stringify(data, null, 2)
emailInfo = JSON.stringify({emailInfo})
console.log(emailInfo)
});
//fetch helpdesk function
//note: this is functionally the same as another fetch request I have written that works perfectly fine
fetch(`${scriptRoot}/helpdesk`, {
method: 'POST',
body: JSON.stringify({ mailInfo: emailInfo }),
headers: {
'Content-Type': 'application/json'
},
})
}
return valid;
}
Flask
(all functions from other file have been imported)
#app.post("/helpdesk")
def sendtodesk():
mailInfo = request.get_json().get("mailInfo")
mailInfo = str(mailInfo)
#extra info - this prints only the first line of mailInfo and not the scraped IP. this doesn't really matter to me, though, because I think I can fix it
print(mailInfo)
SendEmail(mailInfo)
return("Sent! Thank you.")
Python Email Script
note: I have tested this separately and it works
def SendEmail(emailInfo):
port = 465 # For SSL
smtp_server = "smtp.gmail.com"
#sender and reciever address. here its the helpbot's email and the address of the testing environment helpdesk
senderemail = "(email sender goes here)"
password = "(password to email sender goes here)"
recieveremail = "(email reciever goes here)"
#message
message = emailInfo
context = ssl.create_default_context()
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
server.login(senderemail, password)
server.sendmail(senderemail, recieveremail, message)
I've tried a lot of things to get this to work. the best success I've had has been in making it similar to another flask function I've made that works perfectly fine, which has allowed me to get as far as printing the variable. I can't tell whether the variable gets through to the function, however I know that the SendEmail function is called because the sender sends an email. This is the email that is sent, there is nothing in there, which makes me think that flask never passes the variable on properly.
thanks for any and all help you can give me. I've already done a lot for this project (it is also an NLTK chatbot that works perfectly fine, which is how I've done flask before - the code here is very similar to the code used there) and I've really hit a brick wall with this after doing everything in my programming power.
turns out it was due to the format of the emailinfo variable. the smtp and ssl method of sending mail is a string, but requires a certain format. having colons broke it. I need to have colons because that's the format that the IP API returns the information I need in. switching to another email package (I used yagmail) has worked fine :)
First improvment, use string formatting:
let email_info = `User's Name: ${name.val()}\n User's E-Mail: ${email.val()}\n Reported Issue: ${issue.val()}\n More Details: ${start.val()}\n`;
Second improvment, create a second entry in json and seperate the data,
also use fetch api just to get data, and combine this data together on python side.
{ mailInfo: emailInfo, ipapiInfo: <your json data from ip-api>}
then u can better debug wich or where date get lost, maybe this helps u a little bit
I am having trouble with the authentication process for the GoogleAPI. In the end I want to be able to read the users steps using the GoogleFit API and then store that value in a database. Currently I'm using restdb.io and executing javascript in codehooks.
The documentation from Google that I am following can be found here, clicking on the HTTP/REST option in the code examples. At the moment I am at step 5: I have gotten the users authentication code and stored it in the database. Now I have to POST the code along with some other parameters and get the access and refresh tokens.
If the POST is successful (from what I understand) I should get back a 200-OK message that the request was valid. Google will then POST a JSON body with the access and refresh token to the redirect_uri that I have specified in my GoogleAPI credentials page and the initial request. At redirect_uri I have to handle the request and save the two values.
The problem is that I receive a redirect_uri_mismatch - Bad Request message from Google as a response when executing the request. I get it at the log.debug("ERROR HERE: " + [...]); in the code below:
async function mainFunction(){
const authCode = THIS_IS_MY_AUTHENTICATION_CODE;
try {
var answer = await postRequestToGoogle(authCode);
//do stuff with response from Google
} catch (error) {
//do stuff
}
}
async function postRequestToGoogle(authCode){
//body for the request
const params = "code=" + authCode + "&" +
"client_id=THIS_IS_MY_CLIENT_ID" + "&" +
"client_secret=THIS_IS_MY_CLIENT_SECRET" + "&" +
"redirect_uri=THIS_IS_MY_REDIRECT_URI" + "&" +
"grant_type=authorization_code";
try{
const result = await fetch('https://oauth2.googleapis.com/token', {
method: 'POST',
headers: {'Content-Type': 'application/x-www-form-urlencoded'},
body: params})
.then(res => {
log.debug("ERROR HERE: " + JSON.stringify(res.json()));
return res.json();
})
//return JSON back main function
return result;
}catch(error){
//do stuff
}
}
I looked up the error message and tried some things:
Copy and pasted multiple different Authorized redirect URI from the GoogleAPI credentials page into the code to make sure that there is no problem with
http/https
www/no www
trailing slashes
typos or capitalization
Waited for changes to be processed by Google (read that it can more than 30min)
Changed all the other parameters to see if the redirect_uri is actually the problem
If code is changed the message is invalid_grant - Bad Request
If client_id is changed the message is invalid_client - The OAuth client was not found
If client_secret is changed the message is invalid_client - Unauthorized
If the grant_type is changed the message is unsupported_grant_type - Invalid grant_type
That's why I think the issue is the redirect_uri, but it's unclear to me how since I copy&pasted it. Something that came to mind was that maybe the value of redirect_uri gets changed when it's read by Google? Or maybe when the request is being put together? Do some characters have to be replaced?
I tried to analyze the request with Wireshark but didn't think about the fact that it's HTTPS so I would have I would have to decrypt it.. Is that something I should look into?
Thank you for taking the time to read all of this! If you have any advice please let me know :)
Update 16.11.20:
I have created a new OAuth 2.0 Client ID and used the new id/secret in my request. The resulting message the same as before. I will wait and try again tomorrow to see if maybe Google needs some more time. Then I'll try to delete all current IDs and start with a fresh GoogleAPI project.
Update 19.11.20:
Creating a new OAuth 2.0 Client ID did not resolve my problem, neither did creating a whole new GoogleAPI project and adding those credentials into the request. I am in contact with the developers of restdb.io and have asked them to add the Google Auth Library: Node.js Client to the list of supported Node.js packages. Hopefully that will help, I will give it a try as soon as it can be used :)
Update 02.12.20:
No progress so far, but I'm optimistic that the developers will add the package soon. I will post a final update as soon as I am done with this project.
I have the following Node.JS (ran with Express) code :
let app = express();
app.use(cors());
app.get('/callback', function (req, res) {
// your application requests refresh and access tokens
// after checking the state parameter
var code = req.query.code || null;
var authOptions = {
url: 'https://accounts.spotify.com/api/token',
form: {
code: code,
redirect_uri: redirectUri,
grant_type: 'authorization_code'
},
headers: {
'Authorization': 'Basic ' + (new Buffer(clientId + ':' + clientSecret).toString('base64'))
},
json: true
};
request.post(authOptions, function (error, response, body) {
if (!error && response.statusCode === 200) {
var access_token = body.access_token,
refresh_token = body.refresh_token;
fs.writeFile('test.txt', 'HELLO', function (err) {
if (err) return console.log(err);
console.log('Hello World > helloworld.txt');
});
}
}
)
});
console.log('Listening on 8888');
app.listen(8888);
The route is used as a callback for a request to the Spotify Web API, thus I can get an access token.
Spotify then redirects to the callback function above, you can see it in the URI by looking at "redirect_uri".
If you need more information about the Spotify Authorization Flow, see here.
Here's the URI I'm using to authenticate my app to Spotify.
https://accounts.spotify.com/authorize?client_id=CLIENT_ID&response_type=code&redirect_uri=http://localhost:8888/callback&scope=user-read-private%20user-read-email%20playlist-modify-public&state=PexBrjEzISHepTp7&show_dialog=false
CLIENT_ID is replaced by my real CLIENT_ID in the request I make
My problem is located to the file writing part :
fs.writeFile('test.txt', 'HELLO', function (err) {
if (err) return console.log(err);
console.log('Hello World > helloworld.txt');
});
When the callback route is called by Spotify, I have the string "HELLO" wrote in my text file, so the file writing is functional.
But even if it has finished writing the string, the Chrome page is still running and "pending" on the server. It runs for a few minutes and then crash by saying that the page didn't sent any data. Why ?
I've looked at this page talking about the methods of writing to text files, using writeFile and writeFileAsync, but using both of them didn't solved my problem.
EDIT: I don't really want to stop the Express process! I just want to be able to process another request :)
Any idea ? Thanks in advance :)
You aren't returning anything from your route, try adding res.send({})
In your get route you are not sending response, you must send response irrespective of writing a file was successful or not.
Add below code post writing to file (as well as in if error case)
res.send({YOUR_CHOICE_RESPONSE_DATA})
I'm adding tests to an application that already (partially) exists. It was written using angular and php/MariaDB in the backend. I'm now working on the http calls to the server. I plan to use jasmine with request.
I was able to make some simple tests, and can login. But I cannot test the pages that require to be logged in. I cannot find a way to add the token to the calls.
If I understand things correctly, on the received message from the sig in I should get a token that I should then use in the following calls. Who wrote the app followed the instructions given by the angular documentation which handles everything, so we are learning toguether how things really work under the hood.
Going through the received answer on the login, the only thing that looks like a token is a cookie set in the header, whose name is 'PHPSESSID'. I read and parse that cookie to get the token and make the next call like this:
request.get(
{
url: 'http://xxx.xxx.com/php/authentication/session.php',
'auth': {
'bearer': mytoken
}
}, function(err, res) {
console.log(res['body']);
done();
})
the response is what I should get if the user is NOT logged in.
Using Postman, everything works. Aparently it saves the token and uses it for the next call. That is, f I make the sign in and then make the get call to session.php I get the correct answer. I just cannot figure out what exact call postman makes or how to use the token in the next call using jasmine and request.
Since the token was passed as a cookie on the response call, what I was supposed to do was set the same cookie in the next call. Here is the code of the whole test just in case somebody needs a hand. First I sign in, then I made a call to an address that should return my email if indeed I'm logged in.
var request = require('request');
describe("login test", function() {
it("should log in", (done) => {
var user = {'email':'test#xxx.de', 'password':'blablabla'};
request.post(
{
url: 'http://xxx/test/public/php/authentication/login.php',
body: JSON.stringify(user)
},
(err, res) => {
var result = JSON.parse(res['body']);
var cookieString = res['headers']['set-cookie'];
expect(result['success']).toBe(true);
request.get(
{
url: 'http://xxx/test/public/php/authentication/session.php',
headers: {
'Cookie': cookieString
}
}, function(err, res) {
var result = JSON.parse(res['body']);
expect(result.user.email).toBe(user.email);
done();
})
});
});
});
I have a requirement to access multiple devices over IP from an HTML5 web app.
My approach to working around the cross-domain impossibility of doing this all on the client side has been to "cook" requests from the client inside of express' middleware. A route receives a get or post from the client, and then performs a get or post to the 3rd party device identified by the payload from the client.
I'm using the code to get info from a device. It works just fine when I run it directly from file inside of a client I made for testing purposes. Running directly from file avoids the CORS difficulty because the client is also the server I guess.
When I run the same code from within an express route, I get a 500 error.
Am I trying to do something impossible? I'm only about a week into node, express etc so hopefully it's something dumb and easy to solve. I'm taking fact that I haven't been able to find any other questions quite like this as an indication that there's a proper way to achieve what I need.
// post to find a camera
router.post('/find', function(req, res) {
var url = 'http://' + req.body.addr + '/cgi-bin/aw_cam?cmd=QID&res=1';
console.log(url);
$.ajax({
type: 'GET',
url: url,
dataType: 'html',
success: function (result) {
console.log('success: ' + result);
res.send(result);
},
error: function (xhr, textStatus, err) {
console.log('error: ' + textStatus);
}
});
});
Here's what logged to the server console:
http://192.168.0.10/cgi-bin/aw_cam?cmd=QID&res=1
POST /cameras/find 500 126.593 ms - 1656
Thanks in advance!
ok I found how to do this. The trick is to use Node's built-in http messaging capabilities. I found a good article on how to do this here
The code below does exactly what I wanted from within my custom route middleware. I guess I just learned that I can only use AJAX how I wanted on the client side.
This lets me abstract the hairier details of the device control protocol into the server, leaving my client apps to use JSON/AJAX model to interact with them. Success!
var http = require('http');
// post to find a camera
router.post('/find', function(req, res) {
var url = 'http://' + req.body.addr + '/cgi-bin/aw_cam?cmd=QID&res=1';
console.log(url);
http.get(url, (response) => {
console.log(`Got response: ${response.statusCode}`);
var body = '';
response.on ('data', function(d) {
body += d;
});
response.on ('end', function () {
console.log('received: ' + body);
var reply = {};
if (body.indexOf('OID:') == 0) {
reply.msg = body.slice(4);
reply.ok = true;
} else {
reply.msg = body;
reply.ok = false;
}
res.send(reply);
});
// consume response body
response.resume();
}).on('error', (e) => {
console.log(`Got error: ${e.message}`);
});
});