I'm looking at my balance on Venmo.com but they only show you 3 months at a time and I'd like to get my entire transaction history.
Looking at the Chrome Developer Tools, under the network tab, I can see the request to https://api.venmo.com/v1/transaction-history?start_date=2017-01-01&end_date=2017-01-31 which returns JSON.
I'd like to programmatically iterate through time and make several request and aggregate all of the transactions. However, I keep getting 401 Unauthorized.
My initial approach was just using Node.js. I looked at the cookie in the request and copied it into a secret.txt file and then sent the request:
import fetch from 'node-fetch'
import fs from 'fs-promise'
async function main() {
try {
const cookie = await fs.readFile('secret.txt')
const options = {
headers: {
'Cookie': cookie,
},
}
try {
const response = await fetch('https://api.venmo.com/v1/transaction-history?start_date=2016-11-08&end_date=2017-02-08', options)
console.log(response)
} catch(e) {
console.error(e)
}
} catch(e) {
console.error('please put your cookie in a file called `secret.txt`')
return
}
}
That didn't work do I tried copying all of the headers over:
const cookie = await fs.readFile('secret.txt')
const options = {
headers: {
'Accept-Encoding': 'gzip, deflate, sdch, br',
'Accept-Language': 'en-US,en;q=0.8',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Cookie': cookie,
'Host': 'api.venmo.com',
'Origin': 'https://venmo.com',
'Pragma': 'no-cache',
'Referer': 'https://venmo.com/account/settings/balance/statement?end=02-08-2017&start=11-08-2016',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36',
},
}
try {
const response = await fetch('https://api.venmo.com/v1/transaction-history?start_date=2016-11-08&end_date=2017-02-08', options)
console.log(response)
} catch(e) {
console.error(e)
}
This also did not work.
I even tried making the request from the console of the website and got a 401:
fetch('https://api.venmo.com/v1/transaction-history?start_date=2016-11-08&end_date=2017-02-08', {credentials: 'same-origin'}).then(console.log)
So my question here is this: I see a network request in Chrome Developer Tools. How can I make that same request programmatically? Preferably in Node.js or Python so I can write an automated script.
In the Network tab of the Chrome Developer Tools, right click the request and click "Copy" > "Copy as cURL (bash)". You can then either write a script using the curl command directly, or use https://curlconverter.com/ to convert the cURL command to Python, JavaScript, PHP, R, Go, Rust, Elixir, Java, MATLAB, Dart or JSON.
Related
I am using News API in ReactJS for fetching news using fetch API.
CODE FRAGMENT
const url = `https://newsapi.org/v2/top-headlines?country=in&category=${this.props.category}&pageSize=${this.state.pageSize}&page=${this.state.page}`;
let data = await fetch(url);
But when I run it on the localhost, it gave an error 400 saying:
{status: 'error', code: 'userAgentMissing', message: 'Please set your User-Agent header to identify your application. Anonymous requests are not allowed.'}
When I searched the same on Stack Overflow, I got a link of a similar problem here:
User Agent Missing error when making a get request to an API
Then I added the User-Agent header in the fetch API's argument:
CODE FRAGMENT
headers = new Headers({
"X-Api-Key": this.apiKey,
"User-Agent": navigator.userAgent
});
const url = `https://newsapi.org/v2/top-headlines?country=in&category=${this.props.category}&pageSize=${this.state.pageSize}&page=${this.state.page}`;
let data = await fetch(url, {
headers: this.headers
});
But still it shows the same error:
{status: 'error', code: 'userAgentMissing', message: 'Please set your User-Agent header to identify your application. Anonymous requests are not allowed.'}
This error is asking for "User-Agent" in the request header, but it is already there! When I looked into the network section of the chrome browser.
So, what's exactly the problem is? And how to solve this issue?
I am having a similar problem. So far I have been able to pass in one of the web browser User-Agents from the Mozilla Developer websites.
function webCall(url,method,headers,postBody,callBack){
/* build options */
var options = {
method: method,
uri: url,
headers: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59',
body: postBody
}
...
};
This is still in development; however, I was able to return results from my newsapi.org pull.
Edit: For reference, I am using the old 'Requests' npm library (legacy requirement). So I make my request like so:
request(options, function handleResponse(err, response, body){
var errMessage = null;
var headers = {};
if(err) {
errMessage='HTTP ERROR: ' + err;
}
else {
headers = response.headers;
if(response.statusCode!=200){
errMessage='HTTP ' + response.statusCode + ' ERROR: ' + err;
}
}
});
I am working on a project that talks with an API, my script is in nodeJS (it's a big project so I will give you a very simplified version)
Essentially when I do a PUT request to the API, with the postman, it works fine,
When I do it with my Code: GET works great (so I know the URL is correct), but PUT doesn't work with my script, I get no errors on my console, and the API simple sends back the OK status, so I have no clues where it went wrong
Here is a very basic version of my script (that doesn't work either)
(I know the URL is correct)
PUT parameters:
{ method: 'PUT',
baseUrl: 'https://fusionrv.XXXX.XX/fusion/apiservice',
url:
'XXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXX',
qs:
{ auth:
'AUTH_KEY',
value: 1 },
headers: { 'content-type': 'application/json' } }
simple code (without the imports):
request(putParams, (err, res) => {
if (err) console.log(err);
data = JSON.parse(res.body)
console.log(data)
});
In Postman, I have the same baseURL and url, as well as the same params "auth" and "value", the body is set to "none", and headers are Content-type: application/json.
The response (both in Postman and NodeJS):
{
"Status": "Success"
}
So does anyone know why it works with postman but not with the request module?
I appreciate all of you!
Add user-agent as part of the header
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
Turns out the issue was with the API, I needed to do a POST request and use a different URL to access it, it's a very strange API.
I am running a local Springboot server, that when I access it locally in the browser, gives me a valid JSON object properly formatted (I verified this via JSON formatter).
I am also locally running a React application using node. I am attempting to use fetch() to get back that JSON object and running into issues. Finally got around CORs header issues, but not cannot figure out why the JSON object isn't coming back. Here's my code
var headers = new Headers();
headers.append("Content-type", "application/json;charset=UTF-8");
var myInit = { method: 'GET',
headers: headers,
mode: 'no-cors',
cache: 'default',
};
fetch(`http://localhost:3010/getJSON`, myInit)
.then(function(response){
console.log(response.data);
console.log(response);
console.log(JSON.parse(JSON.stringify(response)));
},function(error){
console.log(error);
});
So when I run this in Chrome with the debugger, the responses to the 3 log statements are:
1st logger
undefined
2nd logger
Response {type: "opaque", url: "", redirected: false, status: 0, ok: false,
…}
body
:
(...)
bodyUsed
:
false
headers
:
Headers {}
ok
:
false
redirected
:
false
status
:
0
statusText
:
""
type
:
"opaque"
url
:
""
__proto__
:
Response
3rd logger
{}
I have tried many different JSON parsing, stringify, etc, to no avail.
The next confusing part, is if within the Chrome debugger I go to the "Network" tab, click on the /getJSON, it shows me the entire JSON object just fine in both the "Preview" and "Response" tabs. So clearly Chrome is connecting to it correctly. Here's Chrome's "Headers" tab within "Network":
Request URL:http://localhost:3010/getJSON
Request Method:GET
Status Code:200
Remote Address:[::1]:3010
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Content-Type:application/json;charset=UTF-8
Date:Thu, 12 Oct 2017 16:05:05 GMT
Transfer-Encoding:chunked
Request Headers
view source
Accept:*/*
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Host:localhost:3010
Referer:http://localhost:3000/
User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
I have tried to mimic this header in my request, but not sure how it differs? Any help would be greatly appreciated as I am currently banging my head against the way with this!
You're getting an opaque response, which tells me that maybe you haven't completely resolved the cors headers situation. If you're fetching from the client, I would suggest proxying that through your nodejs so that instead of calling your springboot service, you call node, thus getting rid of the cors issues.
EDIT
You could create something like this:
import express from 'express';
import request from 'request';
const router = express.Router();
router.get('/proxyname', (req, res) => {
// Removing IPv4-mapped IPv6 address format, if present
const requestUrl = [your service's endpoint];
request(requestUrl, (err, apiResponse, body) => {
res.status(apiResponse.statusCode);
try {
res.json(JSON.parse(body));
} catch (e) {
res.send(body);
}
});
});
export default router;
and then on your nodejs server file, add it, like this:
import proxy from '[path to proxy file above]';
app.use('/path-to-endpoint', proxy);
and then call that from the client instead of your SpringBoot service.
I want to login to blogfa.com (a persian blog service) and create a new post by node.js
To do that i use request.js to post login site and go this url "/Desktop/Post.aspx?action=newpost" and post a new content
here is code i got so far :
var request = require('request');
var cheerio = require('cheerio');
var j = request.jar();
request = request.defaults({ jar : j }); //it will make the session default for every request
//...
var headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36 OPR/28.0.1750.48',
'Content-Type': 'application/x-www-form-urlencoded'
};
request({
url:"http://blogfa.com/Desktop/login.aspx",
method:"POST",
form:{uid:"demoblog1",pwd:"test"},
headers: headers,
followRedirect:true
},
function(error,response,body){
console.log(body);
});
Problem i have its, i cant login in to the site .
i dont why not working ! i used jar for cookie , posted user and password , also set headers.
here is runnable code you can test :
http://code.runnable.com/Vc7EnmyVlgRa1Hx-/blog-login-for-node-js
Update
Request Cookies
.ASPXBLOG BC045A0CD184FEA10D91561EB67A302F1E036D88E50CE4264E4ABD003
__utma 36873331.1996897518.1435135939.1437159460.1439563539.7
__utmz 36873331.1437159460.6.6.utmcsr=chat.delgarm.com|utmccn=(referral)
pubset ar=1&z=12600&ds=0&cmt=0&cats=0&tag=0&nu=1&bt=dGVzdCB
ten 67145377
By default request does not follow redirects for non-GET requests, which is what is happening in this case. So set followAllRedirects: true in your request() options and it should work fine.
I 'm making a windows app using HTML5/JAVASCRIPT . What actually i want is to retrieve a captcha from a website and then submitting the captcha along with other form field to the website(.aspx) back and getting the response back.I think that i have to handle cookies for this purpose ,but i do not know how to do this .
100 times salute to the person who will show interest in this .
Here is what i did .
// Retrieving the Captcha.
WinJS.xhr({
url: "http://example.in/main.aspx",
type: "get",
responseType: "document",
headers: {
"CONTENT-TYPE": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; MSAppHost/2.0; rv:11.0) like Gecko",
"CONNECTION": "keep-alive",
},
}).then(
function complete(xhr) {
var image = document.createElement("img");
image.src = xhr.response.querySelector("img[alt='Captcha']").src;
document.getElementById("captcha").appendChild(image);
});
//submit the data ..
document.getElementById("submit").onclick = function () {
WinJS.xhr({
url: "http://example.in/main.aspx/",
type: "post",
responsetype: "document",
headers: {
"CONTENT-TYPE": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; MSAppHost/2.0; rv:11.0) like Gecko",
"CONNECTION" : "keep-alive",
},
data: alldata // all data contain username,password and captcha(entered by user)
}).then(
function complete(xhr) {
document.getElementById("response").innerHTML = toStaticHTML(xhr.response);
});
I know that here i have to handle cookie , but really no idea how to do that.
The WinJS.xhr API is just a wrapper for the HTML XMLHttpRequest API, which unfortunately doesn't let you get to cookies and such.
You'll thus need to use the Windows.Web.Http.HttpClient API instead, which does allow you access to cookies and everything else.
Start with the Windows.Web.Http landing page for an overview, followed by How to connect to an HTTP Server for the next level of details. Refer also to the HttpClient sample, specifically scnearios 8, 9, and 10 that deal with cookies.
I also cover the details of this API (and making HTTP requests in general) in Chapter 4 of my free ebook, Programming Windows Store Apps with HTML, CSS, and JavaScript, 2nd Edition.
That should get you going.