Develop Reference

User Agent missing error in ReactJS fetch api

I am using News API in ReactJS for fetching news using fetch API.
CODE FRAGMENT
const url = `https://newsapi.org/v2/top-headlines?country=in&category=${this.props.category}&pageSize=${this.state.pageSize}&page=${this.state.page}`;
let data = await fetch(url);
But when I run it on the localhost, it gave an error 400 saying:
{status: 'error', code: 'userAgentMissing', message: 'Please set your User-Agent header to identify your application. Anonymous requests are not allowed.'}
When I searched the same on Stack Overflow, I got a link of a similar problem here:
User Agent Missing error when making a get request to an API
Then I added the User-Agent header in the fetch API's argument:
CODE FRAGMENT
headers = new Headers({
"X-Api-Key": this.apiKey,
"User-Agent": navigator.userAgent
});
const url = `https://newsapi.org/v2/top-headlines?country=in&category=${this.props.category}&pageSize=${this.state.pageSize}&page=${this.state.page}`;
let data = await fetch(url, {
headers: this.headers
});
But still it shows the same error:
{status: 'error', code: 'userAgentMissing', message: 'Please set your User-Agent header to identify your application. Anonymous requests are not allowed.'}
This error is asking for "User-Agent" in the request header, but it is already there! When I looked into the network section of the chrome browser.
So, what's exactly the problem is? And how to solve this issue?

I am having a similar problem. So far I have been able to pass in one of the web browser User-Agents from the Mozilla Developer websites.
function webCall(url,method,headers,postBody,callBack){
/* build options */
var options = {
method: method,
uri: url,
headers: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59',
body: postBody
}
...
};
This is still in development; however, I was able to return results from my newsapi.org pull.
Edit: For reference, I am using the old 'Requests' npm library (legacy requirement). So I make my request like so:
request(options, function handleResponse(err, response, body){
var errMessage = null;
var headers = {};
if(err) {
errMessage='HTTP ERROR: ' + err;
}
else {
headers = response.headers;
if(response.statusCode!=200){
errMessage='HTTP ' + response.statusCode + ' ERROR: ' + err;
}
}
});

Receiving JSON between local React App and local Springboot service issues

I am running a local Springboot server, that when I access it locally in the browser, gives me a valid JSON object properly formatted (I verified this via JSON formatter).
I am also locally running a React application using node. I am attempting to use fetch() to get back that JSON object and running into issues. Finally got around CORs header issues, but not cannot figure out why the JSON object isn't coming back. Here's my code
var headers = new Headers();
headers.append("Content-type", "application/json;charset=UTF-8");
var myInit = { method: 'GET',
headers: headers,
mode: 'no-cors',
cache: 'default',
};
fetch(`http://localhost:3010/getJSON`, myInit)
.then(function(response){
console.log(response.data);
console.log(response);
console.log(JSON.parse(JSON.stringify(response)));
},function(error){
console.log(error);
});
So when I run this in Chrome with the debugger, the responses to the 3 log statements are:
1st logger
undefined
2nd logger
Response {type: "opaque", url: "", redirected: false, status: 0, ok: false,
…}
body
:
(...)
bodyUsed
:
false
headers
:
Headers {}
ok
:
false
redirected
:
false
status
:
0
statusText
:
""
type
:
"opaque"
url
:
""
__proto__
:
Response
3rd logger
{}
I have tried many different JSON parsing, stringify, etc, to no avail.
The next confusing part, is if within the Chrome debugger I go to the "Network" tab, click on the /getJSON, it shows me the entire JSON object just fine in both the "Preview" and "Response" tabs. So clearly Chrome is connecting to it correctly. Here's Chrome's "Headers" tab within "Network":
Request URL:http://localhost:3010/getJSON
Request Method:GET
Status Code:200
Remote Address:[::1]:3010
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Content-Type:application/json;charset=UTF-8
Date:Thu, 12 Oct 2017 16:05:05 GMT
Transfer-Encoding:chunked
Request Headers
view source
Accept:*/*
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Host:localhost:3010
Referer:http://localhost:3000/
User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
I have tried to mimic this header in my request, but not sure how it differs? Any help would be greatly appreciated as I am currently banging my head against the way with this!

You're getting an opaque response, which tells me that maybe you haven't completely resolved the cors headers situation. If you're fetching from the client, I would suggest proxying that through your nodejs so that instead of calling your springboot service, you call node, thus getting rid of the cors issues.
EDIT
You could create something like this:
import express from 'express';
import request from 'request';
const router = express.Router();
router.get('/proxyname', (req, res) => {
// Removing IPv4-mapped IPv6 address format, if present
const requestUrl = [your service's endpoint];
request(requestUrl, (err, apiResponse, body) => {
res.status(apiResponse.statusCode);
try {
res.json(JSON.parse(body));
} catch (e) {
res.send(body);
}
});
});
export default router;
and then on your nodejs server file, add it, like this:
import proxy from '[path to proxy file above]';
app.use('/path-to-endpoint', proxy);
and then call that from the client instead of your SpringBoot service.

How to programmatically replicate a request found in Chrome Developer Tools?

I'm looking at my balance on Venmo.com but they only show you 3 months at a time and I'd like to get my entire transaction history.
Looking at the Chrome Developer Tools, under the network tab, I can see the request to https://api.venmo.com/v1/transaction-history?start_date=2017-01-01&end_date=2017-01-31 which returns JSON.
I'd like to programmatically iterate through time and make several request and aggregate all of the transactions. However, I keep getting 401 Unauthorized.
My initial approach was just using Node.js. I looked at the cookie in the request and copied it into a secret.txt file and then sent the request:
import fetch from 'node-fetch'
import fs from 'fs-promise'
async function main() {
try {
const cookie = await fs.readFile('secret.txt')
const options = {
headers: {
'Cookie': cookie,
},
}
try {
const response = await fetch('https://api.venmo.com/v1/transaction-history?start_date=2016-11-08&end_date=2017-02-08', options)
console.log(response)
} catch(e) {
console.error(e)
}
} catch(e) {
console.error('please put your cookie in a file called `secret.txt`')
return
}
}
That didn't work do I tried copying all of the headers over:
const cookie = await fs.readFile('secret.txt')
const options = {
headers: {
'Accept-Encoding': 'gzip, deflate, sdch, br',
'Accept-Language': 'en-US,en;q=0.8',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Cookie': cookie,
'Host': 'api.venmo.com',
'Origin': 'https://venmo.com',
'Pragma': 'no-cache',
'Referer': 'https://venmo.com/account/settings/balance/statement?end=02-08-2017&start=11-08-2016',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36',
},
}
try {
const response = await fetch('https://api.venmo.com/v1/transaction-history?start_date=2016-11-08&end_date=2017-02-08', options)
console.log(response)
} catch(e) {
console.error(e)
}
This also did not work.
I even tried making the request from the console of the website and got a 401:
fetch('https://api.venmo.com/v1/transaction-history?start_date=2016-11-08&end_date=2017-02-08', {credentials: 'same-origin'}).then(console.log)
So my question here is this: I see a network request in Chrome Developer Tools. How can I make that same request programmatically? Preferably in Node.js or Python so I can write an automated script.

In the Network tab of the Chrome Developer Tools, right click the request and click "Copy" > "Copy as cURL (bash)". You can then either write a script using the curl command directly, or use https://curlconverter.com/ to convert the cURL command to Python, JavaScript, PHP, R, Go, Rust, Elixir, Java, MATLAB, Dart or JSON.

Node.js Login to blog site and Create new post

I want to login to blogfa.com (a persian blog service) and create a new post by node.js
To do that i use request.js to post login site and go this url "/Desktop/Post.aspx?action=newpost" and post a new content
here is code i got so far :
var request = require('request');
var cheerio = require('cheerio');
var j = request.jar();
request = request.defaults({ jar : j }); //it will make the session default for every request
//...
var headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36 OPR/28.0.1750.48',
'Content-Type': 'application/x-www-form-urlencoded'
};
request({
url:"http://blogfa.com/Desktop/login.aspx",
method:"POST",
form:{uid:"demoblog1",pwd:"test"},
headers: headers,
followRedirect:true
},
function(error,response,body){
console.log(body);
});
Problem i have its, i cant login in to the site .
i dont why not working ! i used jar for cookie , posted user and password , also set headers.
here is runnable code you can test :
http://code.runnable.com/Vc7EnmyVlgRa1Hx-/blog-login-for-node-js
Update
Request Cookies
.ASPXBLOG BC045A0CD184FEA10D91561EB67A302F1E036D88E50CE4264E4ABD003
__utma 36873331.1996897518.1435135939.1437159460.1439563539.7
__utmz 36873331.1437159460.6.6.utmcsr=chat.delgarm.com|utmccn=(referral)
pubset ar=1&z=12600&ds=0&cmt=0&cats=0&tag=0&nu=1&bt=dGVzdCB
ten 67145377

By default request does not follow redirects for non-GET requests, which is what is happening in this case. So set followAllRedirects: true in your request() options and it should work fine.

Chrome extension OATH 401 error(Unauthorized)

I am using JS to access the rdio plugin. I am using the following for Oauth http://code.google.com/chrome/extensions/tut_oauth.html.
I am able to get the signed token etc. However, when ever I try to send a signedRequest at http://api.rdio.com/1/, I receive 401, un-authorized error.
X-Mashery-Error-Code:ERR_401_INVALID_SIGNATURE
X-Mashery-Responder:mashery-web4.LAX
This is what I am trying to send:
var url = 'http://api.rdio.com/1/';
var request = {
'method': 'POST',
'headers': {
'Content-Type': 'application/x-www-form-urlencoded'
},
'parameters': {
'alt': 'json',
'method':'currentUser'
},
'body': 'Data to send'
};
bgPage.oauth.sendSignedRequest(url, mycallback, request);
I receive the following error in console.
Request URL:http://api.rdio.com/1/?alt=json&method=currentUser&oauth_consumer_key=yv8ehzehdv55**********&oauth_nonce=******&oauth_signature=**********&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1305190893&oauth_token=us6myp99p4qc86umea9p8fp*****************
Request Method:POST
Status Code:401 Unauthorized
Request Headers
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:12
Content-Type:application/x-www-form-urlencoded
Cookie:__qca=P0-158278476-1296771701175; r=eyJfdSI6IDE5MjY1LCAiX2UiOiAzMTU1NjkyNn0.SvN8xd7rIuLzTp7hxqi4eJEdvu8; __utmz=225830489.1305153361.198.18.utmcsr=rdioquiz.ianloic.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utma=225830489.281668250.1296676147.1305184513.1305187119.201; __utmc=225830489
Host:api.rdio.com
Origin:chrome-extension://oiojbkkpmcgmpnjkhjmaggajckamjkap
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24
Query String Parameters
alt:json
method:currentUser
oauth_consumer_key:yv8ehzehdv55pbb74ss9dt23
oauth_nonce:BQF0x
oauth_signature:KttF************tRO 8PL yjPF2Ktk=
oauth_signature_method:HMAC-SHA1
oauth_timestamp:1305190893
oauth_token:us6myp99p4qc86umea9p8fphbgq4dxdd76txvyn***********
Form Data
Data to send:
Response Headers
Accept-Ranges:bytes
Content-Length:30
Content-Type:text/xml
Date:Thu, 12 May 2011 09:01:33 GMT
Server:Mashery Proxy
X-Mashery-Error-Code:ERR_401_INVALID_SIGNATURE
X-Mashery-Responder:mashery-web4.LAX
*I am just trying to mimic what's mentioned here. Its an Oauth library(http://code.google.com/chrome/extensions/tut_oauth.html) from Google to make Chrome extension development easy.
They have an Oauth sample code to get your document list etc. http://code.google.com/chrome/extensions/samples.html#4e35caa9742fb82dbd628892d23a781614f6eff6
I think I am not able to get past send a POST requestto the rdio API. It gives an un-authorized error.*

We found a similar issue with the same service (rdio) and method ("currentUser").
What ended up working was:
(1) make sure you have method=currentUser in the POST body; I'm not sure from the above curl output if that is the case.
And, this is the bit that actually fixed the issue:
(2) we had to also add the method name to the signature itself.
FYI we used this library: https://oauth.googlecode.com/svn/code/javascript/
But the tricky part, as you are seeing, was figuring out how to seed the method in that library that creates the signature. Without the 'method=currentUser' being part of the signature, we experienced the same error condition.

Check your timezone, date, and time on your computer. If any one of these is wrong, OAuth will fail.