Develop Reference

react native fetch not getting the same content as post man

Im having a little problem with my request on getting an html from https://readnovelfull.com/beauty-and-the-beast-wolf-hubby-xoxo/chapter-1-i-would-not-be-responsible.html as example.
I can get all the html on the other url eg novel detalj, latest upgated etc.
but not when im getting the detali for the chapters.
I tested those url on postman and also on https://codebeautify.org/source-code-viewer as well and there is no problem on getting the content of the chapter of which it exist under the div #chr-content
So I am a bit lost now, what am I doing wrong?
Here is my fetch calls which is working on other novel sites.
static async getHtml(
url: string
): Promise<HTMLDivElement> {
console.log(`Sending html request to ${url}`);
var container = parse('<div>test</div>') as any;
try {
let headers = new Headers({
Accept: '*/*',
'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
});
var data = await fetch(url, {
method: 'GET',
headers: headers,
});
if (!data.ok) {
const message = `An error has occured:${data.status}`;
console.log(message);
} else {
var html = await data.text();
console.log('Data is ok. proceed to parse it');
container = parse('<div>' + html + '</div>') as any;
}
} catch (e) {
console.log(e);
}
return container as HTMLDivElement;
}
I should mention that am not getting any error what so ever, its just that the html I am getting is not the same as postman and other site is getting.
Update
Ok so i did some research on the site and this is what i come up with.
the site need X-CSRF-TOKEN and i was able to extract those and find those values
const csrf = 'x09Q6KGqJOJJx2iHwNQUa_mYfG4neV9EOOMsUBKTItKfNjSc0thQzwf2HvCR7SQCqfIpC2ogPj18jG4dQPgVtQ==';
const id = 774791;
which i need to send a request to https://readnovelfull.com/ajax/increase-chapter-views with the values above. and this will send back true/false
now i tried to inc the csrf on my fetch call after but its still the same old same no data.
any idee if i am doing something wrong still?

Looks like you have an issue with CORS. To make sure just try to send request through cors proxy. One of the ways you can quickly do that is add prefix URL:
https://cors-anywhere.herokuapp.com/https://readnovelfull.com/beauty-and-the-beast-wolf-hubby-xoxo/chapter-1-i-would-not-be-responsible.html`
NOTE: Using this CORS proxy on production is not recommended, because it's not secure
If after that you'll receive data, that means that you faced with CORS, and you need to figure out how to solve it in your specific case.
Reproducable example:
const parse = (str) => str;
const getHtml = async (url) => {
console.log(`Sending html request to ${url}`);
var container = parse('<div>No content =(</div>')
try {
let headers = new Headers({
Accept: '*/*',
'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
});
var data = await fetch(url, {
method: 'GET',
headers: headers,
});
if (!data.ok) {
const message = `An error has occured:${data.status}`;
console.log(message);
} else {
var html = await data.text();
console.log('Data is ok. proceed to parse it');
container = parse('<div>' + html + '</div>');
}
} catch (e) {
console.log(e);
}
return container;
}
getHtml('https://cors-anywhere.herokuapp.com/https://readnovelfull.com/beauty-and-the-beast-wolf-hubby-xoxo/chapter-1-i-would-not-be-responsible.html').then(htmlContent => document.querySelector('div').innerHTML = htmlContent);
<div>loading...</div>
If it doesn't help, please provide a reproducible RN example, but I believe there is no difference between RN and web environments in that case.

How to send GET request without downloading response content using node-requests?

I'm currently learning node and i'm looking for HTTP library that would allow me to send GET request, without downloading server response content (body).
I need to send very large amount of http requests every minute. However i do not need to read their content (also to save bandwidth). I can't use HEAD for this purpose.
Is there any way to avoid downloading response body using node-requests, or perhaps any other library - could be used?
My sample code using node-request:
const options = {
url: "https://google.com",
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
}
}
//How to avoid downloading a whole response?
function callback(err, response, body) {
console.log(response.request.uri.host + ' - ' + response.statusCode);
}
request(options, callback);

HTTP GET by standard fetches the file content, you cannot avoid downloading(getting response) it but you can ignore it. Which is basically what you are doing.
request(options, (err, response, body)=>{
//just return from here don't need to process anything
});
EDIT1:
To just use some bytes of the response, you can use http.get and get the data using the data event. From the doc:
http.get('http://nodejs.org/dist/index.json', (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
//this is when the response will end
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});

Nodejs request issue with especific endpoint (Client network socket disconnected)

This is my entire code
const rp = require('request-promise');
(async () => {
const headers = {
Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
Host: 'www.receita.fazenda.gov.br',
Pragma: 'no-cache',
'Upgrade-Insecure-Requests': 1,
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0',
};
const jar = rp.jar();
try {
const firstAccess = await rp.get('https://www.receita.fazenda.gov.br/Aplicacoes/SSL/ATCTA/CPF/ConsultaSituacao/ConsultaPublica.asp', {
headers,
gzip: true,
simple: false,
strictSSL: false,
jar,
});
console.log(firstAccess);
} catch (e) {
console.log(e);
}
})();
When i run my code i receive this error:
UnhandledPromiseRejectionWarning: RequestError: Error: Client network
socket disconnected before secure TLS connection was established
The url works in the browser, but I can not access through nodeJS, apparently it's some problem with the https certificate
Is there any way to ignore the error and access it?

You need to move your call inside a try catch block to handle the promise rejection in case the call fails:
...
try {
const firstAccess = await rp.get(...)
} catch (e) {
console.log(e)
}
...

My problem was resolved after restarting/changing the network connection. The issue is with the network most of the time.

How to use the node.js 'request' library with this http request?

I was trying to make a simple request to site. it should get html text, but it gets ' '
NPM module here: github.com/request/request
Code:
var fs = require('fs');
var request = require('request');
var options = {
url:'https://sample.site/phpLoaders/getInventory/getInventory.php',
encoding : 'utf8',
gzip : true,
forever: true,
headers: {
'Host': 'sample.site',
'Connection': 'keep-alive',
'Content-Length': '58',
'Cache-Control': 'max-age=0',
'Accept': '*/*',
'Origin': 'https://csgosell.com',
'X-Requested-With': 'XMLHttpRequest',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Referer': 'https://sample.site/',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
'Cookie': 'my-cookies from browser'
},
form: {
stage:'bot',
steamId:76561198284997423,
hasBonus:false,
coins:0
}
};
request.post(options,
function(error, response, body){
console.log(response.statusCode);
if (!error) {
fs.writeFileSync('site.html', body);
}
else{
console.log(error);
}
}
);
Chrome request: https://i.stack.imgur.com/zKQo5.png
Nodejs request:https://i.stack.imgur.com/yH9U3.png
the difference is in headers:
:authority:csgosell.com
:method:POST :path:/phpLoaders/getInventory/getInventory.php :scheme:https
after some googling, I anderstood that it is http2, and tried to put it inow another agent's options, but nothing changed.
var spdy = require('spdy');
var agent = spdy.createAgent({
host: 'sample.site',
port: 443,
spdy: {
ssl: true,
}
}).once('error', function (err) {
this.emit(err);
});
options.agent = agent;

To answer your question i will copy/paste a part of my code that enable you to receive a post request from your frontend application(angularJS) to your backend application (NodeJS), and another function that enable you to do the inverse send a post request from nodeJS to another application (that might consume it):
1) receive a request send from angularJS or whatever inside your nodeJS app
//Import the necessary libraries/declare the necessary objects
var express = require("express");
var myParser = require("body-parser");
var app = express();
// we will need the following imports for the inverse operation
var https = require('https')
var querystring = require('querystring')
// we need these variables for the post request:
var Vorname ;
var Name ;
var e_mail ;
var Strasse ;
app.use(myParser.urlencoded({extended : true}));
// the post request is send from http://localhost:8080/yourpath
app.post("/yourpath", function(request, response ) {
// test the post request
if (!request.body) return res.sendStatus(400);
// fill the variables with the user data
Vorname =request.body.Vorname;
Name =request.body.Name;
e_mail =request.body.e_mail;
Strasse =request.body.Strasse;
response.status(200).send(request.body.title);
});
2) Do the inverse send a POST request from a nodeJS application to another application
function sendPostRequest()
{
// prepare the data that we are going to send to anymotion
var jsonData = querystring.stringify({
"Land": "Land",
"Vorname": "Vorname",
"Name": "Name",
"Strasse": Strasse,
});
var post_options = {
host: 'achref.gassoumi.de',
port: '443',
method: 'POST',
path: '/api/mAPI',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': jsonData.length
}
};
// request object
var post_req = https.request(post_options, function(res) {
var result = '';
res.on('data', function (chunk) {
result += chunk;
console.log(result);
});
res.on('end', function () {
// show the result in the console : the thrown result in response of our post request
console.log(result);
});
res.on('error', function (err) {
// show possible error while receiving the result of our post request
console.log(err);
})
});
post_req.on('error', function (err) {
// show error if the post request is not succeed
console.log(err);
});
// post the data
post_req.write(jsonData);
post_req.end();
// ps : I used a https post request , you could use http if you want but you have to change the imported library and some stuffs in the code
}
So finally , I hope this answer will helps anyone who is looking on how to get a post request in node JS and how to send a Post request from nodeJS application.
For further details about how to receive a post request please read the npm documentation for body-parser library : npm official website documentation

Node.js / Socket io video streaming does not seem to work while any other file type works

Okay so i have the following route for when a file is being uploaded:
router.route('/moduleUpload')
.post(function (request, response) {
request.files.file.originalname = request.files.file.originalname.replace(/ +?/g, '');
var media = new Media(request.files.file, './user_resources/module/' + request.body.module_id + '/');
if (!fs.existsSync(media.targetDir)) {
fs.mkdirSync(media.targetDir, 0777, function (err) {
if (err) {
console.log(err);
response.send("ERROR! Can't make the directory! \n"); // echo the result back
}
});
}
moveFile(media);
var token = jwt.encode({
mediaObject: media
}, require('../secret')());
response.status(200).json(token);
});
As you can see this uses the support of a function called moveFile(media);
So here is that function:
function moveFile(media) {
var source = fs.createReadStream(media.file.path);
var dest = fs.createWriteStream(media.targetDir + media.file.originalname);
source.pipe(dest);
source.on('end', function () { /* copied */
console.log('file has been moved!')
});
source.on('error', function (err) { /* error */
console.log('error while moving file')
});
}
This works and the console prints out:
file has been moved!
Now after this i attempt to download / stream the video file i use the following route:
router.route('/resource/:encodedString')
.all(function (req, res) {
var decoded = jwt.decode(req.params.encodedString, require('../secret')());
var mediaObject = decoded.mediaObject;
res.header('content-disposition', 'filename=' + mediaObject.file.originalname, "Content-Length: " + mediaObject.file.size, "mimeType:" + mediaObject.file.mimetype);
var stream = fs.createReadStream(mediaObject.targetDir + mediaObject.file.originalname);
stream.pipe(res);
stream.on('end', function () {
console.log("Reading operation completed.");
res.end();
});
});
Using the encoded string i first decode it to find the media object and then stream it back.
This works WITH EVERY file execpt mp4 / video files.
Here is the complete console print out of when i upload a video:
listening on *:8105
file has been moved!
Reading operation completed.
And here are my browser request headers:
Remote Address:***
Request ****
Request Method:GET
Status Code:200 OK
Response Headers
view source
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://localhost
Connection:keep-alive
Content-Length:5
Content-Type:application/octet-stream
Date:Sun, 21 Jun 2015 17:13:24 GMT
Set-Cookie:io=1T7R1vmJ3xV-k0ApAABH
Request Headers
view source
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:io=1T7R1vmJ3xV-k0ApAABH; lb_login_id=117
Host:angular.learningbankapp.com:8105
Origin:http://localhost
Referer:http://localhost/learningbankapp/
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36
So can someone tell me what im doing wrong?