Im having a strange issue.
My logs are showing that occasionally, a post endpoint returns http 400, because the header object is empty. It seems like its the same user that has the problem.
I am not able to reproduce the issue locally, and the endpoint works for most of the time for most users.
export async function postJson<T>(
url: string,
jsonPayload: string,
caller: string
) {
const headers = new Headers();
headers.append("Content-Type", "application/json");
const payload: RequestInit = {
credentials: "same-origin",
method: "post",
headers: headers,
body: jsonPayload
};
const response = await fetch(url, payload);
if (response.ok) {
//Testing for empty response
const data = await response.text();
if (data) {
return <T>JSON.parse(data);
} else {
return null;
}
} else {
throw new Error(
`Url=${url} returned non-ok status: ${response.status}. Status text:${
response.statusText
}; caller=${caller}, payload=${JSON.stringify(payload)}`
);
}
}
The log is showing the payload object as
payload={"credentials":"same-origin","method":"post","headers":{},"body":"...some json..."}
The useragent that gets the 400 is
Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
Any ideas?
Related
I'm trying to get the specific token value from this response in the outside of the loop so then i can show this token value in the HTML page.
import puppeteer from 'puppeteer-extra'
var http = import('http');
puppeteer.launch({ headless: true }).then(async (browser) => {
const page = await browser.newPage();
const token=[];
await page.goto('https://orbitxch.com/customer/inplay/highlights/1');
page.on('request', req => {
const tok = req.headers()
console.log(tok)
});
});
Response:
'x-csrf-token': 'c7474114cc5e6b5e5deda9f46a9978dd69',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4889.0 Safari/537.36
Wants to get the x-csrf-token value in a variable. right this command returns the entire website HTTP header so there are lots of x-csrf-token but all are the same I just want one token in token varaible. any nodejs expert?
i tried this:
const token=[];
await page.goto('https://orbitxch.com/customer/inplay/highlights/1');
page.on('request', req => {
const tok = req.headers()
token.push(tok) // but this not working it return []
console.log(tok)
});
So I'm trying to scrape mediamarkt.es with this code:
PORT = 8000;
const express = require("express");
const axios = require("axios").default;
const cors = require("cors")({ origin: true });
const app = express();
const ax = axios.create({
headers: {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.9",
},
});
const tc = () => {
ax.get("https://www.mediamarkt.es")
.then((response) => {
console.log(response.status);
})
.catch((err) => {
console.log(err.response.status);
});
};
app.listen(PORT, () => {
tc();
});
Every time I get 403 error. Tried on chrome with disabled javascript and cleaned cache, so it works and returns 200, but in nodejs code I always get 403. With 403 there is html which says captcha, but I believe I can make same request with nodejs as chrome do. Just can't imagine what I'm missing...
Any help would be greatly appreciated.
Im having a little problem with my request on getting an html from https://readnovelfull.com/beauty-and-the-beast-wolf-hubby-xoxo/chapter-1-i-would-not-be-responsible.html as example.
I can get all the html on the other url eg novel detalj, latest upgated etc.
but not when im getting the detali for the chapters.
I tested those url on postman and also on https://codebeautify.org/source-code-viewer as well and there is no problem on getting the content of the chapter of which it exist under the div #chr-content
So I am a bit lost now, what am I doing wrong?
Here is my fetch calls which is working on other novel sites.
static async getHtml(
url: string
): Promise<HTMLDivElement> {
console.log(`Sending html request to ${url}`);
var container = parse('<div>test</div>') as any;
try {
let headers = new Headers({
Accept: '*/*',
'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
});
var data = await fetch(url, {
method: 'GET',
headers: headers,
});
if (!data.ok) {
const message = `An error has occured:${data.status}`;
console.log(message);
} else {
var html = await data.text();
console.log('Data is ok. proceed to parse it');
container = parse('<div>' + html + '</div>') as any;
}
} catch (e) {
console.log(e);
}
return container as HTMLDivElement;
}
I should mention that am not getting any error what so ever, its just that the html I am getting is not the same as postman and other site is getting.
Update
Ok so i did some research on the site and this is what i come up with.
the site need X-CSRF-TOKEN and i was able to extract those and find those values
const csrf = 'x09Q6KGqJOJJx2iHwNQUa_mYfG4neV9EOOMsUBKTItKfNjSc0thQzwf2HvCR7SQCqfIpC2ogPj18jG4dQPgVtQ==';
const id = 774791;
which i need to send a request to https://readnovelfull.com/ajax/increase-chapter-views with the values above. and this will send back true/false
now i tried to inc the csrf on my fetch call after but its still the same old same no data.
any idee if i am doing something wrong still?
Looks like you have an issue with CORS. To make sure just try to send request through cors proxy. One of the ways you can quickly do that is add prefix URL:
https://cors-anywhere.herokuapp.com/https://readnovelfull.com/beauty-and-the-beast-wolf-hubby-xoxo/chapter-1-i-would-not-be-responsible.html`
NOTE: Using this CORS proxy on production is not recommended, because it's not secure
If after that you'll receive data, that means that you faced with CORS, and you need to figure out how to solve it in your specific case.
Reproducable example:
const parse = (str) => str;
const getHtml = async (url) => {
console.log(`Sending html request to ${url}`);
var container = parse('<div>No content =(</div>')
try {
let headers = new Headers({
Accept: '*/*',
'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
});
var data = await fetch(url, {
method: 'GET',
headers: headers,
});
if (!data.ok) {
const message = `An error has occured:${data.status}`;
console.log(message);
} else {
var html = await data.text();
console.log('Data is ok. proceed to parse it');
container = parse('<div>' + html + '</div>');
}
} catch (e) {
console.log(e);
}
return container;
}
getHtml('https://cors-anywhere.herokuapp.com/https://readnovelfull.com/beauty-and-the-beast-wolf-hubby-xoxo/chapter-1-i-would-not-be-responsible.html').then(htmlContent => document.querySelector('div').innerHTML = htmlContent);
<div>loading...</div>
If it doesn't help, please provide a reproducible RN example, but I believe there is no difference between RN and web environments in that case.
I try to set user agent to a npm request. Here is the documentation, but it gives the following error:
Error: Invalid URI "/"
const request = require('async-request')
const run = async (url) => {
const {statusCode} = await request(url)
console.log(statusCode) // 200, works
const options = {
url,
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36'
}
}
await request(options) // Error: Error: Invalid URI "/"
}
run('https://github.com/')
sandbox
I also tried request.get, as it mentioned in here, but it gives "request.get is not a function" error.
The problem is you are looking at the documentation of request, but using the async-request, which dont support calling with the object argument like you do.
const request = require('async-request')
const run = async (url) => {
const {statusCode} = await request(url)
console.log(statusCode) // 200, works
const options = {
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36'
}
}
const res = await request(url, options);
console.log(res.statusCode) // 200, works
}
run('https://github.com/')
I'm trying to login to Amazon via the node.js request module, and seem to be having difficulties.
My aim is to login to the site via their form, here is my code:
const request = require("request");
const rp = require("request-promise");
var querystring = require("querystring");
var cookieJar = request.jar();
var mainUrl = "https://www.amazon.com/";
var loginUrl = "https://www.amazon.co.uk/ap/signin";
let req = request.defaults({
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.61 Safari/537.36"
},
jar: cookieJar,
gzip: true,
followAllRedirects: true
});
var loginData =
"email=email#me.com&create=0&password=password123";
req.post(loginUrl, { data: loginData }, function(err, res, body) {
console.log(body);
});
I ran a debugger in the background, and found this seemed to be the URL called. I'm wondering if anyone knows what I may have done incorrectly.
Thank you.