I'm trying to get the specific token value from this response in the outside of the loop so then i can show this token value in the HTML page.
import puppeteer from 'puppeteer-extra'
var http = import('http');
puppeteer.launch({ headless: true }).then(async (browser) => {
const page = await browser.newPage();
const token=[];
await page.goto('https://orbitxch.com/customer/inplay/highlights/1');
page.on('request', req => {
const tok = req.headers()
console.log(tok)
});
});
Response:
'x-csrf-token': 'c7474114cc5e6b5e5deda9f46a9978dd69',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4889.0 Safari/537.36
Wants to get the x-csrf-token value in a variable. right this command returns the entire website HTTP header so there are lots of x-csrf-token but all are the same I just want one token in token varaible. any nodejs expert?
i tried this:
const token=[];
await page.goto('https://orbitxch.com/customer/inplay/highlights/1');
page.on('request', req => {
const tok = req.headers()
token.push(tok) // but this not working it return []
console.log(tok)
});
Related
I have a simple HTTP request with javascript that uses the await feature, however, I get the following result:
{}
undefined
When I run the function, I have tested the headers and url with python and it works successfully with requests am I missing something here?
import pkg from 'superagent';
const { get } = pkg;
const header = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15'
};
const url = 'https://books.toscrape.com';
class Agent {
constructor(url, headers){
this.url = url;
this.headers = headers;
}
getAgent = async () => {
const res = await get(this.url)
.set(this.headers);
return res.body;
}
};
const request = new Agent(url, header);
console.log(await request.getAgent());
I.e. this works successfully in python:
import requests
header = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15'
};
url = 'https://books.toscrape.com';
print(requests.get(url, header).content)
My code :
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless: true,
args : [
--user-agents='view below'
]
});
const page = await browser.newPage();
await page.goto('website that has sso login', {
waitUntil: 'networkidle0',
});
//tried adding or replacing with waitForSelector
await page.click('#button_to_open_sso_popup');
const [ popup ] = await Promise.all([
new Promise((resolve) => page.once('popup', resolve)),
]);
await popup.waitForNavigation({
waitUntil: 'networkidle0' //tried default load, ...
});
//type in email and password to login then do stuff
await browser.close();
})();;
In non-headless mode it works fine
In headless mode i always get navigation timed out
TimeoutError: Navigation timeout of 30000 ms exceeded
at C:\Users\DW\node_modules\puppeteer\lib\cjs\puppeteer\common\LifecycleWatcher.js:106:111
at async FrameManager.waitForFrameNavigation (C:\Users\DW\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:167:23)
at async Frame.waitForNavigation (C:\Users\DW\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:524:16)
at async Page.waitForNavigation (C:\Users\DW\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:1229:16)
at async handleSSO (C:\Users\DW\Documents\Code Work\puppeteer_test\fptu_schedule_chat_bot\main.js:65:5)
at async main (C:\Users\DW\Documents\Code Work\puppeteer_test\fptu_schedule_chat_bot\main.js:33:5)
I have went through similar question and tried using these user agents ( not solving my problem ):
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36
I added an issue on github repo if you are interested : https://github.com/puppeteer/puppeteer/issues/7775
So I'm trying to scrape mediamarkt.es with this code:
PORT = 8000;
const express = require("express");
const axios = require("axios").default;
const cors = require("cors")({ origin: true });
const app = express();
const ax = axios.create({
headers: {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.9",
},
});
const tc = () => {
ax.get("https://www.mediamarkt.es")
.then((response) => {
console.log(response.status);
})
.catch((err) => {
console.log(err.response.status);
});
};
app.listen(PORT, () => {
tc();
});
Every time I get 403 error. Tried on chrome with disabled javascript and cleaned cache, so it works and returns 200, but in nodejs code I always get 403. With 403 there is html which says captcha, but I believe I can make same request with nodejs as chrome do. Just can't imagine what I'm missing...
Any help would be greatly appreciated.
I try to set user agent to a npm request. Here is the documentation, but it gives the following error:
Error: Invalid URI "/"
const request = require('async-request')
const run = async (url) => {
const {statusCode} = await request(url)
console.log(statusCode) // 200, works
const options = {
url,
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36'
}
}
await request(options) // Error: Error: Invalid URI "/"
}
run('https://github.com/')
sandbox
I also tried request.get, as it mentioned in here, but it gives "request.get is not a function" error.
The problem is you are looking at the documentation of request, but using the async-request, which dont support calling with the object argument like you do.
const request = require('async-request')
const run = async (url) => {
const {statusCode} = await request(url)
console.log(statusCode) // 200, works
const options = {
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36'
}
}
const res = await request(url, options);
console.log(res.statusCode) // 200, works
}
run('https://github.com/')
I'm trying to login to Amazon via the node.js request module, and seem to be having difficulties.
My aim is to login to the site via their form, here is my code:
const request = require("request");
const rp = require("request-promise");
var querystring = require("querystring");
var cookieJar = request.jar();
var mainUrl = "https://www.amazon.com/";
var loginUrl = "https://www.amazon.co.uk/ap/signin";
let req = request.defaults({
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.61 Safari/537.36"
},
jar: cookieJar,
gzip: true,
followAllRedirects: true
});
var loginData =
"email=email#me.com&create=0&password=password123";
req.post(loginUrl, { data: loginData }, function(err, res, body) {
console.log(body);
});
I ran a debugger in the background, and found this seemed to be the URL called. I'm wondering if anyone knows what I may have done incorrectly.
Thank you.