Background
I am using Puppeteer to create some PDFs. I need to inject some data into the page when Puppeteer loads it.
Problem
I have tried using evaluateOnNewDocument() which was successful when using a String only. When I try with an Object it fails. I also tried with evaluate() and it fails regardless of what I pass in.
Example
// Works
await page.evaluateOnNewDocument(() => {
window.pdfData = {};
window.pdfData = "Some String";
});
// Does not work
await page.evaluateOnNewDocument(() => {
window.pdfData = {};
window.pdfData = data;
});
// Fails
await page.evaluate(data => {
window.pdfData = {};
window.pdfData = data;
}, data);
I would like to access this object like this,
const data = window.pdfData;
Question
What is the proper way to pass a data object into window on a loaded Puppeteer page so that it can be accessed within the page to use the data client side?
Passing object to evaluate
You can pass data which will be serialized as JSON.
await page.evaluateOnNewDocument(data => { // <-- pass as parameter
window.pdfData = data; // <-- read it here
}, data); // <-- pass as argument
Passing object to evaluateOnNewDocument
evaluateOnNewDocument works similarly to evaluate, except it will run whenever there is a new window/navigation/frame. This way the data will stay even if you navigate away to another page.
You can pass data and read inside the function.
await page.evaluateOnNewDocument(data => {
window.pdfData = data;
}, data);
Related
I work on a chrome extension which has a const user that is set in the first lines of the background.js, when I change its values and use them later in a different function they always return to their set values in the beginning.
I read online about hoisting and that it is to source of the problem but how can I avoid it so i can actually manipulate the variables and use them later on with no problem.
I also saw the use of globalThis but when i tried to call or set the values in the functions with globalThis it didn't work as well.
Here is my first lines of background.js:
var user={
connection:undefined,
id:"id",
username:"username",
room:["id","password","url","tabid","ishost"],
room_members:[],
in_room:false,
room_process:[false,"intervelId"],
current_tab:["url","id"]
}
I also tried working with globalThis when setting some of the values of user like showen here:
function connect(){
globalThis.user.connection = new WebSocket('ws://192.168.3.16:8765');
globalThis.user.connection.onopen = function(e) {
globalThis.user.connection.send("get_id");
};
globalThis.user.connection.onmessage=function(event){
let data=event.data.split(',');
let temp_i=data[0]
globalThis.user.id=temp_i;
let temp_u=data[1]
globalThis.user.username=temp_u;
console.log("connection: ",user.id,",",user.username);
};
}
and for example if I got from the last function that user.id="Oak" when I try to recall the value in a different function like this:
function test(){
console.log(user.id);
}
it shows "id" and not "Oak"
EDIT
I tried to use the storage.local by putting the var user in the onInstalled event of the program and then by having two functions I can get the current value of user out of the local storage and then update it as well.
here is my code:
first init of the var user:
chrome.runtime.onInstalled.addListener(async () => {
//opens welcome page
let url = chrome.runtime.getURL("htmls/hello.html");
let tab = await chrome.tabs.create({ url });
//setting up the user var in chrome's local storage
chrome.storage.local.get(['userLocal'], async function (result) {
var user={
connection:undefined,
id:"id",
username:"username",
room:["id","password","url","tabid","ishost"],
room_members:[],
in_room:false,
room_process:[false,"intervelId"],
current_tab:["url","id"]
}
chrome.storage.local.set({userLocal: user}, function () {}); // save it in local.
});
//connect to server and recive id
connect()
});
the two functions function:
function get_user(){
chrome.storage.local.get(['userLocal'], async function (result) {
var userLocal = result.userLocal;
return userLocal;
});
}
function update_user(tmp_user){
chrome.storage.local.get(['userLocal'], async function (result) {
var userLocal = tmp_user;
chrome.storage.local.set({userLocal: userLocal}, function () {}); // save it in local.
return userLocal;
});
}
and for example if I use what I've written:
function connect(){
var user=get_user();
user.connection = new WebSocket('ws://192.168.3.16:8765'); //'ws://192.168.3.16:8765'
user=update_user(user);
}
I get this error: "Uncaught (in promise) TypeError: Cannot set properties of undefined (setting 'connection')"
The problem is that a background service worker is loaded when it is needed, and unloaded when it goes idle. So, after a few seconds of being idle, the background script unloads itself, making you lose the changes you made to the user object.
You could save the user object in chrome.storage.local, and then read and modify from there. Something like this:
// background.js; "initial lines" =>
chrome.storage.local.get(['userLocal'], async function (result) {
let userLocal = result.userLocal
if (userLocal === undefined) {
// it means there was nothing before. This way you don't overwrite
// the user object every time the backgorund.js loads.
var user={
connection:undefined,
id:"id",
username:"username",
room:["id","password","url","tabid","ishost"],
room_members:[],
in_room:false,
room_process:[false,"intervelId"],
current_tab:["url","id"]
}
chrome.storage.local.set({userLocal: user}, function () {}); // save it in local.
}})
When you want to modify the user, you retrieve it from local and modify the result before saving it back in the localStorage.
// example of modifying a property =>
chrome.storage.local.get(['userLocal'], async function (result) {
let userLocal = result.userLocal
if (userLocal !== undefined) {
userLocal.connection = new WebSocket('ws://192.168.3.16:8765');
// ...
chrome.storage.local.set({userLocal: userLocal}, function () {}); // save it in local.
}})
You can also just read the user properties, without modifying them and saving them back to the localStorage.
############ EDIT:
because chrome.storage.local is asynchronous, try to transform you code like this:
async function get_user(){
return new Promise(async function (res, rej) {
chrome.storage.local.get(['userLocal'], async function (result) {
var userLocal = result.userLocal;
res(userLocal);
});
});
}
var user= await get_user(); // call it like this.
I'm using playwright to scrape some data from a page. I need to call a page function in order to change the browser's state to collect additional info.
What's the syntax for getting a function name from a data attribute and then calling that function on the page?
I keep getting the error :UnhandledPromiseRejectionWarning: page.evaluate: Evaluation failed: TypeError: cb is not a function
Here's what I have so far:
const { chromium} = require("playwright");
(async()=>{
this.browser = await chromium.launch({
headless: true,
});
this.context = await this.browser.newContext();
this.page = await this.context.newPage();
this.page.goto('http://fakeExample.org')
const callbackHandle = await this.page.$('[data-callback]');
const cbName = await callbackHandle.evaluate(element=>element.getAttribute('data-callback')); //returns actual page function name 'myPageFunction'
this.page.evaluate((cb) => {
cb() //should call myPageFunction() on the page
}, cbName)
})()
I think it comes down to either window[cb]() or eval(cb) since you're passing in a string with the function name.
Some reading on this topic:
How to execute a JavaScript function when I have its name as a string
Call a JavaScript function name using a string?
Call a function whose name is stored in a variable
Fetch is often used to retrieve data from api. But if you want to retrieve this data in every line of the code file, you have to type fetch again and again and iterate through these data. But if i want to call only part of it for example username. Look example:
fetch ('/ API')
.then ((res) => res.json ())
.then ((data) => {
let output = 'Here is my output<br />';
data.forEach (function (item)
{
output + = `$ {item.username}`;
})
document.getElementById ('here'). innerHTML = output;
})
if i want to call $ {item.username} in each place in the file without calling fetch.
How to save these data in a variabel and call it every time i want.
Sorry for my bad English.
But if you want to retrieve this data in every line of the code file, you have to type fetch again and again and iterate through these data.
You can call fetch one time and keep the response in your local state, things like username usually stay the same so you should not call that endpoint again just to fetch a username because you can take it from your previous response.
It depends on your framework, but you can write window.username = usernameFromFetch on your first call, and you will have access to it every where (which is not the best aproach but it will work).
From what I understand through your question is that you want to access variable item.username globally and from the code it looks to be an array of users with username property.
In this case, you can define a variable object
var data = {
usernames = []
};
before your fetch function and the value will be initialized to an empty array and as soon as you fetch('/API') users you can do something like this
data.forEach (function (item)
{
data.usernames.push(item.username);
})
data.usernames will have your usernames throughout your current JS file.
access by
data.usernames[0]
data.usernames[1]
data.usernames[2] or
let output = 'Here is my output<br />';
data.usernames.forEach(function(username){
output + = `$ {username}`;
})
document.getElementById('here').innerHTML = output;`
You can use a memoization function to save the result:
// Stores in memory pairs of request urls (arguments) and its result
const memo = (callback) => {
const cache = new Map();
return (...args) => {
const selector = JSON.stringify(args);
if (cache.has(selector)) return cache.get(selector);
const value = callback(...args);
cache.set(selector, value);
return value;
};
};
// Memoize the fetch function. This function 'cachedRequest' should be used across your application
const cachedRequest = memo(fetch);
// Your url to call
const URL = "/API";
// First time the fetch is called, the result will be stored in the memory of 'memo' function. Second time its called, it will be retrieved from the previous in memory result and the fetch call wont be done
cachedRequest(URL).then((response) => {
console.log({response});
}).catch((error) => {
console.log({error});
})
Try this using localStorage:
fetch ('/ API')
.then ((res) => res.json ())
.then ((data) => {
// Store
localStorage.setItem("localData", data);
// Retrieve
var data = localStorage.getItem("localData");
let output = 'Here is my output<br />';
data.forEach (function (item)
{
output + = `$ {item.username}`;
})
document.getElementById ('here'). innerHTML = output;
})
Or use sessionStorage:
fetch ('/ API')
.then ((res) => res.json ())
.then ((data) => {
// Store
sessionStorage.setItem("localData", data);
// Retrieve
var data = sessionStorage.getItem("localData");
let output = 'Here is my output<br />';
data.forEach (function (item)
{
output + = `$ {item.username}`;
})
document.getElementById ('here'). innerHTML = output;
})
I've an object and I push that into main array just when document is loaded, now after response from server data is pushed into specific arrays which are nested into main array.
I have tried making a new instance of MainData each time when I want to push into array, but object name has to be same (as there are lot of requests and all of them has to be asynchronous) that is why it is mixing up values.
I am doing right now:
// Knockout viewModel
function ViewModel() {
self = this
self.main_array = ko.observableArray([])
}
var pointer = new ViewModel();
ko.applyBidings(pointer);
// Constructor for dummy data
function MainData() {
self = this
self.id = ko.observable()
self.first_val = ko.observableArray()
self.second_val = ko.observableArray()
}
var main_data;
// Main function which is called when document is loaded
num_type = (data) => { // data is 1234
main_data = new MainData();
main_data["id"] = data;
pointer.main_array.push(main_data);
// Fetch Requests
fetch("Num/Func1").then(x => {
x.json().then(b => {
main_data.first_val(b);
char_type(b[0].char); // this has to be called once b is loaded
})
})
// Another fetch request which gives a little late response
fetch("Num/Func2").then(x => {
x.json().then(b => {
// It takes 10 plus seconds meanwhile char_type function has fetched data
main_data.second_val(b);
})
})
}
// char_type function
char_type = (data) => { // data is abc
main_data = new MainData();
main_data["id"] = data;
pointer.main_array.push(main_data);
// Fetch Request
fetch("Char/Func1").then(x => {
x.json().then(b => {
main_data.first_val(b); // This response comes before fetch to 'Num/Func2'
})
})
fetch("Char/Func2").then(x => {
x.json().then(b => {
main_data.second_val(b); // This response also comes before fetch to
//'Num/Func2'
})
})
}
The values from both indices mixed together i.e index that is having main_data.id = 123 carries data from index which is having main_data.id = abc
Also There are some server calls which are outside num_type and char_type function.
How to get rid of that mixing stuff?
You need to move the declaration of main_edata inside the function, so you get a different, local copy each time the function is called. Then, the fetch callbacks will close over the local copy related to the num_type/char_type call that triggered that specific fetch.
Change this:
var main_data;
// Main function which is called when document is loaded
num_type = (data) => { // data is 1234
main_data = new MainData();
// ...
}
to this:
// Main function which is called when document is loaded
num_type = (data) => { // data is 1234
var main_data = new MainData(); // <====== Note `var`
// ...
}
And add var before main_data in char_type, too.
You need to create local variable for each main function and then pass that to other functions.
like:
num_type = (data) => {
var main_data = new MainData();
other_fun(data,main_data)
}
Why I can't see my messages at console.log in page.evaluate, page.$, page.$$, page.$eval, page.$$eval
And can't to get access to variables out that?
let variable = 0;
const divColors = await page.evaluate(() => {
const divs = Array.from(document.querySelectorAll('.map-filters div'));
let text = divs.map((element, index) => {
console.log(element.textContent)
variable =1;
return element.style.color;
})
return text;
})
Why I can't do variable=1 and console.log(element.textContent) in this example?
You're using console.log inside of page.evaluate, so it is logging its output to the Chromium browser and not to node output. To see console messages from browser in node's console one needs to subscribe to them after page object is created and before console.log is used in the script:
const page = await browser.newPage();
page.on('console', consoleObj => console.log(consoleObj.text()));
page.evaluate(...);
As for variable variable, there are in fact two of them in your script.
The first one exists in node.js context:
let variable = 0;
And the other one — in web page context:
page.evaluate( () => {
variable = 1;
})
They are completely different. Think of page.evaluate as of a portal into another world: objects that exist there are only present inside of a javascript runtime on a page open in the web browser that puppeteer is driving. node has its own runtime with its own set of objects.
You may pass data into page.evaluate from node:
let variable = 420;
page.evaluate(variable => {
// it is now passed here from outside
console.log(variable)
}, variable);