I'm trying to pass a variable into a page.evaluate() function in Puppeteer, but when I use the following very simplified example, the variable evalVar is undefined.
I can't find any examples to build on, so I need help passing that variable into the page.evaluate() function so I can use it inside.
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
const evalVar = 'WHUT??';
try {
await page.goto('https://www.google.com.au');
await page.waitForSelector('#fbar');
const links = await page.evaluate((evalVar) => {
console.log('evalVar:', evalVar); // appears undefined
const urls = [];
hrefs = document.querySelectorAll('#fbar #fsl a');
hrefs.forEach(function(el) {
urls.push(el.href);
});
return urls;
})
console.log('links:', links);
} catch (err) {
console.log('ERR:', err.message);
} finally {
// browser.close();
}
})();
You have to pass the variable as an argument to the pageFunction like this:
const links = await page.evaluate((evalVar) => {
console.log(evalVar); // 2. should be defined now
…
}, evalVar); // 1. pass variable as an argument
You can pass in multiple variables by passing more arguments to page.evaluate():
await page.evaluate((a, b c) => { console.log(a, b, c) }, a, b, c)
The arguments must either be serializable as JSON or JSHandles of in-browser objects: https://pptr.dev/#?show=api-pageevaluatepagefunction-args
I encourage you to stick on this style, because it's more convenient and readable.
let name = 'jack';
let age = 33;
let location = 'Berlin/Germany';
await page.evaluate(({name, age, location}) => {
console.log(name);
console.log(age);
console.log(location);
},{name, age, location});
Single Variable:
You can pass one variable to page.evaluate() using the following syntax:
await page.evaluate(example => { /* ... */ }, example);
Note: You do not need to enclose the variable in (), unless you are going to be passing multiple variables.
Multiple Variables:
You can pass multiple variables to page.evaluate() using the following syntax:
await page.evaluate((example_1, example_2) => { /* ... */ }, example_1, example_2);
Note: Enclosing your variables within {} is not necessary.
It took me quite a while to figure out that console.log() in evaluate() can't show in node console.
Ref: https://github.com/GoogleChrome/puppeteer/issues/1944
everything that is run inside the page.evaluate function is done in the context of the browser page. The script is running in the browser not in node.js so if you log it will show in the browsers console which if you are running headless you will not see. You also can't set a node breakpoint inside the function.
Hope this can help.
For pass a function, there are two ways you can do it.
// 1. Defined in evaluationContext
await page.evaluate(() => {
window.yourFunc = function() {...};
});
const links = await page.evaluate(() => {
const func = window.yourFunc;
func();
});
// 2. Transform function to serializable(string). (Function can not be serialized)
const yourFunc = function() {...};
const obj = {
func: yourFunc.toString()
};
const otherObj = {
foo: 'bar'
};
const links = await page.evaluate((obj, aObj) => {
const funStr = obj.func;
const func = new Function(`return ${funStr}.apply(null, arguments)`)
func();
const foo = aObj.foo; // bar, for object
window.foo = foo;
debugger;
}, obj, otherObj);
You can add devtools: true to the launch options for test
I have a typescript example that could help someone new in typescript.
const hyperlinks: string [] = await page.evaluate((url: string, regex: RegExp, querySelect: string) => {
.........
}, url, regex, querySelect);
Slightly different version from #wolf answer above. Make code much more reusable between different context.
// util functions
export const pipe = (...fns) => initialVal => fns.reduce((acc, fn) => fn(acc), initialVal)
export const pluck = key => obj => obj[key] || null
export const map = fn => item => fn(item)
// these variables will be cast to string, look below at fn.toString()
const updatedAt = await page.evaluate(
([selector, util]) => {
let { pipe, map, pluck } = util
pipe = new Function(`return ${pipe}`)()
map = new Function(`return ${map}`)()
pluck = new Function(`return ${pluck}`)()
return pipe(
s => document.querySelector(s),
pluck('textContent'),
map(text => text.trim()),
map(date => Date.parse(date)),
map(timeStamp => Promise.resolve(timeStamp))
)(selector)
},
[
'#table-announcements tbody td:nth-child(2) .d-none',
{ pipe: pipe.toString(), map: map.toString(), pluck: pluck.toString() },
]
)
Also not that functions inside pipe cant used something like this
// incorrect, which is i don't know why
pipe(document.querySelector)
// should be
pipe(s => document.querySelector(s))
Related
I have this code in my action creator:
const getMyFilenames = async () => {
const response = await axios.get(API_URL, {
withCredentials: true,
});
const rawFilenames = response.data;
const filenames = rawFilenames.map((filename) => filename.split("-"));
for (filename in filenames) {
filenames[filename].shift();
filenames[filename] = filenames[filename].join("-");
}
return { filenames, rawFilenames };
};
I don't understand why the 'filename' constant calls are showing as undefined at runtime.
I assume it must be something to do with the asynchronous code at the beginning of the function, but I'm not sure...
Many thanks
for in loops need to have the iterator initialised with const, in React.
for (const filename in filenames) {
is the solution
I'm a novice with Javascript and am struggling to understand how or at least, how best to return array values to another script to assert agains their values.
The context is I want to use Puppeteer to obtain some string values from WebElement attributes and then use the Chai expect library to assert for correct values( or otherwise).
The code I have thus far is:
//app.spec.js
const clothingChoice = await frame.$eval('#option-clothing-5787', e => e.getAttribute('value'));
const groceryChoice = await frame.$eval('#option-clothing-4556', e => e.getAttribute('value'));
const wineChoice = await frame.$eval('#option-clothing-4433', e => e.getAttribute('value'));
const voucherChoice = await frame.$eval('#option-clothing-3454', e => e.getAttribute('value'));
function testFunction() {
return new Promise(function(resolve, reject) {
resolve([clothingChoice, groceryChoice, wineChoice, voucherChoice]);
});
}
async function getChosenItemValues() {
const [clothingChoice, groceryChoice, wineChoice, voucherChoice] = await testFunction();
console.log(clothingChoice, groceryChoice, wineChoice, voucherChoice);
}
getChosenItemValues();
module.exports = getChosenItemValues;
};
I simply need to understand how to import the values that are currently simply printed out as:
1|clothing|option 1|grocery|option 1|wine|option 1|voucher|option
...into another file test.js in which I want to use chai to assert for their presence like so:
const [clothingEmailGrantedValue,emailRewardsNewsletterGrantedValue, emailGroceryOffersGrantedValue,telephoneRewardsDeniedValue ] = await app.spec.js(page, frame);
expect(clothingChoice).to.equal('1|clothing|option');
One way of doing this would be to export directly your function and destructure your array inside your test.js file.
async function getChosenItemValues() {
return await testFunction();
}
module.exports = getChosenItemValues()
inside your test.js file
const [clothingChoice, groceryChoice, wineChoice, voucherChoic] = require('./app.spec.js');
console.log(clothingChoice, groceryChoice, wineChoice, voucherChoice);
I'm a novice with Javascript and am struggling to understand how or at least, how best to return array values to another script to assert agains their values.
The context is I want to use Puppeteer to obtain some string values from WebElement attributes and then use the Chai expect library to assert for correct values( or otherwise).
The code I have thus far is:
//app.spec.js
const clothingChoice = await frame.$eval('#option-clothing-5787', e => e.getAttribute('value'));
const groceryChoice = await frame.$eval('#option-clothing-4556', e => e.getAttribute('value'));
const wineChoice = await frame.$eval('#option-clothing-4433', e => e.getAttribute('value'));
const voucherChoice = await frame.$eval('#option-clothing-3454', e => e.getAttribute('value'));
function testFunction() {
return new Promise(function(resolve, reject) {
resolve([clothingChoice, groceryChoice, wineChoice, voucherChoice]);
});
}
async function getChosenItemValues() {
const [clothingChoice, groceryChoice, wineChoice, voucherChoice] = await testFunction();
console.log(clothingChoice, groceryChoice, wineChoice, voucherChoice);
}
getChosenItemValues();
module.exports = getChosenItemValues;
};
I simply need to understand how to import the values that are currently simply printed out as:
1|clothing|option 1|grocery|option 1|wine|option 1|voucher|option
...into another file test.js in which I want to use chai to assert for their presence like so:
expect(clothingChoice).to.equal('1|clothing|option');
Try this:
// app.spec.js (.spec is normally reserved for test files, you may to change the name to avoid confusion)
const clothingChoice = frame.$eval('#option-clothing-5787', e => e.getAttribute('value'));
const groceryChoice = frame.$eval('#option-clothing-4556', e => e.getAttribute('value'));
const wineChoice = frame.$eval('#option-clothing-4433', e => e.getAttribute('value'));
const voucherChoice = frame.$eval('#option-clothing-3454', e => e.getAttribute('value'));
async function getChosenItemValues() {
return await Promise.all([clothingChoice, groceryChoice, wineChoice, voucherChoice]);
}
module.exports = {
getChosenItemValues
};
NOTE: does frame.$eval definitely return a Promise? I've not had any experience with Puppeteer.
// test file
const app = require('/path/to/app.spec.js');
describe('Suite', function() {
it('Returns expected values', function(done) {
app.getChosenItemValues()
.then(res => {
const [clothingChoice, groceryChoice, wineChoice, voucherChoice] = res;
// assertions here
done();
});
});
});
Excuse me if the test functions aren't in the correct format, I don't use Mocha or Chai.
Background
I am trying to filter an array of objects. Before I filter, I need to convert them to some format, and this operation is asynchronous.
const convert = () => new Promise( resolve => {
setTimeout( resolve, 1000 );
});
So, my first try was to do something like the following using async/await:
const objs = [ { id: 1, data: "hello" }, { id: 2, data: "world"} ];
objs.filter( async ( obj ) => {
await convert();
return obj.data === "hello";
});
Now, as some of you may know, Array.protoype.filter is a function which callback must return either true or false. filter is synchronous. In the previous example, I am returning none of them, I return a Promise ( all async functions are Promises ).
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
So as one can assume, the code before doesn't really work... That assumption is correct.
Problem
To make filter work with an async function, I checked stackoverflow and found this topic:
Filtering an array with a function that returns a promise
Unfortunately, the chosen answer is overly complex and uses classes. This won't do for me. I am instead looking for a more simple solution, using simple functions with a functional approach.
There is one solution at the very end, using a map with a callback to simulate a filter:
https://stackoverflow.com/a/46842181/1337392
But I was hoping to fix my filter function, not to replace it.
Questions
Is there a way to have an async function inside a filter?
If not, what is the simplest replacement I can do?
There is no way to use filter with an async function (at least that I know of).
The simplest way that you have to use filter with a collection of promises is to use Promise.all and then apply the function to your collection of results.
It would look something like this:
const results = await Promise.all(your_promises)
const filtered_results = results.filter(res => //do your filtering here)
Hope it helps.
Adapted from the article How to use async functions with Array.filter in Javascript by Tamás Sallai, you basically have 2 steps:
One that creates the conditions for an object to pass
One that receives the objects and returns true or false according to conditions
Here's an example
const arr = [1, 2, 3, 4, 5];
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
const asyncFilter = async (arr, predicate) => {
const results = await Promise.all(arr.map(predicate));
return arr.filter((_v, index) => results[index]);
}
const asyncRes = await asyncFilter(arr, async (i) => {
await sleep(10);
return i % 2 === 0;
});
console.log(asyncRes);
// 2,4
Use Scramjet fromArray/toArray methods...
const result = await scramjet.fromArray(arr)
.filter(async (item) => somePromiseReturningMethod(item))
.toArray();
as simple as that - here's a ready example to copy/paste:
const scramjet = require('../../');
async function myAsyncFilterFunc(data) {
return new Promise(res => {
process.nextTick(res.bind(null, data % 2));
});
}
async function x() {
const x = await scramjet.fromArray([1,2,3,4,5])
.filter(async (item) => myAsyncFilterFunc(item))
.toArray();
return x;
}
x().then(
(out) => console.log(out),
(err) => (console.error(err), process.exit(3)) // eslint-disable-line
);
Disclamer: I am the author of scramjet. :)
Build a parallel array to your array which you want to call filter on.
Await all of the promises from your filter func, in my eg, isValid.
In the callback in filter, use the 2nd arg, index, to index into your parallel array to determine if it should be filtered.
// ===============================================
// common
// ===============================================
const isValid = async (value) => value >= 0.5;
const values = [0.2, 0.3, 0.4, 0.5, 0.6];
// ===============================================
// won't filter anything
// ===============================================
const filtered = values.filter(async v => await isValid(v));
console.log(JSON.stringify(filtered));
// ===============================================
// filters
// ===============================================
(async () => {
const shouldFilter = await Promise.all(values.map(isValid));
const filtered2 = values.filter((value, index) => shouldFilter[index]);
console.log(JSON.stringify(filtered2));
})();
This behavior makes sense since any Promise instance has a truthy value, but it's not intuitive at a glance.
This answer uses library iter-ops, which handles iterable objects, and supports async filtering:
import {pipeAsync, filter, toAsync} from 'iter-ops';
// your input data:
const objs = [{id: 1, data: 'hello'}, {id: 2, data: 'world'}];
const i = pipeAsync(
objs,
filter(async value => {
await convert(); // any async function
return value.data === 'hello'; // filtering logic
})
);
for await(const a of i) {
console.log(a); // filtered data
}
P.S. I'm the author of iter-ops.
Reduce method can mimic filter and can operate with promises.
const isPositiveNumberAsync = async (number) => number >= 0;
const filterPositiveNumbersAsync = async (numbers) => numbers?.reduce(async (accumulatorPromise, number) => {
const accumulator = await accumulatorPromise;
if (await isPositiveNumberAsync(number)) {
return [...accumulator, number];
}
return accumulator;
}, Promise.resolve([])) || [];
(async () => {
// no numbers argument provided
console.log(await filterPositiveNumbersAsync());
// an empty argument list provided
console.log(await filterPositiveNumbersAsync([]));
// ok, but no positive numbers provided
console.log(await filterPositiveNumbersAsync([-1,-2,-3]));
// ok, positive numbers filtered.
console.log(await filterPositiveNumbersAsync([0,1,-1,-3,2,-2]));
})();
Array.prototype.asyncFilter =function( filterFn) {
const arr = this;
return new Promise(function(resolve){
const booleanArr = [];
arr.forEach(function (e) {
booleanArr.push(filterFn(e))
})
Promise.all(booleanArr).then(function (booleanArr) {
const arr2 = arr.filter(function (e, i) {
return booleanArr[i]
})
resolve(arr2)
})
})
}
/** use it like this**/
const arr=[1,2,3]
arr.asyncFilter(async e=>{}).then(...)
You can use Promise.filter from Bluebird that works similarly to Array.filter but it supports async & await.
Add asyncFilter as an extension to Array:
#available(macOS 10.15.0, *)
extension Array where Element: Any {
public func asyncFilter(closure: (Element) async -> Bool) async -> Array {
var result = [Element]()
for item in self {
if await closure(item) {
result.append(item)
}
}
return result
}
}
Usage:
result = await result.asyncFilter { item in
if <item match> {
return true
}
}
Is it possible in ES6 to set a variable inside of a try{} using const in strict mode?
'use strict';
const path = require('path');
try {
const configPath = path.resolve(process.cwd(), config);
} catch(error) {
//.....
}
console.log(configPath);
This fails to lint because configPath is defined out of scope. The only way this seems to work is by doing:
'use strict';
const path = require('path');
let configPath;
try {
configPath = path.resolve(process.cwd(), config);
} catch(error) {
//.....
}
console.log(configPath);
Basically, is there any way to use const instead of let for this case?
Declaring a variable as const requires you to immediately point it to a value and this reference cannot be changed.
Meaning you cannot define it at one place (outside of try) and assign it a value somewhere else (inside of try).
const test; // Syntax Error
try {
test = 5;
} catch(err) {}
On the other hand, both creating it and giving it a value within the try block is fine.
try {
const test = 5; // this is fine
} catch(err) {}
However, const is block-scoped, like let, so if you do create it and give it a value within your try block, it will only exist within that scope.
try {
const test = 5; // this is fine
} catch(err) {}
console.log(test); // test doesn't exist here
Therefore, if you need to access this variable outside of the try, you must use let:
let configPath;
try {
configPath = path.resolve(process.cwd(), config);
} catch(error) {
//.....
}
console.log(configPath);
Alternatively, although probably more confusingly, you can use var to create a variable within the try and use it outside of it because var is scoped within the function, not the block (and gets hoisted):
try {
var configPath = path.resolve(process.cwd(), config);
} catch(error) {
//.....
}
console.log(configPath);
'use strict';
const path = require('path');
const configPath = (function() {
try {
return path.resolve(process.cwd(), config);
} catch (error) {
//.....
}
})()
console.log(configPath);
I would try to use a temp variable with let and assign that to a const var after the try/catch and 'delete' the temp var.
'use strict';
let temp;
try {
temp = path.resolve(process.cwd(), config);
} catch (error) {
//.....
}
const configPath = temp;
temp = undefined;
console.log(configPath);
There are plenty of good answers here. But it's real annoying to have to manage your lets being inside and out of scope. So if you are like me, and came here searching for a better pattern. I wrote a little utility that helps a ton.
export const tc = <T>(option: { try: () => T; catch: (e: Error) => T }) => {
try {
return option.try()
} catch (e) {
return option.catch(e)
}
}
Here are some unit tests to show how it's useful
import { tc } from './tc'
describe('tc', () => {
it('should return successfully', () => {
const result = tc({
try: () => 1 + 2,
catch: () => -1,
})
expect(result).toEqual(3)
})
it('should catch', () => {
const spy = jest.fn()
const result = tc({
try: (): number => {
throw new Error('Test')
},
catch: (e) => {
spy()
expect(e.message).toEqual('Test')
return -1
},
})
expect(spy).toHaveBeenCalledTimes(1)
expect(result)
})
it('should rethrow', () => {
expect(() =>
tc({
try: (): number => {
throw new Error('Test')
},
catch: (e) => {
throw e
},
}),
).toThrowError()
})
it('should have proper types', () => {
// #ts-expect-error
const result: string = tc({
try: () => 12,
catch: (e) => {
return -1
},
})
})
})
You can avoid the try block altogether if the function is async! Just learned this today, thought I'd share!
More broadly applicable to just your situation as this is the top Google result for "const in a try block"
'use strict';
const path = require('path');
const getPath = async () => {
return path.resolve(process.cwd())
}
const logPath = async () => {
const configPath = await getPath().catch(e => console.log(e)) <--avoid the try
console.log(configPath);
}
logPath()
Works great when you're already in an async function and need to wait for something else:
async function main() {
const result = await asyncTask().catch(error => console.error(error));
}
Learned from: https://stackoverflow.com/a/55024396/2936521
Besides the let options I see here, another option may be to use an object, as the reference to the object is constant, but it's properties can be altered, so something like this:
'use strict';
const path = require('path');
const configPath = { value: null };
try {
configPath.value = path.resolve(process.cwd(), config);
} catch(error) {
//.....
}
console.log(configPath.value);
It would probably be cleaner to stick with let, but I just wanted to point out another possible option.
You can just do:
const result = await promise.catch(err=>{
console.log(err)
})
Use let. You cannot use const. const does not allow you to reassign the declared constant. While it is generally good practice to declare objects like yours with const, the entire point of doing so is to allow objects to be mutated without allowing them to be reassigned. You are reassigning the object (thus, defeating the purpose of const), so use let instead.
let path = require('path');
// Good to go!