Develop Reference

How to correctly pass parameters through nested functions in for loop?

I'm trying to use nodejs to run a loop functions oh_lawd_it_chonky() that is supposed to do the following: Loop target_users array passing each user to initialize() for authorization, for each given user get data using returnbigdatachunk() take that data and refine it or do something to it using process_returnbigdatachunk() push each of the results of that for each user to an array output.
I have each of the initialize(), returnbigdatachunk() and process_returnbigdatachunk() working individually and so I've omitted the details of those functions for the sake of simplicity in this post, I just can't seem to get the darn oh_lawd_it_chonky() function working. I don't quite understand how to pass through the user parameters correctly in nested functions like this. Please help if you can! Sorry this isn't quite a working example.
const target_users = ['user1', 'user2', 'user3'];
//authorize each user
function initialize(user) {
//do the stuff to get authorization
}
//loop through all the users and build an array of all the process_returnbigdatachunk
const oh_lawd_it_chonky = async () => {
for (var f = 0; f < target_users; f++) {
try {
//get data of passed in user
const returnbigdatachunk = async () => {
const auth = initialize(user[target_users[f]]);
let output = [];
//refine data of passed in user
const process_returnbigdatachunk = async () => {
try {
const data = await returnbigdatachunk();
//push refined data into output array
output.push(process_returnbigdatachunk)
console.log("crampus: " + output)
} catch (e) {
console.error('Failed to process returnbigdatachunk', e);
}
};
};
} catch (e) {
console.error('Failed to process returnbigdatachunk', e);
}
}
};

setTimeout in for-loop does not print consecutive values
block-scoped
change you var f = 0 to let f = 0

Asyncronicity in a reduce() function WITHOUT using async/await

I am patching the exec() function to allow subpopulating in Mongoose, which is why I am not able to use async/await here -- my function will be chained off a db call, so there is no opportunity to call await on it, and within the submodule itself, there I can't add async/await outside of an async function itself.
With that out of the way, let's look at what I'm trying to do. I have two separate arrays (matchingMealPlanFoods and matchingMealPlanRecipeFoods) full of IDs that I need to populate. Both of them reside on the same array, foods. They each require a db call with aggregation, and the problem in my current scenario is that only one of the arrays populates because they are happening asynchronously.
What I am trying to do now is use the reduce function to return the updated foods array to the next run of reduce so that when the final result is returned, I can replace the entire foods array once on my doc. The problem of course is that my aggregate/exec has not yet returned a value by the time the reduce function goes into its next run. Is there a way I can achieve this without async/await here? I'm including the high-level structure here so you can see what needs to happen, and why using .then() is probably not viable.
EDIT: Updating code with async suggestion
function execute(model, docs, options, lean, cb) {
options = formatOptions(options);
let resolvedCount = 0;
let error = false;
(async () => {
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (let option of options) {
const path = option.path.split(".");
// ... various things happen here to prep the data
const aggregationOptions = [
// // $match, then $unwind, then $replaceRoot
];
await rootRefModel
.aggregate(aggregationOptions)
.exec((err, refSubDocuments) => {
// more stuff happens
console.log('newFoodsArray', newFoodsArray); // this is to check whether the second iteration is using the updated newFoods Array
const arrToReturn = newFoodsArray.map((food) => {
const newMatchingArray = food[nests[1]].map((matchingFood) => {
//more stuff
return matchingFood;
});
const updatedFood = food;
updatedFood[`${nests[1]}`] = newMatchingArray;
return updatedFood;
});
console.log('arrToReturn', arrToReturn);
newFoodsArray = [...arrToReturn];
});
}
};
console.log('finalNewFoods', newFoodsArray); // this should log after the other two, but it is logging first.
const document = doc.toObject();
document.foods = newFoodsArray;
if (resolvedCount === options.length) cb(null, [document]);
}
})()
EDIT: Since it seems it will help, here is the what is calling the execute function I have excerpted above.
/**
* This will populate sub refs
* #param {import('mongoose').ModelPopulateOptions[]|
* import('mongoose').ModelPopulateOptions|String[]|String} options
* #returns {Promise}
*/
schema.methods.subPopulate = function (options = null) {
const model = this.constructor;
if (options) {
return new Promise((resolve, reject) => execute(model, [this], options, false, (err, docs) => {
if (err) return reject(err);
return resolve(docs[0]);
}));
}
Promise.resolve();
};
};

We can use async/await just fine here, as long as we remember that async is the same as "returning a Promise" and await is the same as "resolving a Promise's .then or .catch".
So let's turn all those "synchronous but callback-based" calls into awaitables: your outer code has to keep obeying the API contract, but since it's not meant to a return a value, we can safely mark our own version of it as async, and then we can use await in combination with promises around any other callback based function calls in our own code just fine:
async function execute(model, docs, options, lean, andThenContinueToThis) {
options = formatOptions(options);
let option, resolvedCount = 0;
for (let doc of docs) {
let newFoodsArray = [...doc.foods];
for (option of options) {
// ...things happen here...
const aggregationOptions = [/*...data...*/];
try {
const refSubDocuments = await new Promise((resolve, reject) => rootRefModel
.aggregate(aggregationOptions)
.exec((err, result) => err ? reject(err) : resolve(result));
// ...do some work based on refSubDocuments...
}
// remember to forward errors and then stop:
catch (err) {
return andThenContinueToThis(err);
}
}
// remember: bind newFoodsArray somewhere so it doesn't get lost next iteration
}
// As our absolutely last action, when all went well, we trigger the call forwarding:
andThenContinueToThis(null, dataToForward);
}

Custom Function Not Defined Puppeteer

I made this custom function and put it outside globally which normally would work. I also tried moving it inside the main async puppeteer function but also doesn't work. Its a simple function. In each page evaluate function I call this and pass the selector. But, its saying not defined and promise rejection which is weird because the function isn't a promise....Please help
const grabDomConvertNodlistToArray = (grabDomHtmlPath) => {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
};
I tried turning the function into an async function adding a new parameter page. I then added async to my evaluate function and then passes the puppeteer page as an argument and still errors and not working.
const grabDomConvertNodlistToArray = async (page, grabDomHtmlPath) => {
try {
// grabbing node list from html selector all
const nList = await page.document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
} catch (error) {
console.log(error);
}
};
So I have your typical puppeteer setup where you awai browser.newPage() then you goto(url). Then i added this;
await page.exposeFunction("grabDomConvertNodlistToArray", grabDomConvertNodlistToArray);
added async to my evaluate callback function aka async() => {}. But still when calling my custom function inside the above evaluate function it doesn't work for some reason.
Found A Solution But, It Doesn't Work For Me. I'm Getting array.forEach is not a method which indicates to me that inside my grabDomConvertNodlistToArray function its not grabbing the nodeList or converting it into an array. If it did then forEach would be a function.
Solution 3
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(someURL);
var functionToInject = function(){
return 1+1;
}
var otherFunctionToInject = function(input){
return 6
}
await page.exposeFunction("functionToInject", functionToInject)
await page.exposeFunction("otherFunctionToInject", otherFunctionToInject)
var data = await page.evaluate(async function(){
console.log('woo I run inside a browser')
return await functionToInject() + await otherFunctionToInject();
});
return data
So erase the two functions above and convert it to use my function below.
const grabDomConvertNodlistToArray = (grabDomHtmlPath) => {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
};
Running my js file results in an error of array.forEach isn't a function which is weird because if the function worked as intended the const array inside my evaluate function would be an array because its = to the above function which is returning an array. So.....idk whats going on think it has something to do with the document.querySelectorAll() line.
const rlData = async () => {
const browser = await puppeteer.launch(
{
headless: true,
},
{
args: ["--flag-switches-begin", "--disable-features=OutOfBlinkCors", "--flag-switches-end"],
}
);
const pageBodies = await browser.newPage();
await pageBodies.goto("https://test.com/bodies", {
waitUntil: "load",
});
const grabDomConvertNodlistToArray = (grabDomHtmlPath) => {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
};
await pageBodies.exposeFunction("grabDomConvertNodlistToArray", grabDomConvertNodlistToArray);
const rlBodyNames = await pageBodies.evaluate(async () => {
// grabs all elements in html to make nodelist & converts it to an array
const array = grabDomConvertNodlistToArray(".testbodies > div > h1");
// push the data collected from array into data array and returned
const data = [];
array.forEach((element) => {
data.push(element.textContent);
});
return data;
});
}
rlData();
Guess I'm going to have to move the document.querySelectorAll functionality out of the custom function and back in the evaluate. However, the whole reason of making that custom function was to reduce the same code being used multiple times since my overall crawler is 238 lines long with a lot of repetitiveness. Not being able to call custom functions like mine is horrible for refactoring same code executions.
I gave up trying to get this to work and decided just to do it this way. Yeah it makes your code repetitive if you have more pages to scrape so you will be using the same code many times which is what I was trying to avoid but, puppeteer is the worse for refactoring your code maybe down the line the developers of said package will add the ability to easily use custom functions like how I was trying too.
const testNames = await pageBodies.evaluate(() => {
const nodeList = document.querySelectorAll(".test > div h2");
const array = Array.from(nodeList);
const data = [];
array.forEach((element) => {
data.push(element.textContent);
});
return data;
});

exposeFunction() is not suitable for your case: the exposed function is intended to tranfer data between browser and Node.js contexts so it can be wrapped under the hood in a code that serialize and deserialize arguments and returned data and some unserializable data (as DOM elements) can be lost. Try this instead:
const rlData = async () => {
const browser = await puppeteer.launch(
{
headless: true,
},
{
args: ["--flag-switches-begin", "--disable-features=OutOfBlinkCors", "--flag-switches-end"],
}
);
const pageBodies = await browser.newPage();
await pageBodies.evaluateOnNewDocument(() => {
window.grabDomConvertNodlistToArray = function grabDomConvertNodlistToArray(grabDomHtmlPath) {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
}
});
await pageBodies.goto("https://test.com/bodies", {
waitUntil: "load",
});
const rlBodyNames = await pageBodies.evaluate(() => {
// grabs all elements in html to make nodelist & converts it to an array
const array = grabDomConvertNodlistToArray(".testbodies > div > h1");
// push the data collected from array into data array and returned
const data = [];
array.forEach((element) => {
data.push(element.textContent);
});
return data;
});
}
rlData();

Making a fetch call within a nested for loop

I'm currently converting my local function into a lambda function and am running into a few hardships. In a previous method, I get an array of url fragments that I have to string along programmatically to a prefix to get the data back.
Logically, I figure the best way of doing this is to do a nested for loop. Go through the series of prefixes adding the url fragment and doing the fetch call.
It works fine in local, but lambda throws errors.
function getVariantData(data, cb) {
for (var i = 0; i < chapters.length; i++) {
let source = chapters[i];
// chapters = url prefix
data.forEach(async element => {
let res = await fetch(chapters[i] + element);
//element = url fragment
let body = await res.text();
createVariantsFile(element, source, body, cb);
});
}
}
this code runs fine but i've learned that lambdas are a bit more strict with forEach and async/awaits so I've changed my code to this and I've been dealing with a mess of issues. I havent gone past writing console.log because...well I haven't gotten past the error.
async function getVariantData(data, cb) {
for (var i = 0; i < chapters.length; i++) {
let source = chapters[i];
const promises = data.map((datum, index) => fetch(source+datum))
const chapterData = await Promise.all(promises)
console.log(chapterData)
// await data.map(async element => {
// return await (chapters[i] + element);
// createVariantsFile(element, source, body, cb);
// });
}
}

Chained and Nested promises with For Loop

I am trying to get each property of my games within chained promises (Each of the property is coming from a different async calls).
Logic of my algorithm:
Check the Network and Get the smart contract address
Register the contract containing the addresses of all the Games
Get the number of Games
For each game, perform one aSync call
per property
Print all the games and details (here I am not able
to get the updated object)
Code:
var games = [];
window.addEventListener('load', function() {
// Check the Network and assign the smart contract address
web3.eth.net.getId()
.then(function(networkId) {
let contractAddressRegistry;
if (networkId == 1) {
contractAddressRegistry = "0xQWERTYUIOPQWERTYUIOPQWERTY"
} else {
contractAddressRegistry = "0x12345678901234567890123456"
}
return contractAddressRegistry;
})
.then(function(contractAddressRegistry) {
let contractRegistry = new web3.eth.Contract(contractAbiRegistry, contractAddressRegistry);
contractRegistry.methods.numberOfGames().call()
.then(function(numberOfGames) {
for (let i = 0; i < numberOfGames; i++) {
let game = {};
game.propertyA = aSyncCallGetPropertyA(i); // Promise
game.propertyB = aSyncCallGetPropertyB(i); // Promise
game.propertyC = aSyncCallGetPropertyC(i); // Promise
}
games.push(game);
})
})
.then(function() {
console.log(games) // Empty
})
})
I tried used Promises.all() but I am not able to sync it properly as some async calls are within a then().
How can I make sure to get the object Games filled with all its properties?

You should use Promise.all like this. Basically, you need to wrap all three aSyncCallGetProperty async calls in Promise.all for waiting until they really finish then assign the results to object game.
whatever
.then(function(contractAddressRegistry) {
let contractRegistry = new web3.eth.Contract(contractAbiRegistry, contractAddressRegistry);
return contractRegistry.methods.numberOfGames().call();
})
.then(function(numberOfGames) {
return Promise.all(numberOfGames.map(() => {
return Promise.all([
aSyncCallGetPropertyA(),
aSyncCallGetPropertyB(),
aSyncCallGetPropertyC()
]).then(results => {
let game = {};
game.propertyA = results[0];
game.propertyB = results[1];
game.propertyC = results[2];
return game;
});
}));
})
.then(function(games) {
console.log(JSON.stringify(games));
})

#Lewis' code seems right but I can not make sure what numberOfGames is. Assuming that it's an integer as used in your question (not an array as treated in the other answer) here is a further rephrased version without nested .then()s.
window.addEventListener('load', function() {
web3.eth.net.getId()
.then(networkId => networkId === 1 ? "0xQWERTYUIOPQWERTYUIOPQWERTY"
: "0x12345678901234567890123456")
.then(contractAddressRegistry => new web3.eth.Contract(contractAbiRegistry, contractAddressRegistry).methods.numberOfGames().call())
.then(numberOfGames => Promise.all(Array(numberOfGames).fill()
.map(_ => Promise.all([aSyncCallGetPropertyA(),
aSyncCallGetPropertyB(),
aSyncCallGetPropertyC()]))))
.then(function(games){
games = games.map(game => ({propertyA: game[0],
propertyB: game[1],
propertyC: game[2]}));
doSomethingWith(games);
});
});