Develop Reference

Alter a object variable inside cloud functions

I am trying to alter a object variable after query in cloud functions, but before sending it to my app, but not sure how to do so! Here is what I am trying to do:
Parse.Cloud.define("getUserData", async(request) => {
// Getting the users data
const userQuery = new Parse.Query("UserData");
userQuery.equalTo("userId", request.params.userId);
const userData = await userQuery.first();
// Getting the groups
const groupQuery = new Parse.Query("GroupMembers");
groupQuery.equalTo("userId", request.params.userId);
groupQuery.include('pointerObject'); // including the pointer object
const groups = await groupQuery.find();
const allGroups = [];
for (let i = 0; i < groups.length; ++i) {
var thisGroup = groups[i].get("pointerObject");
thisGroup.isFavorite = true;
allGroups.push(thisGroup);
}
var returnObject = {
"playerData": userData,
"playerGroups": allGroups
}
const jsonString = JSON.stringify(returnObject);
return jsonString;
});
It is "thisGroup.isFavorite" i am trying to set to true, but when receiving the jsonString, it is still set to false? How do I alter a variable in cloud functions?
Any help is appreciated and thanks in advance :-)

Try with:
thisGroup.set('isFavorite', true);
await thisGroup.save();

nodejs query in query result loop

I'm performing mysql query using nodejs/mysql.
After the first result set, I will loop through the results set and query a second table.
1) Why 'for' works and 'foreach' doesn't? What's the proper looping way to achieve what I need?
2) item = { ...item, images: rows } in the getImage function also doesn't work, why?
3) How to let the console.log show the modified results? If I use 'await', the console.log(rows) should show the modified results right?
const getImages = async (item, key) => {
let sql = await connection.format('SELECT * from `images` WHERE id = ?', [item.id]);
const [ rows , fields ] = await connection.execute(sql);
item['images'] = rows
item = { ...item, images: rows } //'<-- this method doesn't work.
return item
}
let sql = await connection.format(`SELECT * from table ORDER BY a.listing_id LIMIT ?,?`, [0,10])
var [rows, fields] = await connection.execute(sql);
// rows.forEach(getImages) //'<-- this does not work when uncommented
// rows = rows.forEach(getImages) //'<-- this also does not work when uncommented.
for (let i = 0; i < rows.length; i++) { //'<-- but this works
rows[i] = await getImages(rows[i], i);
}
console.log(rows) //<----- this doesn't show modified results on terminal console
res.json(rows) //<----- browser can see the modified results on browser console

You have to pass to the foreach method the proper handler on each item:
let new_rows = Array()
rows.forEach(row => {
new_rows.push(await getImages(row, i))
});
Plus, if you want to get the images on another array you should use map, its cleaner:
let new_rows = rows.map(row => {
return await getImages(row, i)
});

Because forEach not returning anything .So better use Array#map
rows = rows.map(getImages)

ForEach only looping through first item on DataSnapshot

I´m trying to loop through the content of a DataSnapshot and then depending on a condition do some work FOR EACH one of the elements but currently, the ForEach is only doing the work in the first item. The "serverStatus" sometimes is waiting and sometimes in "onCall". When the first item is "onCall" does not go through the rest of the items as I think is supposed to do. Below a snapchot of where I get the information from:
And here is my function:
exports.manageCallRequests = functions.database.ref('/resquests/{userId}').onCreate((snap, context) => {
const event = snap.val();
console.log("function manageCallRequests is being called")
var rootPath = admin.database().ref();
var userOnCall = context.params.userId;
var serversRef = rootPath.child('servers');
var callRequest = event;
var userTime = callRequest["time"];
var waiting= "waiting";
//We first get all the servers in ascending order depending on the last time they were used
var serversSorted = serversRef.orderByChild('lastTimeUsed')
//Gets the children on the "serversSorted" Query
return serversSorted.once("value").then(allServers =>{
//Checks if there is any child
if(allServers.hasChildren()){
allServers.forEach(async function(server) {
//we extract the value from the server variable, this contains all the information
//about each one of the servers we have
var serverInfo = server.val();
var serverKey = server.key;
var serverNumber = serverInfo["serverNumber"];
var serverStatus = serverInfo["serverStatus"];
console.log("server status "+serverStatus)
if(serverStatus === waiting){
const setCallRequest = await serversRef.child(serverKey).child("current").child("callRequest").set(callRequest);
const removeUserOnCall = await rootPath.child("resquests").child(userOnCall).remove();
const setServerStatus = await serversRef.child(serverKey).child("serverStatus").set("onCall");
}
});
}else{
console.log("No servers available")
}
});
});

I had the same behavior because my cloud function was exited before that all iterations were executed in the forEach loop.I get rid of it using this snippet of code:
for (const doc of querySnapshot.docs) {
// Do wathever you want
// for instance:
await doc.ref.update(newData);
}

I found 2 ways of getting this done. The first one is useful if we have a DataSnapshot without any OrderBy* call, in this case, would be:
var allServers = await serversRef.once("value");
for (let serverKey of Object.keys(allServers.val())){
var server = allServers[serverKey];
//Do some work
}
We need to first get the keys of the object to then be able to extract it from within the for loop, as explained here otherwise we´ll get a "TypeError: 'x' is not iterable"
Now the problem with this particular case is that a have a DataSnapshot that was previously sorted at var serversSorted = serversRef.orderByChild('lastTimeUsed') so when we call Object.keys(allServers.val()) the value returned is no longer sorted and that´s where forEach() comes in handy. It guarantees the children of a DataSnapshot will be iterated in their query order as explained here however for some reasons when doing some async work within the forEach loop this seems not to work, that´s why I had to do this:
var serversSorted = serversRef.orderByChild('lastTimeUsed')
var allServers = await serversSorted.once("value");
//Checks if there is any children
if (allServers.hasChildren()) {
//if there is iterate through the event that was passed in containing all
// the servers
var alreadyOnCall = false;
var arrayOfServers = []
var arrayOfKeys = []
allServers.forEach(function(individualServer){
arrayOfKeys.push(individualServer.key)
arrayOfServers.push(individualServer)
})
for (var serveIndex = 0; serveIndex < arrayOfServers.length;serveIndex++){
var serverObj = arrayOfServers[serveIndex]
var serverObject = serverObj.val()
var serverKey = arrayOfKeys[serveIndex]
var serverStatus = serverObject["serverStatus"];
var serverNumber = serverObject["serverNumber"];
console.log("server info "+serverStatus+" "+serverKey);
if (serverStatus === waiting && alreadyOnCall === false) {
const setCallRequest = await serversRef.child(serverKey).child("current").child("callRequest").set(callRequest);
const removeUserOnCall = await rootPath.child("resquests").child(userOnCall).remove();
const setServerStatus = await serversRef.child(serverKey).child("serverStatus").set("onCall");
alreadyOnCall= true
console.log("Call properly set");
}
}
}

Waiting for an iframe to be opened and scraped is too slow to scrape js

I'm trying to scrape an old website built with tr, br and iframe. Everything was going good so far before I started to want to extract data from an iframe, see iFrameScraping setTimeout, but the clicking is too fast for me to be able to get the datas. Would anyone have an idea of how to click, wait for the content to show and be scraped, then continue?
const newResult = await page.evaluate(async(resultLength) => {
const elements = document.getElementsByClassName('class');
for(i = 0; i < resultLength; i++) {
const companyArray = elements[i].innerHTML.split('<br>');
let companyStreet,
companyPostalCode;
// Get company name
const memberNumber = elements[i].getElementsByTagName('a')[0].getAttribute('href').match(/[0-9]{1,5}/)[0];
const companyName = await companyArray[0].replace(/<a[^>]*><span[^>]*><\/span>/, '').replace(/<\/a>/, '');
const companyNumber = await companyArray[0].match(/[0-9]{6,8}/) ? companyArray[0].match(/[0-9]{6,8}/)[0] : '';
// Get town name
const companyTown = await companyArray[1].replace('"', '');
// Get region name
const companyRegion = await companyArray[2].replace(/<span[^>]*>Some text:<\/span>/, '');
// Get phone number
const telNumber = await elements[i].innerHTML.substring(elements[i].innerHTML.lastIndexOf('</span>')).replace('</span>', '').replace('<br>', '');
const iFrameScraping = await setTimeout(async({elements, i}) => {
elements[i].getElementsByTagName('a')[0].click();
const iFrameContent = await document.getElementById('some-id').contentWindow.document.getElementById('lblAdresse').innerHTML.split('<br>');
companyStreet = iFrameContent[0].replace('"', '');
companyPostalCode = iFrameContent[2].replace('"', '');
}, 2000, {elements, i});
console.log(companyStreet, companyPostalCode)
};
}, pageSearchResults.length);

I fixed my issues after a while, so I'll share my solution.
I add to stop getting all the data with a loop from the evaluate because it's going to fast and creating a race condition. Instead I used a combination of page.$$ coupled with a for…of loop. Note that the forEach from es6 are causing race condition as well, since puppeteer does not wait for them to end to continue its execution.
Here is the example from my updated code:
const companies = await page.$$('.repmbr_result_item');
const companiesLinks = await page.$$('.repmbr_result_item a');
for(company of companies) {
const companyEl = await page.evaluate(el => el.innerHTML, company)
const companyElArray = companyEl.split('<br>');

await in nested for ... of loop

async traverse(url) {
const ts = new TournamentScraper()
const ms = new MatchScraper()
const results = []
const tournaments = await ts.run(url)
for(let href of tournaments.map(t => t.href)){
let matches = await ms.run(href)
let pages = ms.getPages()
let seasons = ms.getSeasons()
//console.log(pages)
//console.log(seasons)
results.push(matches)
for(let href of pages) {
//console.log(href)
matches = await ms.run(href)
//console.log(matches)
results.push(matches)
}
}
return results
}
TournamentScraper returns an array of objects, which typically looks like this:
{name: 'Foo', href: 'www.example.org/tournaments/foo/'}
The link points to the tournament's last season's first page. This page contains the links to the other seasons and a paginator (if any).
MatchScraper's run returns some data, and sets the instance's dom property. getPages() and getSeasons() consumes this property and each returns an array of links.
The problem that results contains only the first batch of matches. I can see the 2nd page's matches in the console log, but they are not in the results array when traverse returns.
I found this rule which is against await in for loop. The problem, that I have to wait for ms.run(href), because it sets dom, and getPages() and getSeasons() needs it to be set, to extract the needed links.

I think this should work. It utilizes Promise all rather than for loops
const run = href => ms.run(href);
async function getMatches(href) {
const out = [];
const matches = await run(href);
const pages = ms.getPages();
out.push(matches);
if(pages.length) {
const pageResults = await Promise.all(pages.map(href => run(href)));
out.push(...pageResults);
}
return out;
}
async function traverse(url) {
const ts = new TournamentScraper();
const ms = new MatchScraper();
const tournaments = await ts.run(url)
const matches = await Promise.all(tournaments.map(t => getMatches(t.href)));
return matches.reduce((a, b) => {
a.push(...b);
return a;
}, []);
}