I'm writing a cloud function that uses request-promise and cheerio to scrape a website and then check that information against a user document.
I am not entirely familiar with Javascript and Cloud Functions.
I've come so far that I managed to extract the information I need and navigate to the user's document and compare the data. Now the last piece of this function is to give the user points for each matching data point, so I need to update a map inside the user document.
This function has to loop through all users and change their document if the data point matches. I'm not sure the way I've written the code is the most optimal in terms of performance and billing if the userbase gets huge... Any pointers to how I could minimize the impact on the task would be of great help, as im new with JS.
So this is the code:
exports.getV75Results = functions.pubsub.schedule('every 2 minutes').onRun(async (context) => {
let linkMap = new Map();
const url = `https://www.example.com`
const options = {
uri: url,
headers: { 'User-Agent': 'test' },
transform: (body) => cheerio.load(body)
}
await rp(options)
.then(($) => {
for(let i = 1; i <= 7; i++)
{
//Find player from game
const lopp1 = $(`#mainContentHolder > div > div.mainContentStyleTrot > div > div.panel-body > table:nth-child(1) > tbody > tr:nth-child(${i}) > td:nth-child(2) > span`).text()
const lopp1StrR1 = lopp1.replace("(", "");
const lopp1StrR2 = lopp1StrR1.replace(")", "");
const lopp1StrR3 = lopp1StrR2.replace(" ", "");
linkMap.set(i, lopp1StrR3.toUpperCase());
}
console.log(linkMap);
return linkMap;
}).then(async () => {
//Start lookup users
let usersRef = db.collection('fantasyfotball').doc('users');
usersRef.listCollections().then(collections => {
collections.forEach( collection => {
var user = collection.doc(collection.id);
let batch = new admin.firestore().batch();
user.get().then(function(doc) {
let json = doc.data();
//Look in users collection if players document exist
Object.keys(json).forEach((name) => {
if(name != null) {
//Document with users active fotball players
if(name == 'players') {
let i = 0;
Object.values(json[name]).forEach((value) => {
i++;
if(value.localeCompare(linkMap.get(i)) == 0) {
//Loop through user keys and find owned players if user has the correct player
Object.keys(json).forEach((map) => {
if(map != null)
{
//Document with a map of player owned fotball players, each respective player has a key = 'fotball player' and value = '[price, points]'
if(map == 'ownedplayers')
{
Object.entries(json[map]).forEach((players) => {
if(players[0].localeCompare(value) == 0) {
console.log(players[1][1]);
//Add points to respective player field
//PROBABLY NOT HOW TO CHANGE A DOCUMENT FILED, THIS DOESNT WORK..
players[1][1]++;
}
});
//EACH TIME THIS RUNS IT SAYS: "Cannot modify a WriteBatch that has been committed"
batch.update(user, {'ownedplayers': json[map]});
}
}
});
}
});
}
} else {
console.log('user does not have a playermode document.');
}
});
});
return batch.commit().then(function () {
console.log("Succesfully commited changes.");
return null;
});
});
});
}).catch((err) => {
return err;
});
});
The issues i get in the console are "Cannot modify a WriteBatch that has been committed." and I fail to modify and add points to the player field inside the users document.
This is the console:
This is the firestore document structure:
I'm completely stuck on this.. Feels like I've tried all different approaches, but I think i dont fully understand cloud functions and javascript, so i would gladly recive feedback and help on how to make this work.
Cheers,
Finally.... i managed to update the document successfully. I put the commit outside another ".then()". Thought I tried that, but yay I guess :P
}).then(() => {
return batch.commit().then(function () {
console.log("Succesfully commited changes.");
return null;
});
The problem now is that it commits every loop. I think the most optimal here would be to batch update ALL users before committing?
And again, is there a more optimal way to do this, in terms of minimizing the operation and impact? I'm afraid I go too deep with for loops instead of directly navigating to the document, but haven't found an easier way to do that.
Any thoughts?
Using the Google Geocoder v3, if I try to geocode 20 addresses, I get an OVER_QUERY_LIMIT unless I time them to be ~1 second apart, but then it takes 20 seconds before my markers are all placed.
Is there any other way to do it, other than storing the coordinates in advance?
No, there is not really any other way : if you have many locations and want to display them on a map, the best solution is to :
fetch the latitude+longitude, using the geocoder, when a location is created
store those in your database, alongside the address
and use those stored latitude+longitude when you want to display the map.
This is, of course, considering that you have a lot less creation/modification of locations than you have consultations of locations.
Yes, it means you'll have to do a bit more work when saving the locations -- but it also means :
You'll be able to search by geographical coordinates
i.e. "I want a list of points that are near where I'm now"
Displaying the map will be a lot faster
Even with more than 20 locations on it
Oh, and, also (last but not least) : this will work ;-)
You will less likely hit the limit of X geocoder calls in N seconds.
And you will less likely hit the limit of Y geocoder calls per day.
You actually do not have to wait a full second for each request. I found that if I wait 200 miliseconds between each request I am able to avoid the OVER_QUERY_LIMIT response and the user experience is passable. With this solution you can load 20 items in 4 seconds.
$(items).each(function(i, item){
setTimeout(function(){
geoLocate("my address", function(myLatlng){
...
});
}, 200 * i);
}
Unfortunately this is a restriction of the Google maps service.
I am currently working on an application using the geocoding feature, and I'm saving each unique address on a per-user basis. I generate the address information (city, street, state, etc) based on the information returned by Google maps, and then save the lat/long information in the database as well. This prevents you from having to re-code things, and gives you nicely formatted addresses.
Another reason you want to do this is because there is a daily limit on the number of addresses that can be geocoded from a particular IP address. You don't want your application to fail for a person for that reason.
I'm facing the same problem trying to geocode 140 addresses.
My workaround was adding usleep(100000) for each loop of next geocoding request. If status of the request is OVER_QUERY_LIMIT, the usleep is increased by 50000 and request is repeated, and so on.
And of cause all received data (lat/long) are stored in XML file not to run request every time the page is loading.
EDIT:
Forgot to say that this solution is in pure js, the only thing you need is a browser that supports promises https://developer.mozilla.org/it/docs/Web/JavaScript/Reference/Global_Objects/Promise
For those who still needs to accomplish such, I've written my own solution that combines promises with timeouts.
Code:
/*
class: Geolocalizer
- Handles location triangulation and calculations.
-- Returns various prototypes to fetch position from strings or coords or dragons or whatever.
*/
var Geolocalizer = function () {
this.queue = []; // queue handler..
this.resolved = [];
this.geolocalizer = new google.maps.Geocoder();
};
Geolocalizer.prototype = {
/*
#fn: Localize
#scope: resolve single or multiple queued requests.
#params: <array> needles
#returns: <deferred> object
*/
Localize: function ( needles ) {
var that = this;
// Enqueue the needles.
for ( var i = 0; i < needles.length; i++ ) {
this.queue.push(needles[i]);
}
// return a promise and resolve it after every element have been fetched (either with success or failure), then reset the queue.
return new Promise (
function (resolve, reject) {
that.resolveQueueElements().then(function(resolved){
resolve(resolved);
that.queue = [];
that.resolved = [];
});
}
);
},
/*
#fn: resolveQueueElements
#scope: resolve queue elements.
#returns: <deferred> object (promise)
*/
resolveQueueElements: function (callback) {
var that = this;
return new Promise(
function(resolve, reject) {
// Loop the queue and resolve each element.
// Prevent QUERY_LIMIT by delaying actions by one second.
(function loopWithDelay(such, queue, i){
console.log("Attempting the resolution of " +queue[i-1]);
setTimeout(function(){
such.find(queue[i-1], function(res){
such.resolved.push(res);
});
if (--i) {
loopWithDelay(such,queue,i);
}
}, 1000);
})(that, that.queue, that.queue.length);
// Check every second if the queue has been cleared.
var it = setInterval(function(){
if (that.queue.length == that.resolved.length) {
resolve(that.resolved);
clearInterval(it);
}
}, 1000);
}
);
},
/*
#fn: find
#scope: resolve an address from string
#params: <string> s, <fn> Callback
*/
find: function (s, callback) {
this.geolocalizer.geocode({
"address": s
}, function(res, status){
if (status == google.maps.GeocoderStatus.OK) {
var r = {
originalString: s,
lat: res[0].geometry.location.lat(),
lng: res[0].geometry.location.lng()
};
callback(r);
}
else {
callback(undefined);
console.log(status);
console.log("could not locate " + s);
}
});
}
};
Please note that it's just a part of a bigger library I wrote to handle google maps stuff, hence comments may be confusing.
Usage is quite simple, the approach, however, is slightly different: instead of looping and resolving one address at a time, you will need to pass an array of addresses to the class and it will handle the search by itself, returning a promise which, when resolved, returns an array containing all the resolved (and unresolved) address.
Example:
var myAmazingGeo = new Geolocalizer();
var locations = ["Italy","California","Dragons are thugs...","China","Georgia"];
myAmazingGeo.Localize(locations).then(function(res){
console.log(res);
});
Console output:
Attempting the resolution of Georgia
Attempting the resolution of China
Attempting the resolution of Dragons are thugs...
Attempting the resolution of California
ZERO_RESULTS
could not locate Dragons are thugs...
Attempting the resolution of Italy
Object returned:
The whole magic happens here:
(function loopWithDelay(such, queue, i){
console.log("Attempting the resolution of " +queue[i-1]);
setTimeout(function(){
such.find(queue[i-1], function(res){
such.resolved.push(res);
});
if (--i) {
loopWithDelay(such,queue,i);
}
}, 750);
})(that, that.queue, that.queue.length);
Basically, it loops every item with a delay of 750 milliseconds between each of them, hence every 750 milliseconds an address is controlled.
I've made some further testings and I've found out that even at 700 milliseconds I was sometimes getting the QUERY_LIMIT error, while with 750 I haven't had any issue at all.
In any case, feel free to edit the 750 above if you feel you are safe by handling a lower delay.
Hope this helps someone in the near future ;)
I have just tested Google Geocoder and got the same problem as you have.
I noticed I only get the OVER_QUERY_LIMIT status once every 12 requests
So I wait for 1 second (that's the minimum delay to wait)
It slows down the application but less than waiting 1 second every request
info = getInfos(getLatLng(code)); //In here I call Google API
record(code, info);
generated++;
if(generated%interval == 0) {
holdOn(delay); // Every x requests, I sleep for 1 second
}
With the basic holdOn method :
private void holdOn(long delay) {
try {
Thread.sleep(delay);
} catch (InterruptedException ex) {
// ignore
}
}
Hope it helps
This worked well for me, after intermittent trial and error over the past couple days. I am using react instant-search-hooks via Algolia with Nextjs and Sanity for a new jobs site for a large company.
Postal Code is a facet for filtering/sorting/query matching that is defined in the algolia index. In another script file, I map out all of these facets (postal code, city, etc); Now that I have 100 returned files they can be mapped out by iterating through a mapped asynchronous import and the lat/lng coords matched to the corresponding zip codes defining a job posting (there are ~2500 postings but only ~100 zip codes to narrow down the coordinates of)
import * as dotenv from "dotenv";
dotenv.config();
import {
googleNetwork,
axiosConfig as googleAxiosConfig
} from "../utils/google-axios";
import JSONData from "../../public/data/postalCode/2022/05/26.json";
import fs from "fs";
import { join } from "path";
import type { GeneratedGeolocData } from "../types/algolia";
import { timezoneHelper } from "../utils/timezone-helper";
import { Unenumerate } from "../types/helpers";
let i = 0;
i < JSONData.postalCodes.facetHits.length;
i++;
const getGeoCode = (
record: Unenumerate<typeof JSONData.postalCodes.facetHits>
) =>
function () {
return JSONData.postalCodes.facetHits.map(async (data = record, u) => {
const googleBase = process.env.NEXT_PUBLIC_GOOGLE_MAPS_BASE_PATH ?? "";
const googleApiKey =
process.env.NEXT_PUBLIC_TAKEDA_JOBS_GOOGLE_SERVICES ?? "";
const params: (string | undefined)[][] = [
["address", data.value],
["key", googleApiKey]
];
const query = params
.reduce<string[]>((arr, [k, v]) => {
if (v) arr.push(`${k}=${encodeURIComponent(v)}`);
return arr;
}, [])
.join("&");
return await googleNetwork("GET")
.get(`${googleBase}geocode/json?${query}`, googleAxiosConfig)
.then(dat => {
const geoloc = dat.data as GeneratedGeolocData;
const {
[0]: Year,
[2]: Month,
[4]: Day
} = new Date(Date.now())
.toISOString()
.split(/(T)/)[0]
.split(/([-])/g);
const localizedTimestamp = timezoneHelper({
dateField: new Date(Date.now()),
timezone: "America/Chicago"
});
return setTimeout(
() =>
fs.appendFileSync(
join(
process.cwd(),
`public/data/geoloc/${Year}/${Month}/${Day}-${[i]}.json`
),
JSON.stringify(
{
generated: localizedTimestamp,
_geoloc: {
postalCode: data.value,
geolocation: geoloc
}
},
null,
2
)
),
1000
);
});
});
};
getGeoCode(JSONData.postalCodes.facetHits[i]);
It took a lot less time than anticipated -- under 4 seconds for 100 unique results to generate
Context on the Unenumerate type -- Unenumerate strips the internal repeating unit within an array:
type Unenumerate<T> = T extends Array<infer U> ? U : T;
This Meteor server code needs to send email only if the category property has not been populated in myCollection after 2 minutes.
But it is sending the email every time regardless. Any suggestions so that it only send email if the category is still blank after 2 minutes? thx
let myFunc = function (){
myCollection.inert({lastName: 'john'});
//run other code to populate the myCollection.category
}
myCollection.before.insert(function (userId, doc) {
if (userId === '8mmjdueej') {
lib.alertAdmin(doc);
}
});
'alertAdmin': function (Obj) {
Meteor.setTimeout(function () {
let category = myCollection.find({lastName: Obj.name}).bigCategory;
if (!category) {
lib.sendEmail(null, 'failed to get category for: ' + Obj.name);
}
},
120000);
},
let category = myCollection.find({lastName: Obj.name}).bigCategory;
here could be the problem. A find returns the Mongo's cursor, not the document itself. You can use findOne for this.
I've been battling this test most of the afternoon, I've commented out what I tried to do but it wouldn't work. I have the ability the search for users within the company, a table is then generated and I want to select the one that matches the name of the user. Currently I just use td.text-center as it is the only td in the table, but this seems really poor and prone to breaking if other users with the same name are in the system.
Attempt 2 seems like it would be the best solution but it always gives the element not found error. Has anyone found a reliable way of locating TD within a table? The method below works in other tables, just not this one....
this.editUser = function(name) {
// Original Code Block
//browser.actions().mouseMove(element(by.css('td.text-center'))).perform();
//var editUserBtn = element(by.css('.user-action-open'));
//editUserBtn.click().then(function() {
// var editDetails = element(by.cssContainingText('li > a', 'Edit Details'));
// browser.wait(EC.visibilityOf(editDetails), 3000);
// editDetails.click();
//});
// Attempt 2
// browser.actions().mouseMove(element(by.css('td.text-center'))).perform();
//browser.sleep(3000);
//var edit = element.all(by.repeater('user in users')).filter(function(rowElement){
// return rowElement.element.all(by.css('td[ng-bind="user.Firstname"]')).getText().then(function(text){
// return text.trim() == name;
// });
//}).first();
//browser.actions().mouseMove(edit).perform();
// Currently just times out and doesn't find the element.
var editUserButton = $('.user-action-open');
browser.wait(EC.visibilityOf(editUserButton), 20000).then(function() {
editUserButton.click().then(function() {
var editDetails = element(by.cssContainingText('li > a', 'Edit Details'));
browser.wait(EC.visibilityOf(editDetails), 3000);
editDetails.click();
});
});
}
You could use ExpectedConditions to do some waiting instead of arbitrarily sleeping for 3 seconds. You could instead do a browser wait. This will wait up to the timeout if the object is in a desired state.
let users = element.all(by.repeater('user in users');
// the original post shows that this is an element.all
// this would only be true if the user can have multiple first names
let firstName = element(by.css('td[ng-bind="user.Firstname"]'));
browser.actions().mouseMove(element(by.css('td.text-center'))).perform();
// wait to see if the repeater is present or timeout in 3 seconds
browser.wait(users).isPresent(), 3000);
let edit = element.all(users).filter((rowElement) => {
// wait for first name, get the first name of the row's element
// if the text matches the name, resolve the promise true.
return browser.wait(rowElement.firstName, 3000).then(() => {
return rowElement.firstName.getText().then((text) => {
return text.trim() == name;
});
});
}).first();
browser.actions().mouseMove(edit).perform();
The rowElement.firstName might not be the correct syntax.
Chaining the promises
// wait to see if the repeater is present or timeout in 3 seconds
browser.wait(users).isPresent(), 3000).then(() => {
let edit = element.all(users).filter((rowElement) => {
// wait for first name, get the first name of the row's element
// if the text matches the name, resolve the promise true.
return browser.wait(rowElement.firstName, 3000).then(() => {
return rowElement.firstName.getText().then((text) => {
return text.trim() == name;
});
});
}).first();
browser.actions().mouseMove(edit).perform();
}).catch(err => {
// we could also have asserted the length was at least 1
// maybe we want to catch on something else? maybe this isn't a
// fail case so maybe do not use `fail(`
fail("did not find users");
});
So I am working on an API using WebdriverIO to automate some stuff for sites that have no API support. Currently facing an issue with trying to dynamically scrape x number of pages. The scenario is only say 20 items are listed on each page and I can click next to jump to the next, scrape it and repeat until there are no more pages.
The issue; There are not a specific number of pages, at one time it could be 2 and another it could be 12 so I need to do this dynamically.
My problem (I think); Judging from console output I believe the selenium session is being killed before the recursive scraping is finished and that's because it's proceeding to the execute I have there to test things.
My question; How can I recursively scrape those pages without the script continuing and terminating the session before I am finished?
Here is a slightly redacted version of what I have so far. Thanks!
...webdriver doing stuff that works...
.then(function() {
console.log('Entering the initial THIS...');
function recursive() {
console.log('This is inside the RECURSIVE Function!');
client
.execute(function() {
return document.title;
})
.then(function(ret) {
console.log('The thing I asked it for is: ' + ret.value);
})
.isExisting('#lastPage').then(function(lastPage) {
console.log(lastPage);
if (!lastPage) {
console.log('Its not the last page!');
client
.click('//a[text()="Next"]')
.saveScreenshot('/src/api/debug_images/05.png')
recursive();
}
})
}
recursive();
})
.execute(function() {
var table = document.getElementById("transactionHistoryTable");
var result = [];
for (var i = 1, row; row = table.rows[i]; i++) {
result.push({
date: row.cells[0].innerText,
details: row.cells[1].innerText,
debt: row.cells[2].innerText,
credit: row.cells[3].innerText,
balance: row.cells[4].innerText
});
}
return result;
})
.then(function(ret) {
return ret.value;
})
.then(function(ledger) {
console.log('This should not run yet?');
res.json({
status: 'success',
request: { bsb: bsb, account: account, period: period },
ledger: ledger.length
});
})
.end();