Develop Reference

Is there a way to select only certain fields from Firestore?

I am working on a performance issue of a function, that takes 15sec to response, which makes a request to firebase for all documents that are
"ErrorID" "==" "0"
The problem is that there are many documents and they are kind of very large objects, and I only need TWO FIELDS (Order and Amount) of each document, there are any way to request only those two fields that accomplish the condition?
Something like :
firestore.collection("purchases").where("ErrorID", "==", "0").get(Order, Amount);
The function that im talking about:
const totalEarn = async (req, res, next) => {
const DAY = 86400000;
const WEEK = 604800016;
const MONTH = 2629800000;
try {
let snap = await firestore.collection("purchases").where("ErrorID", "==", "0").get(); // CONDITION
let totalToday = 0;
let totalYesterday = 0;
let totalLastWeek = 0;
let totalLastMonth = 0;
let now = Date.now();
let Yesterday = now - 86400000;
await snap.forEach((doc) => { // THIS FOR EACH TAKES TOO MUCH TIME
let info = doc.data();
let time = info.Order.split("-")[2]; // FIRESTORE FIELD -> ORDER
let amount = info.AmountEur * 1; // FIRESTORE FIELD -> AMOUNT
if (time > now - DAY) {
totalToday = totalToday + amount;
}
if (time < now - DAY && time > Yesterday - DAY) {
totalYesterday = totalYesterday + amount;
}
if (time > now - WEEK) {
totalLastWeek = totalLastWeek + amount;
}
if (time > now - MONTH) {
totalLastMonth = totalLastMonth + amount;
}
});
res.send({
status: true,
data: {
totalToday: totalToday.toFixed(2),
totalYesterday: totalYesterday.toFixed(2),
totalLastWeek: totalLastWeek.toFixed(2),
totalLastMonth: totalLastMonth.toFixed(2),
},
});
} catch (error) {
res.status(410).send({
status: false,
error: "some error occured counting the numbers",
e: error.message,
});
}
};
The document im talking about

If you use Firestore Node.JS client or Firebase Admin SDK, then you can use select() to select fields:
import { Firestore } from "#google-cloud/firestore";
const firestore = new Firestore();
const snap = firestore
.collection("purchases")
.where("ErrorID", "==", "0")
.select("Order", "Amount")
.get();

Value Undefined: Loops in async await with Promise.race?

I am working on a project where I need to brute force PDF password.
For that PDF.js is used for verifying password and promise.race to run parallel functions to make the overall work fast.
and this is how i implemented it:
var sfile = "KRIS.pdf"
var dBuf = fs.readFileSync(sfile);
const tryCases = getCombos(alphaArray, 4); // permutation having total length 456976
var finishFunction = false;
async function getPass(startAt = Number(), endAt = Number()) {
var isDone = false;
var currentPass = ''
for (let index = startAt; index < endAt; index++) {
if (finishFunctions) { return; }
currentPass = tryCases[index].join("");
await pdfjsLib.getDocument({ data: dBuf, password: currentPass }).promise
.then((r) => {
console.log('Worked: ' + currentPass);
isDone = true;
pdfjsLib.distroy();
return new Promise.resolve();
})
.catch((e) => { })
if (isDone) {
finishFunctions = true;
return currentPass;
}
}
console.log(`Hey Nothing From ${startAt} - ${endAt}`);
}
console.time('Found ');
Promise.race([
getPass(0, 100000),
getPass(100000, 200000),
getPass(200000, 300000),
getPass(300000, 400000),
getPass(400000, 456976)
])
.then((s) => { console.timeEnd('Found '); console.log('HeyThen ' + s) })
.catch((e) => console.log('Hey Error ' + e));
now it works to get the 4 letter password but There are problems preventing it from being complete.
First Current Function is very very slow, it takes forever even after running parallel functions.
Second I added a flag to stop other parallel functions but it does not work as expected with 4 Letter Forcing.
Third Resource usage is really high. My Linux system stops responding.
for visualizing the time and flag issue I used 3 letters forcing and here is log of it:
Worked: KRIS
Found: 13950.743ms
HeyThen KRIS
Hey Nothing From 4934 - 8788
Hey Nothing From 0 - 4394
Hey Nothing From 13182 - 17576
(node:3068) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

By calling return new Promise.resolve(currentPass); you return a result to then and not from ATryCase. Write the result of await myFunc.getDocument to some variable and return it
const res = await myFunc.getDocument({ data: dBuf, pos: currentPass })
.promise
.then(() => {
console.log('Data is: ' + currentPass);
console.timeEnd('GotData ');
return new Promise.resolve(currentPass);
}).catch(e => { });
return res;

if the currentPass is correct, i need to break the loop and return the value
None of your code snippets did that so far. You will want to use something like
async function ATryCase(indexStart, indexEnd) {
for (let index = indexStart; index < indexEnd; index++) {
var currentPass = startLetters + index;
try {
await myFunc.getDocument({ data: dBuf, pos: currentPass }).promise;
console.log('Data is: ' + currentPass);
return currentPass; // this breaks the loop and `return`s from ATryCase
} catch {
continue;
}
}
throw new Error("No pass did find a result");
}
for that.
Alternatively, have a look at the proposed Promise.any, which would do exactly what you are looking for, but concurrently. (You will also want to use it in place of Promise.race if you go for 4 concurrent sequential searches). You could use it like
function ATryCase(indexStart, indexEnd) {
const promises = [];
for (let index = indexStart; index < indexEnd; index++) {
var currentPass = startLetters + index;
promises.push(myFunc.getDocument({ data: dBuf, pos: currentPass }).promise.then(() => {
console.log('Data is: ' + currentPass);
return currentPass;
}));
}
return Promise.any(promises);
}

How to use multiple promises in recursion?

I am trying to solve the problem where the script enters a website, takes the first 10 links from it and then goes on those 10 links and then goes on to the next 10 links found on each of these 10 previous pages. Until the number of visited pages will be 1000.
This is what it looks like:
I was trying to get this by using for loop inside promise and recursion, this is my code:
const rp = require('request-promise');
const url = 'http://somewebsite.com/';
const websites = []
const promises = []
const getOnSite = (url, count = 0) => {
console.log(count, websites.length)
promises.push(new Promise((resolve, reject) => {
rp(url)
.then(async function (html) {
let links = html.match(/https?:\/\/(www\.)?[-a-zA-Z0-9#:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()#:%_\+.~#?&//=]*)/g)
if (links !== null) {
links = links.splice(0, 10)
}
websites.push({
url,
links,
emails: emails === null ? [] : emails
})
if (links !== null) {
for (let i = 0; i < links.length; i++) {
if (count < 3) {
resolve(getOnSite(links[i], count + 1))
} else {
resolve()
}
}
} else {
resolve()
}
}).catch(err => {
resolve()
})
}))
}
getOnSite(url)

I think you might want a recursive function that takes three arguments:
an array of urls to extract links from
an array of the accumulated links
a limit for when to stop crawling
You'd kick it off by calling it with just the root url, and await all of the returned promises:
const allLinks = await Promise.all(crawl([rootUrl]));
On the initial call the second and third arguments could assume default values:
async function crawl (urls, accumulated = [], limit = 1000) {
...
}
The function would fetch each url, extract its links, and recurse until it hit the limit. I haven't tested any of this, but I'm thinking something along these lines:
// limit the number of links per page to 10
const perPageLimit = 10;
async function crawl (urls, accumulated = [], limit = 1000) {
// if limit has been depleted or if we don't have any urls,
// return the accumulated result
if (limit === 0 || urls.length === 0) {
return accumulated;
}
// process this set of links
const links = await Promise.all(
urls
.splice(0, perPageLimit) // limit to 10
.map(url => fetchHtml(url) // fetch the url
.then(extractUrls)); // and extract its links
);
// then recurse
return crawl(
links, // newly extracted array of links from this call
[...accumulated, links], // pushed onto the accumulated list
limit - links.length // reduce the limit and recurse
);
}
async fetchHtml (url) {
//
}
const extractUrls = (html) => html.match( ... )

Repeat sending the request to another server until the expected result is achieved

I'm requesting to server "S" to get some data, but this data may not be ready.
When the data is not yet ready, server S responds with {"data":null,"state": "pending"} but when the data has been prepared the response will be something like {"data": {...somedata}, "state": "done"}.
I have to repeat the request until the data is ready. What I'm doing now is something like this:
let wait = function* () {
let t = 500;
for (let j = 1; j < 10; j++) {
yield new Promise((resolve) => {
setTimeout(() => resolve(), t*=2);
});
}
}();
let result = await sendRequestToS();
status = result;
for (let i = 0; i < 4 && result.state==='pending'; i++) {
await wait.next().value;
result = await sendRequestToS();
}
As you can see, I send the request up to 4 times with a delay of 1, 2, 4 and 8 seconds.
Am I doing this the right way?
Isn't that (using setTimeout to delay between requests) a bad practice?

I'd write this as such:
function wait(ms) {
return new Promise(res => setTimeout(res, ms));
}
async function requestAndRetry() {
let retries = 10;
let timeout = 1000;
while(retries>0) {
const response = await sendRequestToS();
if (result?.state === 'done') {
return result;
}
await wait(timeout);
retries--;
timeout*=2;
}
throw new Error('Request failed after 10 retries');
}
I don't think it's a bad idea. It's called exponential back-off and you're not blocking the script engine.

Instead of using generators directly, you could simply do this using async/await and recursion. Here's an example which tries to get the response a limited number of times in order to prevent an endless recursion and with a timeout between retries:
async function wait(timeInMs) {
console.log('Waiting ...');
return new Promise((resolve => setTimeout(() => resolve(), timeInMs)));
}
async function tryRequestToS(numberOfTries, timeout) {
if (numberOfTries <= 0) {
throw new Error("could net get result");
}
const result = await sendRequestToS();
if (result && result.state === "done") {
return result;
}
await wait(timeout); // wait for the defined timeout before recurring
return tryRequestToS(numberOfTries - 1, timeout);
}
(async () => {
try {
const result = await tryRequestToS(10, 500); // max. 10 retries, 500 ms delay between retries
console.log(result);
} catch(err) {
console.log(err);
}
})();

When babel processes async / await code does it bundle unrelated calls?

Is babels async / await code smart enough to see the code below:
async function alpha () {
let resultOne = await processNumber(5)
let resultTwo = await processNumber(5 + 8)
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
As something like the following, where the first two promises within this function can happen together because the values needed to perform those operations does not need to wait for anything.
import Promise from 'bluebird'
async function beta () {
let {resultOne, resultTwo} = await Promise.props({
resultOne: processNumber(5),
resultTwo: processNumber(5 + 8)
})
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
I would understand the alpha function as waiting for each async function call before it moves on to the next, where in beta resultOne and resultTwo are happening simultaneously, this is only possible because they don't need to wait on any other calls. I was wondering if this was truly the case or if babel does something behind the scenes to bundle these together.

I setup a benchmark between the two and it seems it does not account for this on it's own.
Here's the test:
import Promise from 'bluebird'
async function processNumber (int) {
await Promise.delay(500)
return {number: int + 3}
}
async function alpha () {
let resultOne = await processNumber(5)
let resultTwo = await processNumber(5 + 8)
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
async function beta () {
let {resultOne, resultTwo} = await Promise.props({
resultOne: processNumber(5),
resultTwo: processNumber(5 + 8)
})
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
async function main () {
const TEST_ALPHA = 'test alpha'
const TEST_BETA = 'test beta'
console.time(TEST_ALPHA)
let resultAlpha = await alpha()
console.log(resultAlpha)
console.timeEnd(TEST_ALPHA)
console.time(TEST_BETA)
let resultBeta = await beta()
console.log(resultBeta)
console.timeEnd(TEST_BETA)
return true
}
main()
.then(console.log)
.catch(console.error)
Here's the results:
thomasreggi#zx:PAS-api$ babel-node test.js
{ number: 22 }
test alpha: 2025ms
{ number: 22 }
test beta: 1508ms
true
thomasreggi#zx:PAS-api$ babel-node test.js
{ number: 22 }
test alpha: 2033ms
{ number: 22 }
test beta: 1511ms
true

In JS it is nearly impossible to make any strict statements whether a given expression is "unrelated" to another arbitrary expression (especially statically).
It happens because due to its highly-dynamic nature almost every expression may cause hidden (or not so hidden) side effects that may break the expected flow of the program.
For your very code it is really easy to break the code in case if both "unrelated" calls are triggered "simultaneously":
let isFirst = true;
async function processNumber(v) {
await Promise.delay(2000 - v * 100);
if (v < 10) {
if (!isFirst) {
throw new Error();
}
}
isFirst = false;
return { number: v + 3 };
}
This would work for alpha, but throw for beta.
If you know it would be fine and want to run them "in parallel" just use awaits correspondingly:
async function alpha () {
let one = processNumber(5)
let two = processNumber(5 + 8)
const resultOne = await one;
let resultThree = await processNumber(resultOne.number)
let resultFour = await processNumber(resultOne.number + resultThree.number)
return resultFour
}
Also note, that the resultTwo is not used anywhere in your code.