I have spent 6 hours already trying to figure out how to do this in NodeJS.
I am using NodeJS with Express and MongoDB.
I have a database that has two collections viz. Listings and Categories. Each listing has a "category" which is an ID that maps to a category inside the Categories collection.
What I want to do is, fetch all the listings, then loop through all of them, and get the category title from the Categories collection using the category id.
This is what I had for getting all the listings:
const listings_raw = [];
await db.collection("listings").find().forEach(listing => listings_raw.push(listing));
The code above works fine. The trouble I am having is with this:
const listings = [];
listings_raw.forEach(async listing => {
const category_id = listing.category;
const category_title = await db.collection('categories').findOne({_id: objectId(category_id)});
listing.category_title = category_title;
listings.push(listing);
});
response.send(listings);
The response.send(listings); gets executed before the listings_raw.forEach completes.
I want to send the data only after all listings have been populated with the category titles.
After spending 6 hours, I have come up with the following hack which I am sure is nasty!
const listings_raw = [];
const em = new events.EventEmitter();
await db.collection("listings").find().forEach(listing => listings_raw.push(listing));
const listings = [];
let counter = 0;
listings_raw.forEach(async (listing) => {
const category_id = listing.category;
const category = await db.collection('categories').findOne({_id: objectId(category_id)});
listing.category_title = category.title;
listings.push(listing);
if (counter === listings_raw.length - 1) {
em.emit('listings:processing:done');
}
counter++;
});
em.on('listings:processing:done', () => {
response.send(listings);
});
Please, can someone explain or guide me on how this should be done in JavaScript?
Basically, I am not able to figure out how to know if all promises have been resolved or not.
Thank you!
The listings_raw.forEach function executes synchronously on the array, even though you are then performing an asynchronous operation within that.
Promise.all will allow you to await for the result of an array of promises. Therefore you can .map the listings to an array of promises which return the updated listing.
const listings = await Promise.all(listings_raw.map(async listing => {
const category_id = listing.category;
const category_title = await db.collection('categories').findOne({_id: dependencies.objectId(category_id)});
listing.category_title = category_title;
return listing;
});
response.send(listings);
Related
So I've been working on my project that involves scraper.
So workflow is next: There are two scrapers right now. Data is being parsed and pushed into array for each individual scraper and passed to the merge component.
So the merge component looks like this:
let mergedApartments = []; //Creating merged list of apartments
exports.mergeData = (apartments) => {
//Fetching all apartments that are passed from scraper(s)
mergedApartments.push(...apartments); //Pushing apartments into the list
console.log(mergedApartments.length);
};
So right now output of mergedApartments.length is 9 39. So the first function that calls mergeData() and pass it an array have 9 objects inside it, and the other scraper have 30 objects inside it's array, who is again passed to the mergeData.
Now this is not what I've expected. I've expected one array with all merged objects from the scrapers. Right now, scraperno1 send apartments and it's added to the mergedApartments, then scraperno2 sends apartments and it's overwriting that array by adding new apartments objects into the array.
Now I want different output: I just want to get one list with all merged objects from the arrays. Because this data will be passed to the storing component, and I don't want to query DB multiple times, because for each new mergedApartments list, data will be inserted and creating duplicate values - throwing an error.
So what I've tried: I've tried creating some kind of a counter which counts number of time that function mergeData is called, and then do the logic about merging but no success.
So I just want my array to have one output of mergedApartments.length - in this case 39.
Thanks!
EDIT
Here how one of the scraper looks:
const merge = require('../data-functions/mergeData');
const axios = require('axios');
const cheerio = require('cheerio');
//function for olx.ba scraper. Fetching raw html data and pushing it into array of objects. Passing data to merge function
exports.santScraper = (count) => {
const url = `https://www.sant.ba/pretraga/prodaja-1/tip-2/cijena_min-20000/stranica-${count}`;
const santScrapedData = [];
const getRawData = async () => {
try {
await axios.get(url).then((response) => {
const $ = cheerio.load(response.data);
$('div[class="col-xxs-12 col-xss-6 col-xs-6 col-sm-6 col-lg-4"]').each(
(index, element) => {
const getLink = $(element).find('a[class="re-image"]').attr('href');
const getDescription = $(element).find('a[class="title"]').text();
const getPrice = $(element)
.find('div[class="prices"] > h3[class="price"]')
.text()
.replace(/\.| ?KM$/g, '')
.replace(',', '.');
const getPicture = $(element).find('img').attr('data-original');
const getSquaremeters = $(element)
.find('span[class="infoCount"]')
.first()
.text()
.replace(',', '.')
.split('m')[0];
const pricepersquaremeter =
parseFloat(getPrice) / parseFloat(getSquaremeters);
santScrapedData[index] = {
id: getLink.substring(42, 46),
link: getLink,
descr: getDescription,
price: Math.round(getPrice),
pictures: getPicture,
sqm: Math.round(getSquaremeters),
ppm2: Math.round(pricepersquaremeter),
};
}
);
merge.mergeData(santScrapedData); //here i'm calling function and passing array to function
});
} catch (error) {
console.log(error);
}
};
getRawData();
};
Other scraper looks the same(it's same calling of the function)
For this, you need to use concat function from the Array prototype
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat
exports.mergeData = (apartments) => {
mergedApartment = mergedApartments.concat(apartments);
};
exports.sendData = () => {
console.log(mergedApartment.length);
}
and in your main script
getRawData().then(merge.sendData);
I want to do more optimal my code. I summeries to following code.I have a data access layer named
ICDataAccess.I making that get all inventorys firts line.Then I get locations of inventory response and I want to get all barcodes with locations information from ServiceCaller data access.Then this barcodes adds to each inventory of inventorys array.How can i do more optimal.Is there a logic error. Help me please...
const ic = await this.ICDataAccess.getActiveIC({ warehouse, locations, status: statuses }); // An object array
await Promise.all(ic.map(async (item) => {
const inventory = item;
const response = await ServiceCaller.filterStore({ ids: inventory.locations, fields: 'barcode' }); // An object array
inventory.barcodes = response.map(res =>res.barcode);
}));
I am stuck in what I thought was a very simple use case: I have a list of client ids in an array. All I want to do is fetch all those clients and "watch" them (using the .onSnapshot).
To fetch the client objects, it is nice and simple, I simply go through the array and get each client by their id. The code looks something like this:
const accessibleClients = ['client1', 'client2', 'client3']
const clients = await Promise.all(
accessibleClients.map(async clientId => {
return db
.collection('clients')
.doc(clientId)
.get()
})
)
If I just needed the list of clients, it would be fine, but I need to perform the .onSnapshot on it to see changes of the clients I am displaying. Is this possible to do? How can I get around this issue?
I am working with AngularFire so it is a bit different. But i also had the problem that i need to listen to unrelated documents which can not be queried.
I solved this with an object which contains all the snapshot listeners. This allows you to unsubscribe from individual client snapshots or from all snapshot if you do not need it anymore.
const accessibleClients = ['client1', 'client2', 'client3'];
const clientSnapshotObject = {};
const clientDataArray = [];
accessibleClients.forEach(clientId => {
clientSnapshotArray[clientId] = {
db.collection('clients').doc(clientId).onSnapshot(doc => {
const client = clientDataArray.find(client => doc.id === client.clientId);
if (client) {
const index = clientDataArray.findIndex(client => doc.id === client.clientId);
clientDataArray.splice(index, 1 , doc.data())
} else {
clientDataArray.push(doc.data());
}
})
};
})
With the clientIds of the accessibleClients array, i create an object of DocumentSnapshots with the clientId as property key.
The snapshot callback function pushes the specific client data into the clientDataArray. If a snapshot changes the callback function replaces the old data with the new data.
I do not know your exact data model but i hope this code helps with your problem.
I have been doing this for an hour. I simply want to get the number of children in the child "Success" in the database below. The answers in similar stackoverflow questions are not working. I am new in Javascript Programming.
So far I have tried this
var children = firebase.database().ref('Success/').onWrite(event => {
return event.data.ref.parent.once("value", (snapshot) => {
const count = snapshot.numChildren();
console.log(count);
})
})
and also this
var children = firebase.database().ref('Success/').onWrite(event => {
return event.data.ref.parent.once("value", (snapshot) => {
const count = snapshot.numChildren();
console.log(count);
})
})
Where might I be going wrong.
As explained in the doc, you have to use the numChildren() method, as follows:
var ref = firebase.database().ref("Success");
ref.once("value")
.then(function(snapshot) {
console.log(snapshot.numChildren());
});
If you want to use this method in a Cloud Function, you can do as follows:
exports.children = functions.database
.ref('/Success')
.onWrite((change, context) => {
console.log(change.after.numChildren());
return null;
});
Note that:
The new syntax for Cloud Functions version > 1.0 is used, see https://firebase.google.com/docs/functions/beta-v1-diff?authuser=0
You should not forget to return a promise or a value to indicate to the platform that the Cloud Function execution is completed (for more details on this point, you may watch the 3 videos about "JavaScript Promises" from the Firebase video series: https://firebase.google.com/docs/functions/video-series/).
const db = getDatabase(app)
const questionsRef = ref(db, 'questions')
const mathematicalLiteracy = child(questionsRef, 'mathematicalLiteracy')
onValue(mathematicalLiteracy, (snapshot) => {
const data = snapshot.val()
const lenML = data.length - 1
console.log(lenML)
})
This method worked for me. I wanted to get the children's count of the mathematicalLiteracy node in my database tree. If I get its value using .val() it returns an array that contains that node's children and an extra empty item. So, I subtracted that one empty item's count. Finally, I get my needed children's count.
I had a function which successfully queried a child node called /mobile_user and ranked the top 10 users based on the value on "earned_points" and also wrote those top 10 users to a new node called /leaderboard which is on the same level as /mobile_user
I switched to firestore because i want to use the powerful queries and i;m unable to change the syntax from firebase rtdb to firestore
My function looks like this
exports.leaderboardUpdate2 = functions.https.onRequest((req, res) =>{
const updates = [];
const leaderboard = {};
const ref = admin.firestore().collection('mobile_user');
const leaderboardRef = admin.firestore().collection('/leaderboard');
return ref.orderBy("earned_points").limit(10).get().then(dataSnapshot => {
let i = 0;
dataSnapshot.forEach(function(childSnapshot) {
const r = dataSnapshot.data.count() - i;
console.log(r)
updates.push(childSnapshot.ref.update({rank: r}));
leaderboard[childSnapshot.key] = Object.assign(childSnapshot.val(), {rank: r});
i++;
});
updates.push(leaderboardRef.set(leaderboard));
return Promise.all(updates);
}).then(() => {
res.status(200).send("Mobile user ranks updated");
}).catch((err) => {
console.error(err);
res.status(500).send("Error updating ranks.");
});
});
I've been stuck for hours trying to get the count of the snapshot and then iterating to add a rank.
Any help or advice is appreciated