So I've been working on my project that involves scraper.
So workflow is next: There are two scrapers right now. Data is being parsed and pushed into array for each individual scraper and passed to the merge component.
So the merge component looks like this:
let mergedApartments = []; //Creating merged list of apartments
exports.mergeData = (apartments) => {
//Fetching all apartments that are passed from scraper(s)
mergedApartments.push(...apartments); //Pushing apartments into the list
console.log(mergedApartments.length);
};
So right now output of mergedApartments.length is 9 39. So the first function that calls mergeData() and pass it an array have 9 objects inside it, and the other scraper have 30 objects inside it's array, who is again passed to the mergeData.
Now this is not what I've expected. I've expected one array with all merged objects from the scrapers. Right now, scraperno1 send apartments and it's added to the mergedApartments, then scraperno2 sends apartments and it's overwriting that array by adding new apartments objects into the array.
Now I want different output: I just want to get one list with all merged objects from the arrays. Because this data will be passed to the storing component, and I don't want to query DB multiple times, because for each new mergedApartments list, data will be inserted and creating duplicate values - throwing an error.
So what I've tried: I've tried creating some kind of a counter which counts number of time that function mergeData is called, and then do the logic about merging but no success.
So I just want my array to have one output of mergedApartments.length - in this case 39.
Thanks!
EDIT
Here how one of the scraper looks:
const merge = require('../data-functions/mergeData');
const axios = require('axios');
const cheerio = require('cheerio');
//function for olx.ba scraper. Fetching raw html data and pushing it into array of objects. Passing data to merge function
exports.santScraper = (count) => {
const url = `https://www.sant.ba/pretraga/prodaja-1/tip-2/cijena_min-20000/stranica-${count}`;
const santScrapedData = [];
const getRawData = async () => {
try {
await axios.get(url).then((response) => {
const $ = cheerio.load(response.data);
$('div[class="col-xxs-12 col-xss-6 col-xs-6 col-sm-6 col-lg-4"]').each(
(index, element) => {
const getLink = $(element).find('a[class="re-image"]').attr('href');
const getDescription = $(element).find('a[class="title"]').text();
const getPrice = $(element)
.find('div[class="prices"] > h3[class="price"]')
.text()
.replace(/\.| ?KM$/g, '')
.replace(',', '.');
const getPicture = $(element).find('img').attr('data-original');
const getSquaremeters = $(element)
.find('span[class="infoCount"]')
.first()
.text()
.replace(',', '.')
.split('m')[0];
const pricepersquaremeter =
parseFloat(getPrice) / parseFloat(getSquaremeters);
santScrapedData[index] = {
id: getLink.substring(42, 46),
link: getLink,
descr: getDescription,
price: Math.round(getPrice),
pictures: getPicture,
sqm: Math.round(getSquaremeters),
ppm2: Math.round(pricepersquaremeter),
};
}
);
merge.mergeData(santScrapedData); //here i'm calling function and passing array to function
});
} catch (error) {
console.log(error);
}
};
getRawData();
};
Other scraper looks the same(it's same calling of the function)
For this, you need to use concat function from the Array prototype
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat
exports.mergeData = (apartments) => {
mergedApartment = mergedApartments.concat(apartments);
};
exports.sendData = () => {
console.log(mergedApartment.length);
}
and in your main script
getRawData().then(merge.sendData);
Related
I have spent 6 hours already trying to figure out how to do this in NodeJS.
I am using NodeJS with Express and MongoDB.
I have a database that has two collections viz. Listings and Categories. Each listing has a "category" which is an ID that maps to a category inside the Categories collection.
What I want to do is, fetch all the listings, then loop through all of them, and get the category title from the Categories collection using the category id.
This is what I had for getting all the listings:
const listings_raw = [];
await db.collection("listings").find().forEach(listing => listings_raw.push(listing));
The code above works fine. The trouble I am having is with this:
const listings = [];
listings_raw.forEach(async listing => {
const category_id = listing.category;
const category_title = await db.collection('categories').findOne({_id: objectId(category_id)});
listing.category_title = category_title;
listings.push(listing);
});
response.send(listings);
The response.send(listings); gets executed before the listings_raw.forEach completes.
I want to send the data only after all listings have been populated with the category titles.
After spending 6 hours, I have come up with the following hack which I am sure is nasty!
const listings_raw = [];
const em = new events.EventEmitter();
await db.collection("listings").find().forEach(listing => listings_raw.push(listing));
const listings = [];
let counter = 0;
listings_raw.forEach(async (listing) => {
const category_id = listing.category;
const category = await db.collection('categories').findOne({_id: objectId(category_id)});
listing.category_title = category.title;
listings.push(listing);
if (counter === listings_raw.length - 1) {
em.emit('listings:processing:done');
}
counter++;
});
em.on('listings:processing:done', () => {
response.send(listings);
});
Please, can someone explain or guide me on how this should be done in JavaScript?
Basically, I am not able to figure out how to know if all promises have been resolved or not.
Thank you!
The listings_raw.forEach function executes synchronously on the array, even though you are then performing an asynchronous operation within that.
Promise.all will allow you to await for the result of an array of promises. Therefore you can .map the listings to an array of promises which return the updated listing.
const listings = await Promise.all(listings_raw.map(async listing => {
const category_id = listing.category;
const category_title = await db.collection('categories').findOne({_id: dependencies.objectId(category_id)});
listing.category_title = category_title;
return listing;
});
response.send(listings);
I am trying to create an add to cart button which fetches the data from product database using the id of specific product which I selected. I am trying to push the object found using the same Id into a normal javascript array and then to display it using ejs methods. While I was tring I found I am unable to push the data in object form.
Summary:
On 7th line I have declared an array and in that array I want to store some objects which I have fetched frome a db model.
On 15th line I am trying to push the object form into my array so that I could iterate through the objects to display them on my page using ejs. But I am unable to do that.
screenshots:
Here's the final result I'm getting even after trying to push objects in array:
empty array logged
Here are the objects I'm trying to push:
Objects
Code:
app.get("/cart", (req, res) => {
if (req.isAuthenticated()) {
const findcartdata = req.user.username;
userData.findOne({email: findcartdata}, (err, BookId) => {
// console.log(BookId.cartItemId);
const idArray = BookId.cartItemId;
var bookArray = [];
idArray.forEach((data) => {
productData.findOne({_id: data}, (err, foundBookData) =>{
// console.log(foundBookData);
if(err){
console.log(err);
}
else{
bookArray.push(foundBookData);
}
})
});
console.log(bookArray);
// res.render("cart", {
// cartBookArray: BookId.cartItemId
// })
});
} else {
res.redirect("/login");
}
})
In above code i found the user's email using passport authentication user method and using that email I wanted to add the products in a different javascript array (which I am goint to pass to my ejs file of cart and then iterate it on list) using those array of Id which I got from another model called userData. The problem is I am able to find userData of each Id but unable to store them as an array of objects.
Looks like a timing issue, your code completes before the database downloads the objects and pushes them to your array.
This should fix your issue:
// ...
const idArray = BookId.cartItemId;
var bookArray = [];
for (const data of idArray) {
const foundBookData = await productData.findOne({_id: data}).catch(console.error);
if (!foundBookData) continue;
bookArray.push(foundBookData);
}
console.log(bookArray);
// ...
By the way, make sure to make the whole function asynchronous as well, which would be done by changing this line:
userData.findOne({email: findcartdata}, async (err, BookId) => { // ...
Within my function, through interaction from the user, I aim slowly build up an array of responses which I then pass off to an API. However, different approaches to append to the array, simply return a single position array (overwrite).
My current code as follows:
const contribution: Array = [];
const handlePress = () => {
var col = {
response,
user: 1,
update: update.id,
question: q.id,
};
contribution = [...contribution, col];
}
My understanding is that contribution = [...contribution, col] is the correct way to add to the array.
What is the best practice approach for doing this inside a function called each time the user interacts?
Although it is not clear from the question, I suspect, this code is inside a component. If so, then a new contribution array is created on every render. You need to use useState to store this array so that a new array is not created on every render.
const [contribution, setContribution] = React.useState([]);
const handlePress = () => {
var col = {
response,
user: 1,
update: update.id,
question: q.id,
};
setContribution([...contribution, col]);
}
I'm trying to pass a property, that is inside the first position of an array of objects, to another module so I can use this value later. I've tried to pass it as module(args), but it keeps reading the default value which is 0. Is there a way to do this?
I tried to implement some React.context but the Bot framework Emulator is refusing it.
/////////////////Module that ll acquire the value/////////////////////////////
getCard(bot, builder, params) {
let configValues = { ...params[0] }
bot.dialog(`${configValues.path}`, function (session) {
var msg = new builder.Message(session);
const cardItem = (obj) => {
return (new builder.HeroCard(session)
.title(`${obj.title}`)
.text(`R$ ${obj.price}`)
.images([builder.CardImage.create(session, `${obj.img}`)])
.buttons([
builder.CardAction.imBack(session, `${obj.price} Item adicionado!`, 'add to cart')
// !onClick event must add the current obj.price to
// the configValues.total(Ex: configValues.total += obj.price)!
])
)
}
msg.attachmentLayout(builder.AttachmentLayout.carousel)
msg.attachments(
eval(params.map(obj => cardItem(obj)))
);
//!in here before end the dialog is where i want to update
// the configValues.total so i can show it in the -> Checkout module
session.send(msg).endDialog()
}).triggerAction({ matches: configValues.regex });
}
}
//////////////CheckOut.Module///////////////////////////////
{...}
let configValues = { ...params[0] }
let state = {
nome: "",
endereco: "",
pagamento: "",
total: configValues.total // this is the value to be read
}
bot.dialog('/intent', [
{...},
(session, results) => {
state.pagamento = results.response
session.send(
JSON.stringify(state) // here is the place to be printed
)
{...}
]
).triggerAction({ matches: /^(finalizar|checar|encerrar|confirmar pedido|terminar)/i })
Since you solved your original problem, I'll answer the one in your comment.
Your problem is here:
cartId.map((obj, i , arr) => {
// if (!obj.total) {
// obj.total.reduce(i => i += i)
// }
const newtotal = new total
newtotal.getTotals(bot, builder, obj, arr)
})
cartId contains the totals for each of your items. When you call map on it, you're passing each item individually to getTotals, which passes each item to checkout()
The reason you can't sum all of the totals and can only sum one item's total is that you pass cartId to checkout and cartId has been changed to just a single item. Instead, there's a couple of different things you could do:
Pass the whole cartId from cartItems and use something like for (var key in cartItems) in totalConstructor() and checkoutConstructor(). This is probably the easiest, but not very memory efficient.
Use BotBuilder's State Storage to store your totals array in userData, then sum that at the end. This might be more difficult to implement, but would be a much better route to go. Here's a sample that can help you get started.
I have been doing this for an hour. I simply want to get the number of children in the child "Success" in the database below. The answers in similar stackoverflow questions are not working. I am new in Javascript Programming.
So far I have tried this
var children = firebase.database().ref('Success/').onWrite(event => {
return event.data.ref.parent.once("value", (snapshot) => {
const count = snapshot.numChildren();
console.log(count);
})
})
and also this
var children = firebase.database().ref('Success/').onWrite(event => {
return event.data.ref.parent.once("value", (snapshot) => {
const count = snapshot.numChildren();
console.log(count);
})
})
Where might I be going wrong.
As explained in the doc, you have to use the numChildren() method, as follows:
var ref = firebase.database().ref("Success");
ref.once("value")
.then(function(snapshot) {
console.log(snapshot.numChildren());
});
If you want to use this method in a Cloud Function, you can do as follows:
exports.children = functions.database
.ref('/Success')
.onWrite((change, context) => {
console.log(change.after.numChildren());
return null;
});
Note that:
The new syntax for Cloud Functions version > 1.0 is used, see https://firebase.google.com/docs/functions/beta-v1-diff?authuser=0
You should not forget to return a promise or a value to indicate to the platform that the Cloud Function execution is completed (for more details on this point, you may watch the 3 videos about "JavaScript Promises" from the Firebase video series: https://firebase.google.com/docs/functions/video-series/).
const db = getDatabase(app)
const questionsRef = ref(db, 'questions')
const mathematicalLiteracy = child(questionsRef, 'mathematicalLiteracy')
onValue(mathematicalLiteracy, (snapshot) => {
const data = snapshot.val()
const lenML = data.length - 1
console.log(lenML)
})
This method worked for me. I wanted to get the children's count of the mathematicalLiteracy node in my database tree. If I get its value using .val() it returns an array that contains that node's children and an extra empty item. So, I subtracted that one empty item's count. Finally, I get my needed children's count.