How to purge duplicate objects from MongoDB - javascript

I have an array inside an object in my database that accumulates duplicate objects over time (a drawback of working with the unreliable Instagram API). I'm attempting to routinely purge this array of duplicates and replace it with the cleaned up array. I'm having a difficult time understanding why the below code run on my node server does not work. The trimArray function works perfectly, but the 'photos' array within my MongoDB object is never updated.
// Takes in an array and outputs an array with only unique objects
function trimArray(bloatedArray) {
var seen = {};
var trimmedArray = [];
var len = bloatedArray.length;
var j = 0;
for(var i = 0; i < len; i++) {
var imageLink = bloatedArray[i].link;
var image = bloatedArray[i];
if(seen[imageLink] !== 1) {
seen[imageLink] = 1;
trimmedArray[j++] = image;
}
}
return trimmedArray;
}
Event.find( { $and: [{latitude: latitude}, {radius: distance}] },
function(err,event){
if (err) {
} else {
var array = event[0].photos;
Event.update( { $and: [{latitude: latitude}, {radius: distance}] },
{ 'photos': trimArray(array) }
);
}
}
);

I think update will simply update existing records, not remove them. It looks for items in the returned array and updates them accordingly. If you want to use your trimArray, you'll have to empty the collection and then reinsert the trimArray results (terrible idea).
Instead, you should set your collection up correctly to not store duplicates in the first place. You'll want to set up a unique index, and then you'll never have to purge.
See the docs for details.
db.things.ensureIndex({'photos.make_some_key' : 1}, {unique : true, dropDups : true})

Related

Replicating MySQL array formatting in Javasript

I have 5 mysql tables that i need a variety of data from in several different scripts that all reference each other using an id's located in 1 or more column.
I need to create a master query that replicates the array structure exactly as it was imported from mysql AND ALSO needs error handling for each field before it writes to an array to determine if it needs to write the value to an array, or write it as null.
So far the script is looking like this:
const items = [];
// Items
for (let i = 0; i < gameItems.length; i++) {
if (gameItems[i].id) {
items.push({ id: gameItems[i].id });
} else {
items.push({ id: null });
}
if (gameItems[i].identifier) {
items.push({ identifier: if (gameItemParams[i].custom_name)
{
items.push({ custom_name: gameItemParams[i].custom_name });
}
else {
items.push({ custom_name: null }); }
}
}
The problem, or my lack of ability to figure out the logic on how to execute the code correctly, is that in order to attach multiple fields of data to the same row in an array the values must be comma separated.
Individual pushes like above add data to the next row instead of the same object which renders the array.length properly useless because there is a new row for every single field so instead of having 1 row with 10 pieces of data attached, i would have 10 rows each with 1 piece of data.
Is there a way to perform error handling for each field i need to call from the tables or is there another way to add data to the same object after a row has already been pushed.
This is how the newly created array must be structured:
https://puu.sh/E7ogn/61c3117d3b.png
This is how the array is currently being structured with individual pushes:
https://puu.sh/E7oh7/422541a70d.png
Maybe if it is possible to break in the middle of an array.push i can then add error handling in the push block but was unable to find if it can be done.
The problem is that you are pushing an object every time. Instead of that, you need to create an object with all of fields and then push it to the array.
Other problem of your code is that you can use an if statement into a assigment statement. You need to use a conditional operator to do that or extract this conditional from the assigment.
const items = [];
// Items
for (let i = 0; i < gameItems.length; i++) {
var object = {};
if (gameItems[i].id) {
object.id = gameItems[i].id;
}
else {
object.id = null;
}
if (gameItems[i].identifier) {
object.identifier = (gameItemParams[i].custom_name) ? items.push({ custom_name: gameItemParams[i].custom_name }); : items.push({ custom_name: null });
}
items.push(object);
}
As per the data mentioned in https://puu.sh/E7oh7/422541a70d.png.
You have data like
gameItems=[{id:0}, {identifier:"master-ball"}, {category_id:34}, {"custom_name":"Master Ball"}];
I suggest that instead of making items as an array, Please create temporary object item and then push it to items.
let items = [];
let item = {
id:null,
identifier: null,
custom_name: null
};
for (let i = 0; i < gameItems.length; i++) {
if (gameItems[i].id !== undefined) {
item.id = gameItems[i].id;
}
if (gameItems[i].identifier !== undefined) {
item.identifier = gameItems[i].identifier;
}
if (gameItems[i].custom_name !== undefined) {
item.custom_name = gameItems[i].custom_name;
}
}
items.push(item);

Updating the value of an object inside a loop using javascript

I'm currently facing a difficulty in my codes.
First i have an array of objects like this [{Id:1, Name:"AML", allowedToView:"1,2"}, {Id:2, Name:"Res", allowedToView:"1"}...] which came from my service
I assign it in variable $scope.listofResource
Then inside of one of my objects I have that allowedToView key which is a collection of Id's of users that I separate by comma.
Then I have this code...
Javascript
$scope.listofResource = msg.data
for (var i = 0; i < msg.data.length; i++) {
First I run a for loop so I can separate the Id's of every user in allowedToView key
var allowed = msg.data[i].allowedToView.split(",");
var x = [];
Then I create a variable x so I can push a new object to it with a keys of allowedId that basically the Id of the users and resId which is the Id of the resource
for (var a = 0; a < allowed.length; a++) {
x.push({ allowedId: allowed[a], resId: msg.data[i].Id });
}
Then I put it in Promise.all because I have to get the Name of that "allowed users" base on their Id's using a service
Promise.all(x.map(function (prop) {
var d = {
allowedId: parseInt(prop.allowedId)
}
return ResourceService.getAllowedUsers(d).then(function (msg1) {
msg1.data[0].resId = prop.resId;
Here it returns the Id and Name of the allowed user. I have to insert the resId so it can pass to the return object and it will be displayed in .then() below
return msg1.data[0]
});
})).then(function (result) {
I got the result that I want but here is now my problem
angular.forEach(result, function (val) {
angular.forEach($scope.listofResource, function (vv) {
vv.allowedToView1 = [];
if (val.resId === vv.Id) {
vv.allowedToView1.push(val);
I want to update $scope.listofResource.allowedToView1 which should hold an array of objects and it is basically the info of the allowed users. But whenever I push a value here vv.allowedToView1.push(val); It always updates the last object of the array.
}
})
})
});
}
So the result of my code is always like this
[{Id:1, Name:"AML", allowedToView:"1,2", allowedToView:[]}, {Id:2, Name:"Res", allowedToView:"1", allowedToView:[{Id:1, Name:" John Doe"}]}...]
The first result is always blank. Can anyone help me?
Here is the plunker of it... Plunkr
Link to the solution - Plunkr
for (var i = 0; i < msg.length; i++) {
var allowed = msg[i].allowedToView.split(",");
msg[i].allowedToView1 = [];
var x = [];
Like Aleksey Solovey correctly pointed out, the initialization of the allowedToView1 array is happening at the wrong place. It should be shifted to a place where it is called once for the msg. I've shifted it to after allowedToView.split in the first loop as that seemed a appropriate location to initialize it.

how can I filter an array without losing the index?

I have two really long arrays containing "picture names" and "picture files". The first one represents the actual name of the pictures, while the second one is just the file name. For example:
picturenames[0] = '0 - zero';
picturenames[1] = '1 - one';
picturenames[2] = '1 o\'clock';
...
picturefiles[0] = 'numbers-zero.jpg';
picturefiles[1] = 'numbers-one.jpg';
picturefiles[2] = 'time-1.jpg';
...
I have about 1000 items in each array in several languages (the picture files are always the same). I'm "recycling" these arrays from the previous application to save some time and avoid rewriting everything anew.
Desirable functionality: using the user's input in a textbox I want to filter the picturenames array and then show the correspondant picturefiles image.
The issue I'm facing: when I filter the picturenames array I lose the index and I can't "reach" the picture file name.
This is the code I'm using to filter the picturenames array.
var matches = picturenames.filter(function(windowValue){
if(windowValue) {
return windowValue.indexOf(textToFindLower) >= 0;
}
});
What would be the best way to do this?
UPDATE: the solution proposed by Ahmed is the best one, but for time reasons and negligible performance issues I'm just using a for loop to search trough the array, as follows:
var matchesCounter = new Array();
for (i = 0; i < picturenames.length; i++) {
if (picturenames[i].indexOf(textToFindLower) >= 0) {
matchesCounter.push(i);
}
}
console.log(matchesCounter);
for (i = 0; i < matchesCounter.length; i++) {
console.log(picturenames[i]);
console.log(picturefiles[i]);
}
Try this:
const foundIndicies = Object.keys(picturenames).filter(pictureName => {
pictureName.includes(textToFindLower)
});
// reference picturefiles[foundIndicies[0]] to get the file name
Though, it would be far nicer to have both the name and the file in a single object, like so:
const pictures = [
{
name: '0 - zero',
file: 'numbers-zero.jpg',
},
{
name: '1 - one',
file: 'numbers-one.jpg',
}
];
const foundPictures = pictures.filter(picture => picture.name.includes('zero'));
if (foundPictures[0]) console.log(foundPictures[0].file);
You can add one property index during the filtering time, then later on you can use the index.
var matches = picturenames.filter(function(windowValue, index){
if(windowValue) {
windowValue.index = index;
return windowValue.comparator(textToFindLower) >= 0;// Need to define comparator function
}
});
Later on you can access by using like follows:
picturefiles[matches[0].index]
However, the solution will work on object, not primitive type string.
If your data type is string, then you have to convert as object and put the string as a property value like name. The snippet is given below:
var picturenames = [];
var picturefiles = [];
picturenames.push({name:'0 - zero'});
picturenames.push({name:'1 - one'});
picturenames.push({name:'1 o\'clock'});
picturefiles.push({name:'numbers-zero.jpg'});
picturefiles.push({name:'numbers-one.jpg'});
picturefiles.push({name: 'time-1.jpg'});
var textToFindLower = "0";
var matches = picturenames.filter(function(windowValue, index){
if(windowValue) {
windowValue.index = index;
return windowValue.name.indexOf(textToFindLower) >= 0;
}
});
console.log(matches);

MongoDB Documents : Find number of Repetitions and Sort In Ascending Order

So I'm working on an agent that's supposed get data from MongoDB, iterate through it, and then find the keys which have the same values, and sort that list in ascending order. Basically, find out how many repetitions there are in the documents from Mongo, and then sort it.
So I've gotten to the part where I get repetitions in the data, and count them, but I'm stuck with getting the ascending order part.
The code for getting documents out of mongodb and processing it is as follows.
function matchRoomAndAgent() {
var url = 'mongodb://192.168.10.10:27017';
MongoClient.connect(url, function (err, db) {
db.collection("whiteboardRooms").find({}).toArray(function (err, res) {
var roomsArray = [];
var count = {};
var temp_count = {};
if (err) {
throw err;
}
db.close();
// roomsArray.push(res);
for (var i = 0; i < res.length; i++) {
roomsArray.push(res[i]);
// console.log('agentId: '+ res[i][agentId].agentId);
}
// console.log(roomsArray);
for (var i = 0; i < 15; i++) //supposed to use length of roomsArray
{
var currentAgent = roomsArray[i].agentId;
for (var j = 0; j < 15; j++)
{
if (roomsArray[j].agentId == currentAgent)
{
count[j] = (count[j] || 0) + 1;
temp_count[j] = count[j];
}
}
console.log(temp_count);
}
Also please find attached, a sample pic of the mongodb documents.
MongoDB Documents
Please let me know if you require more information, and I'll do my best to get it. I'm kind of a noob at programming. Sorry!
Thanks in advance for your help :)
At some point you might want to limit the material being grouped, for example by lastUpdateTime between two timestamps and you don't want to drag all that out of the DB just to throw it away. Here's the general shape of the solution:
db.foo.aggregate([
{$match: {agentID: {$ne: " "}}} // guessing this is noise so filter out
// other $match could be used here to further shrink the data...
,{$group: {_id:"$agentID", n:{$sum:1}}}
,{$match: {n: {$gt:1}}} // ignore non-dupes
,{$sort: {_id: 1}} // and finally sort.
]);
So I've figured out the answer to my own problem. Its pretty much got nothing to do with MongoDB.
I basically had an array called roomsArray which contained objects. I used groupBy and map to project these objects to a format I wanted, and then used sort as I would an array, using its length to get the count.
I should have used groupBy() and map() with sort() to get the values I needed.
var groupBy = function (array, key) {
return array.reduce(function(acc,val){
(acc[val[key]] = acc[val[key]] || []).push(val);
return acc;
}, {});
};
var result = groupBy(roomsArray, 'agentId');
// console.log(result);
var mapped = Object.keys(result).map(function(x){
return { agentId : x, rooms: parseInt(result[x].length)}
});
// console.log(mapped);
var sorted = mapped.sort((a,b) => a.rooms - b.rooms);
console.log(sorted);

How to check if a item/value already exists in Local Storage

I am working on a shopping cart application were users can click an Add to Cart button that will then add the specific item to local storage. When the user is adding a product to their shopping cart I need to be able to check to see if that specific item/value/product already exists in local storage. If it does, I need to increase only the count of that item/value/product by 1. If not, I need to add an entirely new item/value/product to local storage with a count of 1.
How can I find if an item already exists in local storage and compare it to the id of the current product that a user is attempting to add to their cart? My first few attempts failed miserably and have yet to find anything online that is helping with this issue. Is there a better way of going about this? Any assistance is appreciated. Even a good link to a good page would be extremely helpful.
Below is the code I have to attempt in checking for if the productid being added matches any of the productids in local storage. Basically if the productId that is being added matches the productId of an item in local storage simply add 1 to the quantity.
var retrieverObject = localStorage.getItem('Products');
var retrieveObject = JSON.parse(retrieverObject);
var data = {};
var productId = currentNode.name;
var product = currentNode;
data.productPrice = product.parentNode.previousSibling.previousSibling.id;
data.productId = productId;
var length = retrieveObject.length;
console.log(length);
for(var i = 0; i<length; i++){
if(retrieveObject[i].productId == data.productId){
var quantity = retrieveObject[i].quantity;
retrieveObject.push({"productPrice": data.productPrice, "productId": data.productId, "quantity": quantity++});
}else{
retrieveObject.push({"productPrice": data.productPrice, "productId": data.productId, "quantity": 1});
}
}
console.log(retrieveObject);
localStorage.setItem('Products', JSON.stringify(retrieveObject));
var retrievedObject = localStorage.getItem('Products');
var obj = JSON.parse(retrieverObject);
var len = obj.length;
console.log(len);
for(var i=0; i<len;i++){
console.log(obj[i]['productPrice']+", "+obj[i]['productId']);
}
}
}
There are a few issues. First, I am not entirely certain that the productId of the retrieved object is being compared to the one that is being added. Secondly, the for(var i = 0; i<length; i++){} definitely does not seem to be doing what was expected and is multiplying the number of items being added by 2. Thirdly, which may relate to the second issue, the retrieveObject.push() is not updating the quantity of the product but is adding an entire new entry to local storage. The given answers did not seem to be working for me before so this is what I have been working on. Any new answers or general help would be great.
PT 2. : So I am having an issue with the first entry into the local storage. Without noting that when there is nothing in local storage and you make a call to get the items in it, it returns null or undefined. So currently I have it set up like this:
if(localStorage.getItem("Products") === null || localStorage.getItem("Products") === undefined){
var data = {};
var productId = currentNode.name;
var product = currentNode;
data.productPrice = product.parentNode.previousSibling.previousSibling.id;
data.productId = productId;
var obj = [];
obj = obj[data.productId] = {
productPrice: data.productPrice,
count: 1
};
console.log(obj);
localStorage.setItem('Products', JSON.stringify(obj));
}
else{
var retrieverObject = localStorage.getItem('Products');
var retrieveObject = JSON.parse(retrieverObject);
var data = {};
var productId = currentNode.name;
var product = currentNode;
data.productPrice = product.parentNode.previousSibling.previousSibling.id;
data.productId = productId;
if(retrieveObject[data.productId]){
retrieveObject[data.productId].count++;
}else{
retrieveObject[data.productId] = {
productPrice: data.productPrice,
count: 1
};
}
console.log(retrieveObject);
localStorage.setItem('Products', JSON.stringify(retrieveObject));
}
This creates a first entry in local storage that looks like this : {"productPrice":"78.34","count":1}, and then when adding others looks like this: {"productPrice":"78.34","count":1,"rJUg4uiGl":{"productPrice":"78.34","count":3}} and works perfectly fine. The issue is getting the first entry to b formatted properly. When I change the code in the first if statement like so:
var obj = [];
obj[data.productId] = {
productPrice: data.productPrice,
count: 1
}
I get an empty [] in local storage but when I console.log the obj it is in the proper format. [rJUg4uiGl: Object]. I have been stuck on this and haven't been able to get it working. Again, any help would be really appreciated.
Once you have your data structure in obj, I would suggest using a dictionary with product IDs as keys.
To add the order or whatever, where you have:
obj.push({"productPrice": data.productPrice, "productId": data.productId});
Use:
if (obj[data.productId]) { // if the entry exists,
// increment the count
obj[data.productId].count++;
} else { // if it doesn't,
// add a new entry with count = 1
obj[data.productId] = {
productPrice: data.productPrice,
count: 1
};
}
Here is a complete function, including localStorage handling:
function addToCart(productID, productPrice) {
// get the current cart, or an empty object if null
var cart = JSON.parse(localStorage.getItem("Products")) || {};
// update the cart by adding an entry or incrementing an existing one
if (cart[productId]) {
cart[productId].count++;
} else {
cart[productId] = {
productPrice, // shorthand for `productPrice: productPrice,`
count: 1
};
}
// put the result back in localStorage
localStorage.setItem("Products", JSON.stringify(cart));
}
The solution above is preferable because you can check for a productId without looping through the whole list. If you really want to keep your current data structure of an array of objects, you could update it like this:
var length = retrieveObject.length;
console.log(length);
for (var i = 0; i < length; i++) {
if (retrieveObject[i].productId == data.productId) {
retrieveObject[i].quantity++; // update the entry in the array
} else {
retrieveObject.push({
productPrice: data.productPrice,
productId: data.productId,
quantity: 1
});
}
}
Note that you shouldn't push a new entry into the array; just update the existing one.
Just use localstorage.getItem; it returns null if the key doesn't already exist.
Assuming you are using localStorage node package you could do
if (localStorage.getItem('Products') !== null) {
localStorage.setItem('Products', JSON.stringify(obj));
}
Here is your reference:
https://www.npmjs.com/package/localStorage
Regards
Update:
Searching within your objet is a different story... so you want to check if the Product id is there then you can search for it using lodash
var _ = require('lodash');
// the rest of your code to get the data.productId set
if (localStorage.getItem('Products') !== null) {
var arrayOfProducts = localStorage.getItem('Products');
var existingProducts = _.filter(arrayOfProducts, function (product) { return product.productId === data.productId });
if (existingProducts.length > 0) {
// product found, do your logic
}
}
Here's lodash info https://www.npmjs.com/package/lodash
The other option is using a dictionary and having the productId as key and then using Object.keys to search for it... I've offered an approach that does not change your json structure.

Categories

Resources