Develop Reference

mongoose find() sorting and organizing returned results from a product database in js

I have a problem with organizing my mongoDB data to send to my page in my res and cant figure out how to do the correct js. Here is a simplified version of my schema
var productSchema = new mongoose.Schema({
medium: String,
brand: String,
group: String
});
Here is what a typical entry looks like
medium :"Acrylic",
brand :"liquitex",
group :"heavy body"
there are many more entries in the schema, but these are the only ones I need to be able to sort and organize the returned results with. The problem is I have a route that returns all colors in my database and I want to be able to display them in sections on my page that are grouped under Brand, and then has the individual colors listed under the correct group.
The problem is there are paints from other brands that fall into the heavy body group and so when I use a filter function to sort my data by group, some brands get mixed together. I cant filter by brand, because some brands have acrylic and watercolor so then those get lumped together.
I need some way to filter the returned results of a
mongoose.find({})
that can use the group data as a filter, but then filter those results by the brands so they get separated into the correct brand categories.
I have this so far:
this is all a stripped down version of my app.js file:
//finds all colors in the DB
Color.find({}).lean().exec(function( err, colors)
var groups = [];
// find all groups in the databse
colors.forEach( function(color){
groups.push(color["group"]);
});
//returns only unique names to filter out duplicates
var groupTypes = Array.from(new Set(groups));
var tempVariableBrands = [];
// this sorts all returned paints into their respective group, but we get paints from multiple brands under the same group and that is not good
groupTypes.forEach( function(group){
var name = group;
var result = colors.filter(obj => { return obj.group === group });
tempVariable.push( {group : name, result } );
});
// the tempVariable gets sent to my page like so
res.render("landing", {colorEntry:tempVariable} );
and this works fine to allow me to display each paint by its grouping, but that fails when there is more than one paint from a different manufacturer that is considered the same group like a "heavy body". This is my ejs on my page that works fine:
<% colorEntry.forEach( function(entry){ %>
<div class="brandBlock">
<div class="brandTitle">
<span><%=entry.result[0].brand%> - <%=entry.result[0].group%></span>
I for the life of me cant seem to figure out the combination of filter() and maybe map() that would allow this kind of processing to be done.
My database has like 600 documents, colors from a number of different manufacturers and I don't know how to get this as a returned structure: lets say this is a few colors in the DB that get returned from a mongoose find:
[{ medium: "Oil",
brand: "Gamblin",
group: "Artists oil colors"},
{ medium: "Acrylic",
brand: "Liquitex",
group: "Heavy Body"},
{ medium: "Acrylic",
brand: "Golden",
group: "Heavy Body"}
]
i need to organize it like this or something similar. It can be anything that just sorts this data into a basic structure like this, I am not confined to any set standard or anything, this is just for personal use and a site I am trying to build to learn more.
returnedColors = [ { brand: "Gamblin", group: "Artists oil colors", { 50 paints colors returned} },
{ brand: "liquitex" , group: "heavy body", { 20 paint colors returned } },
{ brand: "golden" , group: "heavy body",{ 60 paint colors returned} }
];
I am not a web developer and only write some web code every 6 months or so and have been trying how to figure this out for the last 2 days. I can't wrap my head around some of the awesome filter and map combo's i have seen and cant get this to work.
Any help or advice would be great. I am sure there are many areas for improvement in this code, but everything was working up until I entered paints that were from different brands that had the same group type and i had to try to rewrite this sorting code to deal with it.
It boils down to needing to be able to iterate over the entire set of returned documents from the DB and then sort them based off 2 values.
UPDATE:
I was able to get something that works and returns the data in the format that I need to be able to send it to my ejs file and display it properly. The code is rather ugly and probably very redundant, but it technically works. It starts off by using the group value to run over paints since each set of paints will have a group name, but can sometimes share a group name with a paint from another brand like "heavy body".
groupTypes.forEach( function(group){
var name = group;
var result = colors.filter(obj => { return obj.group === group });
// this gets brand names per iteration of this loop so that we will know if more than one brand of paint
// has the same group identity.
var brands = [];
result.forEach( function(color){
brands.push(color["brand"]);
});
// This filters the brand names down to a unique list of brands
var brandNames = Array.from(new Set(brands));
// if there is more than one brand, we need to filter this into two separate groups
if( brandNames.length > 1){
//console.log("You have duplicates");
brandNames.forEach( x => {
var tmpResult = [...result];
var resultTmp = result.filter(obj => { return obj.brand === x });
result = resultTmp;
//console.log("FILTERED RESULT IS: ", result);
tempVariable.push( {brand: x ,group : name, result } );
result = [...tmpResult];
});
}else{
tempVariable.push( {brand: result[0].brand ,group : name, result } );
}
});
if anyone can reduce this to something more efficient, I would love to see the "better" way or "right" way of doing something like this.
UPDATE2
Thanks to the answer below, I was put on the right track and was able to rewrite a bunch of that long code with this:
Color.aggregate([
{
$sort: { name: 1}
},
{
$group: {
_id: { brand: '$brand', group: '$group' },
result: { $push: '$$ROOT' }
}
},
{ $sort: { '_id.brand': 1 } }
], function( err, colors){
if(err){
console.log(err);
}else{
res.render("landing", {colorEntry:colors, isSearch:1, codes: userCodes, currentUser: req.user, ads: vs.randomAds()} );
}
});
Much cleaner and appears to achieve the same result.

Since you're using MongoDB, "right" way is to utilize an Aggregation framework, precisely, $group stage.
Product.aggregate([{
$group: {
_id: { group: '$group', brand: '$brand' },
products: { $push: '$$ROOT' }
}
}])
This will output array of objects containing every combination of brand and group, and push all relevant products to corresponding subarray.
Combine it with $project and $sort stages to shape your data further.

Removing duplicates elements with conditionality

I have an array of objects that look like this:
[
{ project: "test1", date: "8/14/2018" },
{ project: "test1", date: "8/15/2018" },
{ project: "test2", date: "8/14/2018" }
]
I want to remove duplicate objects based on the project name, BUT, keep the one with the most recent date.
So my final array would like this:
[
{ project: "test1", date: "8/15/2018" },
{ project: "test2", date: "8/14/2018" }
]
I've come up with a disgruntled recursive solution that I'm unhappy with.
Looking for suggestions on a painless way to do this.

The approach I generally take for problems like this is to maintain a "dictionary" object to track uniqueness while iterating through, then extract the final result from that result. reduce is the go-to tool for creating one value from multiple values:
const dict = objects.reduce((result, item) => {
const currentForProject = result[item.project];
const isMostRecent =
!currentForProject
|| new Date(item.date) > new Date(currentForProject.date);
if (isMostRecent) {
result[item.project] = item;
}
return result;
}, {});
const mostRecents = Object.values(dict);

I will propose the general algorithm to solve your task but the implementation is up to you.
1) sort the array based on date field;
2) create an empty Set for storing project names and empty array for the result;
3) iterate over the sorted array and check:
3.1) if the Set does not contain current project name, then:
3.2) add the project name to the Set and add current item to your result
array;

query an array of objects and return a single object - mongodb

I'm having some trouble searching an array I have in my model which currently contains dummy posts.
My posts array looks like this
posts: [
{
image: <filename>,
comments: [],
joined: <number>,
}
]
The posts array atm holds about 10 objects. I need a way to query this array and return a single object. I already tried the answers given on other similar questions, but they all return the whole user which is not what I want.
I tried this:
model
.find(
{ $match : { "posts.image": req.params.image } },
{ $unwind : "$posts" },
{ $match : { "posts.image": req.params.image } }
)
This also returns the whole object including the password, username, etc. I also tried $elemMatch, and no luck.
I'm only expecting it to return one object (not multiple objects) since ill be querying the array with req.params.

Unsure
Not entirely certain if this is what you are asking for, but a way to get a single property value of an array would be done this way
If I am misunderstood please clarify what you are after and I will amend my answer.
var posts = [{
image: "x.jpg",
comments: [],
joined: 12
},
{
image: "x1.jpg",
comments: [],
joined: 121
},
{
image: "ax.jpg",
comments: [],
joined: 2
}
];
for (a = 0; a < posts.length; a++){
console.log(posts[a].image);
}

Methodology Approach - json string/output

Hope this question is up to par for SO. I've never posted before but looking for some info in regards to approaching a task I have been assigned.
I have a page with Form information, some simple fields that I've got going.
The page outputs a json string of the form fields and writes them to a text file.
What I need to do is use the key: values to fill in a bash script to output to a hotfolder for ingestion into the production environment.
The "addmore" array needs to determine the number of filename fields produced. I should add that this is dynamic, the "addmore" array may contain 1 entry, or up to technically an unlimited number. This number will fluctuate, though.
JSON Output:
{"formatcode":"JFM","serialnumber":"555","callletters":"555","rotator":"555","addmore":["555","444","333",""]}
How can I use the key : value pairs to output this into a file like below:
{"type":"sequential","items":[
{"filename": "/assets/$formatcode/$serialnumber/$addmore1", "description":"First item"},
{"filename": "/assets/$formatcode/$serialnumber/$addmore2", "description": "Second item"},
{"filename": "/assets/$formatcode/$serialnumber/$addmore3", "description": "Third item"}
]}

This should do the trick, I'm using ES6 string templates to create the dynamic filename and description. The resulting object should have addmore.length items, so we reduce based on data.addmore, creating the basic object as our starting point ({ type: "sequential", items: [] }) and pushing the dynamic strings into output.items on each iteration.
const data = {
"formatcode":"JFM",
"serialnumber":"555",
"callletters":"555",
"rotator":"555",
"addmore":[
"555",
"444",
"333",
]
};
const desiredOutput = data.addmore.reduce((p, c, index) => {
p.items.push({
filename: `/assets/${data.formatcode}/${data.serialnumber}/${c}`,
description: `Item ${index + 1}`
});
return p;
}, {
type: "sequential",
items: []
});
console.log(desiredOutput);
If you don't have ES6 available, you can turn the arrow function into a regular function and use the + concat operator instead of string templates.

How can I select the latest version of an object from a ForerunnerDb collection

I have a collection which contains a series of objects generated over time. Since I have disparate types stored in the same collection, I have a TypeId and a UID per object (where the UID identifies objects that refer to the same entity over time). I am trying to choose the most recent object from the collection, and running into serious difficulties grasping how to do so without manually iterating a query result - something I'd rather avoid since I think it could become expensive when the collection gets larger.
For example:
var db; // assigned elsewhere
var col = db.collection("res");
col.primaryKey("resId");
col.insert({
resId: 1,
TypeId: "Person",
UID: "Bob",
Data: {Age: 20, Name:Bob}
});
col.insert({
resId: 2,
TypeId: "Person",
UID: "Bob",
Data: {Age: 25, Name:Bob}
});
col.insert({
resId: 3,
TypeId: "Car",
UID: "TeslaModelX",
Data: {Manufacturer: "Tesla", Owner:"Bob"}
});
col.insert({
resId: 4,
TypeId: "Person",
UID: "Bill",
Data: {Age: 22, Name:Bill}
});
From col, I want the query to select all objects with TypeId="Person" ranked by resId descending, i.e. I'd expect to select objects 4 and 2, in that order.
The collection above is contrived, but in reality I'd expect there to be certainly 000s of entries and potentially 0000s, with maybe 00s of versions of each UID. In other words, I'd rather not return the full collection of objects, grouped or otherwise, and iterate it.
I have tried the following, but since the $distinct operator is applied before the $orderBy one, this doesn't help:
col.find(
{
TypeId : {$eq : "Person"}
$distinct: { UID: 1}
},
{
$orderBy: {
resId : -1
}
}
);
I have in mind that I should be able to use the $groupBy, $limit and $aggregate clauses to identify the per group desired IDs, and then use a subquery to find the precise (non-aggregated) elements, but as yet I haven't managed to get anything to do what I want. Any ideas?
My current solution is to include a Deleted property amongst my objects, and set it to true for all existing non-deleted objects in the DB before I insert new entries. This lets me do what I want but also stops me from, for instance, choosing the best available within a known timeframe or similar.

You can do this like:
var tmpObj = {};
col.sort({resId: -1}, coll.find({
"TypeId": "Person"
})).filter(function (doc) {
return col._match(doc, {
$distinct: {
UID: 1
}
}, {}, 'and', tmpObj);
});
It's a bit dirty since it's not neatly wrapped up in a single command, but it's as clean as you'll get it in ForerunnerDB v1.x.
Version 2 will have a new query pipeline system that would allow for exactly this sort of usage, something like this:
col.pipeline()
.find({"TypeId": "Person"})
.orderBy({"resId": 1})
.distinct({"UID": 1})
.then(function (err, data) {
console.log(data);
});
Source: I'm the developer of ForerunnerDB.