Related
Im just learning indexing with Mongoose/MongoDB and I dont know why this isnt working.
this is my schema
const timeSchema = new mongoose.Schema({
actionId:{
type:String,
required:true
},
start: {
type: Date
},
end: {
type: Date
},
user:{type : mongoose.Schema.Types.ObjectId, ref : 'User'},
task:{type : mongoose.Schema.Types.ObjectId, ref : 'Task'},
pausedSeconds:{
type: Number,
default: 0
}
});
const Time = mongoose.model('Time', timeSchema)
i want to have a text index in two populated fields user and task, i created the index this way
timeSchema.index({"user.name":"text","task.taskName":"text"})
Here is an example of the documents
{
"pausedSeconds": 18,
"_id": "5db1dde8d5bc93526c26fa38",
"actionId": "5feaebcf-6b90-45be-8104-452d643472a0",
"user": {
"_id": "5d4af77e4b6cbf3dd8c5f3ac",
"name": "admin"
},
"task": {
"_id": "5d4aff2f61ad755154b8a1c6",
"taskName": "task 1 updated!"
},
"start": "2019-10-24T17:22:48.000Z",
"end": "2019-10-24T17:30:00.000Z"
},
I have one issue and one question
The issue is:
What im trying to do is get all the documents that have "task 1 updated" (for task.taskName) or
"admin" (for user.name) doing it this way
Time.find({ '$text': { '$search': "admin" } })
Time.find({ '$text': { '$search': "task 1 updated" } })
but it doesnt seem to work
The question is:
If I want to do a text search for the fields start,end being a Date type or for the field pausedSeconds being a Number type what should I do?
Thanks in advance
In your query, you aren't specifying what property to search on. Do this: Time.find({taskName: { '$text': { '$search': "admin" }}}).
Also, I'm not sure if you're just not showing all the code or if you're actually doing your query wrong, but it should be written like this:
Time.find({taskName: { '$text': { '$search': "admin" }}}).exec(function(err, times) {
if(err) return console.log(err);
console.log(times);
});
I'm new to mongodb, and I'm using mongoose to validate and order the data (I'm open to change it to MySQL if this doesn't work).
The app will be an e-shop, to buy merchandising related to movies, games, ext.
My schema is as follows:
var productSchema = {
id: {
type: String,
required: true
},
name: {
type: String,
required: true
},
img: {
type: String,
required: true
},
price: {
type: Number,
required: true
},
stock: {
type: Number,
required: true
},
category: {
object: {
type: String,
required: true
},
group: {
type: String,
required: true
},
name: {
type: String,
required: true
}
}
};
This is what I would like to do:
If I have the following data in category:
category.object = "ring"
category.group = "movies"
category.name= "lord of the rings"
I want the id to be made of the first letters of every field in category and a number (the number of the last item added plus 1). In this case, It would be RMLOTR1.
What I'm doing right now
I'm adding a lot of data at the same time, so every time I do it, I made a function that iterates through all the items added and does what I want but...
My question is
Is there a built-in way to do this with mongodb or mongoose, adding the data and creating the id at the same time? I know I can do a virtual, but I want the data to be stored.
Extras
If it's not posible to do this with mongodb, is there a way to do this with MySQL?
Is doing this kind of thing considered a correct/wrong approach?
You are basically looking for a "pre" middleware hook on the "save" event fired by creating new documents in the collection. This will inspect the current document content and extract the "strings" from values in order to create your "prefix" value for _id.
There is also another part, where the "prefix" needs the addition of the numeric counter when there is already a value present for that particular "prefix" to make it distinct. There is a common technique in MongoDB used to "Generate an auto-incrementing sequence field", which basically involves keeping a "counters" collection and incrementing the value each time you access it.
As a complete and self contained demonstration, you combine the techniques as follows:
var async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/warehouse');
var counterSchema = new Schema({
"type": { "type": String, "required": true },
"prefix": { "type": String, "required": true },
"counter": Number
});
counterSchema.index({ "type": 1, "prefix": 1 },{ "unique": true });
counterSchema.virtual('nextId').get(function() {
return this.prefix + this.counter;
});
var productSchema = new Schema({
"_id": "String",
"category": {
"object": { "type": String, "required": true },
"group": { "type": String, "required": true },
"name": { "type": String, "required": true }
}
},{ "_id": false });
productSchema.pre('save', function(next) {
var self = this;
if ( !self.hasOwnProperty("_id") ) {
var prefix = self.category.object.substr(0,1).toUpperCase()
+ self.category.group.substr(0,1).toUpperCase()
+ self.category.name.split(" ").map(function(word) {
return word.substr(0,1).toUpperCase();
}).join("");
mongoose.model('Counter').findOneAndUpdate(
{ "type": "product", "prefix": prefix },
{ "$inc": { "counter": 1 } },
{ "new": true, "upsert": true },
function(err,counter) {
self._id = counter.nextId;
next(err);
}
);
} else {
next(); // Just skip when _id is already there
}
});
var Product = mongoose.model('Product',productSchema),
Counter = mongoose.model('Counter', counterSchema);
async.series(
[
// Clean data
function(callback) {
async.each([Product,Counter],function(model,callback) {
model.remove({},callback);
},callback);
},
function(callback) {
async.each(
[
{
"category": {
"object": "ring",
"group": "movies",
"name": "lord of the rings"
}
},
{
"category": {
"object": "ring",
"group": "movies",
"name": "four weddings and a funeral"
}
},
{
"category": {
"object": "ring",
"group": "movies",
"name": "lord of the rings"
}
}
],
function(data,callback) {
Product.create(data,callback)
},
callback
)
},
function(callback) {
Product.find().exec(function(err,products) {
console.log(products);
callback(err);
});
},
function(callback) {
Counter.find().exec(function(err,counters) {
console.log(counters);
callback(err);
});
}
],
function(err) {
if (err) throw err;
mongoose.disconnect();
}
)
This gives you output like:
[ { category: { name: 'lord of the rings', group: 'movies', object: 'ring' },
__v: 0,
_id: 'RMLOTR1' },
{ category:
{ name: 'four weddings and a funeral',
group: 'movies',
object: 'ring' },
__v: 0,
_id: 'RMFWAAF1' },
{ category: { name: 'lord of the rings', group: 'movies', object: 'ring' },
__v: 0,
_id: 'RMLOTR2' } ]
[ { __v: 0,
counter: 2,
type: 'product',
prefix: 'RMLOTR',
_id: 57104cdaa774fcc73c1df0e8 },
{ __v: 0,
counter: 1,
type: 'product',
prefix: 'RMFWAAF',
_id: 57104cdaa774fcc73c1df0e9 } ]
To first understand the Counter schema and model, you are basically defining something where you are going to look up a "unique" key and also attach a numeric field to "increment" on match. For convenience this just has a two fields making up the unique combination and a compound index defined. This could just also be a compound _id if so wanted.
The other convenience is the virtual method of nextId, which just does a concatenation of the "prefix" and "counter" values. It's also best practice here to include something like "type" here since your Counter model can be used to service "counters" for use in more than one collection source. So here we are using "product" whenever accessing in the context of the Product model to differentiate it from other models where you might also keep a similar sequence counter. Just a design point that is worthwhile following.
For the actual Product model itself, we want to attach "pre save" middleware hook in order to fill the _id content. So after determining the character portion of the "prefix", the operation then goes off and looks for that "prefix" with the "product" type data in combination in the Counter model collection.
The function of .findOneAndUpdate() is to look for a document matching the criteria in the "counters" collection and then where a document is found already it will "increment" the current counter value by use of the $inc update operator. If the document was not found, then the "upsert" option means that a new document will be created, and at any rate the same "increment" will happen in the new document as well.
The "new" option here means that we want the "modified" document to be returned ( either new or changed ) rather than what the document looked like before the $inc was applied. The result is that "counter" value will always increase on every access.
Once that is complete and a document for Counter is either incremented or created for it's matching keys, then you now have something you can use to assign to the _id in the Product model. As mentioned earlier you can use the virtual here for convenience to get the prefix with the appended counter value.
So as long as your documents are always created by either the .create() method from the model or by using new Product() and then the .save() method, then the methods attached to your "model" in your code are always executed.
Note here that since you want this in _id, then as a primary key this is "immutable" and cannot change. So even if the content in the fields referenced was later altered, the value in _id cannot be changed, and therefore why the code here makes no attempt when an _id value is already set.
I've build a form in Angular to add GeoJSON features to MongoDB using Mongoose. The features include several properties and a GeometryCollection with a point and a lineString.
Here comes the trouble: I was able to create features with just a single point in my geometry but I'm unable to create features with a geometry collection that uses a lineString. I get either:
16755 Can't extract geo keys from object, malformed geometry?
or:
{ [CastError: Cast to number failed for value "0,0,1,1" at path "coordinates"]
message: 'Cast to number failed for value "0,0,1,1" at path "coordinates"',
name: 'CastError',
type: 'number',
value: [[0,0],[1,1]],
path: 'coordinates' }'
I do realize it says type: 'number' while my schema is set to an array of arrays:
var featureSchema = new mongoose.Schema({
'type': {
type: String,
default: "Feature"
},
geometry: {
'type': {
type: String,
default: 'GeometryCollection',
}, geometries: [{
'type': {
type: String,
default: 'Point'
},
coordinates: [Number]
}, {
'type': {
type: String,
default: 'LineString'
},
coordinates: {
type: [Array],
default: [[0,0], [1,1]]
}
}]
},
properties: {
title: String
}
});
So my first question is: does anyone know how to properly add features using GeometryCollections with Mongoose?
My second question is how to add an array of arrays when using forms? When I use a text input now I get my array of arrays delivered as a string. I was able to convert the point coordinates using:
var array = req.body.feature.geometry.geometries.coordinates.split(',');
for(var i=0; i<array.length; i++) {
array[i] = +array[i];
}
Is there a way to convert a string (ie "[ [0,0], [1,1] ]") to an array of arrays to create the lineString coordinates?
Thanks in advance!
The proper way is to split this into multiple schemas, which is much easier to read, use and maintain. For example:
GeoJSON.FeatureCollection = {
"type" : {
"type": String,
"default": "FeatureCollection"
},
"features": [GeoJSON.Feature]
}
GeoJSON.Feature = {
"id": {
"type": "String"
},
"type": {
"type": String,
"default": "Feature"
},
"properties": {
"type": "Object"
},
"geometry": GeoJSON.Geometry
}
GeoJSON.GeometryCollection = {
"type": {
"type": String,
"default": "GeometryCollection"
},
"geometries": [GeoJSON.Geometry]
}
GeoJSON.Geometry = {
"type": {
"type": String,
"enum": [
"Point",
"MultiPoint",
"LineString",
"MultiLineString",
"Polygon",
"MultiPolygon"
]
},
"coordinates": []
}
Taken from: https://github.com/RideAmigosCorp/mongoose-geojson-schema
I am using NodeJs and Mongoose and building a feature to list near by deals.
Deal.db.db.command({
"geoNear": Deal.collection.name,
"near": [23,67],
"spherical": true,
"distanceField": "dis"
}, function (err, documents) {
if (err) {
return next(err);
}
console.log(documents);
});
Deal schema:
var dealSchema = new mongoose.Schema({
title: String,
merchant: {
type: mongoose.Schema.Types.ObjectId,
ref: Merchant,
index: true
}
});
Here, I get all deals and their distances from current location. Inside Deal schema I have a merchant as reference object.
How do I populate merchants with each returned Deal object?
Do I need to iterate through all returned Deal objects and populate manually?
This actually turns out to be kind of interesting. Certainly what you do not want to really do is "dive into" the native driver part, though you possibly could on solution 2. Which leads to that there are a few ways to go about this without actually rolling your own queries to match this manually.
First a little setup. Rather than reproduce your dataset I have just picked something I had lying around from previous testing. The principles are the same, so a basic setup:
var mongoose = require('mongoose'),
async = require('async'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost');
var infoSchema = new Schema({
"description": String
});
var shapeSchema = new Schema({
"_id": String,
"amenity": String,
"shape": {
"type": { "type": String },
"coordinates": []
},
"info": { "type": Schema.Types.ObjectId, "ref": "Info" }
});
var Shape = mongoose.model( "Shape", shapeSchema );
var Info = mongoose.model( "Info", infoSchema );
Essentially one model with the "geo" information and the other that is referenced with just some information we want to populate. Also my lazy alteration of data:
{
"_id" : "P1",
"amenity" : "restaurant",
"shape" : { "type" : "Point", "coordinates" : [ 2, 2 ] }
}
{
"_id" : "P3",
"amenity" : "police",
"shape" : { "type" : "Point", "coordinates" : [ 4, 2 ] }
}
{
"_id" : "P4",
"amenity" : "police",
"shape" : { "type" : "Point", "coordinates" : [ 4, 4 ] }
}
{
"_id" : "P2",
"amenity" : "restaurant",
"shape" : { "type" : "Point", "coordinates" : [ 2, 4 ] },
"info" : ObjectId("539b90543249ff8d18e863fb")
}
So there is one thing in there that we can expect to populate. As it turns out a $near query is pretty straightforward as does sort as expected:
var query = Shape.find(
{
"shape": {
"$near": {
"$geometry": {
"type": "Point",
"coordinates": [ 2, 4 ]
}
}
}
}
);
query.populate("info").exec(function(err,shapes) {
if (err) throw err;
console.log( shapes );
});
That is just a standard .find() with the $near operator invoked. This is MongoDB 2.6 syntax, so there is that. But the results sort correctly and populate as expected:
[
{ _id: 'P2',
amenity: 'restaurant',
info:
{ _id: 539b90543249ff8d18e863fb,
description: 'Jamies Restaurant',
__v: 0 },
shape: { type: 'Point', coordinates: [ 2, 4 ] } },
{ _id: 'P4',
amenity: 'police',
info: null,
shape: { type: 'Point', coordinates: [ 4, 4 ] } },
{ _id: 'P1',
amenity: 'restaurant',
info: null,
shape: { type: 'Point', coordinates: [ 2, 2 ] } },
{ _id: 'P3',
amenity: 'police',
info: null,
shape: { type: 'Point', coordinates: [ 4, 2 ] } }
]
That is pretty nice, and a simple way to invoke. The catch? Sadly changing the operator to $geoNear to take the spherical geometry into account starts throwing errors. So if you want that then you can't just do things as you are used to.
Another approach though is that mongoose has a .geoNear() function that is supported. But just like the db.command invocation, this does not return a mongoose document or other Model type object that will accept .populate(). Solution? Just play with the output a little:
var query = Shape.geoNear({
"type": "Point",
"coordinates": [ 2, 4 ]
},{spherical: true},function(err,shapes) {
if (err) throw err;
shapes = shapes.map(function(x) {
delete x.dis;
var a = new Shape( x.obj );
return a;
});
Shape.populate( shapes, { path: "info" }, function(err,docs) {
if (err) throw err;
console.log( docs );
});
So here the result returned is an array of raw objects. But with a little manipulation you can turn these into something that is going to work with the .populate() method which can also be invoked from the model class as shown.
The result of course is the same, though the field order may be a little different. And you didn't need to iterate an call the queries yourself. This is really all that .populate() is actually doing, but I think we can agree that using the .populate() method at least looks a lot cleaner and does not re-invent the wheel for this purpose.
I know this question has been asked a lot of times but I'm kinda new to mongo and mongoose as well and I couldn't figure it out !
My problem:
I have a which looks like this:
var rankingSchema = new Schema({
userId : { type : Schema.Types.ObjectId, ref:'User' },
pontos : {type: Number, default:0},
placarExato : {type: Number, default:0},
golVencedor : {type: Number, default:0},
golPerdedor : {type: Number, default:0},
diferencaVencPerd : {type: Number, default:0},
empateNaoExato : {type: Number, default:0},
timeVencedor : {type: Number, default:0},
resumo : [{
partida : { type : Schema.Types.ObjectId, ref:'Partida' },
palpite : [Number],
quesito : String
}]
});
Which would return a document like this:
{
"_id" : ObjectId("539d0756f0ccd69ac5dd61fa"),
"diferencaVencPerd" : 0,
"empateNaoExato" : 0,
"golPerdedor" : 0,
"golVencedor" : 1,
"placarExato" : 2,
"pontos" : 78,
"resumo" : [
{
"partida" : ObjectId("5387d991d69197902ae27586"),
"_id" : ObjectId("539d07eb06b1e60000c19c18"),
"palpite" : [
2,
0
]
},
{
"partida" : ObjectId("5387da7b27f54fb425502918"),
"quesito" : "golsVencedor",
"_id" : ObjectId("539d07eb06b1e60000c19c1a"),
"palpite" : [
3,
0
]
},
{
"partida" : ObjectId("5387dc012752ff402a0a7882"),
"quesito" : "timeVencedor",
"_id" : ObjectId("539d07eb06b1e60000c19c1c"),
"palpite" : [
2,
1
]
},
{
"partida" : ObjectId("5387dc112752ff402a0a7883"),
"_id" : ObjectId("539d07eb06b1e60000c19c1e"),
"palpite" : [
1,
1
]
},
{
"partida" : ObjectId("53880ea52752ff402a0a7886"),
"quesito" : "placarExato",
"_id" : ObjectId("539d07eb06b1e60000c19c20"),
"palpite" : [
1,
2
]
},
{
"partida" : ObjectId("53880eae2752ff402a0a7887"),
"quesito" : "placarExato",
"_id" : ObjectId("539d0aa82fb219000054c84f"),
"palpite" : [
2,
1
]
}
],
"timeVencedor" : 1,
"userId" : ObjectId("539b2f2930de100000d7356c")
}
My question is, first: How can I filter the resumo nested document by quesito ? Is it possible to paginate this result, since this array is going to increase. And last question, is this a nice approach to this case ?
Thank you guys !
As noted, your schema implies that you actually have embedded data even though you are storing an external reference. So it is not clear if you are doing both embedding and referencing or simply embedding by itself.
The big caveat here is the difference between matching a "document" and actually filtering the contents of an array. Since you seem to be talking about "paging" your array results, the large focus here is on doing that, but still making mention of the warnings.
Multiple "filtered" matches in an array requires the aggregation framework. You can generally "project" the single match of an array element, but this is needed where you expect more than one:
Ranking.aggregate(
[
// This match finds "documents" that "contain" the match
{ "$match": { "resumo.quesito": "value" } },
// Unwind de-normalizes arrays as documents
{ "$unwind": "$resumo" },
// This match actually filters those document matches
{ "$match": { "resumo.quesito": "value" } },
// Skip and limit for paging, which really only makes sense on single
// document matches
{ "$skip": 0 },
{ "$limit": 2 },
// Return as an array in the original document if you really want
{ "$group": {
"_id": "$_id",
"otherField": { "$first": "$otherField" },
"resumo": { "$push": "$resumo" }
}}
],
function(err,results) {
}
)
Or the MongoDB 2.6 way by "filtering" inside a $project using the $map operator. But still you need to $unwind in order to "page" array positions, but there is possibly less processing as the array is "filtered" first:
Ranking.aggregate(
[
// This match finds "documents" that "contain" the match
{ "$match": { "resumo.quesito": "value" } },
// Filter with $map
{ "$project": {
"otherField": 1,
"resumo": {
"$setDifference": [
{
"$map": {
"input": "$resumo",
"as": "el",
"in": { "$eq": ["$$el.questio", "value" ] }
}
},
[false]
]
}
}},
// Unwind de-normalizes arrays as documents
{ "$unwind": "$resumo" },
// Skip and limit for paging, which really only makes sense on single
// document matches
{ "$skip": 0 },
{ "$limit": 2 },
// Return as an array in the original document if you really want
{ "$group": {
"_id": "$_id",
"otherField": { "$first": "$otherField" },
"resumo": { "$push": "$resumo" }
}}
],
function(err,results) {
}
)
The inner usage of $skip and $limit here really only makes sense when you are processing a single document and just "filtering" and "paging" the array. It is possible to do this with multiple documents, but is very involved as there is no way to just "slice" the array. Which brings us to the next point.
Really with embedded arrays, for paging that does not require any filtering you just use the $slice operator, which was designed for this purpose:
Ranking.find({},{ "resumo": { "$slice": [0,2] } },function(err,docs) {
});
Your alternate though is to simply reference the documents in the external collection and then pass the arguments to mongoose .populate() to filter and "page" the results. The change in the schema itself would just be:
"resumo": [{ "type": "Schema.Types.ObjectId", "ref": "Partida" }]
With the external referenced collection now holding the object detail rather than embedding directly in the array. The use of .populate() with filtering and paging is:
Ranking.find().populate({
"path": "resumo",
"match": { "questio": "value" },
"options": { "skip": 0, "limit": 2 }
}).exec(function(err,docs) {
docs = docs.filter(function(doc) {
return docs.comments.length;
});
});
Of course the possible problem there is that you can no longer actually query for the documents that contain the "embedded" information as it is now in another collection. This results in pulling in all documents, though possibly by some other query condition, but then manually testing them to see if they were "populated" by the filtered query that was sent to retrieve those items.
So it really does depend on what you are doing and what your approach is. If you regularly intend to "search" on inner arrays then embedding will generally suit you better. Also if you really only interesting in "paging" then the $slice operator works well for this purpose with embedded documents. But beware growing embedded arrays too large.
Using a referenced schema with mongoose helps with some size concerns, and there is methodology in place to assist with "paging" results and filtering them as well. The drawback is that you can no longer query "inside" those elements from the parent itself. So parent selection by the inner elements is not well suited here. Also keep in mind that while not all of the data is embedded, there is still the reference to the _id value of the external document. So you can still end up with large arrays, which may not be desirable.
For anything large, consider that you will likely be doing the work yourself, and working backwards from the "child" items to then match the parent(s).
I am not sure that you can filter sub-document directly with mongoose. However you can get the parent document with Model.find({'resumo.quesito': 'THEVALUE'}) (you should also and an index on it)
Then when you have the parent you can get the child by comparing the quesito
Additionnal doc can be found here: http://mongoosejs.com/docs/subdocs.html