single query for fetching data from multiple collection in Mongodb/Mongoose - javascript

In a Node.js App, i use Mongodb/Mongoose. MongoDB doesn't has Joins, and this is a big problem for me. because i store data in separate collections. i have two collections. Users and Books. i need a query like this:
SELECT Books.name FROM Users,Books WHERE Users.name='john' and Books.lang='en'
im sure that we cannot do this by one single Mongoose query. Mongoose populate has also lots of problems. for example if i do this:
Users.find({user_name:'john'}).pupulate({path:'books', match:{lang:'en'}})
and if i change this code to this, it gets to another problem:
Users.find({user_name:'john'}).pupulate({path:'books', match:{lang:'en'}}).limit(6)
it fetches first 6 users item and then populate books, if there no population then result will be null. i need fetch first 6 item that belong to an specific user and has an english book.
With this code, All users with name of john will be fetched and a few of them has a book with english language. imagine users with name john is 1 million and one of them has an english book! we have to fetch that huge data and with a secondary block of code, filter that one item. its not clear and good way.
is there any other way? i has a lot of search about this but no good result. in this scenario what annoys me is fetching all users and then filter it again. this can be an slow architecture. whats the best way to me? what can i do.
i'm wondering a database engine that claims to be an enterprise one has this disadvantage. other people doesn't has this problem? there is something i'm missing?

Related

How to create a search results web page and store a large list of items without SQLDatabase

I made a web page for selling items online. The website has a lot of products but will probably have multiple thousand products in the near future. The website contains a search bar and I want to create a search results page, but I am not sure what the best way of doing this is. I thought about using JavaScript to loop through the list of all the products until it finds a match. But this process is probably too slow. My question is: What is the best way to store a large list of items, and what is the best way to find matches from the list for the search query? I now that many people use SQL databases for storing lists but is that method any better than simply storing everything in a JavaScript list, and why? Also how do I find a match in the list? Can I use JavaScript or is it necessary or better to use a language like PHP?

how to make relation for no sql database?

how to make relation for no sql database?
You can say it for firebase database, where database is in json
format.
A NO SQL database means that database has no relations. Either go with SQL database and convert JSON format or face the truth that in NOSQL there are no relations.
Before anything, I will give you an example of how to try to do this, but I want to tell you that I wouldn't actually do this again. Firebase is no good replace for MySQL.
Making relations for a noSQL database is not possible, but you could always make them "manually" and decide how to work with it.
What I meant with "manually" is that you can duplicate data for that, but that's not a very good option. For example, I made long ago an Android app to manage neightbour communities, and because of the time I had to do it I decided to make it with Firebase.
And I will never do it again, to be honest. I didn't want to lose time on an API, but I lost it anyway trying to structure everything nicely and with all the changes I had to make every 2 days so everything wouldn't fail.
Here you have an example. The database has 2 nodes, the communities and the users.
The users have these fields:
And, meanwhile, the communities have an incidences list, and those store the email of its author as any other field (image not relatable, they are random ones).
So, TLDR: No, you can't make relations. The only way to do what you want is with duplicated data, like the email of its author on the incidence.
PS: I made a chat app with Firebase in my company, I would share the DB with you so you can see the structure, but it's confidential, you know.

Caching query results, to do or not to do, overkill or performance energizer?

Good evening,
my project uses the MEAN Stack and has a few collections and a single database from which the data is retrieved.
Thinking about how the user would interface itself with the webapp I am going to build, I figured that my idea of the application is quite a bit of a waste.
Now, the application is hosted on a private server on the LAN, making it very fast on requests and it's running an express server.
The application is made around employee management, services and places where the services can take place. Just describing, so to have an idea.
The "ring to rule them all" is pretty much the first collection, services, which starts the core of the application. There's a page that let's you add rows, one for each service that you intend to do and within that row you choose an employee to "run the service", based on characteristics that this employee has, meaning that if the service is about teaching Piano, the employee must know how to play Piano. The same logic works for the rest of the "columns" that will build up my row into a full service recognized by the app as such.
Now, what I said above is pretty much information retrieval from a database and logic to make the application model the information retrieved and build something with it.
My question or rather my doubt comes from how I imagined the querying to work for each field that is part of the service row. Right now I'm thinking about querying the database (mongodb) each time I have to pick a value for a field, but if you consider that I might want to add a 100 rows, each of which would have 10 fields, that would make up for a lot of requests to the database. Now, that doesn't seem elegant, nor intelligent to me, but I can't come up with a better solution or idea.
Any suggestions or rule of thumbs for a MEAN newb?
Thanks in advance!
EDIT: Answer to a comment question which was needed.
No, the database is pretty static (unless the user willingly inserts a new value, say a new employee that can do a service). That wouldn't happen very often. Considering the query that would return all the employees for a given service, those employees would (ideally) be inside an associative array, with the possibility to be "pop'd" from it if chosen for a service, making them unavailable for further services (because one person can't do two services at the same time). Hope I was clear, I'm surely not the best person at explaining oneself.
It would query the database on who is available when a user looks at that page and another query if the user assigns an employee to do a service.
In general 1 query on page load and another when data is submitted is standard.
You would only want to use an in memory cache for
frequent queries but most databases will do this automatically.
values that change frequently like:
How many users are connected
Last query sent
Something that happens on almost every query (>95%)

NodeJS, express and mongoDB - mapping logic

Nothing important in this paragraph - I am still new to programming and getting the hang of nodejs, so for this question I am only asking for someone to explain the logical process I should take. I can figure it out from there and I feel as though this would be a great example for beginners like myself so I will post the code for the final solution.
The Situation
-I have one mongo database containing customers(there are 100's of customers. It has 3 fields (First, Last, Age)
-The Customers receive a monthly service. Everyday the company assigns to each of their small number of employees a random list of these customers to take care of.
-I have already created a second Schema in the mongo database that has four fields; the employee assigned to the daily list, the date, an array filled with the ID fields of many customers, and a Boolean value (employee, date, array, boolean) - {i need the second schema for archive purposes}
The problem -
I need to query the employee list (2nd) database for incomplete lists( aka false boolean values); Then create, show or active a link to a view for each of the incomplete lists, the individual views will be populated by querying the customer database(1st database) by the arguement of the ID fields that I retrieve from the array in the 2nd database. And I need to create that link and populate the view for each of incomplete lists. I am using NodeJS, Express and Jade, but like i said in the paragraph you skipped if you could just map out the logic I can get started and will post my final result.
Thank You for any attempts. It is now almost 9pmEST, I will be monitoring my post until atleast 11pmEST if anyone needs any clarification. and again tomorrow morning
-Steven R
Having what I understand, is that they have 2 databases. Parese to me a little more simple to do. Use ObjectId with a field "boolesean" or numeric, this field should modify or change the connection to the database. So solve the problem of having multiple databases and you can relate. If you use ObjectId, you can do research in other collections and in this case in other databases. use Mongoose That will help you build better databases, and search on different levels.

Denormalization of data in MongoDB

I am learning MongoDB and I have a question regarding duplication of data. In the SQL world you try to normalize the data. For instance I have a table with categories and another one with products. Each product may belong to many categories so there is a join between these tables.
However am I right that in MongoDB you don't think like this? Does instead each product have a embedded document(s) of categories? Is that just the way it is? You don't care if the data is duplicated?
In the SQL world you try to normalize the data
Not always, normalising to the point of death inflicts performance hits but it is true that I personally do not apply the same normalisation to MongoDB as I do SQL.
If you are aware of the normalised forms ( http://en.wikipedia.org/wiki/Database_normalization ) I like to think MongoDB as going to 1NF and then back down to denormalised again.
You don't care if the data is duplicated?
Oh yes we do. Updating is a pain if the data is duplicated wrong.
Let me give you an example: category and product would be two separate entities, there is no denying it. These two entities are normalised (the repeating data of product has been spearated from category). Another way of thinking of it is: Are all products only going to exist in one category?
So on top level entities, as you can see, the same rules relatively apply with 1NF easily being applied to MongoDB.
On the front of duplication you, of course, would not want to store each product separately within each category (I answered no to the question above) so you would naturally want to separate catgeories and products.
You would normally have a many-to-many relationship here with a middle normalised table. This is where de-normalisation can come in. You can say that a category will have a list of products that are unique to that category as such you could de-normalise the many-to-many relational table into the category row as a list (or the other way around into the product row). This will not generate duplication since that list is unique to that category (more than likely). This of course means that the category or products would house a list _ids of the related row instead of the object itself.
There are times where duplication is nessecary, mainly for optimisation or work arounds for not having JOINs; this rule also applies to SQL as well if you have ever done a big enough site.
Typical usage scenarios of duplication is aggregation fields of stats like a Facebook posts shares and comments and maybe even the 5 latest comments of that post would also be duplicated onto the post row.
So it is not a case of ignoring schema design but more of tuning it for MongoDBs characteristics. Normally if you do that you will find that you, naturally, design a good schema.
As an added reference you can refer here: http://docs.mongodb.org/manual/core/data-modeling

Categories

Resources