I'm trying to display a forum/category. I need to get the latest posts. The problem is that I also need data on the user for each post as well as the latest reply.
db.post.find({
"inForum": forumID,
},
{
'sort': [['date', -1]]
},
function(err, cursor) {
cursor.count(function(err, count) {
cursor.skip(skip).limit(20).toArray(function(err, posts) {
var complete = _.after(nodes.length, function () {
res.send(posts)
});
// for every post get its author info and the latest post info
posts.forEach(function (post) {
var users = _.pluck(posts, 'user');
user.load(users, function (profiles) {
_.each(posts,
function(post, k) {
if (profiles[post.user]) post.fieldAvatar = profiles[post.user].fieldAvatar;
});
if (post.latestReply) {
post.load(post.latestReply.id, function (latestReply) {
if (latestReply) post.latestReply = latestReply
complete()
})
}
else {
complete()
}
})
})
});
})
})
This is what I'm doing and it seems really slow / really inelegant to me. Am I doing this correctly and is there any advice for speeding this up?
Thanks.
The best thing you should do here is to embed some information for the author of the posts (username & email or avatar) into those posts so that you don't make multiple queries to the database, one should suffice (sure you have some duplicate data, but the performance is optimal).
If you don't want to / can't do that you can also modify your second query to find all authors in [array_of_ids_of_the_posts]. That would reduce your [number_of_posts] queries into only one.
You could use some caching. For example you could save the users in an dictionary during the loop so you only have to fetch it on the first occurrence from mongodb.
Maybe you could create some kind of thread model where you save basic information about the containing posts, so you only have to go through the threads.
You could save the result of the function and delete it when a new post is added .. so won't go through all posts on every call.
You should not use a document storage like a sql database. Maybe it is better to generate the forum page directly when a post is created/edited and save the whole data in a document, so you only have to make one read call to mongo to show it.
Related
I want to access resources from a SQLite database table. I have one table for accounts, one for movies and one for reviews. The reviews-table is constructed like this:
CREATE TABLE IF NOT EXISTS reviews (
id INTEGER PRIMARY KEY,
authorId INTEGER,
movieId INTEGER,
review TEXT,
rating INTEGER,
FOREIGN KEY('authorId') REFERENCES 'accounts'('id'),
FOREIGN KEY('movieId') REFERENCES 'movies'('id')
)
What I want to do is that I want to be able to get all reviews, made by one author. But I also want to be able to get all reviews, made for the same movie. Below is my code for getting all reviews made by the same user/author. The code looks the same when getting the reviews based on the movie, with a few changes.
Both of them does what I want them to do. But of course only the one written first in the file are running.
const authorId = request.params.authorId;
const query = "SELECT * FROM reviews WHERE authorId = ?";
const values = [authorId];
db.all(query, values, function (error, review) {
if (error) {
response.status(500).end();
} else if (!review) {
response.status(404).end();
} else {
response.status(200).json(review);
}
});
});
The url will look the same no matter which of them I want running; http://localhost:3000/reviews/3. How can I differentiate the url so that they know which one should run?
I have tried to experiment with query strings, but I'm not sure how that works, and after hours of searching for something that worked on my code, I gave up.
I have also been thinking about using something like
app.use(function (request, response, next) {
if (request.method == "GET" && request.uri == "/reviews/:authorId") {
response.send("Hello World");
} else {
next;
}
});
This didn't work, and it didn't work if I tried to remove ":authorId" from the url either. The page just keeps loading.
So how do I solve this?
The most dynamic would be a single route /reviews and use the query string with the params like ?author=123 or ?movie=123, they can be combined like ?author=123&movie=123. As you want to return JSON the code will be used via API, so the pretty path is usually not as important as when it is a web-url. To make the implementation effective, most people use a function where you can drop the query object in and get the where-clause, or use an ORM.
In express, when you have routers like '/reviews/:authorId' and then '/reviews/:movieId', then the second one will never be called, because the first one will always match. That is something to be careful about when organizing your express routes.
I have an app that uses firebase, the whole stack pretty much, functions, database, storage, auth, messaging, the whole 9. I want to keep the client end very lightweight. So if a user comments on a post and "tags" another user, let's say using the typical "#username" style tagging, I moved all of the heavy lifting to the firebase functions. That way the client doesn't have to figure out the user ID based on the username, and do everything else. It is setup using triggers, so when the above scenario happens I write to a "table" called "create_notifications" with some data like
{
type: "comment",
post_id: postID,
from: user.getUid(),
comment_id: newCommentKey,
to: taggedUser
}
Where the taggedUser is the username, the postID is the active post, the newCommentKey is retrieved from .push() on the comments db reference, and the user.getUid() is from the firebase auth class.
Now in my firebase functions I have a "onWrite" trigger for that specific table that gets all of the relevant information and sends out a notification to the poster of the post with all the relevant details. All of that is complete, what I am trying to figure out is... how do I delete the incoming event, that way I don't need any sort of cron jobs to clear out this table. I can just grab the event, do my needed calculations and data gathering, send the message, then delete the incoming event so it never even really exists in the database except for the small amount of time it took to gather the data.
A simplified sample of the firebase functions trigger is...
exports.createNotification = functions.database.ref("/create_notifications/{notification_id}").onWrite(event => {
const from = event.data.val().from;
const toName = event.data.val().to;
const notificationType = event.data.val().type;
const post_id = event.data.val().post_id;
var comment_id, commentReference;
if(notificationType == "comment") {
comment_id = event.data.val().comment_id;
}
const toUser = admin.database().ref(`users`).orderByChild("username").equalTo(toName).once('value');
const fromUser = admin.database().ref(`/users/${from}`).once('value');
const referencePost = admin.database().ref(`posts/${post_id}`).once('value');
return Promise.all([toUser, fromUser, referencePost]).then(results => {
const toUserRef = results[0];
const fromUserRef = results[1];
const postRef = results[2];
var newNotification = {
type: notificationType,
post_id: post_id,
from: from,
sent: false,
create_on: Date.now()
}
if(notificationType == "comment") {
newNotification.comment_id = comment_id;
}
return admin.database().ref(`/user_notifications/${toUserRef.key}`).push().set(newNotification).then(() => {
//NEED TO DELETE THE INCOMING "event" HERE TO KEEP DB CLEAN
});
})
}
So in that function in the final "return" of it, after it writes the finalized data to the "/user_notifications" table, I need to delete the event that started the whole thing. Does anyone know how to do that? Thank you.
First off, use .onCreate instead of .onWrite. You only need to read each child when they are first written, so this will avoid undesirable side effects. See the documentation here for more information on the available triggers.
event.data.ref() holds the reference where the event occurred. You can call remove() on the reference to delete it:
return event.data.ref().remove()
The simplest way to achieve this is through calling the remove() function offered by the admin sdk,
you could get the reference to the notification_id through the event, i.e event.params.notification_id then remove it when need be with admin.database().ref('pass in the path').remove(); and you are good to go.
For newer versions of Firebase, use:
return change.after.ref.remove()
I am building a web application, and I am spending so long time to take care of updates between related documents.
For example, I have 'Task' document and 'User' document. When task is made, multiple users will be assigned to it. Thus,
taskA.assigned = ["1321231fsdfsdf"(userA's _id), "12312313asdasdasd"(userB's _id)]
userA.tasks = [..., "1231321"(taskA's _id),...]
userB.tasks = [..., "12313211"(taskB's _id),...]
I could handle it well when it comes to just creating tasks. However, it becomes too tricky when I am going to edit tasks. If user B is deleted from taskA, I have to delete userB's id and go to the userB's tasks property and delete taskA's id too.
Is there any shortcut and automatic way to deal with it? Thank you for your time to read it. Let me know if I was too vague, I will add more detail.
In a relational database like MySQL using foreign keys and cascade updates could be done automatically, but in MongoDB that's not possible.
But I see in the tags you are using Moongose, so using a post save hook could do the trick. You can set a hook that updates automatically the user collection each time a task is updated, or viceversa.
Other option would be changing your data estructure, but this depends on your case, there are some facts to take into account. I think we don't have enough information to judge, but there are many resources speaking about data normalization in MongoDB, you can check for example the official MongoDB manual.
I found a way to do this easily and with less lines of codes. Around 100 lines reduced to around 30 lines by doing this.
Long story short, I used 'update()' method and various mongo operators, such as $in, $push, or $pull.
Here is my final codes that are optimized with use of update method.
var edit = req.body;
edit.assignedTo = edit.assignedTo.split(',');
var old = req.task;
var idQueries = edit.assignedTo.map(function (x) {
return mongoose.Types.ObjectId(x);
});
User.update({tasks: old._id}, {$pull: {tasks: old._id}}, {multi: true}, function () {
// Update to remove original task's id from users assigned to it.
User.update({_id: {$in: idQueries}}, {$push: {tasks: old._id}}, {multi: true}, function () {
// Update to add edited tasks'id to new users assigned to it.
old.lastAction = 'edited';
old.edited = true;
old.editedAt = Date.now();
old.titke = edit.title;
old.desc = edit.desc;
old.dueBy = edit.dueBy;
old.assignedTo = edit.assignedTo;
old.save(function (err, task) {
if (err) return next(err);
User.populate(task, 'assignedTo', function (err, task) {
res.json(task);
});
});
});
});
Wish this help some people!
I have a problem I can't find an answer to in Loopback's docs.
Say I have a model Company and a modelEmployee. There is an 1Xn relation between the Company and its Employees. When /api/Employees is called, server returns all the employees.
I only want to return the list of employees who are in the same company with the user requesting the list.
For this, I created a remote hook
Employee.beforeRemote('find', function(context, modelInstance, next) {
var reject = function() {
process.nextTick(function() {
next(null, false);
});
};
// do not allow anonymous users
var userId = context.req.accessToken.userId;
if (!userId) {
return reject();
}
//I get the details of the user who sent the request
//to learn which company does he belong to
Employee.findById(userId, function(err, user) {
if(!context.req.query.filter) context.req.query.filter={};
context.req.query.filter.where = {brandId:user.companyId};
console.log(context.req.query);
next();
});
});
I thought this should work every time, but appearantly it only works when find already has some query filters like include - although the console.log prints a correct context.req.query object.
What am I missing? Any help would be greatly appreciated!
context.args.filter seems to work for this purpose.
As a side note, instead of replacing where, you might want to merge it with something provided by client. For implementation idea you can refer to: https://github.com/strongloop/loopback-datasource-juggler/blob/master/lib/utils.js#L56-L122
I was searching for an easy and simple database for a little highscore system for a some games I'm developing in javascript.
I saw Orchestrate.io in github's student developer pack. I found a suitable drivermodule nodejs orchestrate and have integrated them.
The problem comes with querying orchestrate for my data. I have managed saving scores and querying them with db.list('collection'), but this seems to not responding with all data. It appered to me that some values are not returned.
I read about the db.search('collection','query') function. But I don't really understand how I could return all data because I don't want to query in a specific way.
My objects are as simple as follows:
{"name":"Jack","score":1337}
As I understand, one has to send a key, when putting such values to an orchestrate-collection. But I'd like to query the whole collection and get the values in a descendant order in regard to the score.
As for now I end up sorting the result on the client-side.
I hope you guys can give me some hints for a query that can sort for specific values!
You have the option to use a SearchBuilder
db.newSearchBuilder() //Build a search object
.collection('collection') //Set the collection to be searched
.sort(score, 'desc') //Set the order of the results
.query("*") //Empty search
.then(function (res) { //Callback function for results
//Do something with the results
})
Source
By default, .list uses a pagination limit of 10. You can either increase that, e.g.:
db.list('collection', { limit: 100 })
Or use .links, .links.next (from the docs):
db.list('collection', { limit: 10 })
.then(function (page1) {
// Got First Page
if (page1.links && page1.links.next) {
page1.links.next.get().then(function (page2) {
// Got Second Page
})
}
})