I'm using bluebird and mongoose, I'm getting an array of users in a previous function. I want to update users to set the property hasCar true or false for each user from my array.
function findCars(users) {
return Promise.all(users.map((user) => {
return Car.count({
_creator: new ObjectId(req.user),
userId: user._id
})
.then((car) => {
user.hasCar = !!car;
return user;
});
}))
.then(console.log);
}
The problem is, in the console.log those users don't have the property hasCar set. I tried different ways with no success.
Found what was wrong, it seems you can't add a property which does not exist in my mongoose model User. To bypass it I did:
let updatedUser = user.toObject();
updatedUser.hasCar = !!car;
return updatedUser;
Thanks everyone
It seems to work as it is. I made some simple changes to fake out Car.count and removed ObjectId, and it works as expected:
const cars = {1: ['car a', 'car b'], 3: ['car c']};
const Car = {
count: (user) => new Promise((resolve, reject) =>
setTimeout(() => resolve(cars[user.userId]), 1000))
};
function findCars(users) {
return Promise.all(users.map((user) =>
Car.count({
userId: user._id
})
.then((car) => {
user.hasCar = !!car;
return user;
})
))
.then(console.log);
}
const users = [{_id: 1}, {_id: 2}, {_id: 3}];
findCars(users);
This returns a Promise, and one second later logs the following to the console:
[{"_id":1,"hasCar":true},{"_id":2,"hasCar":false},{"_id":3,"hasCar":true}]
Note, I originally used the simpler
count: (user) => Promise.resolve(cars[user.userId])
But I wanted to ensure some true async. Neither way had any issues.
You can see this in action on RunKit.
Related
Im trying to figure this out.
I want to get all my users from my database, cache them
and then when making a new request I want to get those that Ive cached + new ones that have been created.
So far:
const batchUsers = async ({ user }) => {
const users = await user.findAll({});
return users;
};
const apolloServer = new ApolloServer({
schema,
playground: true,
context: {
userLoader: new DataLoader(() => batchUsers(db)),// not sending keys since Im after all users
},
});
my resolver:
users: async (obj, args, context, info) => {
return context.userLoader.load();
}
load method requiers a parameter but in this case I dont want to have a specific user I want all of them.
I dont understand how to implement this can someone please explain.
If you're trying to just load all records, then there's not much of a point in utilizing DataLoader to begin in. The purpose behind DataLoader is to batch multiple calls like load(7) and load(22) into a single call that's then executed against your data source. If you need to get all users, then you should just call user.findAll directly.
Also, if you do end up using DataLoader, make sure you pass in a function, not an object as your context. The function will be ran on each request, which will ensure you're using a fresh instance of DataLoader instead of one with a stale cache.
context: () => ({
userLoader: new DataLoader(async (ids) => {
const users = await User.findAll({
where: { id: ids }
})
// Note that we need to map over the original ids instead of
// just returning the results of User.findAll because the
// length of the returned array needs to match the length of the ids
return ids.map(id => users.find(user => user.id === id) || null)
}),
}),
Note that you could also return an instance of an error instead of null inside the array if you want load to reject.
Took me a while but I got this working:
const batchUsers = async (keys, { user }) => {
const users = await user.findAll({
raw: true,
where: {
Id: {
// #ts-ignore
// eslint-disable-next-line no-undef
[op.in]: keys,
},
},
});
const gs = _.groupBy(users, 'Id');
return keys.map(k => gs[k] || []);
};
const apolloServer = new ApolloServer({
schema,
playground: true,
context: () => ({
userLoader: new DataLoader(keys => batchUsers(keys, db)),
}),
});
resolver:
user: {
myUsers: ({ Id }, args, { userLoader }) => {
return userLoader.load(Id);
},
},
playground:
{users
{Id
myUsers
{Id}}
}
playground explained:
users basically fetches all users and then myusers does the same thing by inhereting the id from the first call.
I think I choose a horrible example here since I did not see any gains in performence by this. I did see however that the query turned into:
SELECT ... FROM User WhERE ID IN(...)
I am trying to push a user's choice as a string to their array of choices and return the updated document.
The route and function work successfully however it returns the User with an empty choice array. I believe the problem lies somewhere in the controller function but I cannot figure it out.
Any help is greatly appreciated!
To help, here is a screenshot of my console where you can see an empty choice array being returned.
Here is an image of my console.log
This is where I call the function
handleAnswerInput = (question) => {
let answerTextSelected = question.Text;
let answerTypeSelected = question.Type;
let usersName = this.state.user._id
this.setState({
count: this.state.count + 1
})
saveUserandScore(usersName, answerTextSelected)
.then(
this.loadQuestion(this.state.count)
)
console.log(answerTextSelected)
console.log(answerTypeSelected)
};
This is the controller function (updated from suggestions)
const saveUserAndTheirScore = (req, res) => {
let filter = { _id: req.params.id }
// let update = { choices: req.params.answer] }
console.log(req.params.id)
console.log(req.params.answer)
User.update(
{ filter },
{
$push: { choices: req.params.answer }
},
{
returnOriginal: false,
},
)
.then(dbData => res.json(dbData))
.catch(err => {
console.log(err);
res.json(err);
});
};
here is the axios call
export const saveUserandScore = (id, answer) => {
return axios.post(`/api/user/${id}/${answer}`);
};
you need to change user schema, in that you might have defined choices type as string. It must be an array.
findOneAndUpdate(filter, update, options, callback) has a returnOriginal option, if set to true (which is the default), it will return the document BEFORE the update. In your case, you might want to set it to false [1].
Unfortunately, the respective option for mongoose is named new [2].
[1] https://mongodb.github.io/node-mongodb-native/3.4/api/Collection.html#findOneAndUpdate
[2] https://mongoosejs.com/docs/api.html#query_Query-findOneAndUpdate
I'm trying to push a value inside a const but its in a .then and it's not working do you know how can I do that ?
I get a value in my console.log(newResult) in my if but the data is not pushed in my const newResult in the return
res.status(200).json(newResult);
.then(function (friends) {
if (friends) {
const newResult = [];
friends.forEach((r) => {
if (r.UserID == userFound.id) {
models.User.findOne({
where: {
id: r.idFriend
}
})
.then(function(userFound) {
newResult.push({
id: r.id,
user: {
id: r.User.id,
email: userFound.email,
username: userFound.username
}
});
console.log(newResult)
})
} else
newResult.push({
id: r.id,
user: {
id: r.User.id,
email: r.User.email,
username: r.User.username
}
});
console.log(newResult)
});
res.status(200).json(newResult);
}
}
every test realised return an empty tab when i go in my if condition
It will never work because, you are doing async calls
models.User.findOne inside forEach.
You'll get results on console.log when async call to database for fetching user is complete.
But before this all happens the forEach is done executing and code hits the line res.status(200).json(newResult); and you see no results from your if condition.
Instead of using this approach go for mongoose populate and populate userObject based userID while finding friends this way you won't have to do async call inside the forEach.
Read about mongoose populate at: http://mongoosejs.com/docs/populate.html
I'd like to perform a batch update using Knex.js
For example:
'UPDATE foo SET [theValues] WHERE idFoo = 1'
'UPDATE foo SET [theValues] WHERE idFoo = 2'
with values:
{ name: "FooName1", checked: true } // to `idFoo = 1`
{ name: "FooName2", checked: false } // to `idFoo = 2`
I was using node-mysql previously, which allowed multiple-statements. While using that I simply built a mulitple-statement query string and just send that through the wire in a single run.
I'm not sure how to achieve the same with Knex. I can see batchInsert as an API method I can use, but nothing as far as batchUpdate is concerned.
Note:
I can do an async iteration and update each row separately. That's bad cause it means there's gonna be lots of roundtrips from the server to the DB
I can use the raw() thing of Knex and probably do something similar to what I do with node-mysql. However that defeats the whole knex purpose of being a DB abstraction layer (It introduces strong DB coupling)
So I'd like to do this using something "knex-y".
Any ideas welcome.
I needed to perform a batch update inside a transaction (I didn't want to have partial updates in case something went wrong).
I've resolved it the next way:
// I wrap knex as 'connection'
return connection.transaction(trx => {
const queries = [];
users.forEach(user => {
const query = connection('users')
.where('id', user.id)
.update({
lastActivity: user.lastActivity,
points: user.points,
})
.transacting(trx); // This makes every update be in the same transaction
queries.push(query);
});
Promise.all(queries) // Once every query is written
.then(trx.commit) // We try to execute all of them
.catch(trx.rollback); // And rollback in case any of them goes wrong
});
Assuming you have a collection of valid keys/values for the given table:
// abstract transactional batch update
function batchUpdate(table, collection) {
return knex.transaction(trx => {
const queries = collection.map(tuple =>
knex(table)
.where('id', tuple.id)
.update(tuple)
.transacting(trx)
);
return Promise.all(queries)
.then(trx.commit)
.catch(trx.rollback);
});
}
To call it
batchUpdate('user', [...]);
Are you unfortunately subject to non-conventional column names? No worries, I got you fam:
function batchUpdate(options, collection) {
return knex.transaction(trx => {
const queries = collection.map(tuple =>
knex(options.table)
.where(options.column, tuple[options.column])
.update(tuple)
.transacting(trx)
);
return Promise.all(queries)
.then(trx.commit)
.catch(trx.rollback);
});
}
To call it
batchUpdate({ table: 'user', column: 'user_id' }, [...]);
Modern Syntax Version:
const batchUpdate = (options, collection) => {
const { table, column } = options;
const trx = await knex.transaction();
try {
await Promise.all(collection.map(tuple =>
knex(table)
.where(column, tuple[column])
.update(tuple)
.transacting(trx)
)
);
await trx.commit();
} catch (error) {
await trx.rollback();
}
}
You have a good idea of the pros and cons of each approach. I would recommend a raw query that bulk updates over several async updates. Yes you can run them in parallel, but your bottleneck becomes the time it takes for the db to run each update. Details can be found here.
Below is an example of an batch upsert using knex.raw. Assume that records is an array of objects (one obj for each row we want to update) whose values are the properties names line up with the columns in the database you want to update:
var knex = require('knex'),
_ = require('underscore');
function bulkUpdate (records) {
var updateQuery = [
'INSERT INTO mytable (primaryKeyCol, col2, colN) VALUES',
_.map(records, () => '(?)').join(','),
'ON DUPLICATE KEY UPDATE',
'col2 = VALUES(col2),',
'colN = VALUES(colN)'
].join(' '),
vals = [];
_(records).map(record => {
vals.push(_(record).values());
});
return knex.raw(updateQuery, vals);
}
This answer does a great job explaining the runtime relationship between the two approaches.
Edit:
It was requested that I show what records would look like in this example.
var records = [
{ primaryKeyCol: 123, col2: 'foo', colN: 'bar' },
{ // some other record, same props }
];
Please note that if your record has additional properties than the ones you specified in the query, you cannot do:
_(records).map(record => {
vals.push(_(record).values());
});
Because you will hand too many values to the query per record and knex will fail to match the property values of each record with the ? characters in the query. You instead will need to explicitly push the values on each record that you want to insert into an array like so:
// assume a record has additional property `type` that you dont want to
// insert into the database
// example: { primaryKeyCol: 123, col2: 'foo', colN: 'bar', type: 'baz' }
_(records).map(record => {
vals.push(record.primaryKeyCol);
vals.push(record.col2);
vals.push(record.colN);
});
There are less repetitive ways of doing the above explicit references, but this is just an example. Hope this helps!
The solution works great for me! I just include an ID parameter to make it dynamic across tables with custom ID tags. Chenhai, here's my snippet including a way to return a single array of ID values for the transaction:
function batchUpdate(table, id, collection) {
return knex.transaction((trx) => {
const queries = collection.map(async (tuple) => {
const [tupleId] = await knex(table)
.where(`${id}`, tuple[id])
.update(tuple)
.transacting(trx)
.returning(id);
return tupleId;
});
return Promise.all(queries).then(trx.commit).catch(trx.rollback);
});
}
You can use
response = await batchUpdate("table_name", "custom_table_id", [array of rows to update])
to get the returned array of IDs.
The update can be done in batches, i.e 1000 rows in a batch
And as long as it does it in batches, the bluebird map could be used.
For more information on bluebird map: http://bluebirdjs.com/docs/api/promise.map.html
const limit = 1000;
const totalRows = 50000;
const seq = count => Array(Math.ceil(count / limit)).keys();
map(seq(totalRows), page => updateTable(dbTable, page), { concurrency: 1 });
const updateTable = async (dbTable, page) => {
let offset = limit* page;
return knex(dbTable).pluck('id').limit(limit).offset(offset).then(ids => {
return knex(dbTable)
.whereIn('id', ids)
.update({ date: new Date() })
.then((rows) => {
console.log(`${page} - Updated rows of the table ${dbTable} from ${offset} to ${offset + batch}: `, rows);
})
.catch((err) => {
console.log({ err });
});
})
.catch((err) => {
console.log({ err });
});
};
Where pluck() is used to get ids in array form
I am building a website using Node.js and Sequelize (with a Postgres backend). I have a query that returns many objects with a foreign key, and I want to pass to the view a list of the objects that the foreign key references.
In the example, Attendances contains Hackathon keys, and I want to return a list of hackathons. Since the code is async, the following thing of course does not work in Node:
models.Attendance.findAll({
where: {
UserId: req.user.id
}
}).then(function (data) {
var hacks = [];
for (var d in data) {
models.Hackathon.findOne({
where: {
id: data[d].id
}
}).then(function (data1) {
hacks.append(data1);
});
}
res.render('dashboard/index.ejs', {title: 'My Hackathons', user: req.user, hacks: hacks});
});
Is there any way to do that query in a synchronous way, meaning that I don't return the view untill I have the "hacks" list filled with all the objects?
Thanks!
Use Promise.all to execute all of your queries then call the next function.
models.Attendance.findAll({
where: {
UserId: req.user.id
}
}).then(function (data) {
// get an array of the data keys, (not sure if you need to do this)
// it is unclear whether data is an object of users or an array. I assume
// it's an object as you used a `for in` loop
const keys = Object.keys(data)
// map the data keys to [Promise(query), Promise(query), {...}]
const hacks = keys.map((d) => {
return models.Hackathon.findOne({
where: {
id: data[d].id
}
})
})
// user Promise.all to resolve all of the promises asynchronously
Promise.all(hacks)
// this will be called once all promises have resolved so
// you can modify your data. it will be an array of the returned values
.then((users) => {
const [user1, user2, {...}] = users
res.render('dashboard/index.ejs', {
title: 'My Hackathons',
user: req.user,
hacks: users
});
})
});
The Sequelize library has the include parameter which merges models in one call. Adjust your where statement to bring the Hackathons model into Attendance. If this does not work, take the necessary time to setup Sequelize correctly, their documentation is constantly being improved. In the end, you'll save loads of time by reducing error and making your code readable for other programmers.
Look how much cleaner this can be...
models.Attendance.findAll({
include: [{
model: Hackathon,
as: 'hackathon'
},
where: {
UserId: req.user.id
}
}).then(function (data) {
// hackathon id
console.log(data.hackathon.id)
// attendance id
console.log(data.id)
})
Also..
Hackathon.belongsTo(Attendance)
Attendance.hasMany(Hackathon)
sequelize.sync().then(() => {
// this is where we continue ...
})
Learn more about Sequelize includes here:
http://docs.sequelizejs.com/en/latest/docs/models-usage/
Immediately invoke asynchronous function expression
This is one of the techniques mentioned at: How can I use async/await at the top level? Toplevel await is likely coming soon as of 2021, which will be even better.
Minimal runnable example:
const assert = require('assert');
const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize({
dialect: 'sqlite',
storage: 'db.sqlite',
});
const IntegerNames = sequelize.define(
'IntegerNames', {
value: { type: DataTypes.INTEGER, allowNull: false },
name: { type: DataTypes.STRING, },
}, {});
(async () => {
await IntegerNames.sync({force: true})
await IntegerNames.create({value: 2, name: 'two'});
await IntegerNames.create({value: 3, name: 'three'});
await IntegerNames.create({value: 5, name: 'five'});
// Fill array.
let integerNames = [];
integerNames.push(await IntegerNames.findOne({
where: {value: 2}
}));
integerNames.push(await IntegerNames.findOne({
where: {value: 3}
}));
// Use array.
assert(integerNames[0].name === 'two');
assert(integerNames[1].name === 'three');
await sequelize.close();
})();
Tested on Node v14.16.0, sequelize 6.6.2, seqlite3 5.0.2, Ubuntu 20.10.