I am new to working on a MongoDB and Docker, I am working on an application and couldn't find a more subtle way to seed my database using an npm run command. First I created a file called seed.js and then associated it to npm run seed command on the package.json file.
On the seed.js file I import Mongoose and the models but two things I will need to do is:
Create roles, if they don’t exist yet
Create capabilities, if they don’t exist yet and associate it to the
roles
The Roles that i want to create are:
admin (description: Administrator)
viewer (description: Viewer)
Capabilities
I need to check each endpoint of the Users service that should require authentication and create an adequate capability. Example: updateUser updates the user data. This could be done by the own user (so there must be an updateUserOwn capability) and by an administrator (that will have an updateUsers capability). I will have to analyse each endpoint and judge what is adequate but I cannot still find a way around getting the initial role and capabilities to the database.
UPDATE:
On the seeding itself, the updated solution works, but it requires lot of code and repetition that could probably be fixed by loops. I’d like to start creating the roles first which means creating an array with objects, with the data from the roles to be created. Each role has the fields role and description
const userRole = [{
role: admin
description: Administrator
},
{
role: viewer
description: Viewer
}]
The idea is that if the role exist it doesn't need to update but I don't know how do I loop through the array and create a role only if it doesn’t exist. Something like using updateOne, with the upsert: true option, but with the data on $setOnInsert as this will add the data only if a document is inserted.
I only need create and not update because in the future I’ll edit roles directly through the API. So, if a change was made on the admin role, for example, the seed will not overwrite it
During the loop, I'll need to create an associative array called rolesIds that will store the ObjectId of the created roles. It should result in something like this:
[
"admin": "iaufh984whrfj203jref",
"viewer": "r9i23jfeow9iefd0ew0",
]
Also each capability must have an array of roles it must be associated to. Example:
{
capability: "updateUsers",
description: "Update the data of all users",
roles: ["admin"]
}
How do I loop through the array on each element, prepare it to be inserted using the array with object IDs. Instead of roles: ["admin"]? something like roles: ["iaufh984whrfj203jref"], otherwise there’ll be a cast error. Remember each capability may be associated to more than one role, so I'll probably need to loop through them but I cannot find a way to create that logic.
Users Model
const userSchema = new mongoose.Schema(
{
.......
role: {
ref: "roles",
type: mongoose.Schema.Types.ObjectId,
},
);
module.exports = mongoose.model("User", userSchema);
Role Model:
const roles = new mongoose.Schema({
role: {
type: String,
required: true,
},
capabilities: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "capabilities",
},
],
});
module.exports = mongoose.model("roles", roles);
Capabilities Model:
const capabilities = new mongoose.Schema({
capability: {
type: String,
required: true,
},
name: {
type: String,
},
});
module.exports = mongoose.model("capabilities", capabilities);
UPDATED: seed file:
const seedDB = async () => {
if (!process.env.DB_URI) {
throw new Error("Error connecting to MongoDB: DB_URI is not defined.");
}
try {
await mongoose.connect(process.env.DB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
});
console.log("Connected to MongoDB");
const tasks = [
Capability.findOneAndUpdate(
{ name: "updateUserOwn" },
{ capability: "updateUser" },
{ upsert: true }
).exec(),
Capability.findOneAndUpdate(
{ name: "updateUsers" },
{ capability: "updateUser" },
{ upsert: true }
).exec(),
// Seed more...
];
const [updateUserOwn, updateUsers] = await Promise.all(tasks);
Role.bulkWrite([
{
updateOne: {
filter: { role: "Admin" },
update: { capabilities: [updateUsers] },
upsert: true,
},
},
{
updateOne: {
filter: { role: "Viewer" },
update: { capabilities: [updateUserOwn] },
upsert: true,
},
},
]);
console.log("seeded data", tasks);
} catch (error) {
console.log(`Error connecting to MongoDB: ${error}`);
}
};
seedDB();
You are on the right path overall.
Because capabilities are used as a reference you'd have to fetch or create them (get a ref) before assigning them to a role.
This could be your seed logic:
const tasks = [
Capability.findOneAndUpdate(
{ name: 'updateUserOwn' }, // matches or creates this capability
{ capability: 'updateUser' }, // adds this to the object
{ upsert: true, new: true } // `new` guarantees an object is always returned
}).exec(),
Capability.findOneAndUpdate(
{ name: 'updateUsers' },
{ capability: 'updateUser' },
{ upsert: true, new: true }
}).exec(),
// Seed more...
];
const [
updateUserOwn,
updateUsers,
] = await Promise.all(tasks);
// We can use bulk write for the second transaction so it runs in one go
await Role.bulkWrite([
{
updateOne: {
filter: { role: 'Admin' },
update: { capabilities: [updateUsers] },
upsert: true,
}
},
{
updateOne: {
filter: { role: 'Viewer' },
update: { capabilities: [updateUserOwn] },
upsert: true,
}
}
]);
We seed capabilities one by one using findOneAndUpdate so we can get a reference to each capability we intend to use on the roles
Then we use bulkWrite to seed the roles
I might have swapped the capabilities and their names but I hope you get the general idea
The seed would have been simpler if there weren't references involved - you could just use bulkWrite everything in one go, but in order to create object with inner references or add references to such object you first need to have the actual reference
You can create static mapping and loop through which would reduce the code a bit, and make things easier. This would also allow you to skip seeding items that already exist
Since capabilities are reused through roles I want to create them first, but it's no problem to alter the logic to first create roles and then capabilities, though it might not be as straight forward
Also each capability must have an array of roles it must be associated to.
This is called a "many to many" relationship (as roles also have an array of references to capabilities) which would only complicate logic. Are you sure you really need it - mongoose/monogo won't manage it automatically for you:
when you add a capability to a role you'd also need to sync and add the role inside capability.roles - manually
and the reverse - adding a role inside capability.roles you'd need to sync this and also manually add the capability to role.capabilities
the same thing for deleting capabilities or roles - manual cleanup
it can fail and would need to recover - e.g. a capability is added to role.capabilities but for some reason execution stopped and the role was not added to capability.roles - so the whole handling might need to be wrapped in a transaction
there are ways to cross reference roles and capabilities without have to have a "many to many" relationship
Here's a simple approach using save middleware to sync many to many relationships for create/update
Role.js
const mongoose = require('mongoose');
const roles = new mongoose.Schema({
role: {
type: String,
required: true,
},
description: String,
capabilities: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'capabilities',
},
],
});
roles.pre('save', async function save() {
// Doesn't need to run if there are no capabilities
if (!this.capabilities || this.capabilities.length === 0) return;
const Capability = mongoose.model('capabilities');
await Capability.updateMany(
{ _id: {$in: this.capabilities} },
// Adds only if it's missing
{ $addToSet: { roles: this._id }},
);
});
// Todo: similar logic to remove from capabilities if role is deleted
module.exports = mongoose.model("roles", roles);
Capability.js
const mongoose = require('mongoose');
const capabilities = new mongoose.Schema({
capability: {
type: String,
required: true,
},
description: {
type: String,
},
roles: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'roles',
}
]
});
capabilities.pre('save', async function save() {
if (!this.roles || this.roles.length === 0) return;
const Role = mongoose.model('roles');
await Role.updateMany(
{_id: {$in: this.roles}},
{$addToSet: {capabilities: this._id}},
);
})
// Todo: similar logic to remove from roles if capability is deleted
module.exports = mongoose.model("capabilities", capabilities);
Here's an update seed routine:
Seed.js
const mongoose = require('mongoose');
const Capability = require('./models/Capability');
const Role = require('./models/Role');
const CAPABILITIES = {
UPDATE_USERS: {
capability: 'updateUsers',
description: 'Update the data of all users',
},
VIEW_USERS: {
capability: 'viewUsers',
description: 'View public data of users',
},
UPDATE_OWN_RECORD: {
capability: 'updateUserOwn',
description: 'Update user own data',
}
}
const ROLES_TO_SEED = [
{
role: 'admin',
description: 'Administrator',
capabilities: [CAPABILITIES.UPDATE_USERS, CAPABILITIES.VIEW_USERS],
},
{
role: 'viewer',
description: 'Viewer',
capabilities: [CAPABILITIES.VIEW_USERS, CAPABILITIES.UPDATE_OWN_RECORD],
}
]
const seedDB = async () => {
await connectToDb();
await seedRoles();
};
const connectToDb = async () => {
if (!process.env.DB_URI) throw new Error('DB_URI is not defined.');
console.info('Connecting to database...');
await mongoose.connect(process.env.DB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
useFindAndModify: false,
});
console.info('Connected \n');
}
const seedRoles = async () => {
console.log('Seeding Roles...');
// runs sequentially to skip creating duplicate capabilities
for (const role of ROLES_TO_SEED) {
await findOrCreateRole(role);
}
console.log('Complete \n');
}
const findOrCreateRole = async ({capabilities, role, ...defaults}) => {
console.info('Looking for role: ', role);
const fromDb = await Role.findOne({role}).exec();
if (fromDb) {
console.info('Role already exists skipping... \n');
return fromDb;
}
console.info('Role does not exist - creating new \n');
const doc = new Role({role, ...defaults});
// All capabilities (per role) can be created/found in parallel
const roleCapabilities = await Promise.all(capabilities.map(findOrCreateCapability));
doc.capabilities = roleCapabilities.map(c => c._id);
await doc.save();
console.info('Role created: ', role);
console.info('');
return doc;
}
const findOrCreateCapability = async ({capability, ...defaults}) => {
console.info('Looking for capability: ', capability);
let doc = await Capability.findOne({capability}).exec();
if (doc) {
console.info(`Capability ${capability} found - using existing...`);
}
else {
console.info(`Capability ${capability} does not exist - creating new`);
doc = new Capability({capability, ...defaults});
await doc.save();
}
return doc;
}
seedDB()
.then(() => {
console.info('Exiting...: ');
process.exit(0);
})
.catch(error => {
console.error('Seed failed');
console.error(error);
process.exit(1);
})
We have a dictionary of capabilities and a list of roles that we can map to db operations.
The idea is that each role should contain the full definition of a capability, it can be used to either find the existing capability or create it if it doesn't exist
For each role in the list we make a query to see if it exists.
When it exists we do nothing and move to the next role
When it doesn't exist we have all the data needed to create it and create/find any capabilities that it might need
When you figure out all the roles and capabilities of the application you just put them in the ROLES_TO_SEED and CAPABILITIES static mappings
The script relies on the above mentioned middleware modifications in models
And a small bonus
You don't need many to many relationship to match capabilities to the roles they are used in. Here's how you can aggregate that information if only the Role model have an array of capabilities (refs). Run this after the database is seeded:
const showCapabilitiesUsages = async () => {
const result = await Capability.aggregate([
{
$lookup: {
from: 'roles',
let: {searched: '$_id'},
pipeline: [
{
$match: {
$expr: {
$in: ['$$searched', '$capabilities']
}
}
}
],
as: 'roles'
}
}, {
$project: {
_id: 0,
capability: 1,
description: 1,
usedInRoles: {
$map: {
input: '$roles',
as: 'role',
in: '$$role.role',
}
}
}
}
]).exec();
console.log('Aggregate result: ', result);
}
You should get a result like:
Aggregate result: [
{
capability: 'updateUsers',
description: 'Update the data of all users',
usedInRoles: [ 'admin' ]
},
{
capability: 'viewUsers',
description: 'View public data of users',
usedInRoles: [ 'admin', 'viewer' ]
},
{
capability: 'updateUserOwn',
description: 'Update user own data',
usedInRoles: [ 'viewer' ]
}
]
Try something like this, it should would work:
const roles = [
{
name: 'admin',
description: 'Administrator',
},
{
name: 'viewer',
description: 'Viewer',
},
];
const capabilities = [
// Capabilities
{
name: 'createCapability',
description: 'Create a new capability',
roles: ['admin'],
},
{
name: 'deleteCapability',
description: 'Delete a capability',
roles: ['admin'],
}
// Roles
{
name: 'createRole',
description: 'Create a new role',
roles: ['admin'],
},
{
name: 'deleteRole',
description: 'Delete a role',
roles: ['admin'],
},
// Users
{
name: 'updateUser',
description: 'Update current user data',
roles: ['viewer'],
},
{
name: 'updateUsers',
description: 'Update the data from any user',
roles: ['admin'],
},
];
const seedRoles = async (roles) => {
if (0 == roles.length || !Array.isArray(roles)) {
return;
}
console.log('');
for (const role of roles) {
const savedRole = await Role.findOneAndUpdate(
{name: role.name},
{$setOnInsert: role},
{upsert: true, new: true, rawResult: true},
);
if (!savedRole) {
console.log(`Role “${savedRole.value.name}” already on database.`);
} else {
console.log(`Role “${savedRole.value.name}” added to database.`);
}
}
};
const seedCapabilities = async (capabilities) => {
if (0 == capabilities.length || !Array.isArray(capabilities)) {
return;
}
console.log('');
for (const capability of capabilities) {
const rolesToPush = capability.roles;
delete capability.roles;
const addedCapability = await Capability.findOneAndUpdate(
{name: capability.name},
{$setOnInsert: capability},
{upsert: true, new: true, rawResult: true},
);
if (!addedCapability) {
console.log(
`Capability “${addedCapability.value.name}” ` +
`already on database.`,
);
} else {
console.log(
`Capability “${addedCapability.value.name}” ` +
`added to database.`,
);
if (rolesToPush && Array.isArray(rolesToPush)) {
rolesToPush.forEach(async (role) => {
const roleToPush = await Role.findOne({name: role});
if (roleToPush) {
roleToPush.capabilities.push(addedCapability.value);
await roleToPush.save();
}
});
}
}
}
};
const seedDb = async (roles, capabilities, users) => {
try {
await seedRoles(roles);
await seedCapabilities(capabilities);
console.log('roles', roles);
} catch (error) {
console.error(error);
}
};
module.exports = seedDb;
I am new to Jest, and I am trying to figure out how to to reset the test object after each test.
Current Code
describe.only('POST request - missing entry', () => {
// newBlog is the "test" object
let newBlog = {
title: 'Test Title',
author: 'Foo Bar',
url: 'www.google.com',
likes: 100
}
test('sets "likes" field to 0 when missing', async () => {
delete newBlog.likes // propagates to next test
console.log(newBlog)
})
test('returns 400 error when "title" and "url" fields are missing', async () => {
console.log(newBlog)
})
})
Objective: I am writing test using jest to test for bad POST request. i.e. my POST request will intentionally have missing fields for each test.
likes field will be omitted from first test while title, url field will be missing from second test. Goal is to write newBlog object only once rather than rewriting objects for each tests.
Problem Main issue here is that the result of first test propagates to next test, i.e. when removing likes field for first test, it stays like that and starts the second test without having likes field.
I want to know how I can reset the content of object for each test.
Attempts So far, I tried few things:
I used BeforeEach to reset the newBlog in following manner:
beforeEach(() => {
let newBlog = {
title: 'Test Title',
author: 'Foo Bar',
url: 'www.google.com',
likes: 100
}
return newBlog
})
However, above code does not work since newBlog is in different scope so each test does not recognize newBlog variable.
I also used AfterEach to reset in following manner:
afterEach(() => {
jest.clearAllMocks()
})
This time, it ran but gave me the same results as first code snippet.
I would like to know how to reset objects for each test as many of the solution discussed in stackoverflow seems to focus on resetting functions rather than objects.
Thank you for your help in advance.
Try something like this, declare the variable in the describe and reset it in the beforeEach:
describe.only('POST request - missing entry', () => {
// newBlog is the "test" object
let newBlog;
beforeEach(() => {
newBlog = {
title: 'Test Title',
author: 'Foo Bar',
url: 'www.google.com',
likes: 100
}
});
test('sets "likes" field to 0 when missing', async () => {
delete newBlog.likes // propagates to next test
console.log(newBlog)
})
test('returns 400 error when "title" and "url" fields are missing', async () => {
console.log(newBlog)
})
})
You were correct in needing to use Jest's beforeEach() function; however, the only things that are returnable from beforeEach() are promises and generators—returning newBlog from beforeEach() does nothing. What I would do is create a local variable in the describe function and have beforeEach() reassign that variable before each test runs as seen below.
fdescribe('POST request - missing entry', () => {
let newBlog;
beforeEach(() => {
newBlog = {
title: 'Test Title',
author: 'Foo Bar',
url: 'www.google.com',
likes: 100
}
});
test('sets "likes" field to 0 when missing', async () => {
delete newBlog.likes; // remove likes key from copied object
console.log(newBlog);
});
test('returns 400 error when "title" and "url" fields are missing', async () => {
delete newBlog.title;
delete newBlog.url;
console.log(newBlog);
});
})
Additionally, jest.clearAllMocks() clears all calls and instances of a mocked function which will not reset the newBlog variable in the way you want to use it here.
I usually deep copy the test object in this kind of scenarios. So I have a helper function (you can use e.g. Lodash's deepClone function too)
const deepCopy = (obj: Object) => JSON.parse(JSON.stringify(obj));
In your tests you create a new deep copy of the wanted test object instead of mutating it's state like this:
test('sets "likes" field to 0 when missing', async () => {
let testObj = deepCopy(newBlog)
delete testObj.likes // propagates to next test
console.log(testObj)
})
How to pass parameters to the inquirer questions, so that i can set the values of a question object based on either values from previous questions or from code outside the prompt?
The only way i can see of achieving this if based on answer of a previous question is to nest the inquirer prompt calls
const inquirer = require('inquirer');
function getPath(){
return {
'system1':`system1path`,
'system2':`system2path`,
'system3':`system3path`
}
}
inquirer.prompt([
{
type: 'list',
name: 'testSystem',
message: 'Which system do you want to run?',
choices: ['system1', 'system2', 'system3']
},
{
type: 'fuzzypath',
name: 'searchTestSuite',
excludePath: nodePath => nodePath.startsWith('node_modules'),
itemType: 'any',
rootPath: getPath()['< answer from question(testSystem) >'],
message: 'Select a target directory :',
default: `system1path`,
suggestOnly: false,
depthLimit: 6,
},
]).then(answers => {
console.log(answers);
});
Expected result :
If you select testSystem = system2
You should get rootPath = system2Path , without nesting the inquirer prompts or by using whenfunction (since when seems to be dealing with boolean values)
You can solve it by nesting, but changing from Promise.then to async-await makes it more readable. Inquirer.js uses promises, hence you can use await to capture prompt answers and by issuing multiple prompts you can save the state between prompts. See code below.
PS: I've removed the default: ..., parameter from fuzzypath because it yields the default value despite it beign outside the root path.
const inquirer = require('inquirer');
inquirer.registerPrompt('fuzzypath', require('inquirer-fuzzy-path'))
const system1path = ...;
const system2path = ...;
const system3path = ...;
function getPath(){
return {
'system1': `${system1path}`,
'system2': `${system2path}`,
'system3': `${system3path}`
};
}
(async function () {
const {testSystem} = await inquirer.prompt({
type: 'list',
name: 'testSystem',
message: 'Which system do you want to run?',
choices: ['system1', 'system2', 'system3']
});
const {searchTestSuite} = await inquirer.prompt({
type: 'fuzzypath',
name: 'searchTestSuite',
excludePath: nodePath => nodePath.startsWith('node_modules'),
itemType: 'any',
rootPath: getPath()[testSystem],
message: 'Select a target directory :',
suggestOnly: false,
depthLimit: 6,
});
console.log({testSystem, searchTestSuite});
})();
I have created an uploadSchema (mongoose.Schema) with the fields (among the rest): key and bucket. each of them alone is not unique but together I want them to create a unique id.
in my code, I used the line (right after declaring the uploadSchema and right before the uploadModel):
uploadSchema.index({ key: 1, bucket: 1 }, { unique: true, background: true });
but then, in my tests (mocha and chai), the indexing is not enforced, and so I can create two instances with the same key and bucket (in my case).
for example, in my code:
await uploadModel.create({ key: testUpload.key, bucket: testUpload.bucket,
name: 'name1', ownerID: USER.id, parent: null }).should.eventually.exist;
and right after that:
await uploadModel.create({key: testUpload.key, bucket: testUpload.bucket,
name: 'name1', ownerID: USER.id, parent: null }).should.eventually.be.rejected;
does not throw the right error error:
AssertionError: expected promise to be rejected but it was fulfilled with { Object ($__, isNew, ...) }
Am I not using it correctly? Or is there a problem with indexing and testing?
Most likely you set autoIndex to false in your connection (which is recommended to do).
Either add it to you Schema:
let uploadSchema = mongoose.Schema({ ... }, {autoIndex: true});
But i would recommend just building the index yourself on the database, i think its the safest way around it.
so I figured it out!
Apparently, I used mongoose.connection.dropDatabase(); in my afterEach of the tests. That means that the indexes were reset each time.
So what I did was to recreate the indexes each time in my tests:
before(async () => {
// Remove files from DB
const collections = ['files', 'uploads'];
for (const i in collections) {
mongoose.connection.db.createCollection(collections[i], (err) => {});
}
await mongoose.connection.collections['files'].createIndex({ name: 1, parent: 1, ownerID: 1 }, { unique: true });
await mongoose.connection.collections['uploads'].createIndex({ key: 1, bucket: 1 }, { unique: true });
});
And in the beforeEach:
beforeEach(async () => {
const removeCollectionPromises = [];
for (const i in mongoose.connection.collections) {
removeCollectionPromises.push(mongoose.connection.collections[i].deleteMany({}));
}
await Promise.all(removeCollectionPromises);
});
afterEach is empty.
now it works :)
How do I perform an sql query such as this?
SELECT column_name AS alias_name FROM table_name;
Example: I want the column 'first' to be selected as 'firstname'
Table.findAll({
attributes: [id,"first"]
})
.then(function(posts) {
res.json(posts);
})
Table.findAll({
attributes: ['id', ['first', 'firstName']] //id, first AS firstName
})
.then(function(posts) {
res.json(posts);
});
You need to use row.get('newname') to access columns aliased by attributes
Doing just row.newname, or row.oldname, will not work like it does for non-aliased names for some reason:
https://github.com/sequelize/sequelize/issues/10592
https://sequelize.org/v5/manual/models-usage.html documents it:
Project.findOne({
where: {title: 'aProject'},
attributes: ['id', ['name', 'title']]
}).then(project => {
// project will be the first entry of the Projects table with the title
// 'aProject' || null
// project.get('title') will contain the name of the project
})
but https://sequelize.org/master/class/lib/model.js~Model.html doesn't mention it, which is confusing:
instance.field
// is the same as
instance.get('field')
Related: Sequelize cannot access alias. Alias is undefined
Minimal runnable example:
const assert = require('assert');
const path = require('path');
const { Sequelize, DataTypes, Op } = require('sequelize');
const sequelize = new Sequelize({
dialect: 'sqlite',
storage: path.basename(__filename) + '.sqlite',
});
(async () => {
const Int = sequelize.define('Int', {
value: {
type: DataTypes.INTEGER,
},
name: {
type: DataTypes.STRING,
},
}, {});
await Int.sync({force: true})
await Int.create({value: 2, name: 'two'});
let row;
row = await Int.findOne({
where: { value: 2 },
attributes: [ 'id', [ 'value', 'newvalue' ] ],
});
assert.strictEqual(row.id, 1);
assert.strictEqual(row.value, undefined);
assert.strictEqual(row.newvalue, undefined);
assert.strictEqual(row.get('newvalue'), 2);
await sequelize.close();
})();
The generated query does exactly what we wanted then:
SELECT `id`, `value` AS `newvalue` FROM `Ints` AS `Int`
WHERE `Int`.`value` = 2 LIMIT 1;
tested on sequelize 6.5.1, sqlite3 5.0.2.
Also Sequelize supports defining column names directly on the model definition too.
Sequelize Docs On that they mention about field attribute on column definition.
ex: (Taken from Docs itself)
const { Model, DataTypes, Deferrable } = require("sequelize");
class Foo extends Model { }
Foo.init({
// You can specify a custom column name via the 'field' attribute:
fieldWithUnderscores: {
type: DataTypes.STRING,
field: 'field_with_underscores'
},
}, {
sequelize,
modelName: 'foo'
});
thanks to this answer