Develop Reference

How to seed roles and capabilities in MongoDB

I am new to working on a MongoDB and Docker, I am working on an application and couldn't find a more subtle way to seed my database using an npm run command. First I created a file called seed.js and then associated it to npm run seed command on the package.json file.
On the seed.js file I import Mongoose and the models but two things I will need to do is:
Create roles, if they don’t exist yet
Create capabilities, if they don’t exist yet and associate it to the
roles
The Roles that i want to create are:
admin (description: Administrator)
viewer (description: Viewer)
Capabilities
I need to check each endpoint of the Users service that should require authentication and create an adequate capability. Example: updateUser updates the user data. This could be done by the own user (so there must be an updateUserOwn capability) and by an administrator (that will have an updateUsers capability). I will have to analyse each endpoint and judge what is adequate but I cannot still find a way around getting the initial role and capabilities to the database.
UPDATE:
On the seeding itself, the updated solution works, but it requires lot of code and repetition that could probably be fixed by loops. I’d like to start creating the roles first which means creating an array with objects, with the data from the roles to be created. Each role has the fields role and description
const userRole = [{
role: admin
description: Administrator
},
{
role: viewer
description: Viewer
}]
The idea is that if the role exist it doesn't need to update but I don't know how do I loop through the array and create a role only if it doesn’t exist. Something like using updateOne, with the upsert: true option, but with the data on $setOnInsert as this will add the data only if a document is inserted.
I only need create and not update because in the future I’ll edit roles directly through the API. So, if a change was made on the admin role, for example, the seed will not overwrite it
During the loop, I'll need to create an associative array called rolesIds that will store the ObjectId of the created roles. It should result in something like this:
[
"admin": "iaufh984whrfj203jref",
"viewer": "r9i23jfeow9iefd0ew0",
]
Also each capability must have an array of roles it must be associated to. Example:
{
capability: "updateUsers",
description: "Update the data of all users",
roles: ["admin"]
}
How do I loop through the array on each element, prepare it to be inserted using the array with object IDs. Instead of roles: ["admin"]? something like roles: ["iaufh984whrfj203jref"], otherwise there’ll be a cast error. Remember each capability may be associated to more than one role, so I'll probably need to loop through them but I cannot find a way to create that logic.
Users Model
const userSchema = new mongoose.Schema(
{
.......
role: {
ref: "roles",
type: mongoose.Schema.Types.ObjectId,
},
);
module.exports = mongoose.model("User", userSchema);
Role Model:
const roles = new mongoose.Schema({
role: {
type: String,
required: true,
},
capabilities: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "capabilities",
},
],
});
module.exports = mongoose.model("roles", roles);
Capabilities Model:
const capabilities = new mongoose.Schema({
capability: {
type: String,
required: true,
},
name: {
type: String,
},
});
module.exports = mongoose.model("capabilities", capabilities);
UPDATED: seed file:
const seedDB = async () => {
if (!process.env.DB_URI) {
throw new Error("Error connecting to MongoDB: DB_URI is not defined.");
}
try {
await mongoose.connect(process.env.DB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
});
console.log("Connected to MongoDB");
const tasks = [
Capability.findOneAndUpdate(
{ name: "updateUserOwn" },
{ capability: "updateUser" },
{ upsert: true }
).exec(),
Capability.findOneAndUpdate(
{ name: "updateUsers" },
{ capability: "updateUser" },
{ upsert: true }
).exec(),
// Seed more...
];
const [updateUserOwn, updateUsers] = await Promise.all(tasks);
Role.bulkWrite([
{
updateOne: {
filter: { role: "Admin" },
update: { capabilities: [updateUsers] },
upsert: true,
},
},
{
updateOne: {
filter: { role: "Viewer" },
update: { capabilities: [updateUserOwn] },
upsert: true,
},
},
]);
console.log("seeded data", tasks);
} catch (error) {
console.log(`Error connecting to MongoDB: ${error}`);
}
};
seedDB();

You are on the right path overall.
Because capabilities are used as a reference you'd have to fetch or create them (get a ref) before assigning them to a role.
This could be your seed logic:
const tasks = [
Capability.findOneAndUpdate(
{ name: 'updateUserOwn' }, // matches or creates this capability
{ capability: 'updateUser' }, // adds this to the object
{ upsert: true, new: true } // `new` guarantees an object is always returned
}).exec(),
Capability.findOneAndUpdate(
{ name: 'updateUsers' },
{ capability: 'updateUser' },
{ upsert: true, new: true }
}).exec(),
// Seed more...
];
const [
updateUserOwn,
updateUsers,
] = await Promise.all(tasks);
// We can use bulk write for the second transaction so it runs in one go
await Role.bulkWrite([
{
updateOne: {
filter: { role: 'Admin' },
update: { capabilities: [updateUsers] },
upsert: true,
}
},
{
updateOne: {
filter: { role: 'Viewer' },
update: { capabilities: [updateUserOwn] },
upsert: true,
}
}
]);
We seed capabilities one by one using findOneAndUpdate so we can get a reference to each capability we intend to use on the roles
Then we use bulkWrite to seed the roles
I might have swapped the capabilities and their names but I hope you get the general idea
The seed would have been simpler if there weren't references involved - you could just use bulkWrite everything in one go, but in order to create object with inner references or add references to such object you first need to have the actual reference

You can create static mapping and loop through which would reduce the code a bit, and make things easier. This would also allow you to skip seeding items that already exist
Since capabilities are reused through roles I want to create them first, but it's no problem to alter the logic to first create roles and then capabilities, though it might not be as straight forward
Also each capability must have an array of roles it must be associated to.
This is called a "many to many" relationship (as roles also have an array of references to capabilities) which would only complicate logic. Are you sure you really need it - mongoose/monogo won't manage it automatically for you:
when you add a capability to a role you'd also need to sync and add the role inside capability.roles - manually
and the reverse - adding a role inside capability.roles you'd need to sync this and also manually add the capability to role.capabilities
the same thing for deleting capabilities or roles - manual cleanup
it can fail and would need to recover - e.g. a capability is added to role.capabilities but for some reason execution stopped and the role was not added to capability.roles - so the whole handling might need to be wrapped in a transaction
there are ways to cross reference roles and capabilities without have to have a "many to many" relationship
Here's a simple approach using save middleware to sync many to many relationships for create/update
Role.js
const mongoose = require('mongoose');
const roles = new mongoose.Schema({
role: {
type: String,
required: true,
},
description: String,
capabilities: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'capabilities',
},
],
});
roles.pre('save', async function save() {
// Doesn't need to run if there are no capabilities
if (!this.capabilities || this.capabilities.length === 0) return;
const Capability = mongoose.model('capabilities');
await Capability.updateMany(
{ _id: {$in: this.capabilities} },
// Adds only if it's missing
{ $addToSet: { roles: this._id }},
);
});
// Todo: similar logic to remove from capabilities if role is deleted
module.exports = mongoose.model("roles", roles);
Capability.js
const mongoose = require('mongoose');
const capabilities = new mongoose.Schema({
capability: {
type: String,
required: true,
},
description: {
type: String,
},
roles: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'roles',
}
]
});
capabilities.pre('save', async function save() {
if (!this.roles || this.roles.length === 0) return;
const Role = mongoose.model('roles');
await Role.updateMany(
{_id: {$in: this.roles}},
{$addToSet: {capabilities: this._id}},
);
})
// Todo: similar logic to remove from roles if capability is deleted
module.exports = mongoose.model("capabilities", capabilities);
Here's an update seed routine:
Seed.js
const mongoose = require('mongoose');
const Capability = require('./models/Capability');
const Role = require('./models/Role');
const CAPABILITIES = {
UPDATE_USERS: {
capability: 'updateUsers',
description: 'Update the data of all users',
},
VIEW_USERS: {
capability: 'viewUsers',
description: 'View public data of users',
},
UPDATE_OWN_RECORD: {
capability: 'updateUserOwn',
description: 'Update user own data',
}
}
const ROLES_TO_SEED = [
{
role: 'admin',
description: 'Administrator',
capabilities: [CAPABILITIES.UPDATE_USERS, CAPABILITIES.VIEW_USERS],
},
{
role: 'viewer',
description: 'Viewer',
capabilities: [CAPABILITIES.VIEW_USERS, CAPABILITIES.UPDATE_OWN_RECORD],
}
]
const seedDB = async () => {
await connectToDb();
await seedRoles();
};
const connectToDb = async () => {
if (!process.env.DB_URI) throw new Error('DB_URI is not defined.');
console.info('Connecting to database...');
await mongoose.connect(process.env.DB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
useFindAndModify: false,
});
console.info('Connected \n');
}
const seedRoles = async () => {
console.log('Seeding Roles...');
// runs sequentially to skip creating duplicate capabilities
for (const role of ROLES_TO_SEED) {
await findOrCreateRole(role);
}
console.log('Complete \n');
}
const findOrCreateRole = async ({capabilities, role, ...defaults}) => {
console.info('Looking for role: ', role);
const fromDb = await Role.findOne({role}).exec();
if (fromDb) {
console.info('Role already exists skipping... \n');
return fromDb;
}
console.info('Role does not exist - creating new \n');
const doc = new Role({role, ...defaults});
// All capabilities (per role) can be created/found in parallel
const roleCapabilities = await Promise.all(capabilities.map(findOrCreateCapability));
doc.capabilities = roleCapabilities.map(c => c._id);
await doc.save();
console.info('Role created: ', role);
console.info('');
return doc;
}
const findOrCreateCapability = async ({capability, ...defaults}) => {
console.info('Looking for capability: ', capability);
let doc = await Capability.findOne({capability}).exec();
if (doc) {
console.info(`Capability ${capability} found - using existing...`);
}
else {
console.info(`Capability ${capability} does not exist - creating new`);
doc = new Capability({capability, ...defaults});
await doc.save();
}
return doc;
}
seedDB()
.then(() => {
console.info('Exiting...: ');
process.exit(0);
})
.catch(error => {
console.error('Seed failed');
console.error(error);
process.exit(1);
})
We have a dictionary of capabilities and a list of roles that we can map to db operations.
The idea is that each role should contain the full definition of a capability, it can be used to either find the existing capability or create it if it doesn't exist
For each role in the list we make a query to see if it exists.
When it exists we do nothing and move to the next role
When it doesn't exist we have all the data needed to create it and create/find any capabilities that it might need
When you figure out all the roles and capabilities of the application you just put them in the ROLES_TO_SEED and CAPABILITIES static mappings
The script relies on the above mentioned middleware modifications in models
And a small bonus
You don't need many to many relationship to match capabilities to the roles they are used in. Here's how you can aggregate that information if only the Role model have an array of capabilities (refs). Run this after the database is seeded:
const showCapabilitiesUsages = async () => {
const result = await Capability.aggregate([
{
$lookup: {
from: 'roles',
let: {searched: '$_id'},
pipeline: [
{
$match: {
$expr: {
$in: ['$$searched', '$capabilities']
}
}
}
],
as: 'roles'
}
}, {
$project: {
_id: 0,
capability: 1,
description: 1,
usedInRoles: {
$map: {
input: '$roles',
as: 'role',
in: '$$role.role',
}
}
}
}
]).exec();
console.log('Aggregate result: ', result);
}
You should get a result like:
Aggregate result: [
{
capability: 'updateUsers',
description: 'Update the data of all users',
usedInRoles: [ 'admin' ]
},
{
capability: 'viewUsers',
description: 'View public data of users',
usedInRoles: [ 'admin', 'viewer' ]
},
{
capability: 'updateUserOwn',
description: 'Update user own data',
usedInRoles: [ 'viewer' ]
}
]

Try something like this, it should would work:
const roles = [
{
name: 'admin',
description: 'Administrator',
},
{
name: 'viewer',
description: 'Viewer',
},
];
const capabilities = [
// Capabilities
{
name: 'createCapability',
description: 'Create a new capability',
roles: ['admin'],
},
{
name: 'deleteCapability',
description: 'Delete a capability',
roles: ['admin'],
}
// Roles
{
name: 'createRole',
description: 'Create a new role',
roles: ['admin'],
},
{
name: 'deleteRole',
description: 'Delete a role',
roles: ['admin'],
},
// Users
{
name: 'updateUser',
description: 'Update current user data',
roles: ['viewer'],
},
{
name: 'updateUsers',
description: 'Update the data from any user',
roles: ['admin'],
},
];
const seedRoles = async (roles) => {
if (0 == roles.length || !Array.isArray(roles)) {
return;
}
console.log('');
for (const role of roles) {
const savedRole = await Role.findOneAndUpdate(
{name: role.name},
{$setOnInsert: role},
{upsert: true, new: true, rawResult: true},
);
if (!savedRole) {
console.log(`Role “${savedRole.value.name}” already on database.`);
} else {
console.log(`Role “${savedRole.value.name}” added to database.`);
}
}
};
const seedCapabilities = async (capabilities) => {
if (0 == capabilities.length || !Array.isArray(capabilities)) {
return;
}
console.log('');
for (const capability of capabilities) {
const rolesToPush = capability.roles;
delete capability.roles;
const addedCapability = await Capability.findOneAndUpdate(
{name: capability.name},
{$setOnInsert: capability},
{upsert: true, new: true, rawResult: true},
);
if (!addedCapability) {
console.log(
`Capability “${addedCapability.value.name}” ` +
`already on database.`,
);
} else {
console.log(
`Capability “${addedCapability.value.name}” ` +
`added to database.`,
);
if (rolesToPush && Array.isArray(rolesToPush)) {
rolesToPush.forEach(async (role) => {
const roleToPush = await Role.findOne({name: role});
if (roleToPush) {
roleToPush.capabilities.push(addedCapability.value);
await roleToPush.save();
}
});
}
}
}
};
const seedDb = async (roles, capabilities, users) => {
try {
await seedRoles(roles);
await seedCapabilities(capabilities);
console.log('roles', roles);
} catch (error) {
console.error(error);
}
};
module.exports = seedDb;

User ID unique violation discord.js?

Here is my code for the two functions I have coded.
Reflect.defineProperty(hp, 'add', {
value: async function add(id, amount) {
const user = hp.get(id);
if (user) {
user.health += Number(amount);
return user.save();
}
const newUser = await Users.create({
user_id: id,
health: amount
});
hp.set(id, newUser);
return newUser;
},
});
Reflect.defineProperty(hp, 'getHealth', {
value: function getHealth(id) {
const user = hp.get(id);
return user ? user.health : 0;
},
});
And when I try this: hp.add(message.author.id, 100);, I get this error:
errors: [
ValidationErrorItem {
message: 'user_id must be unique',
type: 'unique violation',
path: 'user_id',
value: '648183573748121610',
origin: 'DB',
instance: [users],
validatorKey: 'not_unique',
validatorName: null,
validatorArgs: []
}
],
fields: ['user_id'],
parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: users.user_id] {
errno: 19,
code: 'SQLITE_CONSTRAINT',
sql: 'INSERT INTO `users` (`user_id`,`balance`) VALUES ($1,$2);'
},
original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: users.user_id] {
errno: 19,
code: 'SQLITE_CONSTRAINT',
sql: 'INSERT INTO `users` (`user_id`,`balance`) VALUES ($1,$2);'
},
sql: 'INSERT INTO `users` (`user_id`,`balance`) VALUES ($1,$2);'
}
Could somebody tell me what is the problem? And also, could you please tell me the solution to the problem? More info: I'm creating a discord bot with discord.js. I am using Sequilize and SQLite to store the data.

I am quoting another user (Lioness100) on another question about ID's. It's actually not possible to get the ID of a discord user. You could make an array and then get the ID of a person you want, and then check from that array. But I think this image will resolve this question.
Quote:
Hoped this solved your question!

jest testing discord bot commands

So I have a file that I use module exports on and it has 4 fields among which an execute field that takes 2 args and is essentially a function. It doesn't return anything instead it uses discord.js and runs this message.channel.send('Pong');. I want to test this using jest
How do I:
1 - Make sure that the message.channel.send was called with 'Pong' as args
2 - How do I mock it so it doesnt actually call it (i just want to make sure that the text inside of it, like the actual argument is 'Pong' since calling it won't work due to the lack of a proper message object)
I can access the actual command and execute it but I am unsure as to how to check the contents of message.channel.send. The message object cannot be reconstructed by me so that might also need mocking.
I'm using discord.js but that shouldn't really matter.
I will also have to test commands that feature functions that do have returns so how should I go about them?

You can try this:
const Discord = require('discord.js')
// replace this with whatever the execute command is
// e.g. const ping = require('./commands/ping').execute
const ping = async (message, args) => {
message.channel.send('Pong')
}
// a counter so that all the ids are unique
let count = 0
class Guild extends Discord.Guild {
constructor(client) {
super(client, {
// you don't need all of these but I just put them in to show you all the properties that Discord.js uses
id: count++,
name: '',
icon: null,
splash: null,
owner_id: '',
region: '',
afk_channel_id: null,
afk_timeout: 0,
verification_level: 0,
default_message_notifications: 0,
explicit_content_filter: 0,
roles: [],
emojis: [],
features: [],
mfa_level: 0,
application_id: null,
system_channel_flags: 0,
system_channel_id: null,
widget_enabled: false,
widget_channel_id: null
})
this.client.guilds.cache.set(this.id, this)
}
}
class TextChannel extends Discord.TextChannel {
constructor(guild) {
super(guild, {
id: count++,
type: 0
})
this.client.channels.cache.set(this.id, this)
}
// you can modify this for other things like attachments and embeds if you need
send(content) {
return this.client.actions.MessageCreate.handle({
id: count++,
type: 0,
channel_id: this.id,
content,
author: {
id: 'bot id',
username: 'bot username',
discriminator: '1234',
bot: true
},
pinned: false,
tts: false,
nonce: '',
embeds: [],
attachments: [],
timestamp: Date.now(),
edited_timestamp: null,
mentions: [],
mention_roles: [],
mention_everyone: false
})
}
}
class Message extends Discord.Message {
constructor(content, channel, author) {
super(channel.client, {
id: count++,
type: 0,
channel_id: channel.id,
content,
author,
pinned: false,
tts: false,
nonce: '',
embeds: [],
attachments: [],
timestamp: Date.now(),
edited_timestamp: null,
mentions: [],
mention_roles: [],
mention_everyone: false
}, channel)
}
}
const client = new Discord.Client()
const guild = new Guild(client)
const channel = new TextChannel(guild)
// the user that executes the commands
const user = {id: count++, username: 'username', discriminator: '1234'}
describe('ping', () => {
it('sends Pong', async () => {
await ping(new Message('ping', channel, user))
expect(channel.lastMessage.content).toBe('Pong')
})
})
You also need to put testEnvironment: 'node' in your jest configuration (see this issue).
Edit
You can also use Discord.SnowflakeUtil.generate() to generate an id if you need to obtain things like the timestamp.

Mongoose Schema.index on multiple fields does not work with tests

I have created an uploadSchema (mongoose.Schema) with the fields (among the rest): key and bucket. each of them alone is not unique but together I want them to create a unique id.
in my code, I used the line (right after declaring the uploadSchema and right before the uploadModel):
uploadSchema.index({ key: 1, bucket: 1 }, { unique: true, background: true });
but then, in my tests (mocha and chai), the indexing is not enforced, and so I can create two instances with the same key and bucket (in my case).
for example, in my code:
await uploadModel.create({ key: testUpload.key, bucket: testUpload.bucket,
name: 'name1', ownerID: USER.id, parent: null }).should.eventually.exist;
and right after that:
await uploadModel.create({key: testUpload.key, bucket: testUpload.bucket,
name: 'name1', ownerID: USER.id, parent: null }).should.eventually.be.rejected;
does not throw the right error error:
AssertionError: expected promise to be rejected but it was fulfilled with { Object ($__, isNew, ...) }
Am I not using it correctly? Or is there a problem with indexing and testing?

Most likely you set autoIndex to false in your connection (which is recommended to do).
Either add it to you Schema:
let uploadSchema = mongoose.Schema({ ... }, {autoIndex: true});
But i would recommend just building the index yourself on the database, i think its the safest way around it.

so I figured it out!
Apparently, I used mongoose.connection.dropDatabase(); in my afterEach of the tests. That means that the indexes were reset each time.
So what I did was to recreate the indexes each time in my tests:
before(async () => {
// Remove files from DB
const collections = ['files', 'uploads'];
for (const i in collections) {
mongoose.connection.db.createCollection(collections[i], (err) => {});
}
await mongoose.connection.collections['files'].createIndex({ name: 1, parent: 1, ownerID: 1 }, { unique: true });
await mongoose.connection.collections['uploads'].createIndex({ key: 1, bucket: 1 }, { unique: true });
});
And in the beforeEach:
beforeEach(async () => {
const removeCollectionPromises = [];
for (const i in mongoose.connection.collections) {
removeCollectionPromises.push(mongoose.connection.collections[i].deleteMany({}));
}
await Promise.all(removeCollectionPromises);
});
afterEach is empty.
now it works :)

Asynchronous operations on Mongo trap: how can I perform kind of synchronously, I have correct logic

What I want to achieve: I want to create a car which has to have a correct mapping with its dealer and then store this car in the list/array of my company document present in my companies collection in myKaarma db.
The problem that I am facing: all the logic is correct but the problem is arising due to the asynchronous nature, even though I am using the callbacks.
I know the issue but don't know how to solve it.
Let me explain what the problem is:
My companies model using Mongoose:
// jshint node :true
"use strict";
const MONGOOSE = require("mongoose"),
DB = MONGOOSE.connection;
let companySchema = MONGOOSE.Schema({
company_name: {
type: String, required: true
},
company_location: {
type: String, require: true
},
cars: [{
model: {
type: String,required: true
},
year: {
type: Number, required: true
},
PriceInINR: {
type: Number, required: true
},
trim: {
type: String, required: true
},
engine: {
type: String, required: true
},
body: {
type: String, required: true
},
color: {
type: String, required: true
},
transmission_type: {
type: String, required: true
},
dealer_id: {
type: String, required: true
}
}]
});
let collection_name = "companies";
let CompanyModel = MONGOOSE.model(collection_name, companySchema);
createAscendingIndex_on_company_name(DB);
module.exports = CompanyModel;
// indexing at schema level --> using node js
function createAscendingIndex_on_company_name(DB, callback) {
let collection = DB.collection(collection_name);
// ? Create the index
collection.createIndex({
company_name: 1, // specifies : indexing type is ascending indexing
}, {
unique: true
}, function (err, result) {
if (err) {
console.log("error while setting up indexing on companies collection");
}
console.log("index created ", result, "<<<<<<<<", collection_name, " collection");
// callback("result");
});
}
//? NOTE : Creating indexes in MongoDB is an idempotent operation. So running db.names.createIndex({name:1}) would create the index only if it didn't already exist.
As you will notice that I have indexed and also made company_name as unique, so no duplicate entry can be there this is the problem
In my code when I do this: // ? check if the company exists : and if not then create one, the issue I have is, as the nodejs being asynchronous & very fast : [so suppose I have 5 car records] so all the five cars actually go into the code where I check:
CompanyModel.find({
company_name: company_name
}, (err, companies) => {.....}
and it is so fast that it like all go at the same time and as of course there does not exist any such company right now in the company document so all of them passes the if condition
if (companies.length === 0) {...});
so now when in my records there are 3 cars with same company all of them enter almost simultaneously and all passes these condition again simultaneously, but as soon as they pass this above condition I ask Mongo to create the company document
let company = new CompanyModel({
company_name: company_name,
company_location: company_location,
cars: [car]
});
company.save((err) => {...}
but now, all the 3 records are here to create a new company object and add to the collection. But here as soon as one of them create the document and added to the collection, at the same time now the other two also created their objects but now as there is already an object created and added so Mongo throws the unique exception here.
What I wanted to happen is when we found an duplicate object then then new car with the same company should be just pushed into the array that the company document has its with the field cars
Note: this scenario is only in the case of when the company does not present in the collection, but if it is already present then my code works fine, it successfully pushes all the cars into the cars field of the respective company.
This is the function that is doing what I want to:
function map_and_save_cars_in_garage() {
let carList = require("./test.json");
let totalCar = carList.length;
console.log(carList);
carList.forEach((carRecord, index) => {
let company_name = carRecord.make.toLowerCase();
let company_location = "USA";
// build a car
let car = {
model: carRecord.model,
year: carRecord.year,
PriceInINR: carRecord.priceInr,
trim: carRecord.trim,
engine: carRecord.engine,
body: carRecord.body,
color: carRecord.color,
transmission_type: carRecord.transmission,
dealer_id: undefined, // --> just for now
};
// ? search for the correct dealer --> for mapping
let dealer_email = "bitBattle_2018_" + carRecord.DealerID + "_#myKarmaa.com";
DealerModel.find({
email: dealer_email
}, (err, dealer) => {
if (err) {
console.log("Error : dealer not found for this car");
throw new Error(err);
}
car.dealer_id = dealer[0]._id; // ? update the dealer_id
// ? check if the company exists : and if not then create one
CompanyModel.find({
company_name: company_name
}, (err, companies) => {
if (err) {
console.log("Error : while finding the compay");
throw new Error(err);
}
console.log(company_name, companies);
if (companies.length === 0) {
console.log("No such Company car exists in the garrage --> creating one");
let company = new CompanyModel({
company_name: company_name,
company_location: company_location,
cars: [car]
});
company.save((err) => {
if (err) {
console.log("Error : while adding company ");
throw new Error(err);
}
console.log(index, "<<<<<<<< INDEX ", totalCar);
if (index === totalCar - 1) {
console.log("done");
res.send("build complete");
// build_complete();
}
});
} else {
console.log("Company already exists in garage : add this car with all other cars of this company");
let company = companies[0]; // ? as its sure that they are unique
let query = {
_id: company._id
};
let updat_command = {
$push: {
cars: car
}
};
CompanyModel.updateOne(query, updat_command, (err) => {
if (err) {
console.log("Error : while pushing car to the compay's cars");
throw new Error(err);
}
console.log(index, "<<<<<<<< INDEX ", totalCar);
if (index === totalCar - 1) {
console.log("done");
res.send("build complete");
// build_complete();
}
});
}
});
});
console.log(index, "<<<<<<<< INDEX--OUTER ", totalCar);
});
}
Output:
[nodemon] restarting due to changes...
[nodemon] starting `node app.js`
[[20:39:12.519]] [LOG] server live
[[20:39:12.626]] [LOG] Connected to DB : SUCCESS
[[20:39:12.642]] [LOG] index created email_1 <<<<<<<< buyers collection
[[20:39:12.647]] [LOG] index created email_1 <<<<<<<< dealers collection
[[20:39:12.795]] [LOG] index created company_name_1 <<<<<<<< companies collection
[[20:39:42.081]] [LOG] start saving cars
[[20:39:42.084]] [LOG] [ { id: '2',
vin: '5GAKRBKD9EJ323900',
make: 'Buick',
model: 'ENCLAVE',
year: '2014',
priceInr: '2537993',
trim: 'Leather FWD',
engine: 'SPORT UTILITY 4-DR',
body: '3.6L V6 DOHC 24V',
color: 'Silver',
transmission: 'Manual',
DealerID: '103' },
{ id: '4',
vin: '2GKALSEKXD6184074',
make: 'GMC',
model: 'TERRAIN',
year: '2013',
priceInr: '3851710',
trim: 'SLE2 FWD',
engine: 'SPORT UTILITY 4-DR',
body: '2.4L L4 DOHC 16V FFV',
color: 'Yellow',
transmission: 'Manual',
DealerID: '103' },
{ id: '6',
vin: '1GC1KXE86EF127166',
make: 'Chevrolet',
model: 'SILVERADO 2500HD',
year: '2014',
priceInr: '840547',
trim: 'LT Crew Cab 4WD',
engine: 'CREW CAB PICKUP 4-DR',
body: '6.6L V8 OHV 32V TURBO DIESEL',
color: 'Grey',
transmission: 'Automatic',
DealerID: '103' },
{ id: '8',
vin: '1GKKRTED1CJ211299',
make: 'GMC',
model: 'Acadia',
year: '2012',
priceInr: '3805008',
trim: 'Denali FWD',
engine: 'SPORT UTILITY 4-DR',
body: '3.6L V6 DOHC 24V',
color: 'Metallic White',
transmission: 'Automatic',
DealerID: '103' },
{ id: '10',
vin: '1GKKVTKD9EJ282303',
make: 'GMC',
model: 'ACADIA',
year: '2014',
priceInr: '1730235',
trim: 'Denali AWD',
engine: 'SPORT UTILITY 4-DR',
body: '3.6L V6 DOHC 24V',
color: 'Black',
transmission: 'Manual',
DealerID: '103' },
{ id: '12',
vin: '1GKS1AKC0FR200193',
make: 'GMC',
model: 'YUKON',
year: '2015',
priceInr: '3129397',
trim: 'SLE 2WD',
engine: 'SPORT UTILITY 4-DR',
body: '5.3L V8 OHV 16V',
color: 'Silver',
transmission: 'Manual',
DealerID: '103' } ]
[[20:39:42.089]] [LOG] 0 '<<<<<<<< INDEX--OUTER ' 6
[[20:39:42.089]] [LOG] 1 '<<<<<<<< INDEX--OUTER ' 6
[[20:39:42.090]] [LOG] 2 '<<<<<<<< INDEX--OUTER ' 6
[[20:39:42.090]] [LOG] 3 '<<<<<<<< INDEX--OUTER ' 6
[[20:39:42.090]] [LOG] 4 '<<<<<<<< INDEX--OUTER ' 6
[[20:39:42.090]] [LOG] 5 '<<<<<<<< INDEX--OUTER ' 6
[[20:39:42.120]] [LOG] gmc []
[[20:39:42.120]] [LOG] No such Company car exists in the garrage --> creating one
[[20:39:42.134]] [LOG] buick []
[[20:39:42.134]] [LOG] No such Company car exists in the garrage --> creating one
[[20:39:42.138]] [LOG] gmc []
[[20:39:42.138]] [LOG] No such Company car exists in the garrage --> creating one
[[20:39:42.143]] [LOG] chevrolet []
[[20:39:42.143]] [LOG] No such Company car exists in the garrage --> creating one
[[20:39:42.146]] [LOG] gmc []
[[20:39:42.146]] [LOG] No such Company car exists in the garrage --> creating one
[[20:39:42.150]] [LOG] 1 '<<<<<<<< INDEX ' 6
[[20:39:42.150]] [LOG] gmc []
[[20:39:42.151]] [LOG] No such Company car exists in the garrage --> creating one
[[20:39:42.153]] [LOG] 0 '<<<<<<<< INDEX ' 6
[[20:39:42.154]] [LOG] Error : while adding company
events.js:183
throw er; // Unhandled 'error' event
^
Error: MongoError: E11000 duplicate key error collection: myKaarma.companies index: company_name_1 dup key: { : "gmc" }
at company.save (/Users/prashant/Desktop/appathon/route/api.js:179:55)
at /Users/prashant/Desktop/appathon/node_modules/mongoose/lib/model.js:4437:16
at $__save.error (/Users/prashant/Desktop/appathon/node_modules/mongoose/lib/model.js:397:16)
at /Users/prashant/Desktop/appathon/node_modules/kareem/index.js:246:48
at next (/Users/prashant/Desktop/appathon/node_modules/kareem/index.js:167:27)
at next (/Users/prashant/Desktop/appathon/node_modules/kareem/index.js:169:9)
at Kareem.execPost (/Users/prashant/Desktop/appathon/node_modules/kareem/index.js:217:3)
at _handleWrapError (/Users/prashant/Desktop/appathon/node_modules/kareem/index.js:245:21)
at _cb (/Users/prashant/Desktop/appathon/node_modules/kareem/index.js:304:16)
at /Users/prashant/Desktop/appathon/node_modules/mongoose/lib/model.js:258:9
at /Users/prashant/Desktop/appathon/node_modules/kareem/index.js:135:16
at _combinedTickCallback (internal/process/next_tick.js:131:7)
at process._tickCallback (internal/process/next_tick.js:180:9)
How can I get out of this?

If you're using higher than node 7 (I hope so..) you can use async/await to make this code much simpler to deal with. You can also use findOne from mongoose so that you don't have to deal with arrays, since you know there is only one of each result.
The trick to this code working is that it waits until the previous car has been inserted into the database before inserting another one.
async function map_and_save_cars_in_garage() {
let carList = require("./test.json");
let totalCar = carList.length;
for (let carRecord of carList) {
let company_name = carRecord.make.toLowerCase();
let company_location = "USA";
// build a car
let car = {
model: carRecord.model,
year: carRecord.year,
PriceInINR: carRecord.priceInr,
trim: carRecord.trim,
engine: carRecord.engine,
body: carRecord.body,
color: carRecord.color,
transmission_type: carRecord.transmission,
dealer_id: undefined, // --> just for now
};
let dealer_email = "bitBattle_2018_" + carRecord.DealerID + "_#myKarmaa.com";
try {
let dealer = await DealerModel.findOne({
email: dealer_email
}).exec();
car.dealer_id = dealer._id;
let company = await CompanyModel.findOne({
company_name: company_name
}).exec();
if (!company) {
console.log("No such Company car exists in the garrage --> creating one");
let company = new CompanyModel({
company_name: company_name,
company_location: company_location,
cars: [car]
});
await company.save();
} else {
console.log("Company already exists in garage : add this car with all other cars of this company");
await CompanyModel.updateOne({
_id: company._id
}, {
$push: {
cars: car
}
}).exec();
}
} catch (err) {
throw new Error(err);
}
}
console.log("done");
res.send("build complete");
}
Another thing that I might try is not waiting for each car to be created but creating an array (which will be accessed instantly, compared to a database), containing the newly inserted companies, like so:
async function map_and_save_cars_in_garage() {
let carList = require("./test.json");
let totalCar = carList.length;
let newCompanies = {};
for (let carRecord of carList) {
(async function () {
let company_name = carRecord.make.toLowerCase();
let company_location = "USA";
// build a car
let car = {
model: carRecord.model,
year: carRecord.year,
PriceInINR: carRecord.priceInr,
trim: carRecord.trim,
engine: carRecord.engine,
body: carRecord.body,
color: carRecord.color,
transmission_type: carRecord.transmission,
dealer_id: undefined, // --> just for now
};
let dealer_email = "bitBattle_2018_" + carRecord.DealerID + "_#myKarmaa.com";
try {
let dealer = await DealerModel.findOne({
email: dealer_email
}).exec();
car.dealer_id = dealer._id;
// Check for company in newCompanies
let company = newCompanies[company_name];
// If company is not in newcompanies it will be undefined so this if statement will be executed
if (!company) {
// If company is not found in database this will be null
await CompanyModel.findOne({
company_name: company_name
}).exec();
}
// If company is null then create a new one
if (!company) {
console.log("No such Company car exists in the garrage --> creating one");
let company = new CompanyModel({
company_name: company_name,
company_location: company_location,
cars: [car]
});
// Add company to newCompanies
newCompanies[company_name] = company;
await company.save();
} else {
console.log("Company already exists in garage : add this car with all other cars of this company");
await CompanyModel.updateOne({
_id: company._id
}, {
$push: {
cars: car
}
}).exec();
}
} catch (err) {
throw new Error(err);
}
})();
}
console.log("done");
res.send("build complete");
}
This will not have to wait for previous cars to be added to the database.