Develop Reference

Rename database field within array in MongoDB

I need to change a few database fields in my backend controller before returning the object to the frontend.
Currently, I am doing it in the front end like this with my returned object:
for (let contribution of contributions) {
contribution["title"] = "You Added One"
contribution["launchName"] = contribution.name
contribution["launchId"] = contribution._id
contribution["date"] = contribution.addedAt
contribution["content"] = contribution.description
}
But I am now trying to do this work in the backend using Mongo.
This is my controller:
const Launch = require('../models/launch')
const User = require('../models/user')
async function getRecentActivityByUserId (req, res) {
const { userId } = req.params
const user = await User.findOne({ userId }).lean() || []
const contributions = await Launch.find({ _id: { $in: user.contributions } })
return res.status(200).send(contributions.reverse())
}
So this correctly returns an object to the frontend but I still need to change the database field names.
So I tried this:
async function getRecentActivityByUserId (req, res) {
let recents = []
const { userId } = req.params
const user = await User.findOne({ userId }).lean() || []
const contributions = await Launch.find({ _id: { $in: user.contributions } }).aggregate([
{
$addFields: {
plans: {
$map:{
input: "$launch",
as: "l",
in: {
title: "You Added One",
launchName: "$$l.name",
launchId: "$$l._id",
date: "$$l.addedAt",
content: "$$l.description",
}
}
}
}
},
{
$out: "launch"
}
])
return res.status(200).send(contributions.reverse())
}
The above throws an error saying that I .aggregrate is not a function on .find. Even if I remove the .find, the object returned is just an empty array so I'm obviously not aggregating correctly.
How can I combine .find with .aggregate and what is wrong with my .aggregate function??
I also tried combining aggregate with find like this and get the error Arguments must be aggregate pipeline operators:
const contributions = await Launch.aggregate([
{
$match: {
_id: { $in: user.contributions }
},
$addFields: {
plans: {
$map:{
input: "$launch",
as: "l",
in: {
title: "You Added a Kayak Launch",
launchName: "$$l.name",
launchId: "$$l._id",
date: "$$l.addedAt",
content: "$$l.description",
}
}
}
}
},
{
$out: "launch"
}
])
EDIT: Just realized that I have the word plans in the aggregate function and that is not relevant to my code. I copied this code from elsewhere so not sure what the value should be.

I figured it out. This is the solution:
async function getRecentActivityByUserId (req, res) {
let recents = []
const { userId } = req.params
const user = await User.findOne({ userId }).lean() || []
const contributions = await Launch.aggregate([
{
$match: {
_id: { $in: user.contributions }
}
},
{
$addFields: {
title: "You Added One" ,
launchName: "$name",
launchId: "$_id",
date: "$addedAt",
content: "$description"
}
}
])
if(contributions) {
recents = recents.concat(contributions);
}
return res.status(200).send(recents.reverse())
}

The actual problem from the question was a small syntax error which has been noted and corrected in the self-reported answer here.
I noted in the comments there that the current approach of issuing two separate operations (a findOne() followed by an aggregate() that uses the results) could be simplified into a single query to the database. The important thing here is that you will $match against the first collection (users or whatever the collection name is in your environment) and then use $lookup to perform the "match" against the subsequent launches collection.
Here is a playground demonstrating the basic approach. Adjust as needed to the specifics of your environment.

Asynchronous verification within the .map function

I am developing the backend of an application using Node JS, Sequelize and Postgres database.
When the course is registered, the user must inform which organizations, companies and teachers will be linked to it.
The organization IDs are passed through an array to the backend, I am trying to do a check to make sure that the passed IDs exist.
What I've done so far is this:
const { organizations } = req.body;
const organizationsArray = organizations.map(async (organization) => {
const organizationExists = await Organization.findByPk(organization);
if (!organizationExists) {
return res
.status(400)
.json({ error: `Organization ${organization} does not exists!` });
}
return {
course_id: id,
organization_id: organization,
};
});
await CoursesOrganizations.bulkCreate(organizationsArray);
This link has the complete controller code, I believe it will facilitate understanding.
When !OrganizationExists is true, I am getting the return that the organization does not exist. The problem is when the organization exists, I am getting the following message error.

The Array.map() is returning an array of promises that you can resolve to an array using Promise.all(). Inside the map you should use throw new Error() to break out of the map - this error will be raised by Promise.all() and you can then catch it and return an error to the client (or swallow it, etc).
This is a corrected version of your pattern, resolving the Promise results.
const { organizations } = req.body;
try {
// use Promise.all to resolve the promises returned by the async callback function
const organizationsArray = await Promise.all(
// this will return an array of promises
organizations.map(async (organization) => {
const organizationExists = await Organization.findByPk(organization, {
attributes: ['id'], // we only need the ID
raw: true, // don't need Instances
});
if (!organizationExists) {
// don't send response inside the map, throw an Error to break out
throw new Error(`Organization ${organization} does not exists!`);
}
// it does exist so return/resolve the value for the promise
return {
course_id: id,
organization_id: organization,
};
})
);
// if we get here there were no errors, create the records
await CoursesOrganizations.bulkCreate(organizationsArray);
// return a success to the client
return res.json({ success: true });
} catch (err) {
// there was an error, return it to the client
return res.status(400).json({ error: err.message });
}
This is a refactored version that will be a bit faster by fetching all the Organizations in one query and then doing the checks/creating the Course inserts.
const { Op } = Sequelize;
const { organizations } = req.body;
try {
// get all Organization matches for the IDs
const organizationsArray = await Organization.findAll({
attributes: ['id'], // we only need the ID
where: {
id: {
[Op.in]: organizations, // WHERE id IN (organizations)
}
},
raw: true, // no need to create Instances
});
// create an array of the IDs we found
const foundIds = organizationsArray.map((org) => org.id);
// check to see if any of the IDs are missing from the results
if (foundIds.length !== organizations.length) {
// Use Array.reduce() to figure out which IDs are missing from the results
const missingIds = organizations.reduce((missingIds, orgId) => {
if (!foundIds.includes(orgId)){
missingIds.push(orgId);
}
return missingIds;
}, []); // initialized to empty array
throw new Error(`Unable to find Organization for: ${missingIds.join(', ')}`);
}
// now create an array of courses to create using the foundIds
const courses = foundIds.map((orgId) => {
return {
course_id: id,
organization_id: orgId,
};
});
// if we get here there were no errors, create the records
await CoursesOrganizations.bulkCreate(courses);
// return a success to the client
return res.json({ success: true });
} catch (err) {
// there was an error, return it to the client
return res.status(400).json({ error: err.message });
}

If you have an array of Ids and you want to check if they exist you should you use the (in) operator, this makes it so that you are hitting the DB only once and getting all the records in one hit (instead of getting them one by one in a loop), after you get these records you can check their lengths to determine if they all exist or not.
const { Op } = require("sequelize");
let foundOrgs = await Organization.findAll({
where: {
id: {
[Op.in]: organizationsArray,
}
}
});

DynamoDB PutItem not awaiting, or executing finally blocks

My Lambda takes in an SQS message containing an ID and address. It parses out those fields, and updates the record associated with that ID in a dynamo table.
The parameters for this update contain the following logic
1.Where the record has a id equal to the ID sent by SQS
2.And where the SortKey has a value equal to “null” (Note that null is specifically a String with a value of “null”)
3.Update the address field with the new address
I'm seeing the following issues with this function
The function is not updating the DynamoDB instance
I am not receiving any kind of feedback from the update call. Looking over the code there are several console.logs that should execute but are not. See the Try,Catch,Finally block after the update. Looking at the logs you can see that these do not output to the console. Something is very wrong here. The finally not executing looks like undefined behavior, my only guess is that the call to dynamodb is not being awaited
I also need to implement the following functionality. This is bonus points, if you have an idea of how to do it please feel free to comment!
Right now the update will only change the fieldname of address from one value to another. Instead, I need the record to contain a set of addresses associated with that record. To do this we need to implement the following logic
If a set of addresses does not exist on the record, create a set with the address as the only element
If a set does exist on the student record, update that set with the address. Duplicate addresses should not be added
The code for this function is below. I’ve also attached the most recent CloudWatch log for this function, and the record I am trying to update (the address field on the record was added manually). You’ll notice that we aren’t getting any console.logs after console.log("starting upload"), and the promise has a state “PENDING” when it is examined. We also don’t get any feedback from the dynamodb update. Right now the function is not updating the record, and not giving me any feedback for why it is failing to do so.
const util = require('util')
const aws = require('aws-sdk');
const docClient = new aws.DynamoDB.DocumentClient();
exports.handler = async(event) => {
event.Records.forEach(async record => {
const { body } = record;
const test = JSON.parse(body);
console.log(test);
const message = JSON.parse(test["Message"]);
console.log(message);
const id = message.id;
const name = message.Name;
const address = message.address;
console.log("parameters parsed");
console.log("record being processed is " + id);
const params = {
TableName: "My_Records",
Key: {
"ID": ":id",
"SortKey": ":sortKey"
},
//KeyConditionExpression: 'SortKey = :sortKey',
UpdateExpression: "set info.address = :address",
ExpressionAttributeValues: {
':id': id,
':address': address,
':sortKey': "null"
},
ReturnValues: "UPDATED_NEW"
};
console.log(params)
console.log("starting upload")
try {
let putObjectPromise = docClient.update(params).promise();
console.log(util.inspect(putObjectPromise, {showHidden: false, depth: null}))
putObjectPromise.then(function(data) {
console.log("UpdateItem succeeded:");
}).catch(function(err) {
console.log("Unable to update item. Error JSON:" + err);
}).finally(() =>
console.log("done with upload")
);
return putObjectPromise
}
catch (err) {
console.err(err)
}
});
};
CloudWatch log of the most recent execution of this function
INFO {
Type: 'Notification',
MessageId: 'ID',
TopicArn: 'ARN',
Subject: 'DB updated',
SignatureVersion: '1',
INFO { id: '11111111', Name: 'Jerms Macgee', address: '102 homeslice lane' }
INFO parameters parsed
INFO record being processed is 11111111
INFO {
TableName: 'my_table',
Key: { ID: ':id', SortKey: ':sortKey' },
UpdateExpression: 'set info.address = :address',
ExpressionAttributeValues: {
':id': '11111111',
':address': '102 homeslice lane',
':sortKey': 'null'
},
ReturnValues: 'UPDATED_NEW'
}
INFO starting upload
INFO Promise { <pending> }
END RequestId
And here's an example of the record I'd expect to be updated
{
"address": "test",
"SortKey": "null",
"id": 11111111
"name": James Mcgee
}
The updated record should be
{
"address": "102 homeslice lane",
"SortKey": "null",
"id": 11111111
"name": James Mcgee
}
And for bonus points I'd really like to do something like
{
"address": {"102 homeslice lane"},
"SortKey": "null",
"id": 11111111
"name": James Mcgee
}
where addresses is a set that can hold other records

First, forEach won't work with async/await callback. The example from https://codeburst.io/javascript-async-await-with-foreach-b6ba62bbf404
const waitFor = (ms) => new Promise(r => setTimeout(r, ms));
[1, 2, 3].forEach(async (num) => {
await waitFor(50);
console.log(num);
});
console.log('Done');
Second, if you wrap a rejected promise in a try/catch block and that promise already has a .catch() set, catch block will never be executed.
const waitFor = (ms) => new Promise((resolve, reject) => {
setTimeout(() => {
reject(123);
}, ms)
});
try {
waitFor(2000).catch(e => { console.log(e) })
} catch (error) {
console.error('error');
}

You're mixing up your async/promise stuff.
First, event.Records.forEach isn't going to wait for the async function you're passing to it to resolve, you can change it to:
await Promise.all(event.Records.map(async record => {
///... the rest of your function body
});
This way your main handler function will actually wait for them all to resolve.
Next, all this stuff:
try {
let putObjectPromise = docClient.update(params).promise();
console.log(util.inspect(putObjectPromise, {showHidden: false, depth: null}))
putObjectPromise.then(function(data) {
console.log("UpdateItem succeeded:");
}).catch(function(err) {
console.log("Unable to update item. Error JSON:" + err);
}).finally(() =>
console.log("done with upload")
);
return putObjectPromise
}
catch (err) {
console.err(err)
}
is weird, you're using .then and callback functions, but you're in an async function so you can just await them. Eg:
try {
const putObjectResponse = await docClient.update(params).promise();
console.log("UpdateItem succeeded:");
console.log(JSON.stringify(putObjectResponse));
}
catch (err) {
console.log("Unable to update item. Error JSON:" + err);
console.err(err)
}
console.log("done with upload")
By awaiting update(params).promise() the return value becomes what the promise resolves to, not the promise. If the promise rejects, it is thrown and caught in your catch block.
This also fixes your weird logging messages because you're now logging the resolved value from the promise rather than the promise itself.

push array in const inside .then

I'm trying to push a value inside a const but its in a .then and it's not working do you know how can I do that ?
I get a value in my console.log(newResult) in my if but the data is not pushed in my const newResult in the return
res.status(200).json(newResult);
.then(function (friends) {
if (friends) {
const newResult = [];
friends.forEach((r) => {
if (r.UserID == userFound.id) {
models.User.findOne({
where: {
id: r.idFriend
}
})
.then(function(userFound) {
newResult.push({
id: r.id,
user: {
id: r.User.id,
email: userFound.email,
username: userFound.username
}
});
console.log(newResult)
})
} else
newResult.push({
id: r.id,
user: {
id: r.User.id,
email: r.User.email,
username: r.User.username
}
});
console.log(newResult)
});
res.status(200).json(newResult);
}
}
every test realised return an empty tab when i go in my if condition

It will never work because, you are doing async calls
models.User.findOne inside forEach.
You'll get results on console.log when async call to database for fetching user is complete.
But before this all happens the forEach is done executing and code hits the line res.status(200).json(newResult); and you see no results from your if condition.
Instead of using this approach go for mongoose populate and populate userObject based userID while finding friends this way you won't have to do async call inside the forEach.
Read about mongoose populate at: http://mongoosejs.com/docs/populate.html

Execute Sequelize queries synchronously

I am building a website using Node.js and Sequelize (with a Postgres backend). I have a query that returns many objects with a foreign key, and I want to pass to the view a list of the objects that the foreign key references.
In the example, Attendances contains Hackathon keys, and I want to return a list of hackathons. Since the code is async, the following thing of course does not work in Node:
models.Attendance.findAll({
where: {
UserId: req.user.id
}
}).then(function (data) {
var hacks = [];
for (var d in data) {
models.Hackathon.findOne({
where: {
id: data[d].id
}
}).then(function (data1) {
hacks.append(data1);
});
}
res.render('dashboard/index.ejs', {title: 'My Hackathons', user: req.user, hacks: hacks});
});
Is there any way to do that query in a synchronous way, meaning that I don't return the view untill I have the "hacks" list filled with all the objects?
Thanks!

Use Promise.all to execute all of your queries then call the next function.
models.Attendance.findAll({
where: {
UserId: req.user.id
}
}).then(function (data) {
// get an array of the data keys, (not sure if you need to do this)
// it is unclear whether data is an object of users or an array. I assume
// it's an object as you used a `for in` loop
const keys = Object.keys(data)
// map the data keys to [Promise(query), Promise(query), {...}]
const hacks = keys.map((d) => {
return models.Hackathon.findOne({
where: {
id: data[d].id
}
})
})
// user Promise.all to resolve all of the promises asynchronously
Promise.all(hacks)
// this will be called once all promises have resolved so
// you can modify your data. it will be an array of the returned values
.then((users) => {
const [user1, user2, {...}] = users
res.render('dashboard/index.ejs', {
title: 'My Hackathons',
user: req.user,
hacks: users
});
})
});

The Sequelize library has the include parameter which merges models in one call. Adjust your where statement to bring the Hackathons model into Attendance. If this does not work, take the necessary time to setup Sequelize correctly, their documentation is constantly being improved. In the end, you'll save loads of time by reducing error and making your code readable for other programmers.
Look how much cleaner this can be...
models.Attendance.findAll({
include: [{
model: Hackathon,
as: 'hackathon'
},
where: {
UserId: req.user.id
}
}).then(function (data) {
// hackathon id
console.log(data.hackathon.id)
// attendance id
console.log(data.id)
})
Also..
Hackathon.belongsTo(Attendance)
Attendance.hasMany(Hackathon)
sequelize.sync().then(() => {
// this is where we continue ...
})
Learn more about Sequelize includes here:
http://docs.sequelizejs.com/en/latest/docs/models-usage/

Immediately invoke asynchronous function expression
This is one of the techniques mentioned at: How can I use async/await at the top level? Toplevel await is likely coming soon as of 2021, which will be even better.
Minimal runnable example:
const assert = require('assert');
const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize({
dialect: 'sqlite',
storage: 'db.sqlite',
});
const IntegerNames = sequelize.define(
'IntegerNames', {
value: { type: DataTypes.INTEGER, allowNull: false },
name: { type: DataTypes.STRING, },
}, {});
(async () => {
await IntegerNames.sync({force: true})
await IntegerNames.create({value: 2, name: 'two'});
await IntegerNames.create({value: 3, name: 'three'});
await IntegerNames.create({value: 5, name: 'five'});
// Fill array.
let integerNames = [];
integerNames.push(await IntegerNames.findOne({
where: {value: 2}
}));
integerNames.push(await IntegerNames.findOne({
where: {value: 3}
}));
// Use array.
assert(integerNames[0].name === 'two');
assert(integerNames[1].name === 'three');
await sequelize.close();
})();
Tested on Node v14.16.0, sequelize 6.6.2, seqlite3 5.0.2, Ubuntu 20.10.