Develop Reference

Asynchronous verification within the .map function

I am developing the backend of an application using Node JS, Sequelize and Postgres database.
When the course is registered, the user must inform which organizations, companies and teachers will be linked to it.
The organization IDs are passed through an array to the backend, I am trying to do a check to make sure that the passed IDs exist.
What I've done so far is this:
const { organizations } = req.body;
const organizationsArray = organizations.map(async (organization) => {
const organizationExists = await Organization.findByPk(organization);
if (!organizationExists) {
return res
.status(400)
.json({ error: `Organization ${organization} does not exists!` });
}
return {
course_id: id,
organization_id: organization,
};
});
await CoursesOrganizations.bulkCreate(organizationsArray);
This link has the complete controller code, I believe it will facilitate understanding.
When !OrganizationExists is true, I am getting the return that the organization does not exist. The problem is when the organization exists, I am getting the following message error.

The Array.map() is returning an array of promises that you can resolve to an array using Promise.all(). Inside the map you should use throw new Error() to break out of the map - this error will be raised by Promise.all() and you can then catch it and return an error to the client (or swallow it, etc).
This is a corrected version of your pattern, resolving the Promise results.
const { organizations } = req.body;
try {
// use Promise.all to resolve the promises returned by the async callback function
const organizationsArray = await Promise.all(
// this will return an array of promises
organizations.map(async (organization) => {
const organizationExists = await Organization.findByPk(organization, {
attributes: ['id'], // we only need the ID
raw: true, // don't need Instances
});
if (!organizationExists) {
// don't send response inside the map, throw an Error to break out
throw new Error(`Organization ${organization} does not exists!`);
}
// it does exist so return/resolve the value for the promise
return {
course_id: id,
organization_id: organization,
};
})
);
// if we get here there were no errors, create the records
await CoursesOrganizations.bulkCreate(organizationsArray);
// return a success to the client
return res.json({ success: true });
} catch (err) {
// there was an error, return it to the client
return res.status(400).json({ error: err.message });
}
This is a refactored version that will be a bit faster by fetching all the Organizations in one query and then doing the checks/creating the Course inserts.
const { Op } = Sequelize;
const { organizations } = req.body;
try {
// get all Organization matches for the IDs
const organizationsArray = await Organization.findAll({
attributes: ['id'], // we only need the ID
where: {
id: {
[Op.in]: organizations, // WHERE id IN (organizations)
}
},
raw: true, // no need to create Instances
});
// create an array of the IDs we found
const foundIds = organizationsArray.map((org) => org.id);
// check to see if any of the IDs are missing from the results
if (foundIds.length !== organizations.length) {
// Use Array.reduce() to figure out which IDs are missing from the results
const missingIds = organizations.reduce((missingIds, orgId) => {
if (!foundIds.includes(orgId)){
missingIds.push(orgId);
}
return missingIds;
}, []); // initialized to empty array
throw new Error(`Unable to find Organization for: ${missingIds.join(', ')}`);
}
// now create an array of courses to create using the foundIds
const courses = foundIds.map((orgId) => {
return {
course_id: id,
organization_id: orgId,
};
});
// if we get here there were no errors, create the records
await CoursesOrganizations.bulkCreate(courses);
// return a success to the client
return res.json({ success: true });
} catch (err) {
// there was an error, return it to the client
return res.status(400).json({ error: err.message });
}

If you have an array of Ids and you want to check if they exist you should you use the (in) operator, this makes it so that you are hitting the DB only once and getting all the records in one hit (instead of getting them one by one in a loop), after you get these records you can check their lengths to determine if they all exist or not.
const { Op } = require("sequelize");
let foundOrgs = await Organization.findAll({
where: {
id: {
[Op.in]: organizationsArray,
}
}
});

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Object

I am using the source code from a security rules tutorial to attempt to do integration testing with Jest for my Javascript async function async_create_post, used for my firebase HTTP function create_post The files involved has a directory structure of the following:
Testing file: root/tests/handlers/posts.test.js
File to be tested: root/functions/handlers/posts.js
Helper code from the tutorial: root/tests/rules/helpers.js
And here is the source code that is involved:
posts.test.js
const { setup, teardown} = require("../rules/helpers");
const {
async_get_all_undeleted_posts,
async_get_post,
async_delete_post,
async_create_post
} = require("../../functions/handlers/posts");
describe("Post Creation", () => {
afterEach(async () => {
await teardown();
});
test("should create a post", async () => {
const db = await setup();
const malloryUID = "non-existent uid";
const firstPost = {
body: "First post from Mallory",
author_id: malloryUID,
images: ["url1", "url2"]
}
const before_post_snapshot = await db.collection("posts").get();
expect(before_post_snapshot.docs.length).toBe(0);
await async_create_post(firstPost); //fails at this point, expected to create a new post, but instead threw an error
const after_post_snapshot = await db.collection("posts").get();
expect(after_post_snapshot.docs.length).toBe(1);
});
});
posts.js
const {admin, db } = require('../util/admin');
//admin.initializeApp(config); //my credentials
//const db = admin.firestore();
const { uuid } = require("uuidv4");
const {
success_response,
error_response
} = require("../util/validators");
exports.async_create_post = async (data, context) => {
try {
const images = [];
data.images.forEach((url) => {
images.push({
uid: uuid(),
url: url
});
})
const postRecord = {
body: data.body,
images: images,
last_updated: admin.firestore.FieldValue.serverTimestamp(),
like_count: 0,
comment_count: 0,
deleted: false,
author_id: data.author_id
};
const generatedToken = uuid();
await db
.collection("posts")
.doc(generatedToken)
.set(postRecord);
// return success_response();
return success_response(generatedToken);
} catch (error) {
console.log("Error in creation of post", error);
return error_response(error);
}
}
When I run the test in Webstorm IDE, with 1 terminal running Firebase emulators:start , I get the following error message.
console.log
Error in creation of post TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Object
at validateString (internal/validators.js:120:11)
at Object.basename (path.js:1156:5)
at GrpcClient.loadProto (/Users/isaac/Desktop/project/functions/node_modules/google-gax/src/grpc.ts:166:23)
at new FirestoreClient (/Users/isaac/Desktop/project/functions/node_modules/#google-cloud/firestore/build/src/v1/firestore_client.js:118:38)
at ClientPool.clientFactory (/Users/isaac/Desktop/project/functions/node_modules/#google-cloud/firestore/build/src/index.js:330:26)
at ClientPool.acquire (/Users/isaac/Desktop/project/functions/node_modules/#google-cloud/firestore/build/src/pool.js:87:35)
at ClientPool.run (/Users/isaac/Desktop/project/functions/node_modules/#google-cloud/firestore/build/src/pool.js:164:29)
at Firestore.request (/Users/isaac/Desktop/project/functions/node_modules/#google-cloud/firestore/build/src/index.js:961:33)
at WriteBatch.commit_ (/Users/isaac/Desktop/project/functions/node_modules/#google-cloud/firestore/build/src/write-batch.js:485:48)
at exports.async_create_post (/Users/isaac/Desktop/project/functions/handlers/posts.js:36:5) {
code: 'ERR_INVALID_ARG_TYPE'
}
at exports.async_create_post (/Users/isaac/Desktop/project/functions/handlers/posts.js:44:13)
Error: expect(received).toBe(expected) // Object.is equality
Expected: 1
Received: 0
<Click to see difference>
at Object.<anonymous> (/Users/isaac/Desktop/project/tests/handlers/posts.test.js:59:45)
Error in creation of post comes from the console.log("Error in creation of post", error); in posts.js, so the error is shown in the title of this post.
I want to know why calling the async_create_post from posts.test.js will cause this error and does not populate my database with an additional record as expected behaviour. Do inform me if more information is required to solve the problem.
Here are some code snippets that may give more context.
helpers.js [Copied from the repository]
const firebase = require("#firebase/testing");
const fs = require("fs");
module.exports.setup = async (auth, data) => {
const projectId = `rules-spec-${Date.now()}`;
const app = firebase.initializeTestApp({
projectId,
auth
});
const db = app.firestore();
// Apply the test rules so we can write documents
await firebase.loadFirestoreRules({
projectId,
rules: fs.readFileSync("firestore-test.rules", "utf8")
});
// write mock documents if any
if (data) {
for (const key in data) {
const ref = db.doc(key); // This means the key should point directly to a document
await ref.set(data[key]);
}
}
// Apply the actual rules for the project
await firebase.loadFirestoreRules({
projectId,
rules: fs.readFileSync("firestore.rules", "utf8")
});
return db;
// return firebase;
};
module.exports.teardown = async () => {
// Delete all apps currently running in the firebase simulated environment
Promise.all(firebase.apps().map(app => app.delete()));
};
// Add extensions onto the expect method
expect.extend({
async toAllow(testPromise) {
let pass = false;
try {
await firebase.assertSucceeds(testPromise);
pass = true;
} catch (error) {
// log error to see which rules caused the test to fail
console.log(error);
}
return {
pass,
message: () =>
"Expected Firebase operation to be allowed, but it was denied"
};
}
});
expect.extend({
async toDeny(testPromise) {
let pass = false;
try {
await firebase.assertFails(testPromise);
pass = true;
} catch (error) {
// log error to see which rules caused the test to fail
console.log(error);
}
return {
pass,
message: () =>
"Expected Firebase operation to be denied, but it was allowed"
};
}
});
index.js
const functions = require('firebase-functions');
const {
async_get_all_undeleted_posts,
async_get_post,
async_delete_post,
async_create_post
} = require('./handlers/posts');
exports.create_post = functions.https.onCall(async_create_post);

The error message means that a method of the path module (like path.join) expects one of its arguments to be a string but got something else.
I found the offending line by binary search commenting the program until the error was gone.
Maybe one of your modules uses path and you supply the wrong arguments.

Best practise to combine multiple rest calls to populate 1 graphQL type in apollo-server

I have graphql User type that needs information from multiple REST api's and different servers.
Basic example: get the user firstname from rest domain 1 and get lastname from rest domain 2. Both rest domain have a common "userID" attribute.
A simplefied example of my resolver code atm:
user: async (_source, args, { dataSources }) => {
try {
const datasource1 = await dataSources.RESTAPI1.getUser(args.id);
const datasource2 = await dataSources.RESTAPI2.getUser(args.id);
return { ...datasource1, ...datasource2 };
} catch (error) {
console.log("An error occurred.", error);
}
return [];
}
This works fine for this simplefied version, but I have 2 problems with this solution:
first, IRL there is a lot of logic going into merging the 2 json results. Since some field are shared but have different data (or are empty). So it's like cherry picking both results to create a combined result.
My second problem is that this is still a waterfall method. First get the data from restapi1, when thats done call restapi2. Basicly apollo-server is reintroducing rest-waterfall-fetch graphql tries to solve.
Keeping these 2 problems in mind.. Can I optimise this piece of code or rewrite is for better performance or readability? Or are there any packages that might help with this behavior?
Many thanks!

With regard to performance, if the two calls are independent of one another, you can utilize Promise.all to execute them in parallel:
const [dataSource1,dataSource2] = await Promise.all([
dataSources.RESTAPI1.getUser(args.id),
dataSources.RESTAPI2.getUser(args.id),
])
We normally let GraphQL's default resolver logic do the heavy lifting, but if you're finding that you need to "cherry pick" the data from both calls, you can return something like this in your root resolver:
return { dataSource1, dataSource2 }
and then write resolvers for each field:
const resolvers = {
User: {
someField: ({ dataSource1, dataSource2 }) => {
return dataSource1.a || dataSource2.b
},
someOtherField: ({ dataSource1, dataSource2 }) => {
return someCondition ? dataSource1.foo : dataSource2.bar
},
}
}

Assuming your user resolver returns type User forsake...
type User {
id: ID!
datasource1: RandomType
datasource1: RandomType
}
You can create individual resolvers for each field in type User, this can reduce the complexity of the user Query, to only the requested fields.
query {
user {
id
datasource1 {
...
}
}
}
const resolvers = {
Query: {
user: () => {
return { id: "..." };
}
},
User: {
datasource1: () => { ... },
datasource2: () => { ... } // i wont execute
}
};
datasource1 & datasource2 resolvers will only execute in parallel, after Query.user executes.

For parallel call.
const users = async (_source, args, { dataSources }) => {
try {
const promises = [
dataSources.RESTAPI1,
dataSources.RESTAPI2
].map(({ getUser }) => getUser(args.id));
const data = await Promise.all(promises);
return Object.assign({}, ...data);
} catch (error) {
console.log("An error occurred.", error);
}
return [];
};

Issues with scope in try/catch while using async/await

My issue is that (seemingly) things are going out of scope, or the scope is being polluted when I enter my catch block in the function below:
export const getOne = model => async (req, res, next) => {
let id = req.params.id
let userId = req.user
try {
let item = await model.findOne({ _id: id, createdBy: userId }).exec()
if (!item) {
throw new Error('Item not found!')
} else {
res.status(200).json({ data: item }) // works perfectly
}
} catch (e) {
res.status(400).json({ error: e }) // TypeError: res.status(...).json is not a function
// also TypeError: next is not a function
// next(e)
}
}
Interestingly enough, using res.status(...).end() in the catch block works just fine, but it bothers me that I am not able to send any detail back with the response. According to the Express Documentation for res.send() and res.json I should be able to chain off of .status(), which, also interestingly enough, works just fine in the try statement above if things are successful - res.status(200).json(...) works perfectly.
Also, I tried abstracting the error handling to middleware, as suggested on the Express documentation, and through closures, I should still have access to next in the catch statement, right? Why is that coming back as not a function?
Why does res.status(...).json(...) work in my try but not catch block?
Why is next no longer a function in the catch block?
Thanks in advance!
Edit
This is failing in unit tests, the following code produces the errors described above:
describe('getOne', async () => {
// this test passes
test('finds by authenticated user and id', async () => {
expect.assertions(2)
const user = mongoose.Types.ObjectId()
const list = await List.create({ name: 'list', createdBy: user })
const req = {
params: {
id: list._id
},
user: {
_id: user
}
}
const res = {
status(status) {
expect(status).toBe(200)
return this
},
json(result) {
expect(result.data._id.toString()).toBe(list._id.toString())
}
}
await getOne(List)(req, res)
})
// this test fails
test('400 if no doc was found', async () => {
expect.assertions(2)
const user = mongoose.Types.ObjectId()
const req = {
params: {
id: mongoose.Types.ObjectId()
},
user: {
_id: user
}
}
const res = {
status(status) {
expect(status).toBe(400)
return this
},
end() {
expect(true).toBe(true)
}
}
await getOne(List)(req, res)
})
})

Why does res.status(...).json(...) work in my try but not catch block?
Seems like you're passing a non-express object that only has status & end methods when running using the unit testing. That's why it fails to find the json method

How to test Meteor.users with Mocha

I have a function as follows:
if(Meteor.isServer) {
Meteor.methods({
addUser: function (newUser) {
check(newUser, { email: String, password: String });
userId = Accounts.createUser(newUser);
return userId;
},
getUser: function (userID) {
check(userID, String);
return Meteor.users.find({_id: userID}).fetch();
}
});
And I am trying to test this function using Mocha:
if (Meteor.isServer) {
let testUser;
describe('Users', () => {
it("Add User", (done) => {
testUser = {email: 'test#test.test', password: 'test'};
try {
testUser._id = Meteor.call('addUser', testUser);
console.log(Accounts.users.find({_id: testUser._id}).fetch());
done();
} catch (err) {
assert.fail();
}
});
it("Get user", (done) => {
try {
Meteor.call('getUser', testUser._id);
done();
} catch (err) {
assert.fail();
}
});
});
And I know that the meteor call with 'addUser' works, because the console.log after that returns the user that I just made and the first test passes when I run it with "meteor test --driver-package practicalmeteor:mocha"
But then I come to the second testing part, where I try to get the user with the meteor call 'getUser', but then I get stuck:
'Cannot call method 'find' of undefined'
Now I know that the difference is that I use 'Meteor.users' instead of 'Account.users' to find the user, but I am totally in the dark what the difference is between these two. Should I replace all the Meteor.users method calls with Accounts.user method calls or not? How would you test this?

I just stumbled on this post, since I have dealt with the same issue some hours ago.
As I can see in your code, your testUser is defined in your first unit ( it("Add User"...){}). I advise you not to use the value from the first unit in the second unit.
You may rather use beforeEach and afterEach to have a clean setup for each unit and then create a new user in the second test unit. You should also clean up your db after each unit:
describe('Users', () => {
// use this for each test
let testUser;
beforeEach(() => {
// let's always create a new clean testUser object
testUser = {email: 'test#test.test', password: 'test'};
});
afterEach(() => {
// Remove the user to keep our db clean
if (testUser._id)
Meteor.users.remove(testUser._id);
});
it("Add User", (done) => {
testUser._id = Meteor.call('addUser', testUser);
const users = Meteor.users.find({_id: testUser._id}).fetch();
const user = users[0];
assert.isNotNull(user);
assert.isDefined(user);
assert.equal(user._id, testUser._id);
done();
});
it("Get user", (done) => {
// get a new id from our previously tested
// (and assume to be working) function
testUser._id = Meteor.call('addUser', testUser);
const user = Meteor.call('getUser', testUser._id);
assert.isNotNull(user);
assert.isDefined(user);
assert.equal(user._id, testUser._id);
done();
});
});
I also found, that your 'getUser' method returns an array, so I changed it to:
getUser: function (userID) {
check(userID, String);
const users = Meteor.users.find({_id: userID}).fetch();
return users[0];
}
All tested and running.