I'm learning AWS so go easy on me if this is an obvious issue. My goal is to use the Cognito Authorizer in API Gateway to authenticate my users. That part works fine. I can get tokens to the front end and back to the api. What I'm trying to do now is pull some user attributes from the user pool so I can put them in some denormalized fields in Dynamo. It seems the best way to do that is to pull the user attributes on post (if there's a better way please feel free to answer with that as well), which requires cognitoidentityserviceprovider.listUsers. The problem is the callback never executes. At first I was getting not authorized errors, so I updated the IAM policy, and that went away, now I get the logs outside the callback, but not inside it.
Snippet
let cognitoidentityserviceprovider = new AWS.CognitoIdentityServiceProvider({apiVersion: '2016-04-18'});
let sub = event.requestContext.authorizer.claims.sub;
let userpool = event.stageVariables.userpool;
const cogParams = {
UserPoolId: userpool,
Filter: "sub=\""+ sub + "\"",
Limit: 1
};
console.log('listing',JSON.stringify(cogParams));
cognitoidentityserviceprovider.listUsers(cogParams, function(err, data) {
console.info('entered callback');
if (err) {
console.log(err, err.stack); }
else {
console.log('data',data);
}
console.info('end callback');
});
console.log('listed');
Expected output
listing {...}
entered callback
data {...}
end callback
listed
Actual Output
listing {...}
listed
From the await usage, it seems you are using an async handler. Also, according to AWS documentation on using async/await, it's noted that functions that take a callback do not return a promise, you must chain them with the .promise() method:
Most functions that take a callback do not return a promise. Since you only use await functions that return a promise, to use the async/await pattern you need to chain the .promise() method to the end of your call, and remove the callback.
So, in order for the await to properly wait for the listUsers to execute instead of moving forward, you should update your code to:
Async handler version
const AWS = require('aws-sdk');
exports.handler = async (event) => { // <-- async handler
let cognitoidentityserviceprovider = new AWS.CognitoIdentityServiceProvider({ apiVersion: '2016-04-18' });
let sub = event.requestContext.authorizer.claims.sub;
let userpool = event.stageVariables.userpool;
const cogParams = {
UserPoolId: userpool,
Filter: "sub=\"" + sub + "\"",
Limit: 1
};
console.log('listing', JSON.stringify(cogParams));
// use `await` due to async handler
let userList = await cognitoidentityserviceprovider.listUsers(cogParams, function(err, data) {
console.info('entered callback');
if (err) {
console.log(err, err.stack);
}
else {
console.log('data', data);
}
console.info('end callback');
}).promise(); // <-- added the .promise() for the `await`
console.log('listed', userList); // <-- got the result also on the variable
};
Note that the callback was kept to show it executing but, as per AWS documentation suggestion, it can be removed.
If you prefer the callback programming model:
Non-async handler version
const AWS = require('aws-sdk');
exports.handler = (event) => { // <-- no `async` keyword
let cognitoidentityserviceprovider = new AWS.CognitoIdentityServiceProvider({ apiVersion: '2016-04-18' });
let sub = event.requestContext.authorizer.claims.sub;
let userpool = event.stageVariables.userpool;
const cogParams = {
UserPoolId: userpool,
Filter: "sub=\"" + sub + "\"",
Limit: 1
};
console.log('listing', JSON.stringify(cogParams));
cognitoidentityserviceprovider.listUsers(cogParams, function(err, data) {
console.info('entered callback');
if (err) {
console.log(err, err.stack);
}
else {
console.log('data', data);
}
console.info('end callback');
});
console.log('listed');
};
On using the cognitoidentityserviceprovider.listUsers method, it might be a bit more correct to use cognitoidentityserviceprovider.adminGetUser since you should also have access to the username and you're always retrieving a single user.
Related
I have a Node.js AWS Lambda function created via the serverless framework. I have multiple helper functions inside it. I am having an issue with one of them due to being async. The function runs and logs out all parts I put comments next to however it doesn't update callDuration. I think that the code is having an issue due to async where it finishes in the wrong order. My goal is to be able to return the callDuration to my main function for further processing. How can I get all code to process/run and be able to meet my goal and have the code run in the right order
Here is the function:
const callAggregate = async (billingData, billingDB) => {
const accountSid = process.env.TWILIO_ACCOUNT_SID
const authToken = process.env.TWILIO_AUTH_TOKEN
const client = require('twilio')(accountSid, authToken)
// Setup model
const Billing = billingDB.model('Billing')
await Billing.findOne({_id: billingData._id}).exec().then(bill => {
const callArray = bill.callSid
console.log(bill) // This logs out
let callDuration = 0
for (const call of callArray) {
console.log(call) // This logs out
client.calls(call)
.fetch()
.then(callDetails => {
console.log(callDetails) // This logs out
callDuration += callDetails.duration
})
}
console.log(`Billing for ${callDuration} minutes of voice calling for ${billingData._id}`) // This logs out
Billing.findOneAndUpdate(
{_id: billingData._id},
{ $inc: { call_duration: callDuration }, callSid: []},
(err, doc) => {
if(err) {
console.log(err)
}
}
)
return callDuration
})
}
This is a case of mixing and matching promises with plain callbacks and mixing await with .then(), both of which make proper flow-control and error handling management difficult.
Inside your function which is async and uses await in some places, you also have a promise you are not awaiting (which means it runs open loop and nothing waits for it) and you have a database function that is using a plain callback, not the promise interface so nothing waits for it either.
More specifically, nothing is waiting for this:
client.calls(call).fetch()
So, because of not waiting for the .fetch() to finish, you were attempting to use the variable callDuration before the code was done modifying that variable (giving you the wrong value for it).
Similarly, nothing is waiting for Billing.findOneAndUpdate(...) to complete either.
A clean solution is to switch everything over to promises and await. This involves, using only promises with your database (no plain callbacks) and converting the .then() handlers into await.
async function callAggregate(billingData, billingDB) {
const accountSid = process.env.TWILIO_ACCOUNT_SID
const authToken = process.env.TWILIO_AUTH_TOKEN
const client = require('twilio')(accountSid, authToken)
// Setup model
const Billing = billingDB.model('Billing')
let bill = await Billing.findOne({ _id: billingData._id }).exec();
const callArray = bill.callSid
console.log(bill) // This logs out
let callDuration = 0
for (const call of callArray) {
console.log(call) // This logs out
let callDetails = await client.calls(call).fetch();
console.log(callDetails) // This logs out
callDuration += callDetails.duration
}
console.log(`Billing for ${callDuration} minutes of voice calling for ${billingData._id}`) // This logs out
let doc = await Billing.findOneAndUpdate({ _id: billingData._id }, { $inc: { call_duration: callDuration }, callSid: [] }).exec();
return callDuration
}
I am using ldapjs to query users from an ldap server.
If I put all the code just in a single script without using functions, the query works and I get the results I need.
I am now trying to use expressjs to serve a rest endpoint to enable querying of the ldap server, so I moved the ldapjs client.search code into a async function with a promise surrounding the actual search code.
After the promise code, I have a line which exercises the promise using await and stores the results of the promise in a variable. I then return that variable to the calling function which will eventually send the results back as a json-formatted string to the requesting browser.
The problem I am seeing is that the console.log() of the returned results is undefined and appears before the console.log statements inside the promise code. So it looks like the async function is returning before the promise is fulfilled, but I don't see why because in all the examples of promises and async/await I have seen this scenario works correctly.
Below is a sample script without the expressjs part to just make sure everything works correctly.
// script constants:
const ldap = require('ldapjs');
const assert = require('assert');
const ldapServer = "ldap.example.com";
const adSuffix = "dc=example,dc=com"; // test.com
const client = getClient();
const fullName = "*doe*";
var opts = {
scope: "sub",
filter: `(cn=${fullName})`,
attributes: ["displayName", "mail", "title", "manager"]
};
console.log("performing the search");
let ldapUsers = doSearch(client, opts);
console.log("Final Results: " + ldapUsers);
function getClient() {
// Setup the connection to the ldap server
...
return client;
}
async function doSearch(client, searchOptions) {
console.log("Inside doSearch()");
let promise = new Promise((resolve, reject) => {
users = '{"users": [';
client.search(adSuffix, searchOptions, (err, res) => {
if (err) {
console.log(err);
reject(err)
}
res.on('searchEntry', function(entry) {
console.log("Entry: " + users.length);
if (users.length > 11) {
users = users + "," + JSON.stringify(entry.object);
} else {
users = users + JSON.stringify(entry.object);
}
});
res.on('error', function(err) {
console.error("Error: " + err.message);
reject(err)
});
res.on('end', function(result) {
console.log("end:");
client.unbind();
users = users + "]}";
resolve(users)
});
});
});
// resolve the promise:
let result = await promise;
console.log("After promise has resolved.");
console.log(result);
return result
}
The output from the console.log statements is as follows:
Setting up the ldap client.
ldap.createClient succeeded.
performing the search
Inside doSearch()
Final Results: [object Promise]
Entry: 11
end:
After promise has resolved.
{"users": [{"dn":"cn=john_doe"}]}
I did strip out the code which creates the ldapjs client and redacted the company name, but otherwise this is my code.
Any ideas on why the doSearch function is returning before the promise is fulfilled would be greatly appreciated.
As #danh mentioned in a comment, you're not awaiting the response from doSearch. Since doSearch is an async function it will always return a promise, and thus must be awaited.
As a quick and dirty way to do that you could wrap your call in an immediately invoked asynchronous function like so:
// ...
(async () => console.log(await doSearch(client, opts)))();
// ...
For more info you might check out the MDN docs on asynchronous functions
I think there are a few issues in the provided code snippet. As #danh pointed out you need to await the doSearch call. However you may have not done that already since you may not be using an environment with a top async. This likely means you'll want to wrap the call to doSearch in an async function and call that. Assuming you need to await for the search results.
// script constants:
const ldap = require('ldapjs');
const assert = require('assert');
const ldapServer = "ldap.example.com";
const adSuffix = "dc=example,dc=com"; // test.com
const client = getClient();
const fullName = "*doe*";
function getClient() {
// Setup the connection to the ldap server
...
return client;
}
async function doSearch(client, searchOptions) {
console.log("Inside doSearch()");
return new Promise((resolve, reject) => {
users = '{"users": [';
client.search(adSuffix, searchOptions, (err, res) => {
if (err) {
console.log(err);
reject(err)
}
res.on('searchEntry', function(entry) {
console.log("Entry: " + users.length);
if (users.length > 11) {
users = users + "," + JSON.stringify(entry.object);
} else {
users = users + JSON.stringify(entry.object);
}
});
res.on('error', function(err) {
console.error("Error: " + err.message);
reject(err)
});
res.on('end', function(result) {
console.log("end:");
client.unbind();
users = users + "]}";
console.log(result);
resolve(users)
});
});
});
}
const opts = {
scope: "sub",
filter: `(cn=${fullName})`,
attributes: ["displayName", "mail", "title", "manager"]
};
(async function runAsyncSearch () {
console.log("performing the search");
try {
const ldapUsers = await doSearch(client, opts); // Await the async results
console.log("After promise has resolved.");
console.log("Final Results: " + ldapUsers);
} catch (err) {
console.error(err.message);
}
})(); // Execute the function immediately after defining it.
I have a simple custom module in which all methods are supposed to return database records.
I am getting all records after running a query but when I try to assign those records to some variable then it says null. Not sure what is going on.
Here is my custom module code:
module.exports = {
mydata: {},
all: function (req, res, next) {
var posts = null;
con.query("SELECT post_id,title, body from `posts`", function (err, rows, fileds) {
if (err) {
console.log('Error' + err);
}
posts = rows[0]; // assigned object object to posts but not working
console.log('names' + posts);
});
console.log('my posts' + posts); // it says null/empty
return posts; // empty return
},
I am calling all methods like this in my route:
console.log("admin.js :" + postModel.all());
All are empty or null. Please guide or suggest. Am I missing anything?
Try using async & await plus it's best practice to wrap the code in try catch, any pending promise will be resolved at the .then method or any error will be caught at the .catch of the caller/final function:
/ * Handler (or) Service (or) Whatever */
const FetchDataFromDB = require('../');
/* caller Function */
let fetchDataFromDB = new FetchDataFromDB();
fetchDataFromDB.getDataFromDB()
.then((res) => console.log(res))
.catch((err) => console.log(err))
/* DB Layer */
class FetchDataFromDB {
/* Method to get data from DB */
async getDataFromDB() {
const getQuery = "SELECT post_id,title, body from `posts`";
try {
let dbResp = await con.query(getQuery);
if (dbResp.length) {
//Do some logic on dbResp
}
return dbResp;
} catch (error) {
throw (error);
}
}
}
module.exports = FetchDataFromDB;
Welcome my friend to the world of Asynchronous function.
In your code the
console.log('my posts' + posts); and return posts;
are executing before the callback assigns values to the posts variable.
Also restrict yourself from using var instead use let for declaring variable works better without error for scoped functions.
Here below:
The async keyword declares that the function is asynchronous.
The await keyword basically says that lets first get the result and then move on to next statement/line. All awaits should be done inside an async functions only.
module.exports = {
mydata: {},
all: async function (req, res, next) { //Let this be an asynchronous function
try{
let [rows, fields] = await con.query("SELECT post_id,title, body from `posts`");
//let us await for the query result so stay here until there is result
let posts = rows[0]; // assign first value of row to posts variable now
console.log('my posts' + posts);
return posts;
}catch(err){
return err;
}
},
}
Please take time reading up nature of Asynchronous, Non-blocking in JavaScript and how to handle them with Promises or Async/await (my personal choice).
I'm using node and express to create server for my app. This is how my code looks like:
async function _prepareDetails(activityId, dealId) {
var offerInfo;
var details = [];
client.connect(function(err) {
assert.equal(null, err);
console.log("Connected correctly to server");
const db = client.db(dbName);
const offers_collection = db.collection('collection_name');
await offers_collection.aggregate([
{ "$match": { 'id': Id} },
]).toArray(function(err, docs) {
assert.equal(err, null);
console.log("Found the following records");
details = docs;
});
})
return details;
}
app.post('/getDetails',(request,response)=>{
var Id = request.body.Id;
var activityId = request.body.activityId;
_prepareDetails(activityId,Id).then(xx => console.log(xx));
response.send('xx');
})
While calling getDetails API I am getting
await is only valid in async function error (At line await offers_collection.aggregate)
I am also getting red underline while declaring async function. Node version I'm using is 11.x. I'm also using firebase API. What am I doing wrong here?
You are missing the async declaration on one of your functions. Here is the working code:
async function _prepareDetails(activityId, dealId) {
var offerInfo;
var details = [];
client.connect(async function(err) {
assert.equal(null, err);
console.log("Connected correctly to server");
const db = client.db(dbName);
const offers_collection = db.collection('collection_name');
await offers_collection.aggregate([
{ "$match": { 'id': Id} },
]).toArray(function(err, docs) {
assert.equal(err, null);
console.log("Found the following records");
details = docs;
});
})
return details;
}
app.post('/getDetails', async (request,response)=>{
var Id = request.body.Id;
var activityId = request.body.activityId;
let xx = await _prepareDetails(activityId,Id);
response.send('xx');
})
Await can only be used in an async function, because await is by definition asynchronous, and therefore must either use the callback or promise paradigm. By declaring a function as async, you are telling JavaScript to wrap your response in a promise. Your issue was on the following line:
client.connect(function(err) {
This is where I added the async as mentioned before.
client.connect(async function(err) {
You will notice I made your route use async also, because you would've had an issue before. Notice the two lines in your original code:
_prepareDetails(activityId,Id).then(xx => console.log(xx));
response.send('xx');
Your response would send before you even made your database call because you are not wrapping the response.send within the .then. You could move the response.send into the .then, but if you are going to use async/await, I would use it all the way. So your new route would look like:
app.post('/getDetails', async (request,response)=>{
var Id = request.body.Id;
var activityId = request.body.activityId;
let xx = await _prepareDetails(activityId,Id);
response.send('xx');
})
I want to pass a list of files (obj) that I'm getting through a google drive API to EJS files.
i.e. I want to write
app.get('/',function(req,res){
res.render('index',obj);
}
The problem is that I'm getting the js object through a few call back functions.
This function is called
fs.readFile('client_secret.json',processClientSecrets );
which in turn calls,
function processClientSecrets(err,content) {
if (err) {
console.log('Error loading client secret file: ' + err);
return;
}else{
authorize(JSON.parse(content),findFiles);
}
}
which calls these two,
function authorise(credentials,callback) {
var clientSecret = credentials.installed.client_secret;
var clientId = credentials.installed.client_id;
var redirectUrl = credentials.installed.redirect_uris[0];
var auth = new googleAuth();
var oauth2Client = new auth.OAuth2(clientId, clientSecret, redirectUrl);
// Check if we have previously stored a token.
fs.readFile(TOKEN_PATH, function(err, token) {
if (err) {
getNewToken(oauth2Client, callback);
} else {
oauth2Client.credentials = JSON.parse(token);
callback(oauth2Client);
}
});
}
[EDIT]
function findFiles(auth){
var obj ={};
var key = 'files';
obj[key]=[];
var drive = google.drive('v3');
drive.files.list({
auth: auth,
folderId: '****************',
q: "mimeType contains 'application/pdf' and trashed = false"
},
function(err,response){
var f = response.files;
if (f.length == 0) {
console.log('No files found.');
}else {
var i;
for (i = 0; i < f.length; i++) {
var file = f[i];
//console.log('%s (%s)', file.name, file.id);
obj[key].push(file.name + ' ' + file.id);
}
console.log(obj);
return obj;
}
});
}
This looks like a very basic question, however im not able to solve it as node.js is asynchronous in nature and all my attempts to return obj have resulted in rendering of obj before retrieving it.
Welcome to callback hell. :-) The old "Node" way would be to do nested callbacks, which gets very ugly very quickly.
The modern approach is to use promises, which makes it easier to compose multiple async operations together. Make your own async functions return promises, and for Node API functions (or add-on libs that don't provide promises yet), use wrappers to make them promise-enabled (manually, or by using something like promisify).
With promise-based functions, for instance, your call would look like this:
app.get('/',function(req,res){
readFilePromise('client_secret.json')
.then(content => JSON.parse(content))
.then(authorise)
.then(findFiles)
.then(files => {
res.render('index', files);
})
.catch(err => {
// Render error here
});
});
or since neither JSON.parse nor findFiles is asynchronous:
app.get('/',function(req,res){
readFilePromise('client_secret.json')
.then(content => authorise(JSON.parse(content)))
.then(auth => {
res.render('index', findFiles(auth));
})
.catch(err => {
// Render error here
});
});
It's fine to use non-async functions with then provided the function expects one parameter and returns the processed result, so the first version was fine too, though there is a bit of overhead involved.
In both cases, readFilePromise is a promisified version of readFile, and authorize looks something like this:
function authorise(credentials) {
var clientSecret = credentials.installed.client_secret;
var clientId = credentials.installed.client_id;
var redirectUrl = credentials.installed.redirect_uris[0];
var auth = new googleAuth();
var oauth2Client = new auth.OAuth2(clientId, clientSecret, redirectUrl);
// Check if we have previously stored a token.
return readFilePromise(TOKEN_PATH)
.then(token => {
oauth2Client.credentials = JSON.parse(token);
return oauth2Client;
});
}
(Also note — subjectivity warning! — that because we don't end up with hellish deeply-nested callback structures, we can use a reasonable indentation width rather than the two spaces so many Node programmers felt the need to adopt.)
Moving forward further, if you're using Node V8.x+, you could use async/await syntax to consume those promises:
app.get('/', async function(req, res){
try {
const credentials = JSON.parse(await readFilePromise('client_secret.json'));
const auth = await authorize(credentials);
const files = findFiles(auth);
res.render('index', files);
} catch (e) {
// Render error here
}
});
Note the async before function and the await any time we're calling a function that returns a promise. An async function returns a promise under the covers, and await consumes promises under the covers. The code looks synchronous, but isn't. Every await is effectively a call to then registering a callback for when the promise completes. Similarly, the try/catch is effectively a call to the catch method on the promise chain.
We could condense that if we wanted:
app.get('/', async function(req, res){
try {
res.render('index', findFiles(await authorize(JSON.parse(await readFilePromise('client_secret.json'))));
} catch (e) {
// Render error here
}
});
...but readability/debuggability suffer. :-)
Important note: When passing an async function into something (like app.get) that isn't expecting the function to return a promise, you must wrap it in a try/catch as above and handle any error, because if the calling code isn't expecting a promise, it won't handle promise rejections, and you need to do that; unhandled rejections are a bad thing (and in future versions of Node will cause your process to terminate).
If what you're passing an async function into does expect a function returning a process, best to leave the try/catch` off and allow errors to propagate.
You asked for help with findFiles. I recommend learning promisify or something similar. The right way (to my mind) to solve this is to give yourself a promisified version of drive.files.list, since drive.files.list uses Node-style callbacks instead.
But without promisifying it, we can do this:
function findFiles(auth) {
var drive = google.drive('v3');
return new Promise(function(resolve, reject) {
drive.files.list({
auth: auth,
folderId: '****************',
q: "mimeType contains 'application/pdf' and trashed = false"
},
function(err, response) {
if (err) {
reject(err);
return;
}
var f = response.files;
if (f.length == 0) {
console.log('No files found.');
}
else {
var key = 'files'; // Why this indirection??
resolve({[key]: f.map(file => file.name + ' ' + file.id)});
// Without the indirection it would be:
// resolve({files: f.map(file => file.name + ' ' + file.id)});
}
});
});
}
If we had a promisified version, and we did away with the key indirection which seems unnecessary, it would be simpler:
function findFiles(auth) {
return drivePromisified.files.list({
auth: auth,
folderId: '****************',
q: "mimeType contains 'application/pdf' and trashed = false"
}).then(files => ({files: files.map(file => file.name + ' ' + file.id)}));
}
Or as an async function using await:
async function findFiles(auth) {
const files = await drivePromisified.files.list({
auth: auth,
folderId: '****************',
q: "mimeType contains 'application/pdf' and trashed = false"
});
return {files: files.map(file => file.name + ' ' + file.id)};
}