Develop Reference

Why can I not return an array of objects in an async function?

In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.

So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.

Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);

DynamoDb put does not write/save to table in lambda test or cron but works in Serverless

I am scraping a bunch of API's and saving the data to a dynamodb table.
Everything works absolutely fine when running serverless invoke local -f runAggregator locally.
However, after I set up the cron, I noticed things were not being saved to the Dynamodb table.
Here is my function:
module.exports.runAggregator = async (event) => {
await runModules({ saveJobs: true });
return {
statusCode: 200,
body: JSON.stringify(
{
message: "Aggregate",
input: event,
},
null,
2
),
};
};
And the runModules function:
module.exports = async ({ saveJobs }) => {
if (saveJobs) {
const flushDb = await flushDynamoDbTable();
console.log("Flushing Database: Complete");
console.log(flushDb);
}
// pseudo code
const allJobs = myLongArrayOfJobsFromApis
const goodJobs = allJobs.filter((job) => {
if (job.category) {
if (!job.category.includes("Missing map for")) return job;
}
});
// This runs absolutely fine locally...
if (saveJobs) goodJobs.forEach(saveJob); // see below for function
const badJobs = allJobs.filter((job) => {
if (!job.category) return job; // no role found from API
if (job.category.includes("Missing map for")) return job;
});
console.log("Total Jobs", allJobs.length);
console.log("Good Jobs", goodJobs.length);
console.log("Malformed Jobs", badJobs.length);
return uniqBy(badJobs, "category");
};
saveJob function
// saveJob.js
module.exports = (job) => {
validateJob(job);
dynamoDb
.put({
TableName: tableName,
Item: job,
})
.promise();
};
I am baffled as to why this works fine locally not when I run a 'test' in the lambda console. I only found out due to the table being empty after the cron had ran.

saveJob performs an async operation (ddb.put().promise()) but you are neither awaiting its completion nor returning the promise.
As the forEach in the runModules function will also not await anything, the function completes before the call to dynamodb is even performed (because of how promises vs synchronous code work) and the process is killed after the lambda's execution.
Locally you are not running lambda but something that looks like it. There are subtle differences, and what happens after the function is done is one of those differences. So it may work locally, but it won't on an actual lambda.
What you need to do is to make sure you await your call to dynamodb. Something like:
// saveJob.js
module.exports = (job) => {
validateJob(job);
return dynamoDb
.put({
TableName: tableName,
Item: job,
})
.promise();
};
and in your main function:
...
if (saveJobs) await Promise.all(...goodJobs.map(job => saveJob(job)))
// or with a Promise lib such as bluebird:
if (saveJobs) await Promise.map(goodJobs, job => saveJob(job))
// (or Promise.each(...) if you need to make sure this happens in sequence and not in parallel)
Note: instead of calling many times dynamodb.put, you could/should call once (or at least fewer times) the batchWriteItem operation, which can write up to 25 items in one call, saving quite a bit of latency in the process.

In Meteor, is it possible to make an async function from a sync function (opposite of wrapAsync)?

I'm aware of how to use Meteor.wrapAsync(), to make a function that takes a callback or returns a promise usable as if it was synchronous.
Is it possible to do the opposite, and if so how? I have a server-side function that calls some Meteor stuff synchronously (inc collections and accounts). I'd like to be able to run it concurrently in a batch, using something like p-all or async.js, to process the items in an array and wait until finished.
The app is using Meteor 1.6.0.1.
Here's some code, which doesn't work, as "Meteor code must always run within a Fiber":
const actions = entries.map(entry =>
() => new Promise((resolve, reject) =>
Meteor.defer(() => {
try {
const result = createUserFromEntry(entry, schoolId, creatorId, recordTypeId, signupYmd);
resolve(result);
} catch (exc) {
reject(exc);
}
})
)
);
Meteor.wrapAsync(callback =>
pAll(actions, { concurrency: 8, stopOnError: false })
.then(res => callback(null, res))
.catch(err => callback(err, null))
)();
Constructive suggestions of a different/proper way to achieve the same aim within Meteor are also welcome.

Meteor.defer requires the passed function to run within a fiber. This can be fixed using Meteor.bindEnvironment:
Meteor.defer(Meteor.bindEnvironment(function () {
try {
const result = createUserFromEntry(entry, schoolId, creatorId, recordTypeId, signupYmd);
resolve(result);
} catch (exc) {
reject(exc);
}
}))
The p-all function does not need to be wrapped as it's already async:
Meteor.startup(async function () {
const done = await pAll(actions, { concurrency: 8, stopOnError: false })
})
However, there is a nice candy that allows you to run async code in sync functions ,as long as you are in a fiber, using Promise.await:
Meteor.startup(function () {
const done = Promise.await(pAll(actions, { concurrency: 8, stopOnError: false }))
})
Note, that Promise.await is Meteor specific and only available on the server.

AWS Lambda returning without waiting for promises

I have a function working perfectly well within a node environment. The function uses promises, with S3 calls and a then and catch to call the callback with relevant 200/500 statusCode and a message body in each of them.
Now I am deploying it as a lambda function with a wrapper around it looking like this:
module.exports.getAvailableDates = (event, context, callback) => {
const lambdaParams = retrieveParametersFromEvent(event);
console.log(`Got the criteria`);
module.exports.getFilteredDates(lambdaParams.startDate,
lambdaParams.endDate, callback);
console.log(`Returning without the internal function results`);
};
The internal function looks like this:
module.exports.function getFilteredDates(startDate, endDate) {
const dateSet = new Set();
return new Promise((resolve, reject) => {
const getAllDates = (isDone) => {
if (isDone) {
const Dates = Array.from(dateSet).join();
resolve(Dates);
return;
}
getTestedDates(startDate, endDate, region, func, memory,
lastDateKey, dateSet).then(getAllDates).catch((error) => {
reject(error);
});
};
lastDateKey = '';
getTestedDates(startDate, endDate, region, func, memory,
lastDateKey, dateSet).then(getAllDates).catch((error) => {
reject(error);
});
});
}
And the even more internal function looks similar, only it actually queries the S3 database and returns the list of keys from it that match the date criteria.
In the AWS CloudWatch logs I see the two prints and only after them the internal function output. My understanding is that the lambda function is not waiting for the internal function with the promises to actually do its work (including the internal waiting on the promises) and returns with a bad status to me. What can I do?

Your last console.log is executed before the callback is executed.
If you want to print the complete statement just before exiting the Lambda, you need to wrap the callback ad wait for the Promise to complete:
import getFilteredDates from '../path/to/file';
module.exports.getAvailableDates = (event, context, callback) => {
const lambdaParams = retrieveParametersFromEvent(event);
console.log(`Got the criteria`);
getFilteredDates(lambdaParams.startDate,lambdaParams.endDate)
.then( result => {
console.log(`Returning the internal function results`);
return callback();
})
.catch(callback);
};
I've update the code to work with Promises given the function below.
Your getFilteredDates needs to be reshaped a bit:
Either you to have a third argument to accept the callback inside and handle the Promise chain internally
Or you expose a promise and handle the callback externally in the main scope.
Let's refactor it to just return a Promise and handle the callback outside:
function getFilteredDates(startDate, endDate) {
const dateSet = new Set();
return new Promise((resolve, reject) => {
const getAllDates = (isDone) => {
if (isDone) {
const Dates = Array.from(dateSet).join();
resolve(Dates);
return;
}
getTestedDates(startDate, endDate, region, func, memory,
lastDateKey, dateSet).then(getAllDates).catch((error) => {
reject(error);
});
};
lastDateKey = '';
getTestedDates(startDate, endDate, region, func, memory,
lastDateKey, dateSet).then(getAllDates).catch((error) => {
reject(error);
});
});
}
module.exports = getFilteredDates;

OK, figured it out, it was my bad. The inner function that called the callback with the status code didn't have a null when returning 200 (success) and that failed lambda over and over again. Anyway, I rewrote my lambda to be:
module.exports.getAvailableDates = (event, context, callback) => {
const lambdaParams = retrieveParametersFromEvent(event);
getFilteredDates(lambdaParams.startDate, lambdaParams.endDate)
.then(Dates => callback(null, { statusCode: 200, body: Dates}))
.catch(error => callback({ statusCode: 500, body: error}));
};
And now it works fine. Thanks for anyone who tried helping!
Oren

How can I stub a Node.js MongoDB chained fn call to mock my end result?

I am trying to test (with Sinon.JS Stubs) the following call with the Node.js MongoDB driver...
collection.find({mood: 'happy'}).toArray((err, result) => {
// array result
cb(null, result);
});
What is hanging me up here is the .toArray() chained function. The above call returns result as the expected Array.
To demonstrate my struggle - and to contrast - I am able to stub the following call which is not chained as such...
collection.findOne({id: 'id'}, (err, result) => {
// single result
cb(null, result);
});
stubb'd =>
findOneStub.yields(null, {username: 'craig'});
Is there a straightforward way to stub my .find call, which returns a function with .toArray on it to finally mock my results?

What I usually do when I want to stub methods of complex nested object like with the Mongo driver case, is to mock an object that mimics the call chain like so:
Stubbing toArray() using callback
let mockDbCollection = {
find: function() {
return {
toArray: function(cb) {
const results = [{
username: 'craig'
}];
cb(null, results);
}
};
}
};
sinon.stub(db, 'collection')
.returns(mockDbCollection);
db.collection('users').find({
id: 'id'
}).toArray((err, docs) => {
console.log(docs);
done();
});
Stubbing toArray() using promise
let mockDbCollection = {
find: function() {
return {
toArray: function() {
return Promise.resolve([{
username: 'craig'
}]);
}
};
}
};
sinon.stub(db, 'collection')
.returns(mockDbCollection);
db.collection('messages').find({
id: 'id'
}).toArray().then((docs) => {
console.log(docs);
done();
});
This paradigm can be used in any case that you want to mock a sequence of calls on a complex object, focusing only on the response of the last call in the chain. You can go as deep as you want without any issues.
If you want something more advanced like setting behavior of the stub or counting calls, etc you can find out some other techniques in this article. The author showcases some examples using complex DOM objects.
Adapting the technique from the tutorial in our example, we could easily do the following:
// here we stub the db.collection().findOne() method
// and fabricate a response
let stubFindOne = this.sandbox.stub().resolves({
_id: 'id'
});
// now, we can set the stub
// function in our mock object
let mockDb = {
collection: () => {
return {
findOne: stubFindOne
};
}
};
This way, we can manipulate and inspect the stubbed method as we would normally do, e.g.
const args = stubFindOne.firstCall.args;
would return the arguments list of the first call, etc.