Develop Reference

Asynchronous code works independently, but not together. Not getting any exceptions, just timing out

I am writing a lambda function to add hosts to a SQS queue for a rolling restart. The code I have written works individually, but not together. Even when I hard code values in the constructor. This doesn't appear to be a memory/CPU. I tried running the function with 1GB of memory, even though it only uses about 80MB. The average execution time for the individual functions is about 0.5 seconds (shouldn't take more than about 1.5 seconds to execute in total). I did trying running this function with a 30 second timeout, but it still timed out.
I work behind a corporate proxy, and have to hand jam the code. I don't have an IDE or intellisense on my internet facing network. There may be typos here, but not in the actual code. I have omitted my module imports and variable declarations to save time. It isn't relevant to the issue at hand.
EDIT: I added the module imports and variable declarations to the first example to hopefully alleviate some confusion.
Here are just a few things I have tried. This does not work (timing out):
// Custom lambda layer
const { marklogic, aws } = require('nodejs-layer-lib');
const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;
const params = [
'format=json'
];
const options = {
port: PORT,
params: params,
httpOptions: {
headers: {
'Authorization': `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
},
method: 'GET'
}
};
const taskServers = (HOSTS.split(',') || []).map(host => {
const _host = host.split(':');
return {
id: _host[0],
name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
};
});
exports.handler = async () => {
let hosts, queueUrl, addToQueueResults;
try {
hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
} catch (e) { console.error('hosts', e); }
try {
queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
} catch (e) { console.error('queueUrl ', e); }
try {
addToQueueResults = await aws.sqs.addMessages(queueURL, hosts);
} catch (e) { console.error('addToQueueResults ', e); }
return {
status: 200,
body: addToQueueResults
};
}
This does not work (timing out):
// Modules imports and variable declarations here...
exports.handler = async () => {
const hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
const queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
const addToQueueResults = await aws.sqs.addMessages(queueURL, hosts);
return {
status: 200,
body: addToQueueResults
};
}
This does not work (timing out):
// Modules imports and variable declarations here...
exports.handler = async () => {
const hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
const queueUrl = await aws.sqs.getQueueUrlByName('my-queue-name');
const addToQueueResults = await aws.sqs.addMessages('http://queueurl.com', ['anything', 'in', 'here']); // Doesn't even need the queueUrl or hosts anymore
return {
status: 200,
body: addToQueueResults
};
}
This works. It will return the host objects I am expecting in the response:
// Modules imports and variable declarations here...
exports.handler = async () => {
const hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
return {
status: 200,
body: hosts
};
}
This works. It will get the queue url, then add messages to my SQS queue and return the SQS response:
// Modules imports and variable declarations here...
exports.handler = async () => {
const queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
const addToQueueResults = await aws.sqs.addMessages(queueUrl , ['anything', 'in', 'here']);
return {
status: 200,
body: addToQueueResults
};
}
I tried implementing the Async handler in AWS Lambda function handler in Node.js and reviewed many AWS Lambda execution troubleshooting documents. The marklogic management API runs on port 8002 by default and I think the aws-sdk module uses http/https (80/443), so I don't think the ports are getting tied up.
What am I missing here?
EDIT 2: This has something to do with how promises are handled with AWS Lambda. I cannot find much information about this. Even following the instructions in AWS Lambda function handler in Node.js for "Async Handlers" I cannot get this to work. It works perfectly fine locally with or without my custom lambda layer.
Node.js runtime: 12.x (I didn't mention this before)
This also doesn't work (timing out):
// Modules imports and variable declarations here...
exports.handler = async function (event) {
const promise = function () {
return new Promise(async function (resolve, reject) {
try {
const hosts = await marklogic.hosts.getHosts(taskServers, options) || [];
const queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
const addToQueueResults = await aws.sqs.addMessages(queueUrl, hosts);
resolve({
status: 200,
body: addToQueueResults
});
} catch (error) {
reject({
status: 500,
error: error
});
}
});
};
return promise(); // Throws error without constructor despite the AWS doc example
}
Unless someone AWS Lambda genius has ran into a similar issue before with Node.js, I am just going to convert it into 2 lambda functions and use Step Functions to process them.

There was a typo in queueUrl (I imagine not that, but worth a try!)
Please run:
// Custom lambda layer
const { marklogic, aws } = require('nodejs-layer-lib');
const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;
const params = [
'format=json'
];
const options = {
port: PORT,
params,
httpOptions: {
headers: {
Authorization: `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
},
method: 'GET'
}
};
const taskServers = (HOSTS.split(',') || []).map(host => {
const _host = host.split(':');
return {
id: _host[0],
name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
};
});
exports.handler = async () => {
let hosts, queueUrl, addToQueueResults;
try {
hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
} catch (e) { console.error('hosts', e); }
try {
queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
} catch (e) { console.error('queueUrl ', e); }
try {
addToQueueResults = await aws.sqs.addMessages(queueUrl, hosts);
} catch (e) { console.error('addToQueueResults ', e); }
return {
status: 200,
body: JSON.stringify(addToQueueResults)
};
};
// keeping the same format.. ^^
If no luck - what is on my mind, is as aws-sdk is included out of the box in lambda.. it's not customary to require it extraneously via a layer and although it may not look to be imported at top level by marklogic, it may be bundled deep within marklogic, then when you import AWS and change config (in the layer) it overwrites it
Let's find out..:
Step 1:
So, this you say should work.. if we ignore the AWS import, and just import marklogic?
// Custom lambda layer
// const { marklogic, aws } = require('nodejs-layer-lib'); // ignoring AWS for now
const { marklogic } = require('nodejs-layer-lib');
const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;
const params = [
'format=json'
];
const options = {
port: PORT,
params,
httpOptions: {
headers: {
Authorization: `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
},
method: 'GET'
}
};
const taskServers = (HOSTS.split(',') || []).map(host => {
const _host = host.split(':');
return {
id: _host[0],
name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
};
});
exports.handler = async () => {
// let hosts, queueUrl, addToQueueResults;
let hosts;
try {
hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
console.log('hosts => ', hosts);
// queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
// addToQueueResults = await aws.sqs.addMessages(queueUrl, hosts);
return {
status: 200,
body: JSON.stringify(hosts)
};
} catch (error) {
console.log('error => ', error);
throw error;
}
};
Ok, so if that works..:
Step 2 (Please set the region for SQS and also hard code in the queueUrl):
// Custom lambda layer
// const { marklogic, aws } = require('nodejs-layer-lib');
const { marklogic } = require('nodejs-layer-lib');
const AWS = require('aws-sdk');
AWS.config.update({ region: 'eu-west-1' }); // Please set region accordingly
const sqs = new AWS.SQS({ apiVersion: '2012-11-05' });
const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;
const params = [
'format=json'
];
const options = {
port: PORT,
params,
httpOptions: {
headers: {
Authorization: `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
},
method: 'GET'
}
};
const taskServers = (HOSTS.split(',') || []).map(host => {
const _host = host.split(':');
return {
id: _host[0],
name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
};
});
exports.handler = async () => {
let hosts, addToQueueResults;
try {
hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
console.log('hosts => ', hosts);
const queueUrl = 'Please hard code the queueUrl for now';
const sqsParams = {
MessageBody: hosts,
QueueUrl: queueUrl
};
addToQueueResults = await sqs.sendMessage(sqsParams).promise();
console.log('addToQueueResults => ', addToQueueResults);
return {
status: 200,
body: JSON.stringify(addToQueueResults)
};
} catch (error) {
console.log('error => ', error);
throw error;
}
};
IF.. that doesn't work.. then Step 3.. move the require of marklogic to below the require of AWS and setting the region in this last example.. (so any deeply nested marklogic AWS logic we're unaware of now overwrites your AWS require..) re-run it.. fingers crossed :-)

Mocking Twilio Client from twilio-node package

I currently use The Twilio Node Helper Library to do various API calls whether it may be to create assistants/services, list them, remove them and various other things when it comes to uploading tasks, samples, fields to create a chatbot on Twilio Autopilot.
An example of one some of these functions include:
async function createAssistant(name, client){
var assistantUid = ""
await client.autopilot.assistants
.create({
friendlyName: name,
uniqueName: name
})
.then(assistant => assistantUid = assistant.sid);
return assistantUid
}
async function getAccount(client){
var valid = true
try {
await client.api.accounts.list()
} catch (e) {
valid = false
}
return valid
}
async function connectToTwilio(twilioAccountSid, twilioAuthToken) {
try{
var client = twilio(twilioAccountSid, twilioAuthToken);
} catch (e){
throw new TwilioRequestError(e)
}
var valid = await getAccount(client)
if(valid && client._httpClient.lastResponse.statusCode === 200){
return client
} else{
throw new Error("Invalid Twilio Credentials")
}
}
where client is the client object returned from require("twilio")(twilioAccountSid, twilioAuthToken).
I was wondering what would the best way of mocking this API to allow me to emulate creating assistants, returning their uniqueNames etc..
I was wondering that I may just define some class like
class TwilioTestClient{
constructor(sid, token){
this.sid = sid
this.token = token
this.assistants = TwilioAssistant()
this.services = TwilioServices()
}
}
Where TwilioAssitant and TwilioServices will be additional classes.
Any help would be greatly appreciated!

I struggled with mocking Twilio for a long time. In fact I previously architected my application such that I could mock a wrapper around the Twilio Node Helper just to avoid mocking the actual library. But recent changes to the architecture meant that was no longer an option. This morning I finally got a mock of the Twilio Node Helper Library working. I'm not familiar with the portions of the Twilio library you are using, but I'm hopeful the example here will help you.
We have a function to check if a phone number is mobile, call it isMobile.js.
const Twilio = require("twilio");
const isMobile = async (num) => {
const TwilioClient = new Twilio(process.env.TWILIO_SID, process.env.TWILIO_AUTH_TOKEN);
try {
const twilioResponse = await TwilioClient.lookups.v1
.phoneNumbers(num)
.fetch({ type: "carrier", mobile_country_code: "carrier" });
const { carrier: { type } = {} } = twilioResponse;
return type === "mobile";
} catch (e) {
return false;
}
};
module.exports = isMobile;
Then build a mock for Twilio in __mocks__/twilio.js
const mockLookupClient = {
v1: { phoneNumbers: () => ({ fetch: jest.fn(() => {}) }) }
};
module.exports = class Twilio {
constructor(sid, token) {
this.lookups = mockLookupClient;
}
};
In the test file isMobile.test.js
jest.mock("twilio");
const Twilio = require("twilio");
const isMobile = require("./isMobile");
const mockFetch = jest.fn();
const mockPhoneNumbers = jest.fn(() => ({
fetch: mockFetch
}));
describe("isMobile", () => {
const TwilioClient = new Twilio(process.env.TWILIO_SID, process.env.TWILIO_AUTH_TOKEN);
const lookupClient = TwilioClient.lookups.v1;
lookupClient.phoneNumbers = mockPhoneNumbers;
beforeEach(() => {
mockFetch.mockReset();
});
test("is a function", () => {
expect(typeof isMobile).toBe("function");
});
test("returns true for valid mobile number", async () => {
const validMobile = "+14037007492";
mockFetch.mockReturnValueOnce({
carrier: { type: "mobile", mobile_country_code: 302 }, // eslint-disable-line camelcase
phoneNumber: validMobile
});
expect(await isMobile(validMobile)).toBe(true);
});
test("returns false for non-mobile number", async () => {
const invalidMobile = "+14035470770";
mockFetch.mockReturnValueOnce({
carrier: { type: "not-mobile", mobile_country_code: null }, // eslint-disable-line camelcase
phoneNumber: invalidMobile
});
expect(await isMobile(invalidMobile)).toBe(false);
});
});

Jest - stub function within function

I'm writing unit-tests, where I need to set a mock response for a function within a function.
This is the function I want to mock:
cassandraDriver.js
module.exports = ({
cassandra_user,
cassandra_password,
cassandra_address
}) => {
if (!cassandra_address.length) throw Error('Cassandra address is not valid')
return new Promise((resolve, reject) => {
try {
const client = new driver.Client({
contactPoints: cassandra_address.split(','),
authProvider: authProvider(cassandra_user, cassandra_password),
queryconfig: {
consistency: driver.types.consistencies.quorum
}
})
return resolve(client)
} catch (e) {
reject(e)
}
})
}
This is the file that uses it:
const {
cassandraDriver
} = require('./lib')
module.exports = async ({
username = 'cassandra', //default values
password = 'cassandra', //default values
address,
keyspace,
replication_factor = 1,
migration_script_path,
logger = require('bunyan').createLogger({name: 'BuildCassandra'})
} = {}) => {
try {
const client = await cassandraDriver(username, password, address)
}).catch(err => {
throw Error(err)
})
} catch (e) {
logger.error(e)
throw e
}
}
How can I mock the call to 'cassandraDriver' in unit-tests? I tried using rewire, but the method is not exposed as it normally would be.
Thanks in advance.

let's modify your function so that it can accept a mock driver instead of cassandraDriver
const {
cassandraDriver
} = require('./lib')
module.exports = async ({
username = 'cassandra',
password = 'cassandra',
address,
keyspace,
replication_factor = 1,
migration_script_path,
logger = require('bunyan').createLogger({
name: 'BuildCassandra'
}),
driver = cassandraDriver
} = {}) => {
try {
const client = await driver(
username,
password,
address
})
} catch (e) {
logger.error(e)
throw e
}
}
(i also removed a superfluous .catch block)
next, you should create a "cassandra-driver-mock.js" which emulates the behaviour of the cassandra driver for your unit tests
the unit tests, of course, would pass the mock instead of the real driver as an option parameter

You can stub the module which exports cassandraDriver in your test file:
import cassandraDriver from "<path-to-cassandraDriver.js>";
jest.mock("<path-to-cassandraDriver.js>", () => jest.mock());
cassandraDriver.mockImplementation(() => {
// Stub implementation and return value
});
See Manual Mocks for more information.

How to check test case response using jasemine?

I am writing test case for API where I have passed params and main file method is calling http request to get the data, So core() is basically calling the API and getting the response. This is just backend code that we have to test using jasemine.
First how to see response in test case if that's matched with success that is defined in test.
What is a correct way to write test in below scenario.
balance.spec.ts
import { GetAccountBalance } from './GetAccountBalance.node'
import { Promise } from 'es6-promise'
import { HttpRequest } from '../../../core/http/HttpRequest.node'
import * as sinon from 'sinon'
import { getValidationError } from '../../../common/ValidationErrorFactory'
// for easy mocking and cleanup
const sandbox = sinon.createSandbox()
afterAll(function afterTests () {
sandbox.restore()
})
describe('getAccountBalance', function () {
// the module under test
const module = new GetAccountBalance()
const testURL = 'https://essdd.max.com:4535'
const urlPath = '/webServices/instrument/accountBalance'
const fullURL = testURL + urlPath
const options = { isJSON: true }
let result
const stubbedEC = sandbox.spy(getValidationError)
const stubbedHttp = sandbox.createStubInstance(HttpRequest)
const success = {
header: {
serviceName: 'accountBalance',
statusCode: '0000',
statusDesc: 'SUCCESS',
},
response: {
balanceAccount: '6346.44',
},
}
const params = { Id: '21544', appName: 'instrucn', channelName: 'Web' }
describe('core() is called with correct params', function () {
beforeEach(function () {
result = module.core(params, stubbedHttp)
})
it('it should return the response from server', function () {
// Invoke the unit being tested as necessary
result.then((data) => {
expect(data).toBe(success)
})
})
})
})
getaccountBalnce.ts
public core(args: IAccountBalanceParam, httpRequest: HttpRequestBase): Promise<any> {
console.log('enter core');
const DBPLurl: string = this.constructDBPLurl(args);
const DBPLrequest: IAccountBalanceDBPLParam = this.constructDBPLrequest(args);
return Promise.resolve(httpRequest.makeRequest(DBPLurl, DBPLrequest, {isJSON: true}));
}

How can I mock a fake database for when unit testing against Knex?

I've been using Knex successfully to connect to a backend database. But I want to be able to unit test my code. Is there a way to mock the database connection?
I've tried using proxyquire but I can't seem to get it to work.
The problem seems to be with the way Knex is initialized.
var knex = require('knex')({
client: 'mysql',
connection: {}
});
I setup knex to be mocked in my unit test.
myService = proxyquire('../app/myService', {
'knex': knexProxy
});
My service includes knex.
var knex = require('knex').knex,
When my service runs a query, it fails.
var sql = knex("table_name");
sql.insert(rowToInsert, "auto_increment_id");
sql.then(function (insertId) {
resolve();
}, function (err) {
reject(err);
});
For some reason I just can't seem to capture the request before it attempts the connection.
I've also, tried to create a custom Knex Client, but that hasn't worked yet either.

Using jest:
Create the file /__mocks__/knex.js in your app root:
module.exports = () => ({
select: jest.fn().mockReturnThis(),
from: jest.fn().mockReturnThis(),
where: jest.fn().mockReturnThis(),
first: jest.fn().mockReturnThis(),
then: jest.fn(function (done) {
done(null)
})
})
Pass the desired return value to done

I have been using in-memory Sqlite3 databases for automated testing with great success. It's not true unit testing however it does run much faster than MySQL or PostgreSQL. I have posted more details about this solution on a different question.

I used jest to mock knex but I had to define an object that contains the method that I used.
not the most elegant solution but is working
let knexMock = () => {
const fn = () => {
return {
returning: function() {
return {
insert: jest.fn().mockImplementation(() => [123123])
}
},
insert: jest.fn()
}
}
fn.raw = jest.fn()
return fn
}
knex.mockImplementation(knexMock)

I'm using jest and you can do something like this:
jest.mock('knex', () => {
const fn = () => {
return {
select: jest.fn().mockReturnThis(),
from: jest.fn().mockReturnThis(),
where: jest.fn().mockReturnThis(),
first: jest.fn().mockReturnThis(),
insert: jest.fn().mockReturnThis(),
raw: jest.fn().mockReturnThis(),
then: jest.fn(function (done) {
done(null)
})
}
}
return fn
})

I've written this tiny lib called knex-mock-client which does exactly this, it allows you to setup your db "connection" with a mockClient which will track your calls & help you with responses.
For example:
// my-cool-controller.ts
import { db } from '../common/db-setup';
export async function addUser(user: User): Promise<{ id }> {
const [insertId] = await db.insert(user).into('users');
return { id: insertId };
}
// my-cool-controller.spec.ts
import { expect } from '#jest/globals';
import knex, { Knex } from 'knex';
import { getTracker, MockClient } from 'knex-mock-client';
import faker from 'faker';
jest.mock('../common/db-setup', () => {
return knex({ client: MockClient });
});
describe('my-cool-controller tests', () => {
let tracker: Tracker;
beforeAll(() => {
tracker = getTracker();
});
afterEach(() => {
tracker.reset();
});
it('should add new user', async () => {
const insertId = faker.datatype.number();
tracker.on.insert('users').response([insertId]);
const newUser = { name: 'foo bar', email: 'test#test.com' };
const data = await addUser(newUser);
expect(data.id).toEqual(insertId);
const insertHistory = tracker.history.insert;
expect(insertHistory).toHaveLength(1);
expect(insertHistory[0].method).toEqual('insert');
expect(insertHistory[0].bindings).toEqual([newUser.name, newUser.email]);
});
});

This works for me, hope it helps someone:
//db.ts
import knex from 'knex';
const db = knex({
client: 'pg',
connection: {},
pool: { min: 0, max: 1 }
});
export default db('someTableName');
//myFunction.ts
//somewhere inside a function
const data = await db
// 👇 Knex query builders are mutable so when re-using them .clone() is necessary.
.clone()
.where({ pk: 'someId', sk: 'someId2' })
.select('data')
.orderBy('inserted_at', 'desc')
.first()
//myFunction.test.ts
describe("myFunction", () => {
beforeEach(() => {
jest.spyOn(db, "clone").mockImplementation(() => db);
jest.spyOn(db, "select");
jest.spyOn(db, "where");
jest.spyOn(db, "orderBy");
});
afterEach(() => {
jest.clearAllMocks();
});
it("should work as expected", async () => {
jest.spyOn(db, "first").mockResolvedValueOnce("desiredReturnValue");
await myFunction();
expect(db.where).toHaveBeenCalledWith({
pk: "someId",
sk: "someId2",
});
expect(db.select).toHaveBeenCalledWith("data");
expect(db.orderBy).toHaveBeenCalledWith("inserted_at", "desc");
expect(db.first).toHaveBeenCalledTimes(1);
});
});