Develop Reference

How do I properly route data through my Node API?

I have the following files:
My routes - where the orders_count route lives:
routes/index.js
const express = require('express');
const router = express.Router();
const transactionsController = require('../controllers/transactionsController');
const ordersController = require('../controllers/ordersController');
const ordersCountController = require('../controllers/ordersCountController');
router.get('/transactions', transactionsController);
router.get('/orders', ordersController);
router.get('/orders_count', ordersCountController);
module.exports = router;
I then have my orders count controller living in the controllers directory:
controllers/ordersCountController.js
const ordersCountService = require('../services/ordersCountService');
const ordersCountController = (req, res) => {
ordersCountService((error, data) => {
if (error) {
return res.send({ error });
}
res.send({ data })
});
};
module.exports = ordersCountController;
My controller then calls my order count service which fetches data from another API.
services/ordersService.js
const fetch = require('node-fetch');
// connect to api and make initial call
const ordersCountService = (req, res) => {
const url = ...;
const settings = { method: 'Get'};
fetch(url, settings)
.then(res => {
if (res.ok) {
res.json().then((data) => {
return data;
});
} else {
throw 'Unable to retrieve data';
}
}).catch(error => {
console.log(error);
});
}
module.exports = ordersCountService;
I'm trying to return the JSON response. I initially had it setup with requests but looking at the NPM site, it appears that it's depreciated so have been digging through how to use node-fetch.
I have tried both 'return data' and res.send({data}), but neither are solving the problem.
I am still new to this so I am likely missing something very obvious, but how come I am not sending the JSON back through so that it displays at the /api/orders_count endpoint?
I keep thinking I messed something up in my controller but have been looking at it for so long and can't seem to figure it out.
Any help would be greatly appreciated and if there is anything I can add for clarity, please don't hesitate to ask.
Best.

please learn promises and await syntax. life will be easier.
never throw a string. always prefer a real error object, like that : throw new Error('xxx'); that way you will always get a stack. its way easier to debug.
avoid the callback hell : http://callbackhell.com/
you need to decide if you want to catch the error in the controller or in the service. no need to do in both.
in the controller you call the service that way :
ordersCountService((error, data) => {
but you declare it like that :
const ordersCountService = (req, res) => {
which is not compatible. it should look like this if you work with callback style :
const ordersCountService = (callback) => {
...
if (error) return callback(error)
...
callback(null, gooddata);
here is an example to flatten your ordersCountService function to await syntax, which allows the "return data" you were trying to do :
const fetch = require('node-fetch');
// connect to api and make initial call
const ordersCountService = async (req, res) => {
const url = ...;
const settings = { method: 'Get'};
try {
const res = await fetch(url, settings);
if (!res.ok) throw new Error('Unable to retrieve data');
return await res.json();
} catch(error) {
console.log(error);
}
}
module.exports = ordersCountService;
in fact i would prefer to error handle in the controller. then this woud be sufficient as a service
const fetch = require('node-fetch');
// connect to api and make initial call
const ordersCountService = async () => {
const url = ...;
const settings = { method: 'Get'};
const res = await fetch(url, settings);
if (!res.ok) throw new Error('Unable to retrieve data');
return await res.json();
}
module.exports = ordersCountService;
then you can call this funtion like this :
try {
const data = await ordersCountService(req, res);
} catch(err) {
console.log(err);
}
//or
ordersCountService(req, res).then((data) => console.log(data)).catch((err) => console.error(err));

Jest Mock changes to undefined from one line to another

I want to test my AWS Lambda functions written in typescript. The problem is that I'm mocking a service, which initially works fine and has all expected functions, but from one line to the other it turns to undefined, even though in the console.log right before it is defined.
My lambda function:
export const changePassword = LambdaUtils.wrapApiHandler(
async (event: LambdaUtils.APIGatewayProxyEvent, context: Context) => {
const accessToken = event?.body?.accessToken;
const previousPassword = event?.body?.previousPassword;
const proposedPassword = event?.body?.proposedPassword;
// let authenticationService = Injector.getByName("AuthenticationService"); // I tried mocking this object but given the dependency injection it is always undefined
//Request
const authenticationService = new AuthenticationService(); // Here it is perfectly mocked
console.log("-> authenticationService", authenticationService); // Here it is also defined with all functions
const response = await authenticationService.changePassword(
accessToken,
previousPassword,
proposedPassword
); //Here authenticationService is undefined
console.log("response in auth_handler", response);
if (response?.statusCode === 200) {
return new LambdaUtilities().createHttp200Response(response);
}
}
);
My test component:
jest.mock("../../../controllers/auth.service");
const MockedAuthService = AuthenticationService as jest.Mock<AuthenticationService>;
.....
test(`Test successful response of changePassword Repo function"`, async () => {
const accessToken = "accessToken";
const previousPassword = "123456";
const proposedPassword = "654321";
const mockAuthService = new MockedAuthService() as jest.Mocked<AuthenticationService>;
console.log("-> mockAuthService", mockAuthService);
const event = {
body: {
accessToken,
proposedPassword,
previousPassword
}
};
const response = await changePassword(event, {}, {});
console.log("response", response);
expect(response).not.toBeNull();
});
Here are screenshots from a debug with PyCharm:

Node.js: Awaiting a Require

I'm new the Node.js and I've been working with a sample project by a third party provider and I'm trying to use Azure Key Vault to store configuration values.
I'm having trouble getting a process to wait before executing the rest. I'll try to detail as much as I know.
The sample project has a file named agent.js which is the start page/file. On line 16 (agent_config = require('./config/config.js')[process.env.LP_ACCOUNT][process.env.LP_USER]) it calls a config file with values. I'm trying to set these value using Key Vault. I've tried many combinations of calling functions, and even implementing async / await but the value for agent_config always contains a [Promise] object and not the data returned by Key Vault.
If I'm right, this is because the Key Vault itself uses async / await too and the config file returns before the Key Vault values are returned.
How can Key Vault be added/implemented in a situation like this?
Here's what I've tried:
First updated agent.js to
let agent_config = {};
try {
agent_config = require('./config/config.js')['123']['accountName'];
} catch (ex) {
log.warn(`[agent.js] Error loading config: ${ex}`)
}
console.log(agent_config);
Test 1
./config/config.js
const KeyVault = require('azure-keyvault');
const msRestAzure = require('ms-rest-azure');
const KEY_VAULT_URI = 'https://' + '{my vault}' + '.vault.azure.net/' || process.env['KEY_VAULT_URI'];
function getValue(secretName, secretVersion) {
msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' }).then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
module.exports = {
'123': {
'accountName': {
accountId: getValue('mySecretName', '')
}
}
};
Results
{ accountsId: undefined }
Test 2
Made getValue an async function and wrapped it around another function (tried without the wrapping and didn't work either)
./config/config.js
const KeyVault = require('azure-keyvault');
const msRestAzure = require('ms-rest-azure');
const KEY_VAULT_URI = 'https://' + '{my vault}' + '.vault.azure.net/' || process.env['KEY_VAULT_URI'];
async function getValue(secretName, secretVersion) {
msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' }).then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
async function config() {
module.exports = {
'123': {
'accountName': {
accountId: await getValue('mySecretName', '')
}
}
};
}
config();
Results
{}
Test 3
Made getValue an async function and wrapped it around another function (tried without the wrapping and didn't work either)
./config/config.js
const KeyVault = require('azure-keyvault');
const msRestAzure = require('ms-rest-azure');
const KEY_VAULT_URI = 'https://' + '{my vault}' + '.vault.azure.net/' || process.env['KEY_VAULT_URI'];
async function getValue(secretName, secretVersion) {
return msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' })
.then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
return client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
module.exports = {
'123': {
'accountName': {
accountId: getValue('mySecretName', '')
}
}
};
config();
Results
{ accountId: { <pending> } }
Other
I've tried many others ways like module.exports = async (value) =< {...} (found through other questions/solutions without success.
I'm starting to think I need to do some "waiting" on agent.js but I haven't found good info on this.
Any help would be great!

One issue is that your getValue function is not returning anything as your returns need to be explicit.
(and without the promise being returned, there's nothing to await on)
async function getValue(secretName, secretVersion) {
return msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' })
.then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
return client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
You could also get away with less explicit returns using arrow functions..
const getValue = async (secretName, secretVersion) =>
msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' })
.then(credentials => {
const client = new KeyVault.KeyVaultClient(credentials);
return client.getSecret(KEY_VAULT_URI, secretName, secretVersion)
.then(response => response.Value);
});
Introducing the Azure Key Vault read, which is async, means your whole config read is async. There' nothing you can do to get around that. This will mean that the code that uses the config will need to handle it appropriately. You start by exporting an async function that will return the config..
async function getConfig() {
return {
'123': {
'accountName': {
accountId: await getValue('mySecretName', '')
}
}
};
}
module.exports = getConfig;
In your agent code you call that function. This will mean that your agent code will need to be wrapped in a function too, so maybe something like this..
const Bot = require('./bot/bot.js');
const getConfig = require('./config/config.js');
getConfig().then(agentConfig => {
const agent = new Bot(agentConfig);
agent.on(Bot.const.CONNECTED, data => {
log.info(`[agent.js] CONNECTED ${JSON.stringify(data)}`);
});
});

The package azure-keyvault has been deprecated in favor of the new packages to deal with Keyvault keys, secrets and certificates separately. For your scenario, you can use the new #azure/keyvault-secrets package to talk to Key Vault and the new #azure/identity package to create the credential.
const { SecretClient } = require("#azure/keyvault-secrets");
const { DefaultAzureCredential } = require("#azure/identity");
async function getValue(secretName, secretVersion) {
const credential = new DefaultAzureCredential();
const client = new SecretClient(KEY_VAULT_URI, credential);
const secret = await client.getSecret(secretName);
return secret.value;
}
The DefaultAzureCredential assumes that you have set the below env variables
AZURE_TENANT_ID: The tenant ID in Azure Active Directory
AZURE_CLIENT_ID: The application (client) ID registered in the AAD tenant
AZURE_CLIENT_SECRET: The client secret for the registered application
To try other credentials, see the readme for #azure/identity
If you are moving from the older azure-keyvault package, checkout the migration guide to understand the major changes

Cannot read property of undefined thrown from running NPM and PACT

I'm trying to follow the PACT workshop example with some alternate data.
This may be more of a Javascript/Node question but I'm a but stumped, as a novice.
Given a consumer.spec.js file of:
const chai = require('chai');
const nock = require('nock');
const chaiAsPromised = require('chai-as-promised');
const expect = chai.expect;
const API_PORT = process.env.API_PORT || 9123;
chai.use(chaiAsPromised);
const API_HOST = `http://localhost:${API_PORT}`;
describe('Consumer', () => {
describe('when a call to the Provider is made', () => {
const clothingStatus = 'hello';
const {emailClothingOfferStatus} = require('../client');
it('can process the HTML payload from the provider', () => {
nock(API_HOST)
.get('/provider')
.query({validPermStatus:'hello'})
.reply(200, {
test:'NO',
validPermStatus: clothingStatus,
count: 1000,
});
const response = emailClothingOfferStatus(clothingStatus);
return expect(response.body.clothingStatus).to.eventually.equal('hello')
})
})
});
and a client .js file of:
const request = require('superagent');
const API_HOST = process.env.API_HOST || 'http://localhost';
const API_PORT = process.env.API_PORT || 9123;
const API_ENDPOINT = `${API_HOST}:${API_PORT}`;
// Fetch provider data
const emailClothingOfferStatus = emailPermChoice => {
let withEmailClothing = {};
const emailClothingGrantedRegex = 'hello';
if(emailPermChoice){
console.log(emailPermChoice);
withEmailClothing = {validPermStatus: emailPermChoice}
}
return request
.get(`${API_ENDPOINT}/provider`)
.query(withEmailClothing)
.then(
res => {
if (res.body.validPermStatus.match(emailClothingGrantedRegex)) {
return {
clothingStatus: (res.body.validPermStatus),
}
} else {
throw new Error('Could not verify email clothing offer status')
}
},
err => {
throw new Error(`Error from response: ${err.body}`)
}
)
};
module.exports = {
emailClothingOfferStatus,
};
and I have the following in my package.json scripts:
"test:consumer": "./node_modules/.bin/mocha --timeout 150000 pact/consumer/test/consumer.spec.js",
When I run npm run test:consumer, I get:
1) Consumer
when a call to the Provider is made
can process the HTML payload from the provider:
TypeError: Cannot read property 'clothingStatus' of undefined
at Context.it (pact/consumer/test/consumer.spec.js:29:35)
I'm sure it's something obvious but can anyone help?

Two things stand out to me as a problem:
The test above is a normal unit test designed to show how unit tests won't catch contract issues, and leads you into why Pact is useful (In case this wasn't clear). In short, it's not a Pact test at all - I can tell because it's using Nock, meaning the expected requests will never reach Pact. I can also tell because the Pact package doesn't appear to be imported. You want to model from this file https://github.com/DiUS/pact-workshop-js/blob/master/consumer/test/consumerPact.spec.js
The response value is a Promise, which means you can't do return expect(response.body.clothingStatus).to.eventually.equal('hello') because response is a promise, so body will be undefined and clothingStatus is not a property of that. The chai eventually API is useful for this sort of test, but as I understand, it has to work directly with a Promise - you could do expect(response).to... and then chai can go to work.

Your function emailClothingOfferStatus returns response.then() which is a promise and not an actual response.
Therefore response.body is undefined.
You should be able to test the result like this:
const response = emailClothingOfferStatus(clothingStatus);
response.then((res) => {
expect(res.body.clothingStatus).to.eventually.equal('hello')
})

Parse Deep Security Logs - AWS Lambda 'splunk-logger' node.js

I am trying to modify a Node.js function called 'splunk-logger'. The problem is that when the SNS Message comes into the function, the events from the Anti-Virus (Trend Micro DeepSecurity) console are grouped together. I already contacted their support and they said this is just the way events are sent and they can't help.
Example: {Message {Event_1} {Event_2} {Event_3}}
Now the JavaScript function works great and the events are forwarded to Splunk. However, since they are grouped together BEFORE they even hit the Lambda function, Splunk sees them as 1 single event instead of 3.
My thought is to take the 'event' variable (since it contains the sns 'message') and parse through that to separate each event (probably using regex or something). Then, I can either create another function to send each event immediately or simply call the "logger.flushAsync" function to send them.
Link to splunk-dev explaining the funciton: http://dev.splunk.com/view/event-collector/SP-CAAAE6Y#create.
Here is the code from the index.js:
const loggerConfig = {
url: process.env.SPLUNK_HEC_URL,
token: process.env.SPLUNK_HEC_TOKEN,
};
const SplunkLogger = require('./lib/mysplunklogger');
const logger = new SplunkLogger(loggerConfig);
exports.handler = (event, context, callback) => {
console.log('Received event:', JSON.stringify(event, null, 2));
// Log JSON objects to Splunk
logger.log(event);
// Send all the events in a single batch to Splunk
logger.flushAsync((error, response) => {
if (error) {
callback(error);
} else {
console.log(`Response from Splunk:\n${response}`);
callback(null, event.key1); // Echo back the first key value
}
});
};
Here is the code from the mysplunklogger.js file.
'use strict';
const url = require('url');
const Logger = function Logger(config) {
this.url = config.url;
this.token = config.token;
this.addMetadata = true;
this.setSource = true;
this.parsedUrl = url.parse(this.url);
// eslint-disable-next-line import/no-dynamic-require
this.requester = require(this.parsedUrl.protocol.substring(0, this.parsedUrl.protocol.length - 1));
// Initialize request options which can be overridden & extended by consumer as needed
this.requestOptions = {
hostname: this.parsedUrl.hostname,
path: this.parsedUrl.path,
port: this.parsedUrl.port,
method: 'POST',
headers: {
Authorization: `Splunk ${this.token}`,
},
rejectUnauthorized: false,
};
this.payloads = [];
};
// Simple logging API for Lambda functions
Logger.prototype.log = function log(message, context) {
this.logWithTime(Date.now(), message, context);
};
Logger.prototype.logWithTime = function logWithTime(time, message, context) {
const payload = {};
if (Object.prototype.toString.call(message) === '[object Array]') {
throw new Error('message argument must be a string or a JSON object.');
}
payload.event = message;
// Add Lambda metadata
if (typeof context !== 'undefined') {
if (this.addMetadata) {
// Enrich event only if it is an object
if (message === Object(message)) {
payload.event = JSON.parse(JSON.stringify(message)); // deep copy
payload.event.awsRequestId = context.awsRequestId;
}
}
if (this.setSource) {
payload.source = `lambda:${context.functionName}`;
}
}
payload.time = new Date(time).getTime() / 1000;
this.logEvent(payload);
};
Logger.prototype.logEvent = function logEvent(payload) {
this.payloads.push(JSON.stringify(payload));
};
Logger.prototype.flushAsync = function flushAsync(callback) {
callback = callback || (() => {}); // eslint-disable-line no-param-reassign
console.log('Sending event(s)');
const req = this.requester.request(this.requestOptions, (res) => {
res.setEncoding('utf8');
console.log('Response received');
res.on('data', (data) => {
let error = null;
if (res.statusCode !== 200) {
error = new Error(`error: statusCode=${res.statusCode}\n\n${data}`);
console.error(error);
}
this.payloads.length = 0;
callback(error, data);
});
});
req.on('error', (error) => {
callback(error);
});
req.end(this.payloads.join(''), 'utf8');
};
module.exports = Logger;

import requests
import re
import json
import os
def lambda_handler(event, context):
data = json.dumps(event)
EventIds = re.findall(r'{\\\".+?\\\"}', data)
EventLength = len(EventIds)
headers = {'Authorization': 'Splunk ' + os.environ['SPLUNK_HEC_TOKEN']}
i = 0
while i < EventLength:
response = requests.post(os.environ['SPLUNK_HEC_URL'], headers=headers, json={"event":EventIds[i]}, verify=True)
i+=1

Arrays are the data type used when Deep Security 10.0 or newer sends events to Amazon SNS. But Splunk wants one event per message. So don't send the array directly.
Instead, use the Splunk logger or Lambda to iterate through the array, sending each item as an individual message. You can modify this sample Lambda script for Node.js:
https://github.com/deep-security/amazon-sns/blob/master/lambda-save-ds-event-to-s3.js
It sends events to S3 individually (which is what you need). Just change it to send to Splunk instead.
Disclosure: I work for Trend Micro.