I'm using a Lambda function which gets me the user email from a user id within a dynamoDB table. I use the dynamoDB scan command to scan over all items within the dynamoDB table. I use the new v3 AWS JS SDK.
Question: Why does ExpressionAttributeNames not work properly in my case?
This works:
const params = {
FilterExpression: "user_info.user_id = :userid",
ExpressionAttributeValues: {
":userid": { S: user_id }
},
ProjectionExpression: "user_email",
TableName: aws_table,
}
But this does NOT work, why?
const params = {
FilterExpression: "#xyz = :userid",
ExpressionAttributeNames: {
"#xyz": "user_info.user_id" // <- filter does not work like this (returns 0 findings)
},
ExpressionAttributeValues: {
":userid": { S: user_id }
},
ProjectionExpression: "user_email",
TableName: aws_table,
};
My Lambda scan operation code itself looks like:
const { DynamoDBClient } = require("#aws-sdk/client-dynamodb");
const { ScanCommand } = require("#aws-sdk/client-dynamodb");
const ddbClient = new DynamoDBClient({ region: aws_region });
...
const run = async () => {
try {
const data = await ddbClient.send(new ScanCommand(params));
data.Items.forEach(function (element, index, array) {
console.log(element);
});
return data;
} catch (err) {
console.log("Error", err);
}
}
await run();
npm i #aws-sdk/client-dynamodb
Documentation:
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/interfaces/scancommandinput.html#expressionattributenames
This is the expected behaviour:
DynamoDB interprets a dot in an expression attribute name as a character within an attribute's name
// use multiple expression attribute names if you need a dynamic path:
FilterExpression: "#user_info.#user_id = :userid"
Related
I am trying to perform unit tests with Jest for the new version of Strapi, v4 which was just released a couple of weeks ago. In accordance with their documentation, the old guide for unit testing no longer runs as expected. I have, however, modified the code to work to a certain extent. Currently I have the following:
./test/helpers/strapi.js:
const Strapi = require("#strapi/strapi");
let instance;
async function setupStrapi() {
if (!instance) {
/** the following code in copied from `./node_modules/strapi/lib/Strapi.js` */
await Strapi().load();
instance = strapi; // strapi is global now
await instance.server
.use(instance.server.router.routes()) // populate KOA routes
.use(instance.server.router.allowedMethods()); // populate KOA methods
await instance.server.mount();
}
return instance;
}
module.exports = {
setupStrapi
};
./tests/app.test.js:
const fs = require("fs");
const { setupStrapi } = require("./helpers/strapi");
beforeAll(async () => {
await setupStrapi();
});
afterAll(async () => {
const dbSettings = strapi.config.get("database.connection.connection");
//close server to release the db-file
await strapi.server.destroy();
//DATABASE_FILENAME=.tmp/test.db
//delete test database after all tests
if (dbSettings && dbSettings.filename) {
const tmpDbFile = `${dbSettings.filename}`;
if (fs.existsSync(tmpDbFile)) {
fs.unlinkSync(tmpDbFile);
}
}
});
it("should return hello world", async () => {
await request(strapi.server.httpServer).get("/api/hello").expect(200); // Expect response http code 200
});
./config/env/test/database.js
const path = require("path");
module.exports = ({ env }) => ({
connection: {
client: "sqlite",
connection: {
filename: path.join(
__dirname,
"../../../",
env("DATABASE_FILENAME", ".tmp/test.db")
),
},
useNullAsDefault: true,
},
});
The route /api/hello is a custom API endpoint. This works perfectly when running strapi develop, and all permissions are set correctly.
The tests run, but every endpoint that is not / or /admin returns 403 Forbidden, meaning there is a problem with the permissions. It would seem that the database file .tmp/data.db (used in development) is not replicated correctly in .tmp/test.db. In other words, this is close to working, but the permissions for API endpoints are not set correctly.
I have been searching through StackOverflow and the Stapi Forums over the past few days but to no avail. I would greatly appreciate some pointers as to how to fix this :)
It seems you need to grant the right privileges to your routes on your test DB.
For that you can create a function, lets call it grantPriviledge, and call it in your test in the function beforeAll.
// Here I want to grant the route update in my organization collection
beforeAll(async () => {
await grantPrivilege(1, 'permissions.application.controllers.organization.update');
});
And here is the function grantPriviledge:
// roleID is 1 for authenticated and 2 for public
const grantPrivilege = async (roleID = 1, value, enabled = true, policy = '') => {
const updateObj = value
.split('.')
.reduceRight((obj, next) => ({ [next]: obj }), { enabled, policy });
const roleName = roleID === 1 ? 'Authenticated' : 'Public';
const roleIdInDB = await strapi
.query('role', 'users-permissions')
.findOne({ name: roleName });
return strapi.plugins['users-permissions'].services.userspermissions.updateRole(
roleIdInDB,
updateObj,
);
};
Let me know if that helps
So in order for that to work in v4, this is how I did.
This was based in this and this but Stf's posting in this saying "inject them in your database during the bootstrap phase like it is made in the templates" was what really set me in the right track.
So if you look here you will see this function:
async function setPublicPermissions(newPermissions) {
// Find the ID of the public role
const publicRole = await strapi
.query("plugin::users-permissions.role")
.findOne({
where: {
type: "public",
},
});
// Create the new permissions and link them to the public role
const allPermissionsToCreate = [];
Object.keys(newPermissions).map((controller) => {
const actions = newPermissions[controller];
const permissionsToCreate = actions.map((action) => {
return strapi.query("plugin::users-permissions.permission").create({
data: {
action: `api::${controller}.${controller}.${action}`,
role: publicRole.id,
},
});
});
allPermissionsToCreate.push(...permissionsToCreate);
});
await Promise.all(allPermissionsToCreate);
}
Later on the code, this function is called like this:
await setPublicPermissions({
article: ["find", "findOne"],
category: ["find", "findOne"],
author: ["find", "findOne"],
global: ["find", "findOne"],
about: ["find", "findOne"],
});
So in my case I modified this function a bit to accept between authenticated (1) and public (2) roles inspired by Sidney C answer above.
This is how I did it:
const grantPrivilege = async (roleID = 1, newPermissions) => {
const roleName = roleID === 1 ? "authenticated" : "public";
// Find the ID of the public role
const roleEntry = await strapi
.query("plugin::users-permissions.role")
.findOne({
where: {
type: roleName,
},
});
// Create the new permissions and link them to the public role
const allPermissionsToCreate = [];
Object.keys(newPermissions).map((controller) => {
const actions = newPermissions[controller];
const permissionsToCreate = actions.map((action) => {
return strapi.query("plugin::users-permissions.permission").create({
data: {
action: `api::${controller}.${controller}.${action}`,
role: roleEntry.id,
},
});
});
allPermissionsToCreate.push(...permissionsToCreate);
});
await Promise.all(allPermissionsToCreate);
};
And then in my beforeAll block I call it like this:
await grantPrivilege(1, {
"my-custom-collection": ["create", "update"],
category: ["find", "findOne"],
author: ["find", "findOne"],
});
I am writing a lambda function that fetches data from DynamoDB and stores it in an array. Now I want to create a CSV file from this array and return it. (preferably directly from the lambda function, rather than uploading it to s3 and then sharing the link). Any idea, how to do this?
My code until now -
import AWS from "aws-sdk";
import createError from "http-errors";
import commonMiddleware from "../lib/commonMiddleware";
const dynamodb = new AWS.DynamoDB.DocumentClient();
async function getFile(event, context) {
const { id: someId } = event.pathParameters;
let data;
const params = {
TableName: process.env.TABLE_NAME,
IndexName: "GSIsomeId",
KeyConditionExpression: "someId = :someId",
ExpressionAttributeValues: {
":someId": someId,
},
};
try {
const result = await dynamodb.query(params).promise();
data = result.Items;
} catch (error) {
console.error(error);
throw new createError.InternalServerError(error);
}
// data is array of objects which I can change to 2d array using Object.values()
// I want to create and return a CSV from this array
return {
statusCode: 200,
body: JSON.stringify(data),
};
}
export const handler = commonMiddleware(getFile);
Once you generated the csv using any of the approaches mentioned here Convert JSON array to CSV in Node
I guess you can try sending back the byte array of the file.
I'm attempting to make an economy discord bot using node.js and I'm trying to move the commands into modules so that I can have a generic/dynamic command handler. How do I reference the currency collection and the models that I created within the main file within the command modules?
index.js file:
const currency = new Discord.Collection();
//defining methods for the currency collection
Reflect.defineProperty(currency, 'add', {
/* eslint-disable-next-line func-name-matching */
value: async function add(id, amount) {
const user = currency.get(id);
if (user) {
user.balance += Number(amount);
return user.save();
}
const newUser = await Users.create({ user_id: id, balance: amount });
currency.set(id, newUser);
return newUser;
},
});
Reflect.defineProperty(currency, 'getBalance', {
/* eslint-disable-next-line func-name-matching */
value: function getBalance(id) {
const user = currency.get(id);
return user ? user.balance : 0;
},
});
(In a subfolder) balance.js:
module.exports = {
name: 'balance',
description: 'get balance',
execute(message, args) {
const target = message.mentions.users.first() || message.author;
return message.channel.send(`${target.tag} has ${currency.getBalance(target.id)}🍉`);
},
};
Which currently throws an error on currency, since it's not defined. However, I don't know how to reference the currency collection I made in index.js, which also has methods created for it.
Thank you in advance.
To do this you could attach the Collection to your Client. Something like
Client.currency = new Discord.Collection()
Every time you then reference the collection, instead of doing currency you would then run Client.currency.
As for accessing the currency object across the files, I'd add another parameter to your execute method, something like this:
module.exports = {
name: 'balance',
description: 'get balance',
execute(client, message, args) { // Notice the added "client"
const target = message.mentions.users.first() || message.author;
return message.channel.send(`${target.tag} has ${client.currency.getBalance(target.id)}🍉`); // Added "client." in front of "currency", because currency is a property of your client now
},
};
Then, when executing the execute method, you'd run execute(Client, message, arguments);. Your client would then be passed into the command and be usable in there.
I have a Lambda that I have created using the AWS Toolkit for Visual Studio Code. I have a small lambda.js file that exports the handler for API Gateway events.
const App = require('./app');
const AWS = require('aws-sdk');
exports.handler = async (event, context) => {
let config = {
aws_region: 'us-east-1',
//endpoint: 'http://localhost:8000',
};
let dynamo = new AWS.DynamoDB.DocumentClient();
let app = new App(dynamo, config, event);
try {
let response = await app.run();
return response;
} catch(err) {
return err;
}
};
The app.js file represents the logic of my Lambda. This is broken up to improve testing.
const AWS = require('aws-sdk');
class App {
constructor(datastore, configuration, event) {
this.datastore = datastore;
this.httpEvent = event;
this.configuration = configuration;
}
async run() {
AWS.config.update({
region: this.configuration.aws_region,
endpoint: this.configuration.endpoint,
});
let params = {
TableName: 'test',
Key: {
'year': 2015,
'title': 'The Big New Movie',
},
};
const getRequest = this.datastore.get(params);
// EXCEPTION THROWN HERE
var result = await getRequest.promise();
let body = {
location: this.configuration.aws_region,
url: this.configuration.endpoint,
data: result
};
return {
'statusCode': 200,
'body': JSON.stringify(body)
};
}
}
module.exports = App;
I can write the following mocha test and get it to pass.
'use strict';
const AWS = require('aws-sdk');
const chai = require('chai');
const sinon = require('sinon');
const App = require('../app');
const expect = chai.expect;
var event, context;
const dependencies = {
// sinon.stub() prevents calls to DynamoDB and allows for faking of methods.
dynamo: sinon.stub(new AWS.DynamoDB.DocumentClient()),
configuration: {
aws_region: 'us-west-2',
endpoint: 'http://localhost:5000',
},
};
describe('Tests handler', function () {
// Reset test doubles for isolating individual test cases
this.afterEach(sinon.reset);
it('verifies successful response', async () => {
dependencies.dynamo.get.returns({ promise: sinon.fake.resolves('foo bar')});
let app = new App(dependencies.dynamo, dependencies.configuration, event);
const result = await app.run()
expect(result).to.be.an('object');
expect(result.statusCode).to.equal(200);
expect(result.body).to.be.an('string');
console.log(result.body);
let response = JSON.parse(result.body);
expect(response).to.be.an('object');
expect(response.location).to.be.equal(dependencies.configuration.aws_region);
expect(response.url).to.be.equal(dependencies.configuration.endpoint);
expect(response.data).to.be.equal('foo bar');
});
});
However, when I run the Lambda locally using the Debug Locally via the Code Lens option in VS Code the results from the AWS.DynamoDB.DocumentClient.get call throws an exception.
{
"message":"The provided key element does not match the schema",
"code":"ValidationException",
...
"statusCode":400,
"retryable":false,
"retryDelay":8.354173589804192
}
I have a table created in the us-east-1 region where the non-test code is configured to go to. I've confirmed the http endpoint being hit by the DocumentClient as being dynamodb.us-east-1.amazonaws.com. The table name is correct and I have a hash key called year and a sort key called title. Why would this not find the key correctly? I pulled this example from the AWS documentation, created a table to mirror what the key was and have not had any luck.
The issue is that the Key year was provided a number value while the table was expecting it to be a string.
Wrapping the 2015 in quotes let the Document know that this was a string and it could match to the schema in Dynamo.
let params = {
TableName: 'test',
Key: {
'year': '2015',
'title': 'The Big New Movie',
},
};
I'm new the Node.js and I've been working with a sample project by a third party provider and I'm trying to use Azure Key Vault to store configuration values.
I'm having trouble getting a process to wait before executing the rest. I'll try to detail as much as I know.
The sample project has a file named agent.js which is the start page/file. On line 16 (agent_config = require('./config/config.js')[process.env.LP_ACCOUNT][process.env.LP_USER]) it calls a config file with values. I'm trying to set these value using Key Vault. I've tried many combinations of calling functions, and even implementing async / await but the value for agent_config always contains a [Promise] object and not the data returned by Key Vault.
If I'm right, this is because the Key Vault itself uses async / await too and the config file returns before the Key Vault values are returned.
How can Key Vault be added/implemented in a situation like this?
Here's what I've tried:
First updated agent.js to
let agent_config = {};
try {
agent_config = require('./config/config.js')['123']['accountName'];
} catch (ex) {
log.warn(`[agent.js] Error loading config: ${ex}`)
}
console.log(agent_config);
Test 1
./config/config.js
const KeyVault = require('azure-keyvault');
const msRestAzure = require('ms-rest-azure');
const KEY_VAULT_URI = 'https://' + '{my vault}' + '.vault.azure.net/' || process.env['KEY_VAULT_URI'];
function getValue(secretName, secretVersion) {
msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' }).then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
module.exports = {
'123': {
'accountName': {
accountId: getValue('mySecretName', '')
}
}
};
Results
{ accountsId: undefined }
Test 2
Made getValue an async function and wrapped it around another function (tried without the wrapping and didn't work either)
./config/config.js
const KeyVault = require('azure-keyvault');
const msRestAzure = require('ms-rest-azure');
const KEY_VAULT_URI = 'https://' + '{my vault}' + '.vault.azure.net/' || process.env['KEY_VAULT_URI'];
async function getValue(secretName, secretVersion) {
msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' }).then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
async function config() {
module.exports = {
'123': {
'accountName': {
accountId: await getValue('mySecretName', '')
}
}
};
}
config();
Results
{}
Test 3
Made getValue an async function and wrapped it around another function (tried without the wrapping and didn't work either)
./config/config.js
const KeyVault = require('azure-keyvault');
const msRestAzure = require('ms-rest-azure');
const KEY_VAULT_URI = 'https://' + '{my vault}' + '.vault.azure.net/' || process.env['KEY_VAULT_URI'];
async function getValue(secretName, secretVersion) {
return msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' })
.then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
return client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
module.exports = {
'123': {
'accountName': {
accountId: getValue('mySecretName', '')
}
}
};
config();
Results
{ accountId: { <pending> } }
Other
I've tried many others ways like module.exports = async (value) =< {...} (found through other questions/solutions without success.
I'm starting to think I need to do some "waiting" on agent.js but I haven't found good info on this.
Any help would be great!
One issue is that your getValue function is not returning anything as your returns need to be explicit.
(and without the promise being returned, there's nothing to await on)
async function getValue(secretName, secretVersion) {
return msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' })
.then((credentials) => {
const client = new KeyVault.KeyVaultClient(credentials);
return client.getSecret(KEY_VAULT_URI, secretName, secretVersion).then(
function (response) {
return response.Value;
});
});
}
You could also get away with less explicit returns using arrow functions..
const getValue = async (secretName, secretVersion) =>
msRestAzure.loginWithAppServiceMSI({ resource: 'https://vault.azure.net' })
.then(credentials => {
const client = new KeyVault.KeyVaultClient(credentials);
return client.getSecret(KEY_VAULT_URI, secretName, secretVersion)
.then(response => response.Value);
});
Introducing the Azure Key Vault read, which is async, means your whole config read is async. There' nothing you can do to get around that. This will mean that the code that uses the config will need to handle it appropriately. You start by exporting an async function that will return the config..
async function getConfig() {
return {
'123': {
'accountName': {
accountId: await getValue('mySecretName', '')
}
}
};
}
module.exports = getConfig;
In your agent code you call that function. This will mean that your agent code will need to be wrapped in a function too, so maybe something like this..
const Bot = require('./bot/bot.js');
const getConfig = require('./config/config.js');
getConfig().then(agentConfig => {
const agent = new Bot(agentConfig);
agent.on(Bot.const.CONNECTED, data => {
log.info(`[agent.js] CONNECTED ${JSON.stringify(data)}`);
});
});
The package azure-keyvault has been deprecated in favor of the new packages to deal with Keyvault keys, secrets and certificates separately. For your scenario, you can use the new #azure/keyvault-secrets package to talk to Key Vault and the new #azure/identity package to create the credential.
const { SecretClient } = require("#azure/keyvault-secrets");
const { DefaultAzureCredential } = require("#azure/identity");
async function getValue(secretName, secretVersion) {
const credential = new DefaultAzureCredential();
const client = new SecretClient(KEY_VAULT_URI, credential);
const secret = await client.getSecret(secretName);
return secret.value;
}
The DefaultAzureCredential assumes that you have set the below env variables
AZURE_TENANT_ID: The tenant ID in Azure Active Directory
AZURE_CLIENT_ID: The application (client) ID registered in the AAD tenant
AZURE_CLIENT_SECRET: The client secret for the registered application
To try other credentials, see the readme for #azure/identity
If you are moving from the older azure-keyvault package, checkout the migration guide to understand the major changes