I have a function that looks like this
let fileCache = {};
async function generateFromFile(fullPath) {
if (!fileCache[fullPath]) {
let jsonResource = await fs.readFile(fullPath);
fileCache[fullPath] = JSON.parse(jsonResource);
}
return fileCache[fullPath];
}
When I use it in the same module like this
await generateFromFile('myfile.json');
it works fine.
I created a separate file, put the function there and exported it like this
let fileCache = {};
async function generateFromFile(fullPath) {
if (!fileCache[fullPath]) {
let jsonResource = await fs.readFile(fullPath);
fileCache[fullPath] = JSON.parse(jsonResource);
}
return fileCache[fullPath];
}
module.exports = {
generateFromFile: generateFromFile
}
I imported it like this
const utils = require('../util/utils.js');
and wanted to use like await utils.generateFromFile('myfile.json'); but it fails with the error
TypeError [ERR_INVALID_CALLBACK]: Callback must be a function. Received undefined
What is the problem?
How did you import your fs module. To use readFile with await your should
either write fs.promise.readFile
or import fs as
import { promises as fs } from 'fs'
Are you sure that your newly created file is in the relative directory ../util/utils.js? Your export looks fine. Maybe try to print out the imported utils object to see what you actually imported.
EDIT: You are also passing relative filenames to your function which is now in a different diretory than before. When accessing fs.readFile(fullPath) it tries to read the file inside the directory "util/". You should call your function with a full filepath like util.generateFromFile(path.join(__dirname, "myfile.json")) using the path node package.
EDIT 2: Like Raeesaa state you are using fs.readFile wrong. It does not return a Promise but expects a callback. To is it with await you have to promisify it like so:
const fs = require("fs");
const { promisify } = require("util");
const readFileAsync = promisify(fs.readFile);
let fileCache = {};
async function generateFromFile(fullPath) {
if (!fileCache[fullPath]) {
let jsonResource = await readFileAsync(fullPath);
fileCache[fullPath] = JSON.parse(jsonResource);
}
return fileCache[fullPath];
}
module.exports = {
generateFromFile: generateFromFile
}
EDIT 3: If you only import JSON files you can do it by using require. Internally, require has also a cache so imports are only executed once. You can import JSON files as objects like so: const obj = require("myfile.json");
Related
I don't understand why my spy is not being used. I have used this code elsewhere and it has worked fine.
Here is my test:
const {DocumentEngine} = require('../documentEngine')
const fileUtils = require('../utils/fileUtils')
const request = {...}
const fieldConfig = {...}
test('If the Carbone addons file is not found, context is built with the carboneAddons property as an empty object', async () => {
const expectedResult = {
carboneAddons: {},
}
const fileExistSpy = jest
.spyOn(fileUtils, 'checkFileExists')
.mockResolvedValue(false)
const result = await docEngine.buildContext(request, fieldConfig)
expect(fileExistSpy).toHaveBeenCalledTimes(1)
})
Here is the code that it is being tested:
async function buildContextForLocalResources(request, fieldConfig) {
/* other code */
const addonFormatters = await getCarboneAddonFormatters()
const context = {
sourceJson,
addonFormatters,
documentFormat,
documentTemplateId,
documentTemplateFile,
responseType,
jsonTransformContext
}
return context
}
async function getCarboneAddonFormatters() {
const addOnPath = path.resolve(
docConfig.DOC_GEN_RESOURCE_LOCATION,
'library/addon-formatters.js'
)
if (await checkFileExists(addOnPath)) {
logger.info('Formatters found and are being used')
const {formatters} = require(addOnPath)
return formatters
}
logger.info('No formatters were found')
return {}
}
This is the code from my fileUtils file:
const fs = require('fs/promises')
async function checkFileExists(filePath) {
try {
await fs.stat(filePath)
return true
} catch (e) {
return false
}
}
My DocumentEngine class calls the buildContext function which in turn calls the its method getCarboneAddonFormatters. The fileUtils is outside of DocumentEngine class in a utilities folder. The original code I had this working on was TypeScript as opposed to this which is just NodeJS Javascript. The config files for both are the same. When I try to step through the code (VSCode debugger), as soon as I hit the line with await fs.stat(filePath) in the checkFileExists function, it kicks me out of the test and moves on to the next test - no error messages or warnings.
I've spent most of the day trying to figure this out. I don't think I need to do an instance wrapper for the documentEngine, because checkFileExists is not a class member, and that looks like a React thing...
Any help in getting this to work would be appreciated.
I’ve logger which I initiate using a constractor in the index.js file. Now I need
To pass the logger instance to other files, and I do it like this
index.js
const books = require(“./books”);
books(app, logger);
logger = initLogger({
level: levels.error,
label: “app”,
version: "0.0.1",
});
app.listen(port, () => logger.info(`listening on port ${port}`));
And inside the books.js file I use it like following, get the logger from the index.js file and use it
inside the books.js file, also pass it to another file with the function isbn.get(books, logger);,
Is it recommended to do it like this? Is there a cleaner way in nodes ?
books.js
const isbn = require(“./isbn”);
module.exports = async function (app, logger) {
…
try {
Let books = await getBooks();
logger.info(“get “books process has started”);
} catch (err) {
logger.error("Failed to fetch books", err);
return;
}
…
// this function is from the file “isbn” and I should pass the logger to it also
try {
let url = await isbn.get(books, logger);
} catch (e) {
res.send(e.message);
}
}
Try creating a module specifically for your logger configuration, then you can import that into your modules instead of using a side-effect of your business module to create a logger.
This will help if you ever need/want to change your logger configuration - instead of following a chain of business methods, you can just update the log configuration.
Example
logger.js
'use strict';
// Any setup you need can be done here.
// e.g. load log libraries, templates etc.
const log = function(level, message) {
return console.log(level + ": " + message);
};
module.exports = log;
business-logic.js
'use strict';
var log = require('./logger');
var stuff = require('./stuff');
const do_stuff = function (thing) {
// do stuff here
log("INFO", "Did stuff");
}
This is a pretty clean way of doing it, however it could be awkward when trying to share more variables or adding more requires. So, you could put all the variables in an object and destructure only the variables you need in books.js:
index.js:
const state = {app, logger, some, other, variables};
require("./books")(state);
require("./another_file")(state);
books.js:
module.exports = async function ({app, logger}) {
};
Before top-level await becomes a thing, loading secrets asynchronously from AWS Secrets Manager upon startup is a bit of a pain. I'm wondering if anyone has a better solution than what I currently have.
Upon starting up my Node.JS server I'm loading all secrets from AWS Secrets manager and setting them in config files where I have a mix of hardcoded variables and secrets. Here's an example:
In aws.js
import AWS from 'aws-sdk';
const region = "eu-north-1";
AWS.config.setPromisesDependency();
const client = new AWS.SecretsManager({
region
});
export const getSecret = async(secretName) => {
const data = await client.getSecretValue({SecretId: secretName}).promise();
return data.SecretString;
}
Then in sendgridConfig.js
import { getSecret } from "./aws";
export default async() => {
const secret = JSON.parse(await getSecret("sendgridSecret"));
return {
APIKey: secret.sendgridKey,
fromEmail: "some#email.com",
toEmail: "some#email.com"
}
}
Then in some file where the config is used:
import { sendgridConfig } from "./sendgridConfig";
const myFunc = () => {
const sendgridConf = await sendgridConfig();
... do stuff with config ...
}
This works okay in async functions, but what if I'd like to use the same setup in non-async functions where I use my hardcoded variables? Then the secrets haven't been fetched yet, and I can't use them. Also I have to always await the secrets. IMO a good solution in the future could be top level await, where upon booting the server, the server will await the secrets from AWS before proceeding. I guess I could find a way to block the main thread and set the secrets, but that feels kind of hacky.
Does anyone have a better solution?
So I ended up doing the following. First I'm setting the non-async config variables in an exported object literal. Then I'm assigning values to the object literal in the sendgridConfigAsync IIFE (doesn't have to be an IFEE). That way I don't have to await the config promise. As long as the app awaits the IIFE on startup, the keys will be assigned before being accessed.
In sendgridConfig.js
import { getSecret } from "./aws";
export const sendgridConfig = {
emailFrom: process.env.sendgridFromEmail,
emailTo: process.env.sendgridToEmail
}
export const sendgridConfigAsync = (async() => {
const secret = JSON.parse(await getSecret("Sendgrid-dev"));
sendgridConfig.sendgridKey = secret.key;
})()
Then in the main config file _index.js where I import all the config files.
import { sendgridConfigAsync } from "./sendgrid";
import { twilioConfigAsync } from "./twilio";
import { appConfigAsync } from "./app";
export const setAsyncConfig = async() => {
await Promise.all([
appConfigAsync,
sendgridConfigAsync,
twilioConfigAsync
]);
}
Then in the main index.js file I'm awaiting the setAsyncConfig function first. I did also rebuild the app somewhat in order to control all function invocations and promise resolving in the desired order.
import { servicesConnect } from "../src/service/_index.js";
import { setAsyncConfig } from '$config';
import { asyncHandler } from "./middleware/async";
import { middleware } from "./middleware/_index";
import { initJobs } from "./jobs/_index"
import http from 'http';
async function startServer() {
await setAsyncConfig();
await servicesConnect();
await initJobs();
app.use(middleware);
app.server = http.createServer(app);
app.server.listen(appConfig.port);
console.log(`Started on port ${app.server.address().port}`);
}
asyncHandler(startServer());
Yup I have the same problem. Once you start with a promise, all dependencies down the line require await.
One solution is to do your awaits and only after that have all your downstream code run. Requires a slightly different software architecture.
E.g.
export const getSecretAndThenDoStuff = async(secretName) => {
const data = await client.getSecretValue({SecretId: secretName}).promise();
// instead of return data.SecretString;
runYourCodeThatNeedsSecret(data);
}
A more generic solution to the top-level await that I tend to use:
async function main() {
// Do whatever you want with await here
}
main();
Clean and simple.
For example, if I have main.js calling a defined in src/lib/a.js, and function a calls node-uuid.v1, how can I stub node-uuid.v1 when testing main.js?
main.js
const a = require("./src/lib/a").a
const main = () => {
return a()
}
module.exports = main
src/lib/a.js
const generateUUID = require("node-uuid").v1
const a = () => {
let temp = generateUUID()
return temp
}
module.exports = {
a
}
tests/main-test.js
const assert = require("assert")
const main = require("../main")
const sinon = require("sinon")
const uuid = require("node-uuid")
describe('main', () => {
it('should return a newly generated uuid', () => {
sinon.stub(uuid, "v1").returns("121321")
assert.equal(main(), "121321")
})
})
The sinon.stub(...) statement doesn't stub uuid.v1 for src/lib/a.js as the above test fails.
Is there a way to globally a library function so that it does the specified behavior whenever it gets called?
You should configure the stub before importing the main module. In this way the module will call the stub instead of the original function.
const assert = require("assert")
const sinon = require("sinon")
const uuid = require("node-uuid")
describe('main', () => {
it('should return a newly generated uuid', () => {
sinon.stub(uuid, "v1").returns("121321")
const main = require("../main")
assert.equal(main(), "121321")
})
})
Bear in mind that node-uuid is deprecated as you can see by this warning
[Deprecation warning: The use of require('uuid') is deprecated and
will not be supported after version 3.x of this module. Instead, use
require('uuid/[v1|v3|v4|v5]') as shown in the examples below.]
About how to stub that for testing would be a bit more harder than before as actually there is no an easy way to mock a standalone function using sinon
Creating a custom module
//custom uuid
module.exports.v1 = require('uuid/v1');
Requiring uuid from the custom module in your project
const uuid = require('<path_to_custom_module>');
Sinon.stub(uuid, 'v1').returns('12345');
I need to call module from main index.js File
Here is my module
const request = require('./rq.js');
const callback = require('./callback.js')
const url = `https://localhost.3000/${id}`;
request(url, callback)
.then(res => {
console.log(res);
})
.catch(err => {
console.log(err);
})
module.exports = page; //Tell me how to export all code from module
So here is my index.js file
const Methods = {
page: require('./page.js'),
}
module.exports = //What i need to code here?
File from what i give a call a module :
const main = require('./index.js');
main.page({id: 'id'})
.then(console.log);
So what I should change to call page.js file like that ?
Make the following changes to page.js since in your main file you expect a promise to be returned.
const request = require('./rq.js');
const callback = require('./callback.js')
function page({id}) {
const url = `https://localhost.3000/${id}`;
return request(url, callback)
}
module.exports = {page: page} //Tell me how to export all code from module
Make the following changes to Mehods.js
const Methods = {
page: require('./page.js').page,
}
module.exports = Methods;
Check if this works.