I am trying to use pino for logging in to my node app Server and I do have some large logs coming, so rotating the files every day would be more practical.
So I used pino.multistream() and require('file-stream-rotator')
My code works, but for performance reasons, I would not like to use the streams in the main thread.
according to the doc, I should use pino.transport():
[pino.multistream()] differs from pino.transport() as all the streams will be executed within the main thread, i.e. the one that created the pino instance.
https://github.com/pinojs/pino/releases?page=2
However, I can't manage to combine pino.transport() and file-stream-rotator.
my code that does not work completely
-> logs the first entries, but is not exportable because it blocks the script with the error
throw new Error('the worker has exited')
Main file
const pino = require('pino')
const transport = pino.transport({
target: './custom-transport.js'
})
const logger = pino(transport)
logger.level = 'info'
logger.info('Pino: Start Service Logging...')
module.exports = {
logger
}
custom-transport.js file
const { once } = require('events')
const fileStreamRotator = require('file-stream-rotator')
const customTransport = async () => {
const stream = fileStreamRotator.getStream({ filename: 'myfolder/custom-logger.log', frequency: 'daily' })
await once(stream, 'open')
return stream
}
module.exports = customTransport
Related
I create for my project a log module and i am actually creating a new instance in all module to allow them to log in cli with the right syntax,color conf etc.
For example (a simplified example)
// index.js
const {Log,Ansi} = require("./class/log.js");
const Tool = require("./class/tool.js");
const argv = require("yargs").argv;
let log = new Log({
levelIcon:true,
powerlineRoot:{
name:"root",
backgroundColor:Ansi.BLACK_BRIGHT,
text: "myappName"
}
});
let tool = new Tool(argv.toolName,argv.envName)
tool.run().then(() => {
log.print("Tool is running","info");
}).catch((err) => {
log.print(err,"critical");
});
// tool.js
const {Log,Ansi} = require("./log.js");
class Tool {
let log = new Log({
levelIcon:true,
powerlineRoot:{
name:"root",
backgroundColor:Ansi.BLACK_BRIGHT,
text: "myappName"
}
});
run(){
return new Promise((resolve, reject) => {
resolve()
}
}
}
module.exports = Tool
I am wondering if there is a way to create only one instance in my index.js and share it with the instance of modules like Tools. I don't know if it's possible but i think that it will be less memory consumption to share one instance of Log than creating multiple one
I hope that my question is enough clear. Feel free to ask me more information if needed
Yes you can do that absolutely. Since the entry is index.js you can consider it will finally run as if it was a single file, in a single thread. You can create one more module logger.js like:
const {Log,Ansi} = require("./class/log.js");
const logger = new Log({
levelIcon:true,
powerlineRoot:{
name:"root",
backgroundColor:Ansi.BLACK_BRIGHT,
text: "myappName"
}
});
module.exports = logger;
Now you can just import logger and use it like:
const logger = require("./logger")
logger.print("hello world!");
Like #jjsingh says i use global variable to store the object.
Not sure it's the better way but for the moment it resolve my issue.
global.log = new Log({
levelIcon:true,
powerlineRoot:{
name:"root",
color:{
background:Ansi.BLUE_SEA,
foreground:Ansi.RED,
},
text:global.conf.get("infos").name
}
});
first and foremost, I'm very new to this. I've been following the tutorials at the Discord.js Site, with the goal being to make a discord bot for the Play by Post DnD server I'm in where everyone wants to gain experience via word count.
I mention I'm new to this because this is my first hands-on experience with Javascript, a lot of the terminology goes over my head.
So, the problem seems to be where I've broken away from the tutorial. It goes over command handlers, which I want to stick with because it seems to be good practice and easier to work with down the line when something most breaks (And I know it will). But the tutorial for Databases (Currency/Sequelizer) doesn't really touch on command handlers beyond "Maintain references".
But that's enough foreword, the problem is in trying to get a command that checks the database for a player's current experience points and level.
I have the seemingly relevant files organized with the index.js and dbObjects.js together, a models folder for the Users, and LevelUp(CurrencyShop in the tutorial) and a separate folder for the Commands like the problematic one, xpcheck.js
I can get the command to function without breaking, using the following,
const { Client, Collection, Formatters, Intents } = require('discord.js');
const { SlashCommandBuilder } = require('#discordjs/builders');
const experience = new Collection();
const level = new Collection();
Reflect.defineProperty(experience, 'getBalance', {
/* eslint-disable-next-line func-name-matching */
value: function getBalance(id) {
const user = experience.get(id);
return user ? user.balance : 0;
},
});
Reflect.defineProperty(level, 'getBalance', {
/* eslint-disable-next-line func-name-matching */
value: function getBalance(id) {
const user = level.get(id);
return user ? user.balance : 1;
},
});
module.exports = {
data: new SlashCommandBuilder()
.setName('xpcheck')
.setDescription('Your current Experience and Level'),
async execute(interaction) {
const target = interaction.options.getUser('user') ?? interaction.user;
return interaction.reply(`${target.tag} is level ${level.getBalance(target.id)} and has ${experience.getBalance(target.id)} experience.`);;
},
};
The problem is that the command doesn't reference the database. It returns default values (1st level, 0 exp) every time.
I tried getting the command to reference the database, one of many attempts being this one;
const { Client, Collection, Formatters, Intents } = require('discord.js');
const { SlashCommandBuilder } = require('#discordjs/builders');
const Sequelize = require('sequelize');
const { Users, LevelUp } = require('./DiscordBot/dbObjects.js');
module.exports = {
data: new SlashCommandBuilder()
.setName('xpcheck')
.setDescription('Your current Experience and Level'),
async execute(interaction) {
const experience = new Collection();
const level = new Collection();
const target = interaction.options.getUser('user') ?? interaction.user;
return interaction.reply(`${target.tag} is level ${level.getBalance(target.id)} and has ${experience.getBalance(target.id)} experience.`);;
},
};
However, when I run node deploy-commands.js, it throws
Error: Cannot find module './DiscordBot/dbObjects.js'
It does the same thing even if I remove the /DiscordBot, or any other way I've attempted to make a constant for it. I'm really uncertain what I should do to alleviate this issue.
My file structure, for reference, is:
v DiscordBot
v commands
xpcheck.js
v models
LevelUp.js
UserItems.js
Users.js
dbInit.js
dbObjects.js
deploy-commands.js
index.js
As was pointed out in the comments, the problem was simple, the solution simpler.
Correcting
const { Users, LevelUp } = require('./dbObjects.js');
to
const { Users, LevelUp } = require('../dbObjects.js');
allows it to search the main directory for the requisite file.
I have an app which tallies the number of visits to the url. The tallying is done in Redis. I'm using redis-mock which simulates commands like INCR in memory.
The following test visits the page 3 times and expects the response object to report current as 3:
let app = require('./app');
const supertest = require("supertest");
jest.mock('redis', () => jest.requireActual('redis-mock'));
/* Preceeded by the exact same test */
it('should report incremented value on multiple requests', (done) => {
const COUNT = 3;
const testRequest = function (cb) { supertest(app).get('/test').expect(200, cb) };
async.series([
testRequest,
testRequest,
testRequest
], (err, results) => {
if (err) console.error(err);
const lastResponse = _.last(results).body;
expect(
lastResponse.current
).toBe(COUNT);
done();
});
});
The issue is that if I keep reusing app, the internal "redis" mock will continue getting incremented between tests.
I can side-step this a bit by doing this:
beforeEach(() => {
app = require('./app');
jest.resetAllMocks();
jest.resetModules();
});
Overwriting app seems to do the trick but isn't there a way to clean-up the "internal" mocked module somehow between tests?
My guess is that somehow the '/test' endpoint gets invoked in some other tests in the suite, you could try to run specific parts of your suite using .only or even trying to run the entire suite serially.
To answer the original questions the entire suite must be isolated and consistent either if you are running a specific test case scenario or if you are trying to run the entire suite, thus you need to clear up any leftovers that they could actually affect the results.
So you can actually use the .beforeEach or the .beforeAll methods, provided by Jest in order to "mock" Redis and the .afterAll method for clearance.
A dummy implementation would look like this:
import redis from "redis";
import redis_mock from "redis-mock";
import request from "supertest";
jest.mock("redis", () => jest.requireActual("redis-mock"));
// Client to be used for manually resetting the mocked redis database
const redisClient = redis.createClient();
// Sometimes order matters, since we want to setup the mock
// and boot the app afterwards
import app from "./app";
const COUNT = 3;
const testRequest = () => supertest(app).get("/test");
describe("testing", () => {
afterAll((done) => {
// Reset the mock after the tests are done
jest.clearAllMocks();
// You can also flush the mocked database here if neeeded and close the client
redisClient.flushall(done);
// Alternatively, you can also delete the key as
redisClient.del("test", done);
redisClient.quit(done);
});
it("dummy test to run", () => {
expect(true).toBe(true);
});
it("the actual test", async () => {
let last;
// Run the requests in serial
for (let i = 0; i < COUNT - 1; i++) {
last = await testRequest();
}
// assert the last one
expect(last.status).toBe(200);
expect(last.body.current).toBe(COUNT);
});
});
I have a lot of function in my index.js for my webhook, and i wish split functions to different js files to my code is more clean.
I have not problem to use agent, but I don't know how I can have an access of variable "request" (to have the parameters receipt) in another file.
I tried this :
Index.js
// See https://github.com/dialogflow/dialogflow-fulfillment-nodejs
// for Dialogflow fulfillment library docs, samples, and to report issues
'use strict';
const functions = require('firebase-functions');
const { WebhookClient } = require('dialogflow-fulfillment');
const { Payload } = require('dialogflow-fulfillment');
const Test = require("./Test"); //File with an function
process.env.DEBUG = 'dialogflow:debug'; // enables lib debugging statements
exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
const agent = new WebhookClient({ request, response });
function fallback(agent) {
agent.add(`I didn't understand`);
agent.add(`I'm sorry, can you try again?`);
}
}
let intentMap = new Map();
intentMap.set("intent-test", Test.welcome);
intentMap.set("intent-test", fallback);
agent.handleRequest(intentMap);
});
Test.js
exports.welcome = (agent) => {
agent.add("Hello World !");
console.log(request); // How can I have a access to the variable "request" / "response
};
Do you have a solution please ?
While I understand that you are not having any problems with your agent, there are some limitations when using the DialogFlow Inline editor. As you can see below, one of the limitations is that you can only work with two files: index.js and package.json. That means that all your Fulfillments should be inside your index.js file and you can't split it in more files.
I'm relatively new to Cloud Functions and have been trying to solve this issue for a while. Essentially, the function I'm trying to write is called whenever there is a complete upload onto Firebase Cloud Storage. However, when the function runs, half the time, it runs to the following error:
The following error occured: { Error: ENOENT: no such file or directory, open '/tmp/dataprocessing/thisisthefilethatiswritten.zip'
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/tmp/dataprocessing/thisisthefilethatiswritten.zip' }
Here's the code:
const functions = require('firebase-functions');
const admin = require('firebase-admin')
const inspect = require('util').inspect
const path = require('path');
const os = require('os');
const fs = require('fs-extra');
const firestore = admin.firestore()
const storage = admin.storage()
const runtimeOpts = {
timeoutSeconds: 540,
memory: '2GB'
}
const uploadprocessing = functions.runWith(runtimeOpts).storage.object().onFinalize(async (object) => {
const filePath = object.name
const fileBucket = object.bucket
const bucket_fileName = path.basename(filePath);
const uid = bucket_fileName.match('.+?(?=_)')
const original_filename = bucket_fileName.split('_').pop()
const bucket = storage.bucket(fileBucket);
const workingDir = path.join(os.tmpdir(), 'dataprocessing/');
const tempFilePath = path.join(workingDir, original_filename);
await fs.ensureDir(workingDir)
await bucket.file(filePath).download({destination: tempFilePath})
//this particular code block I included because I was worried that the file wasn't
//being uploaded to the tmp directly, but the results of the function
//seems to confirm to me that the file does exist.
await fs.ensureFile(tempFilePath)
console.log('success!')
fs.readdirSync(workingDir).forEach(file => {
console.log('file found: ', file);
});
console.log('post success')
fs.readdirSync('/tmp/dataprocessing').forEach(file => {
console.log('tmp file found: ', file);
})
fs.readFile(tempFilePath, function (err, buffer) {
if (!err) {
//data processing comes here. Please note that half the time it never gets into this
//loop as instead it goes into the else function below and outputs that error.
}
else {
console.log("The following error occured: ", err);
}
})
fs.unlinkSync(tempFilePath);
return
})
module.exports = uploadprocessing;
I've been trying so many different things and the weird thing is that when I add code into the "if (!err)" (which doesn't actually run because of the err) it just arbitrarily starts working sometimes quite consistently, but then it stops working when I add different code. I would have assumed that the issue arises from the code that I added, but then the error comes up literally when I just change/add/remove comments as well... Which should technically have no effect on the function running...
Any thoughts? Thank you in advance!!! :)
fs.readFile is asynchronous and returns immediately. Your callback function is invoked some time later with the contents of the buffer. This means that fs.unlinkSync is going to delete the file at the same time it's being read. This means you effectively have a race condition, and it's possible that the file will be removed before it's ever read.
Your code should wait until the read is complete before moving on to the delete. Perhaps you want to use fs.readFileSync instead.