I have a scenario where I have to run a loop and each iteration call redis set function (async) and then I have to close the connection to redis.
Sequence...
Run the foreach loop.
For each element use the redis set command.
When loop is completed, close the redis connection.
Now redis connection is getting closed before all the set operation is completed inside the for loop.
Additional detail...
I'm using the node.js redis client.
I know why this happening but I'm not sure how to resolve this situation.
I'm new newbie in NodeJS.
You don't need to hack around structure - the node.js client supports techniques to do this out of the box. Check out Multi.exec
Commands are queued up inside the Multi object until Multi.exec()
// iterate and construct array of set commands
let commands = items.map(i => ['set', i.key, i.value]);
client
.multi(commands)
.exec((err, replies) => {
// disconnect here
});
If you don't actually need the transactions, you can batch all your commands at once via client.batch. You then of course can organize your connect and disconnect strategy around this pattern accordingly.
You can use asyncLoop for this type of problem, it helped me a lot in similar circumstances:-
var asyncLoop = require('node-async-loop');
asyncLoop(arr, function (item, next){
// foreach loop on the array "arr"
//the each element is "item"
//call next when the iteration is done
//execute redis command on "item" and then do
next();
}, function(err)
if(err==null)
{
//do something on loop complete like closing connection etc
});
This is the basic way in which you can do what you want, you can isntall async loop likes this:-
npm install --save node-async-loop
First you create a array of promise and then use Promise.all eg:
function connectPromise (paramers) {
return new Promise((resolve, reject) => {
// if connectToRedisAndSetFunc is a promise
// conectToRedisAndSetFunc(params).then(data => {
// resolve(data)
//}).catch(err => {
// reject(err)
// })
conectToRedisAndSetFunc(params, function(err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})
}
// create promiseArray
let connectsPromises = params-array.map(p => connectPromise(p))
Promise.All(connectsPromises).then(data => {
// close redis here
}).catch(err => {
console.log(err)
})
Related
I have a node app that uses express for routing and pdfmake npm for generating a pdf document. On the click of a button, I make an http request that retrieves data from a database, generates a pdf document, and saves to disk. However, my async/await functions only seem to work before I create a write stream using fs.createWriteStream(path). All async/awaits after that seem to be ignored. Also, this only happens on a prod server. When debugging my app locally, ALL async/await functions seem to work. Any ideas as to why this could be happening?
Express route:
router.patch('/:id(\\d+)/approve', async function (req, res) {
try {
let id = req.params.id
const invoice = await db.fetchInvoiceById(id)
const harvestInvoice = await harvest.getInvoiceById(invoice.harvest_id)
// generate invoice pdf
await pdf.generateInvoice(invoice, harvestInvoice)
res.status(200).json({ id: id })
} catch (error) {
res.status(400).json({ error: 'something went wrong' })
}
})
Functions:
async function SLEEP5() {
await new Promise((resolve, reject) => {
setTimeout(() => {
resolve('DONE');
}, 5000);
});
}
function test(doc, invoicePath) {
return new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(invoicePath)
writeStream.on("finish", () => { resolve(true) })
writeStream.on("error", () => { reject(false) })
doc.pipe(writeStream)
doc.end()
})
}
exports.generateInvoice = async function generateInvoice(invoice, harvestInvoice) {
const invoicePath = `${__dirname}\\invoice_${invoice.number}.pdf`
let printer = new PdfPrinter(fonts)
let def = { // pdf defined here }
// generate invoice PDF
let doc = printer.createPdfKitDocument(def, {})
await SLEEP5() // THIS IS AWAITED
await test(doc, invoicePath)
await SLEEP5() // THIS IS NOT AWAITED FOR SOME REASON
}
I am using PM2 to run this node app on an aws ec2 server and Im using version 0.2.4 of pdfmake
I figured out what my issue was. It turns out that I was using pm2 start appName --watch to run my app. I was writing the pdf to a directory within the app. PM2 was detecting a change when the the pdf was being written and would restart the app (because of the --watch flag), causing all the issues i was seeing.
I don't know what printer.createPdfKitDocument(def, {}) does exactly, but
let doc = printer.createPdfKitDocument(def, {})
await sleep(5)
await writeStreamToFile(doc, invoicePath)
certainly looks problematic. If doc is not paused at creation, it might run and finish while you're still sleeping, and then pipe nothing into your write stream which will never emit a finish or error event. So remove the await sleep(5), and immediately do doc.pipe(writeStream) and immediately start listening for the events.
If you insist on waiting, either do
let doc = printer.createPdfKitDocument(def, {})
await Promise.all([
sleep(5),
writeStreamToFile(doc, invoicePath),
])
or try
const doc = printer.createPdfKitDocument(def, {})
doc.pause()
await sleep(5)
await writeStreamToFile(doc, invoicePath)
(The other explanation of course would be that createPdfKitDocument creates a never-ending stream, or errors without emitting an error event, etc. that would lead to the promise not being resolved).
I am trying to get the results of my simple SELECT command to the index.js file, where I would like to have all records separated in a array. If I print the results in the database.js the JSON.parse just work fine. But if I want to return them and get them into the index.js where I need them, I always get undefined when I print it.
index.js CODE
const express = require('express');
const app = express();
const database = require('./database');
app.use(express.json());
app.use(express.urlencoded());
app.use(express.static('public'));
app.get('/form', (req,res) =>{
res.sendFile(__dirname + '/public/index.html' );
console.log(req.url);
console.log(req.path);
})
app.listen(4000, () =>{
console.log("Server listening on port 4000");
database.connection;
database.connected();
//console.log(database.select());
let results = [];
//results.push(database.select('username, password'));
let allPlayer = database.select('username');
console.log(allPlayer);
});
database.js CODE
let mysql = require('mysql');
const connection = mysql.createConnection({
host: 'localhost',
database: 'minigames',
user: 'root',
password: 'root'
});
function connected(){
connection.connect((err) => {
if(err) throw err;
console.log("Connected...");
})
}
function select(attribute){
let allPlayer = [];
let sql = `SELECT ${attribute} FROM player`;
let query = connection.query(sql, (err, result, field) => {
if(err) throw err;
return Object.values(JSON.parse(JSON.stringify(result)));
})
}
module.exports = {connection, connected, select};
Understand that one of the main things that make JavaScript different from other languages is that it's asynchronous, in simple terms meaning code doesn't "wait" for the code before it to finish executing. Because of this, when you're trying to query a database, which takes some time, the code after it gets impatient and executes regardless of how to the query is doing. To solve this problem, the mysql package utilizes callbacks, which allows you to pass a function to it to execute once the query is finished with the queries result.
Because the library operates on callbacks, it doesn't return anything; that seems quite problematic for using it somewhere else, doesn't it?
To solve this problem, we can make our own callback. Or better yet, use the newer JavaScript feature called promises, where you can basically "return" anything from a function, even when you're in a callback.
Let's implement it with the query:
function select(attribute) {
return new Promise((resolve, reject) => {
let sql = `SELECT ${attribute} FROM player`;
let query = connection.query(sql, (err, result, field) => {
if(err) return reject(err);
resolve(Object.values(JSON.parse(JSON.stringify(result))));
});
});
}
To "return" from a promise, we pass a value to the resolve function. To throw an error, we call the reject function with the error as the argument.
Our new function is rather easy to use.
select("abcd").then(result => {
console.log("Result received:", result);
}).catch(err => {
console.error("Oops...", err);
});
You might look at this code and go, "Wait a minute, we're still using callbacks. This doesn't solve my problem!"
Introducing async/await, a feature to let you work just with that. We can call the function instead like this:
// all 'await's must be wrapped in an 'async' function
async function query() {
const result = await select("abcd"); // woah, this new await keyword makes life so much easier!
console.log("Result received:", result);
}
query(); // yay!!
To implement error catching, you can wrap you stuff inside a try {...} catch {...} block.
I just started out learning Node.js / Express and I still have difficulties with the Asynch functions. I made some functions to interact with a postgresql database (with some tutorials), and selecting rows from data is going fine but for some reason something is going from with deleting the rows. Here is an example of a function that is going well:
const getPlayers = () => {
return new Promise(function(resolve, reject) {
pool.query('SELECT * FROM Players ORDER BY p_id ASC', (error, results) => {
if (error) {
reject(error)
}
resolve(results.rows);
})
})
}
Now the following function is not going well. Console.log(id) gives the right number, but it seems that id is undefined when executing the query and I suspect that it has to do with Asynch/synch. Now Asynch is new for me, so I am also not an expert on what is going wrong.
Here is the function that is nog going good:
const deletePlayer = (id) => {
return new Promise(function(resolve, reject) {
pool.query('DELETE FROM Players WHERE player_id = ?' , [id], (error,results) => {
if (error) {
reject(error)
}
resolve(`Player deleted with ID: ${id}`)
})
})
}
The function call:
app.delete('/laProjects/:id', (req, res) => {
players_model.deletePlayers(req.params.id)
.then(response => {
res.status(200).send(response);
})
.catch(error => {
res.status(500).send(error);
})
})
How to debug this situation
Don't automatically assume it is an async issue. First try some simple console.log steps:
const deletePlayer = (id) => {
console.log("Started deletePlayer with id: ",id) ///////////
return new Promise(function(resolve, reject) {
console.log("Inside the promise, id is the same, namely: ",id) ///////////
pool.query('DELETE FROM Players WHERE player_id = ?' , [id], (error,results) => {
if (error) {
reject(error)
}
resolve(`Player deleted with ID: ${id}`)
})
})
}
See if 'id' is what you expect it to be
If you don't see any console.log messages printed, maybe, as #steve16351 suggests, you are editing deletePlayer but actually calling another functiondeletePlayers?
In general, avoid using the word "async" for promises used without the "async" keyword
When Promises first arrived in Javascript, they were the practical tool for asynchronous programming, so people sometimes used the term "async" for them in speech.
However since the async keyword arrived, it is better not to use the word "async" for things that are not that keyword.
Simple glossary
Blocking code: a program that simply waits (preventing any other code running) while some external process happens.
Callbacks: the oldest form of asynchronous programming in JS. You tell JS to run a certain function only after some external event has happened.
Promises: a more convenient and readable way to achieve the same effect as callbacks. You can imagine the process to be constructed out of hidden callbacks.
async keyword: an even more convenient and readable way to achieve the same effects as callbacks and promises. It is implicitly made out of promises, although it protects you from having to think about how to construct a new promise.
In response to you reporting that id is 5 inside the promise
This tells us that your original assumption, that the error is due to a variable being undefined, is incorrect. id has the expected value.
Therefore the error is that the API is giving an error in response to this call:
pool.query(
'DELETE FROM Players WHERE player_id = ?' ,
[5],
(error,results) => {
if (error) {
reject(error)
} else {
resolve(`Player deleted with ID: ${id}`);
}
}
)
So why don't you run exactly that, in simplified form like this:
pool.query(
'DELETE FROM Players WHERE player_id = ?' ,
[5],
(error,results) => {
console.log("Error:",JSON.stringify(error,null,2),"Results:",JSON.stringify(error,null,2))
}
)
This sends an explicit request to pool.query, and explicitly reveals the output. I don't know the format the pool.query API is expecting: perhaps it is not quite right?
I am scraping a bunch of API's and saving the data to a dynamodb table.
Everything works absolutely fine when running serverless invoke local -f runAggregator locally.
However, after I set up the cron, I noticed things were not being saved to the Dynamodb table.
Here is my function:
module.exports.runAggregator = async (event) => {
await runModules({ saveJobs: true });
return {
statusCode: 200,
body: JSON.stringify(
{
message: "Aggregate",
input: event,
},
null,
2
),
};
};
And the runModules function:
module.exports = async ({ saveJobs }) => {
if (saveJobs) {
const flushDb = await flushDynamoDbTable();
console.log("Flushing Database: Complete");
console.log(flushDb);
}
// pseudo code
const allJobs = myLongArrayOfJobsFromApis
const goodJobs = allJobs.filter((job) => {
if (job.category) {
if (!job.category.includes("Missing map for")) return job;
}
});
// This runs absolutely fine locally...
if (saveJobs) goodJobs.forEach(saveJob); // see below for function
const badJobs = allJobs.filter((job) => {
if (!job.category) return job; // no role found from API
if (job.category.includes("Missing map for")) return job;
});
console.log("Total Jobs", allJobs.length);
console.log("Good Jobs", goodJobs.length);
console.log("Malformed Jobs", badJobs.length);
return uniqBy(badJobs, "category");
};
saveJob function
// saveJob.js
module.exports = (job) => {
validateJob(job);
dynamoDb
.put({
TableName: tableName,
Item: job,
})
.promise();
};
I am baffled as to why this works fine locally not when I run a 'test' in the lambda console. I only found out due to the table being empty after the cron had ran.
saveJob performs an async operation (ddb.put().promise()) but you are neither awaiting its completion nor returning the promise.
As the forEach in the runModules function will also not await anything, the function completes before the call to dynamodb is even performed (because of how promises vs synchronous code work) and the process is killed after the lambda's execution.
Locally you are not running lambda but something that looks like it. There are subtle differences, and what happens after the function is done is one of those differences. So it may work locally, but it won't on an actual lambda.
What you need to do is to make sure you await your call to dynamodb. Something like:
// saveJob.js
module.exports = (job) => {
validateJob(job);
return dynamoDb
.put({
TableName: tableName,
Item: job,
})
.promise();
};
and in your main function:
...
if (saveJobs) await Promise.all(...goodJobs.map(job => saveJob(job)))
// or with a Promise lib such as bluebird:
if (saveJobs) await Promise.map(goodJobs, job => saveJob(job))
// (or Promise.each(...) if you need to make sure this happens in sequence and not in parallel)
Note: instead of calling many times dynamodb.put, you could/should call once (or at least fewer times) the batchWriteItem operation, which can write up to 25 items in one call, saving quite a bit of latency in the process.
Using sequelize.js in a nodejs app, and I have a promise.all that takes two promises (a user query, and a color query):
router.get(`/someEndPoint`, (req, res) => {
let userAccount = user.findOne({
where: {
id: //some ID
}
});
let colorStuff = color.findOne({
where: {
colorName: //some color
}
})
Promise.all([userAccount , colorStuff ]).then(([result1, result2]) => {
//do stuff, such as:
res.send('success');
}).catch(err => {
console.log(err)
});
});
At the part that says //do stuff, my console keeps giving me this warning:
a promise was created in a handler at... but was not returned from it,
see (URL that I can't post) at Function.Promise.attempt.Promise.try
I'm not sure how to resolve this. I thought after the .then that the promises are resolved?
Hard to tell without other context, but perhaps you need to return the Promise.all
return Promise.all([user, color])...
From the bluebird docs here: https://github.com/petkaantonov/bluebird/blob/master/docs/docs/warning-explanations.md#warning-a-promise-was-created-in-a-handler-but-was-not-returned-from-it
if there are any other promises created in the // do stuff area, be sure to return those as well.