I'm using Nodemailer to send mailings in my NodeJS / Express server. Instead of sending the mail directly I want to wait 20 minutes before sending the mail. I think this feels more personal then sending mail directly.
But I have no idea how to achieve this. I guess I don't need something like a NodeJS cronjob like this NodeCron package, or do I?
router.post('/', (req, res) => {
const transporter = nodemailer.createTransport(smtpTransport({
host: 'smtp.gmail.com',
port: 465,
auth: {
user: 'noreply#domain.nl',
pass: 'pass123'
}
}));
const mailOptions = {
from: `"${req.body.name}" <${req.body.email}>`,
to: 'info#domain.nl',
subject: 'Form send',
html: `Content`
};
transporter.sendMail(mailOptions, (error, info) => {
if (error) res.status(500).json({ responseText: error });
res.status(200).json({ responseText: 'Message send!' });
});
}
});
My router looks like as shown above. So if post is called I want this request to wait 20 minutes. Instead of with a cronjob I want to execute the post just once, but with a bit of a delay. Any suggestions on how to do this?
Well some folks may come here and tell you to use an external queue system and bla bla... But you could simply use plain old Javascript to schedule the sending 20*60*1000 milliseconds into the future to get things started. :)
There's however a problem with your code: you're waiting for the mailer to succeed before sending the 200 - 'Message sent' response to the user. Call me a madman but I'm pretty sure the user won't be staring at the browser window for 20 minutes, so you'll probably have to answer as soon as possible and then schedule the mail. Modifying your code:
router.post('/', (req, res) => {
const DELAY = 20*60*1000 // min * secs * milliseconds
const transporter = nodemailer.createTransport(smtpTransport({
host: 'smtp.gmail.com',
port: 465,
auth: {
user: 'noreply#domain.nl',
pass: 'pass123'
}
}));
const mailOptions = {
from: `"${req.body.name}" <${req.body.email}>`,
to: 'info#domain.nl',
subject: 'Form send',
html: `Content`
};
res.status(200).json({ responseText: 'Message queued for delivery' });
setTimeout(function(){
transporter.sendMail(mailOptions, (error, info) => {
if (error)
console.log('Mail failed!! :(')
else
console.log('Mail sent to ' + mailOptions.to)
}),
DELAY
);
}
});
There are however many possible flaws to this solution. If you're expecting big traffic on that endpoint you could end up with many scheduled callbacks that will eat the stack. In addition, if something fails the user of course won't be able to know.
If this is a big / serious project, consider using that cronjob package or using an external storage mechanism where you can queue this "pending" messages (Redis would do and it's incredible simple), and have a different process read tasks from there and perform the email sending.
EDIT: saw some more things on your code.
1) You probably don't need to create a new transport inside your POST handler, create it outside and reuse it.
2) In addition to the mentioned problems, if your server crashed no email will be ever sent.
3) If you still want to do it in a single Node.js app, instead of scheduling an email on every request to this endpoint, you'd be better storing the email data (from, to, subject, body) somewhere and schedule every 20 minutes a function that will get all pending emails, send them one by one, and then reschedule itself to re-run 20 minutes later. This will keep you memory usage low. Server crash still make all emails lost, but if you add REDIS into the mix then you can simply grab all pending emails from REDIS when your app start.
Probably too much for an answer, sorry if it wasn't needed! :)
I think CharlieBrown's answer is correct and since I had two answers in my mind while reading the question, I thank him for simplifying my answer to be the alternative of his.
setTimeout is actually a good idea, but it has a drawback: in the case when there is any reason to stop the server code (server restart, module installation, file management, etc.) your callbacks scheduled at the end of the setTimeout's time parameter will not be executed and some users will not receive emails.
If the problem above is serious-enough, then you might want to store scheduled emails to be sent in the database or into Redis and use a cron job to periodically check the email set and send the emails if there are some.
I think that either this answer or CharlieBrown's should suffice for you, depending on your preferences and needs.
Related
I'm developing a webapp using ReactJS for the frontend and express for the backend. I'm deploying my app to azure.
To test if my requests are going through I wrote two different API requests.
The first one is very simple:
router.get('/test', (req, res) => {
res.send('test was a success');
});
Then in the frontend I have a button which when clicked makes the request and I get the response 'test was a success'. This works every time.
The second test is:
router.post('/test-email', (req, res) => {
let current_template = 'reset';
readHTMLFile(__dirname + '/emails/' + current_template + '.html', function(err, html) {
let htmlSend;
let template = handlebars.compile(html);
let = replacements = {
name: req.body.name
};
htmlSend = template(replacements);
let mailOptions = {
from: 'email#email.com',
to: 'someone#email.com',
subject: 'Test Email',
html: htmlSend
};
transporter.sendMail(mailOptions)
.then(response => {
res.send(response);
})
.catch(console.error);
});
});
Then when I've deployed the app I make a call to each one of these tests. The first one, like I mentioned always succeeds. The second one which is supposed to send a very simple email fails most of the time with the error "iisnode encountered an error when processing the request. HRESULT: 0x6d HTTP status: 500 HTTP subStatus: 1013". The strange thing is that every once in a while the email will send but this happens very rarely. Most times the request will take exactly two minutes before sending a response with an error.
I should note that when in development in localhost both tests work all the time with no issues whatsoever, it's only when in production (deployment to azure) that this happens.
I've been digging around for the last few days and came up with nothing. Any help or directions would be greatly appreciated.
I found out what the problem was. I'm using gmail to send my test emails, by default gmail will block any attempts to use an account if it thinks the app making the request is not secure. This can be easily fixed by simply clicking the link they automatically send you when you make your first attempt. What is not immediately obvious is when you go in production mode they add another level of security which in this case I believe is a captcha, and while you'll be able to send emails in development as soon as you deploy your app this no longer becomes the case.
Anyway, after digging around a little more I found the option to disable the captcha and now my emails send fine!
Link to that option https://accounts.google.com/b/0/DisplayUnlockCaptcha
Hopefully this will help someone.
I would like to send ~50,000 SMS with Twilio, and I was just wondering if my requests are going to crash if I loop through a phone number array of this size. The fact is that Twilio only allows 1 message for each request, so I have to make 50,000 of them.
Is it possible to do it this way or do I have to find another way?
50,000 seems too much but I have no idea of how many requests I can do.
phoneNumbers.forEach(function(phNb)
{
client.messages.create({
body: msgCt,
to: phNb,
from: ourPhone
})
.then((msg) => {
console.log(msg.sid);
});
})
Thanks in advance
Twilio developer evangelist here.
API Limits
First up, a quick note on our limits. With a single number, Twilio has a limit of sending one message per second. You can increase that by adding more numbers, so 10 numbers will be able to send 10 messages per second. A short code can send 100 messages per second..
We also recommend that you don't send more than 200 messages on any one long code per day.
Either way I recommend using a messaging service to send messages like this.
Finally, you are also limited to 100 concurrent API requests. It's good to see other answers here talking about making requests sequentially rather than asynchronously as that will eat up the memory on your server as well as start to find requests are turned down by Twilio.
Passthrough API
We now have an API that allows you to send more than one message with a single API call. It's known as the passthrough API, as it lets you pass many numbers through to the Notify service. You need to turn your numbers into "bindings" and send them via a Notify service, which also uses a messaging service for number pooling.
The code looks a bit like this:
const Twilio = require('twilio');
const client = new Twilio(accountSid, authToken);
const service = client.notify.services('ISXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX');
service.notifications
.create({
toBinding: [
JSON.stringify({
binding_type: 'sms',
address: '+15555555555',
}),
JSON.stringify({
binding_type: 'facebook-messenger',
address: '123456789123',
}),
],
body: 'Hello Bob',
})
.then(notification => {
console.log(notification);
})
.catch(error => {
console.log(error);
})
The only drawbacks in your situation is that every message needs to be the same and the request needs to be less than 1 megabyte in size. We've found that typically means about 10,000 numbers, so you might need to break up your list into 5 API calls.
Let me know if that helps at all.
There are two factors here.
You need to consider Twilio Api usage Limits.
Performing 50.000 parallel http requests (actually your code do it) is not a good idea: you will have memory problems.
Twilio sms limits change based on source and destination.
You have two solution:
Perform 50k http requests sequentially
phoneNumbers.forEach(async function(phNb){
try {
let m = await client.messages.create({
body: msgCt,
to: phNb,
from: ourPhone
})
console.log(a)
} catch(e) {
console.log(e)
}
})
Perform 50k http requests concurrently with concurrency level
This is quite easy to do with the awesome bluebird sugar functions. Anyway, the twilio package uses native promise. You can use async module with mapLimit method for this purpose
You send your requests asynchronous due to non-blocking forEach body calls, I guess it's fastest for the Client. But the question is: does Twilio allow such a load from a single source? it needs to be tested... And if no, you should build some kind of requests queue, e.g. promise based, something like
function sendSync(index = 0) {
if(index === phoneNumbers.length) {
return;
}
client.messages.create({
body: msgCt,
to: phoneNumbers[index],
from: ourPhone
})
.then(function(msg) {
console.log(msg.sid);
sendSync(index + 1);
})
.catch(function(err) {
console.log(err);
});
}
sendSync();
Or if you like async/await –
async function sendSync() {
for (let phNb of phoneNumbers) {
try {
let msg = await client.messages.create({
body: msgCt,
to: phNb,
from: ourPhone
});
console.log(msg);
} catch(err) {
console.log(err);
}
})
}
sendSync();
I have more than 2000 user in my database , when I try to broadcast a message to all users, it barely sends about 200 request then my server stops and I get an error as below :
{ Error: connect ETIMEDOUT 31.13.88.4:443
at Object.exports._errnoException (util.js:1026:11)
at exports._exceptionWithHostPort (util.js:1049:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1090:14)
code: 'ETIMEDOUT',
errno: 'ETIMEDOUT',
syscall: 'connect,
address: '31.13.88.4',
port: 443 }
Sometimes I get another error that says :
Error!: Error: socket hang up
This is my request :
function callSendAPI(messageData) {
request({
uri: 'https://graph.facebook.com/v2.6/me/messages',
qs: { access_token: '#####' },
method: 'POST',
json: messageData
}, function (error, response, body) {
if (!error && response.statusCode == 200) {
var recipientId = body.recipient_id;
var messageId = body.message_id;
if (messageId) {
console.log("Successfully sent message with id %s to recipient %s",
messageId, recipientId);
} else {
console.log("Successfully called Send API for recipient %s",
recipientId);
}
} else {
console.error("Failed calling Send API");
console.log(error)
}
});
}
I have tried
setTimeout to make the the API calling wait for a while:
setTimeout(function(){callSendAPI(data)},200);
Can anyone help if he/she faced a similar error ?
EDITED
I'm using Messenger Platform which support high rate of calls to the Send API and it is not limited with 200 calls .
You may be hitting Facebook API limits. To throttle the requests you should send every request after some interval from the previous one. You didn't include where you're iterating over all users but I suspect that you maybe do it in a loop and if you use setTimeout to delay every request with flat 200ms delay then you have all requests done at the same time like you did before - just 200ms later.
What you can do is:
You can use setTimeout and add variable delay for every request (not recommended)
You can use Async module's series or parallelLimit (using callbacks)
You can use Bluebird's Promise.mapSeries or Promise.map with concurrency limit (using promises)
The 1 is not recommended because it will still be fire-and-forget (unless you add more complexity to that) and you still risk that you have too much concurrency and go over limit because you only control when the requests start, not how many of outstanding requests are there.
The 2 and 3 are mostly the same but differ by using callbacks or promises. In your example you're using callbacks but your callSendAPI doesn't take its own callback which it should if you want option 2 to work - or, alternatively, it should return a promise if you want option 3 to work.
For more info see the docs:
https://caolan.github.io/async/docs.html#parallelLimit
https://caolan.github.io/async/docs.html#series
http://bluebirdjs.com/docs/api/promise.map.html
http://bluebirdjs.com/docs/api/promise.mapseries.html
Of course there are more ways to do it but those are the most straightforward.
Ideally, if you want to fully utilize the 200 requests per hour limit then you should queue the requests yourself and make the requests at certain intervals that correspond to that limit. Sometimes if you didn't do a lot of requests in an hour then you won't need delays, sometime you will. What you should really do here is to queue all requests centrally and empty the queue at intervals corresponding to the already used up portion to the limit which you should track yourself - but that can be tricky.
It sounds like you are hitting a rate limit.
From the Facebook documentation:
Your app can make 200 calls per hour per user in aggregate.
You can check the dashboard to see if you are hitting the rate limiting in these cases.
I built a Slack slash command that communicates with a custom Node API and POSTS acronym data in some way, shape, or form. It either gets the meaning of an acronym or adds/removes a new acronym to a Mongo database.
The command works pretty well so far, but Slack occasionally returns a timeout error because it expects a response within 3 seconds. As a result, I'm trying to implement delayed responses. I'm not sure that I am implementing delayed responses properly for my Slack slash command & Node API.
This resource on Slack slash commands has information on delayed responses. The idea is that I want to send a 200 response immediately to let the Slack user know that their request has been processed. Then I want to send a delayed response to slackReq.response_url that isn't constrained by the 3-second time limit.
The Code
let jwt = require('jsonwebtoken');
let request = require('request');
let slackHelper = require('../helpers/slack');
// ====================
// Slack Request Body
// ====================
// {
// "token":"~",
// "team_id":"~"
// "team_domain":"~",
// "channel_id":"~",
// "channel_name":"~",
// "user_id":"~",
// "user_name":"~",
// "command":"~",
// "text":"~",
// "response_url":"~"
// }
exports.handle = (req, res) => {
let slackReq = req.body;
let token = slackReq.token;
let teamId = slackReq.team_id;
if (!token || !teamId || !slackHelper.match(token, teamId)) {
// Handle an improper Slack request
res.json({
response_type: 'ephemeral',
text: 'Incorrect request'
});
} else {
// Handle a valid Slack request
slackHelper.handleReq(slackReq, (err, slackRes) => {
if (err) {
res.json({
response_type: 'ephemeral',
text: 'There was an error'
});
} else {
// NOT WORKING - Immediately send a successful response
res.json({
response_type: 'ephemeral',
text: 'Got it! Processing your acronym request...'
})
let options = {
method: 'POST',
uri: slackReq.response_url,
body: slackRes,
json: true
};
// Send a delayed response with the actual acronym data
request(options, err => {
if (err) console.log(err);
});
}
});
}
};
What's Happening Right Now
Say I want to find the meaning of acronym NBA. I go on Slack and shoot out the following:
/acronym NBA
I then hit the 3-second timeout error - Darn – that slash command didn't work (error message: Timeout was reached). Manage the command at slash-command.
I send a request a few more times (2 to 4 times), and then the API finally returns, all at once:
Got it! Processing your acronym request...
NBA means "National Basketball Association".
What I Want to Happen
I go on Slack and shoot out the following:
/acronym NBA
I immediately get the following:
Got it! Processing your acronym request...
Then, outside of the 3-second window, I get the following:
NBA means "National Basketball Association".
I never hit a timeout error.
Conclusion
What am I doing wrong here? For some reason, that res.json() with the processing message isn't immediately being sent back. What can I do to fix this?
Thank you in advance!
Edit 1
I tried to replace the res.json() call with res.sendStatus(200).json(), but unfortunately, that only returned an 'OK' without actually processing the request.
I subsequently tried res.status(200).send({..stuff..}) but that resulted in the same problem I was having before.
I think res.json() sends a 200 automatically anyway, but its just not responding fast enough for some reason.
Solution
I eventually figured this one out. I was implementing the delayed responses right all along.
Since I'm using the free plan for Heroku, the dyno that's hosting my app would go down after 30 minutes of inactivity. When the app went down, the first few requests would time out on Slack before properly responding to a request.
The solution to this is either 1) upgrade to a new plan that keeps the dyno active at all times, or 2) ping the app with a simple get request every 15 or so minutes, like so:
const intervalMins = 15;
setInterval(() => {
http.get("<insert app url here>");
console.log('Ping!');
}, intervalMin * 60000)
I decided to go with the latter option. I don't run into the issue of the dyno sleeping anymore. I'd check this article for more details.
I'm attempting to create a one to one chat using Sails.js and sails.io.js on the client side.
I can get the io.socket.get and io.socket.post to work, but I haven't been able to receive anything from either sails.sockets.broadcast or Model.publish as instructed here:
Personalized chat using Sails.js
or here
Sails.js + socket.io: Sending messages from server to clients
Server Side Code:
UserController.js
module.exports = {
listen: function(req, res) {
console.log("about to join " + userId);
sails.sockets.join(req.socket, req.param('userId'));
}
};
From: http://beta.sailsjs.org/#/documentation/reference/sails.sockets/sails.sockets.join.html
MessageController.js
// Some code to get userId and message model here
console.log("about to broadcast on " + userId);
sails.sockets.broadcast(userId, 'conversation_message', message);
// Did not JSONify the message model, not sure if I need to?
From: http://beta.sailsjs.org/#/documentation/reference/sails.sockets/sails.sockets.broadcast.html
Client Side Code:
// Some code to get userId...
io.socket.get('/user/listen', {
userId: userId
}, function() {
io.socket.on('conversation_message', function(message) {
console.log("we have a message");
});
});
When I hit the route that triggers the broadcast with userId, nothing is sent to this client code to log we have a message. Any ideas?
I have tried the Model.subscribe / Model.publish methods as well with no luck--same issue.
Update: The listen function doesn't actually return -- as someone suggested in a comment that they later deleted. Adding res.send(200); to the end of the listen function was enough to make it work.
If that person would add their comment again as an answer I'll accept it.
(That was me. I thought my comment might of been not worthwhile because i reread your post and you said you had the get working so I thought I was wrong. My comment was something to the effect below, but I don't remember verbatim)
Have you checked to make sure the you are actually getting a response on the client-side listen? What log statements are you actually getting throughout your code? Put a console.log('get user listen') or something inside of the Client-side socket.get.