Develop Reference

How to programatically send SMS notifications to 1 million users using queues?

What we have?
An API build in Node.js (using Moleculer.js for micro-services and PostgreSQL for storing data) which has the functionality of keeping track of users and user groups. We have in average 3k users per group, and a user can be part of multiple groups.
What we want to achieve?
We want to create a special service which will send text messages. The admins will select multiple groups, the code will remove the duplicated users and send them an SMS.
After a selection we can have around 1 million users. How can we send them text messages in an efficient way?
What have we tried?
Paginate the users and for each page send a request to the SMS service.
const users = db.getPage(1); // [{ id: 1, phone: '+123456789' }, ...]
smsClient.sendBulk(users);
PROBLEM: The user list in the database can change in the process and can affect the pagination by giving us duplicates or skipping some users.
Load all the results in the memory and send all the users to the SMS service.
const users = db.getAll(); // [..., { id: 988123, phone: '+987654321' }]
smsClient.sendBulk(users);
PROBLEM: We think it's a bad idea, resource wise, to make this kind of queries to the database and keep them in the memory. In the same time, we don't want to send 1 million entities through an HTTP request to the SMS service.
How can we select a 1 million users and send them an SMS message without worry about duplicates, skipped data or any other alteration to the admin's selection? We were thinking about queues as a necessary step but after we find a solution for the cases mentioned above. Or, is the queue part of the solution?

How can we select a 1 million users and send them an SMS message without worry about duplicates, skipped data, or any other alteration to the admin's selection?
For managing duplicates You could use an additional DB to save a Hash Table for the users that been handled already. This is a bit more expensive because you will need to check the user before each SMS send.
Managing not skipping is a bit tricky because you will need to add more recipients to an ongoing SMS transaction. You will need the ability to detect (hook) when a user is added to a group and add it as recipients to the ongoing transactions accordingly.
You will need to find a fast DB and save that user in a HashSet for a fast set and get (O(1))
We were thinking about queues as a necessary step but after we find a solution for the cases mentioned above. Or, is the queue part of the solution?
Defenently. Queue is the correct way to go for this scenario (queueing many small tasks). Some queues come with a re-queue features that will re-queue any task that didn't get acknowledgment.

you need to check out RabbitMQ.message-driven microservices

Have you considered creating an indirect state between the user and sent SMS? Something like SmsRequest / SmsTask / however you'd call it.
It'd consist of necessary user-data, message content, status of the request (to-send, sending, sent, failed, ...) and some additional metadata depending on your needs.
Then the first step you'd do is to prepare these request and store them in db, effectively making a queue out of a table. You can add some constraints on user and message type that'd prevent any duplicates and then start second asynchronous process that simply fetches requests in to-send state, sets the state to sending and then saves the outcome.
This also gives you the benefit of audit + you can batch the outgoing messages and.
Of course it'd increase your data volume significantly but I guess it's cheap nowadays anyway.

How to make a Server perform Tasks behind a normal PHP / SQL Website?

How do i realise Tasks being calculated behind a Website, that is a normal php / Mysql Website, without being triggered by a Website user, like deleting a chat room automatical from the sql db, when the user that created it goes offline, in an simple Webchat with userchatrooms? Another example is some Text and Image based Browsergame, where every 5 Minutes (In Servertime) all Unit Movements and eventual fights between Users on a Image Based Game Map where calculated, when 2 Users or more meet on the same Map Tile? The calculated damage is sent back to the PHP / Ajax Website

To fire off tasks at an interval (ie: every five minutes) you would use a CRON job. To set that up is going to be dependent on what kind of server you're running. If you have a server GUI like CPANEL or PLESK there should be controls for setting CRON jobs. Essentially you input a time in this format
minute, hour, day, month, day of week
so 1 * * * * would run on the 1st minute of every hour, every day.
You point that to a command - probably a shell script, which will run whatever you need it to do.
To communicate that back to the "server" you would basically just update your database or whatever datastore you're using within your cron job.
So in your first example, to do something every five minutes, you would do
*/5 * * * * /path/to/script.sh
Then in that script, do whatever operations you need and save back to the DB.
If you are using a server admin GUI as previously mentioned, it's easiest to just hop in there and find the cron jobs tab and enter it.
If you're just managing your server with shell access you need to go put that in your crontab. The command for this might be dependent on the OS you're using but it's probably
crontab -e

How to correlate Stripe Token with items purchased; prevent spoofing

I am using Stripe Checkout for the front-end with the back-end creating the charge with the produced token.
How can I check in the back end that the token created for the purchase of a set of items is indeed for that set of items? Here is a more detailed scenario:
Item A costs $10
Item B costs $1,000
User chooses B and uses the front-end to generate a Token that was created with an amount of $1,000.
The front-end sends the token and the purchased item list (just B) to the back-end.
The back-end can calculate the given purchased item list to calculate a cost (but where can it verify with the token?) of $1,000 for verification that the user bought B and then a charge is made with the token.
My concern is if at step 2 the user spoofs the sent purchase list to be just A (I know that it is worth less than B but this should be rejected still). Then at step 3, how should the back-end figure out to reject making a Stripe Charge with the token?
Attempt:
Front-end passes Token, totalCost, listOfItems.
Back-end calculates the cost from the listOfItems and checks that it matches the totalCost before creating a Stripe Charge for an amount totalCost.
Is this as secure as it can get?

That's why you should use webhook
webhook documentation
To validate your payment has been made and the amount is correct, you can use webhook instead of user action to trigger. It's common practice nowadays.
After you receive the webhook, you may trigger change of status of the order after your checking (amount, client_info, other checking).

How can I track "online" statuses of users if a server crashes?

I have multiple heroku dynos and a chat app. When a user logs in, their status is set to "online" in MongoDB. However, if a server crashes, their status will still be set as online. How can I update the user status to be "offline" when a server crashes?
If I only had one dyno, this would be easy. I'd just update every user to be "offline" when the server starts. Unfortunately, this is not possible with multiple servers.

As per our chat and comments.
The best option is to go with checking against last activity. So seeing when the last message was sent and if it happened within the last let's say 5 minutes they are online if there were no activity mark them as offline.
Like I mentioned in the comments, if you are not storing a date_created on the messages documents you will not have to change anything because _id stores the timestamp
ObjectId("507f191e810c19729de860ea").getTimestamp()
that returns this Date object
ISODate("2012-10-17T20:46:22Z")
This answer is another option (if you are wanting to keep them as online even if they are not sending messages):
If you would like to know they are still active even when they're not jumping from page to page, include a bit of javascript to ping your server every 60 seconds or so to let you know they are still alive. It'll work the same way as my original suggestion, but it will update your records without requiring them to be frantically browsing your site at least once every five minutes.
var stillAlive = setInterval(function () {
/* XHR back to server
Example uses jQuery */
$.get("stillAlive.php");
}, 60000);

How to Prevent/Manage Stripe Webhook Sending Invoice for $0 on Trial Signup?

I'm working on an app that comes with a 14 day trial for free.
To handle payments, I'm using Stripe and listening for webhooks so I can perform functions on the backend when events happen.
One thing I've noticed, though, is that Stripe is sending me invoice data with an amount charged of $0 for the trial period. So, if a customer signs up, they get an invoice from Stripe for $0 (I have my webhook setup to fire off an email for each invoice I receive).
This isn't terrible, but from a UX perspective, I'd like to avoid the shock of getting an immediate invoice when someone is expecting a trial (even if that invoice is for $0).
I've considered just checking the data Stripe sends over and filtering out $0 invoices, but if I offer a discount or something, this doesn't seem like the best way.
Any thoughts/notes on how to implement this better?

A couple options here:
When you create the customer/subscription, the API returns both the customer and the subscription data to you in its response. You can use data from either or both of these to filter intelligently. Of particular interest:
current_period_start: This will also be the timestamp of the invoice.
trial_end: Until this timestamp, any invoice including a subscription is for a trial.
customer: If you don't like the others, you can always query the customer record when processing a $0 invoice. Customers in their trial period have a status of trialing.
If you're sending the email on invoice.created events, only the initial subscription invoice is created as closed. All other subscription invoices are open when Stripe creates them. (This is so you can make adjustments before the invoice is processed.) An invoice that's both $0 and closed has a high probability of being a trial—100%, in fact, if you're not otherwise creating already-closed invoices.

Try to listen for charges events. The event charge.succeeded Only fired when user has ben charge successfully.