I currently have an SPA built with MERN and I want to improve it further by adding a scheduled update to a particular collection in my MongoDB database by setting a boolean field in all of the documents in a collection to false every midnight.
Can someone point me to the right direction on how to accomplish this?
I want to be able to scale it as well at some point - for example, have a value saved in a document in another collection to indicate the time where these boolean fields will be invalidated in the front end?
I'm using a MERN stack. Thanks for your help!
you can use cron job
const moment = require('moment');
const CronJob = require('cron').CronJob;
const updateCollections = async ()=>{
await someQueriesServices()
}
new CronJob('0 0 * * *', async () => {
await updateCollections()
}, null, true, 'America/Los_Angeles');
or you can use setInterval
const timeInSec = moment().endOf('day').valueOf()
const Interval = Date.now() - timeInSec;
setInterval(async ()=>{
await updateCollections()
},Interval)
I usually use node-schedule
const schedule = require('node-schedule');
const j = schedule.scheduleJob('42 * * * *', function(){
console.log('The answer to life, the universe, and everything!');
});
Related
What I am trying to create is a script in node.js that will be added in a server of a web page that we are building. I want the script to do the following: In the first of every month to count the users inside my database and then create a set of tokens based on the amount of users logged in that current date. Specifically: 0,8 * numberOfUsers * 100. Then in the last day of every month count the users again and refund the tokens that were generated in the first of the month back to all the users (old and new). I also want to try that this runs automatically every time the server goes live and not have to execute it every time.
I am using the cron.schedule method and to check if it works I run for every minute the block that counts the first users(the one that is supposed to be in the first of the month) and every two minutes the block that gets again the new number of users and refunds the Tokens (the one that is supposed to be in the last of the month) . Here is my code:
const { Pool } = require('pg');
const cron = require('node-cron');
// Connection configuration
const config = {
user: '',
password: '',
host: '',
database: ''
};
// Create a new connection pool
const pool = new Pool(config);
async function countUsers() {
// Get number of users
const query = `SELECT COUNT(*) as count FROM users`;
const res = await pool.query(query);
const numUsers = +res.rows[0].count;
console.log(numUsers);
return numUsers;
}
async function refundTokens(tokensToRefund,numUsers) {
console.log("I am here");
console.log(numUsers);
if (numUsers !== 0 && !isNaN(tokensToRefund)) {
// Calculate amount of tokens to give to each user
const tokensPerUser = Math.round(tokensToRefund / numUsers);
// Update tokens for all users
const query = `UPDATE users SET tokens = tokens + $1`;
await pool.query(query, [tokensPerUser]);
}
}
cron.schedule('* * * * *', async () => {
const numUsers = await countUsers();
// Calculate number of tokens to refund (80% of tokens)
const tokensToRefund = 0.8 * numUsers * 100;
// Store number of tokens to refund
localStorage.setItem('tokensToRefund', tokensToRefund);
});
cron.schedule('*/2 * * * *', async () => {
// Get number of tokens to refund from local storage
const tokensToRefund = localStorage.getItem('tokensToRefund');
// Get number of users
const numUsers = await countUsers();
refundTokens(tokensToRefund, numUsers);
});
The first block seems to be running correctly every minute but the second one never does run. I see it because the console.log i have inside the refundTokens() function are not given. What seems to be the problem?
Your problem might be that there's no localStorage in NodeJS, as Marc said in comments. Try saving your tokens in some tmp file using Node's.
const fs = require('fs')
const numUsers = await countUsers();
// Calculate number of tokens to refund (80% of tokens)
const tokensToRefund = 0.8 * numUsers * 100;
// Store number of tokens to refund
await fs.writeFile('tmp', tokensToRefund);
// Get number of tokens to refund from local storage
const tokensFromFile = await fs.readFile('tmp');
const tokensToRefund = parseInt(tokensFromFile);
// Get number of users
const numUsers = await countUsers();
refundTokens(tokensToRefund, numUsers);
I would also write some separate helper function for this and add an Error handling, because both write and read functions could throw an error
P.S.: I would also recommend not to schedule your cronjobs in code for those small tasks, but to set them directly in console or in your backend panel. It would save you a huge amount of time and it will be much easier to configure them later. For your case, I consider it will be better. Just write your code as normal and add a cronjob to execute your js whenever you want
I have a website that allows users to send themselves a message at a date they choose, but I have no idea how to send it at that specific time. I know there exist CronJobs, but here, I'm not doing anything recurring. It's a one-time event trigger that I need.
I first tried using the native setTimeout like this:
const dueTimestamp = ...;
const timeLeft = dueTimestamp - Date().now();
const timeoutId = setTimeout(() => sendMessage(message), timeLeft);
It works perfectly for short periods, however, I'm not sure if it is reliable for long periods such as years or even decades. Moreover, it doesn't offer much control because if I'd like to modify the dueDate or the message's content, I'd have to stop the Timeout and start a new one.
Is there any package, a library, or a service that allows you to run a NodeJS function at a scheduled time? or do you have any solutions? I've heard of Google Cloud Schedule or Cronhooks, but I'm not sure.
You can use node-schedule library. for example :
you want to run a funcation at 5:30am on December 21, 2022.
const schedule = require('node-schedule');
const date = new Date(2022, 11, 21, 5, 30, 0);
const job = schedule.scheduleJob(date, function(){
console.log('The world is going to end today.');
});
As recommended by user3425506, I simply used a Cron job to fetch the messages from a database and to send the message of those whose timestamps have passed.
Dummy representation:
import { CronJob } from "cron";
import { fakeDB } from "./fakeDB";
const messages = fakeDB.messages;
const job = new CronJob("* * * * * *", () => {
const currentTimestamp = new Date().getTime();
messages.forEach((message, index) => {
if (message.timestamp > currentTimestamp) return;
console.log(message.message);
messages.splice(index, 1);
});
});
job.start();
I've created a cron-job to run a task and I want it to stop at some specific condition inside the scheduler, but it is not working.
How can I stop cron job inside the scheduler?
Here is my code:
// cron-job.js
const cron = require('node-cron');
const scheduler = cron.schedule('* * * * *', () => {
let result = fetchResultAfterSomeOperation();
if (result < 20) {
scheduler.stop(); //------------------------ It doesn't work
}
}, {
scheduled: false
});
scheduler.start();
How cron jobs created ?
Actually, Cron jobs are created from parent process. So parent process can have ability or feature to kill or suspend the child processes. So here node-cron may works in this way.
Now come to your issue, you have submitting cron task using cron.schedule(time,callback). Now callback is going to run on separate child process. So even if you're using scheduler object to stop the cron task, it wont work. The scheduler can stop the child process from main process (i.e cron.js file).
So I advise you to refactor your code.
As suggested by #Faizul Ahemed, you may change the code to something like this:
const cron = require('node-cron');
let result = null
const seachFunction = () => {
if (!result) fetchResults();
}
const scheduler = cron.schedule('*/10 * * * * *', searchFunction, { scheduled: false });
scheduler.start();
setTimeout(() => { scheduler.stop() }, 5 * 60 * 1000)
The above code will cause the fetchResults function to fetch/get data every x time for a duration of y set by the setTimeout.
I've already done this on the Firebase Realtime Database and it's worked perfectly before, but I decided to switch to Firestore and am trying to have the same functionality with the cloud functions.
My goal: Have a cloud function that deletes messages after 24 hours from Firestore
I've tried to use the same function that I had and convert it over to Firestore, but the syntax isn't correct and I'm not very familiar with Javascript.
This is the code for the cloud function which worked form my RTDB:
'use strict';
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
// Cut off time. Child nodes older than this will be deleted.
const CUT_OFF_TIME = 24 * 60 * 60 * 1000; // 24 Hours in seconds.
exports.deleteOldMessages = functions.database.ref('/Message/{chatRoomID}/{messageId}').onWrite(function(change) {
var ref = change.after.ref.parent;
var now = Date.now()
var cutoff = (now - CUT_OFF_TIME) / 1000;
var oldItemsQuery = ref.orderByChild('seconds').endAt(cutoff);
return oldItemsQuery.once('value').then(function(snapshot) {
var updates = {};
snapshot.forEach(function(child) {
updates[child.key] = null;
console.log(cutoff)
});
return ref.update(updates);
});
});
Path: My path in the Firestore is different from the RTDB. Collection('Message').Document('userId' (the actual userId of that user)).Collection('chatRoomId' (the actual chatRoomId of that chat)).Document('messageId') and then one of the fields is Seconds: 1587334120
I would greatly appreciate it if someone could help me try to convert/have the same functionality as my previous Cloud Function, but works for Firestore and incorporates the new path. All help is greatly appreciated
Edit: This is what I was able to try and figure out on my own. I understand that I have to query the collection based off of the seconds, and from there see if it's past 24 hours, then delete the document, but I'm unsure on the syntax on how to do it. In the compiler I get the error "Unexpected token query aslant [28, 25]"
exports.deleteOldFirestoreMessages = functions.firestore.document('Message/{userId}/{chatRoomId}/{messageId}').onWrite((change, context) => {
var now = Date().now
var cutoff = (now - CUT_OFF_TIME) / 1000;
const query = admin.firestore().collection('Message/{userId}/{chatRoomId}').where('seconds', '>=', cutoff)
const snapshots = await query.get();
const batch = firestore.batch();
snapshots.forEach(v => batch.delete(v.ref));
await batch.commit();
});
These are the resources I've tried looking through:
Firebase Cloud Functions - Delete Entries where timestamp field is over a month old and run daily
Firebase - Cloud Functions - make queries on collection
Extend Cloud Firestore with Cloud Functions
Delete firebase data older than 2 hours
I would like to delete data that is older than two hours. Currently, on the client-side, I loop through all the data and run a delete on the outdated data. When I do this, the db.on('value') function is invoked every time something is deleted. Also, things will only be deleted when a client connects, and what might happen if two clients connect at once?
Where can I set up something that deletes old data? I have a timestamp inside each object created by a JavaScript Date.now().
Firebase does not support queries with a dynamic parameter, such as "two hours ago". It can however execute a query for a specific value, such as "after August 14 2015, 7:27:32 AM".
That means that you can run a snippet of code periodically to clean up items that are older than 2 hours at that time:
var ref = firebase.database().ref('/path/to/items/');
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var old = ref.orderByChild('timestamp').endAt(cutoff).limitToLast(1);
var listener = old.on('child_added', function(snapshot) {
snapshot.ref.remove();
});
As you'll note I use child_added instead of value, and I limitToLast(1). As I delete each child, Firebase will fire a child_added for the new "last" item until there are no more items after the cutoff point.
Update: if you want to run this code in Cloud Functions for Firebase:
exports.deleteOldItems = functions.database.ref('/path/to/items/{pushId}')
.onWrite((change, context) => {
var ref = change.after.ref.parent; // reference to the items
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var oldItemsQuery = ref.orderByChild('timestamp').endAt(cutoff);
return oldItemsQuery.once('value', function(snapshot) {
// create a map with all children that need to be removed
var updates = {};
snapshot.forEach(function(child) {
updates[child.key] = null
});
// execute all updates in one go and return the result to end the function
return ref.update(updates);
});
});
This function triggers whenever data is written under /path/to/items, so child nodes will only be deleted when data is being modified.
This code is now also available in the functions-samples repo.
I have a http triggered cloud function that deletes nodes, depending on when they were created and their expiration date.
When I add a node to the database, it needs two fields: timestamp to know when it was created, and duration to know when the offer must expire.
Then, I have this http triggered cloud function:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
/**
* #function HTTP trigger that, when triggered by a request, checks every message of the database to delete the expired ones.
* #type {HttpsFunction}
*/
exports.removeOldMessages = functions.https.onRequest((req, res) => {
const timeNow = Date.now();
const messagesRef = admin.database().ref('/messages');
messagesRef.once('value', (snapshot) => {
snapshot.forEach((child) => {
if ((Number(child.val()['timestamp']) + Number(child.val()['duration'])) <= timeNow) {
child.ref.set(null);
}
});
});
return res.status(200).end();
});
You can create a cron job that every X minutes makes a request to the URL of that function: https://cron-job.org/en/
But I prefer to run my own script, that makes a request every 10 seconds:
watch -n10 curl -X GET https://(your-zone)-(your-project-id).cloudfunctions.net/removeOldMessages
In the latest version of Firebase API, ref() is changed to ref
var ref = new Firebase('https://yours.firebaseio.com/path/to/items/');
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var old = ref.orderByChild('timestamp').endAt(cutoff).limitToLast(1);
var listener = old.on('child_added', function(snapshot) {
snapshot.ref.remove();
});
If someone will have the same problem, but in Firestore. I did a little script that at first read documents to console.log and then delete documents from a collection messages older than 24h. Using https://cron-job.org/en/ to refresh website every 24h and that's it. Code is below.
var yesterday = firebase.firestore.Timestamp.now();
yesterday.seconds = yesterday.seconds - (24 * 60 * 60);
console.log("Test");
db.collection("messages").where("date",">",yesterday)
.get().then(function(querySnapshote) {
querySnapshote.forEach(function(doc) {
console.log(doc.id," => ",doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
db.collection("messages").where("date","<",yesterday)
.get().then(function(querySnapshote) {
querySnapshote.forEach(element => {
element.ref.delete();
});
})
You could look into Scheduling Firebase Functions with Cron Jobs. That link shows you how to schedule a Firebase Cloud Function to run at a fixed rate. In the scheduled Firebase Function you could use the other answers in this thread to query for old data and remove it.