Develop Reference

Node.js script running automatically on the first and last of the month

What I am trying to create is a script in node.js that will be added in a server of a web page that we are building. I want the script to do the following: In the first of every month to count the users inside my database and then create a set of tokens based on the amount of users logged in that current date. Specifically: 0,8 * numberOfUsers * 100. Then in the last day of every month count the users again and refund the tokens that were generated in the first of the month back to all the users (old and new). I also want to try that this runs automatically every time the server goes live and not have to execute it every time.
I am using the cron.schedule method and to check if it works I run for every minute the block that counts the first users(the one that is supposed to be in the first of the month) and every two minutes the block that gets again the new number of users and refunds the Tokens (the one that is supposed to be in the last of the month) . Here is my code:
const { Pool } = require('pg');
const cron = require('node-cron');
// Connection configuration
const config = {
user: '',
password: '',
host: '',
database: ''
};
// Create a new connection pool
const pool = new Pool(config);
async function countUsers() {
// Get number of users
const query = `SELECT COUNT(*) as count FROM users`;
const res = await pool.query(query);
const numUsers = +res.rows[0].count;
console.log(numUsers);
return numUsers;
}
async function refundTokens(tokensToRefund,numUsers) {
console.log("I am here");
console.log(numUsers);
if (numUsers !== 0 && !isNaN(tokensToRefund)) {
// Calculate amount of tokens to give to each user
const tokensPerUser = Math.round(tokensToRefund / numUsers);
// Update tokens for all users
const query = `UPDATE users SET tokens = tokens + $1`;
await pool.query(query, [tokensPerUser]);
}
}
cron.schedule('* * * * *', async () => {
const numUsers = await countUsers();
// Calculate number of tokens to refund (80% of tokens)
const tokensToRefund = 0.8 * numUsers * 100;
// Store number of tokens to refund
localStorage.setItem('tokensToRefund', tokensToRefund);
});
cron.schedule('*/2 * * * *', async () => {
// Get number of tokens to refund from local storage
const tokensToRefund = localStorage.getItem('tokensToRefund');
// Get number of users
const numUsers = await countUsers();
refundTokens(tokensToRefund, numUsers);
});
The first block seems to be running correctly every minute but the second one never does run. I see it because the console.log i have inside the refundTokens() function are not given. What seems to be the problem?

Your problem might be that there's no localStorage in NodeJS, as Marc said in comments. Try saving your tokens in some tmp file using Node's.
const fs = require('fs')
const numUsers = await countUsers();
// Calculate number of tokens to refund (80% of tokens)
const tokensToRefund = 0.8 * numUsers * 100;
// Store number of tokens to refund
await fs.writeFile('tmp', tokensToRefund);
// Get number of tokens to refund from local storage
const tokensFromFile = await fs.readFile('tmp');
const tokensToRefund = parseInt(tokensFromFile);
// Get number of users
const numUsers = await countUsers();
refundTokens(tokensToRefund, numUsers);
I would also write some separate helper function for this and add an Error handling, because both write and read functions could throw an error
P.S.: I would also recommend not to schedule your cronjobs in code for those small tasks, but to set them directly in console or in your backend panel. It would save you a huge amount of time and it will be much easier to configure them later. For your case, I consider it will be better. Just write your code as normal and add a cronjob to execute your js whenever you want

How to update a MongoDB collection automatically every midnight?

I currently have an SPA built with MERN and I want to improve it further by adding a scheduled update to a particular collection in my MongoDB database by setting a boolean field in all of the documents in a collection to false every midnight.
Can someone point me to the right direction on how to accomplish this?
I want to be able to scale it as well at some point - for example, have a value saved in a document in another collection to indicate the time where these boolean fields will be invalidated in the front end?
I'm using a MERN stack. Thanks for your help!

you can use cron job
const moment = require('moment');
const CronJob = require('cron').CronJob;
const updateCollections = async ()=>{
await someQueriesServices()
}
new CronJob('0 0 * * *', async () => {
await updateCollections()
}, null, true, 'America/Los_Angeles');
or you can use setInterval
const timeInSec = moment().endOf('day').valueOf()
const Interval = Date.now() - timeInSec;
setInterval(async ()=>{
await updateCollections()
},Interval)

I usually use node-schedule
const schedule = require('node-schedule');
const j = schedule.scheduleJob('42 * * * *', function(){
console.log('The answer to life, the universe, and everything!');
});

Function returned undefined, expected Promise or value and unable to delete old data from firebase database using cloud functions

I'm trying to delete multiple nodes on my database that are older than 12hrs. I"m using a pub/sub function to trigger this event. I don't know if my code is actually looping through all nodes as I'm not using the onWrite, onCreate database triggers on specific. Here is the image sample of the database
this is the pub/sub code
exports.deletejob = functions.pubsub.topic('Oldtask').onPublish(() => {
deleteOldItem();
})
and the deleteOldItem function
function deleteOldItem(){
const CUT_OFF_TIME = 12 * 60 * 1000; // 12 Hours in milliseconds.
//var ref = admin.database().ref(`/articles/${id}`);
const ref = admin.database().ref(`/articles`);
const updates = {};
ref.orderByChild('id').limitToLast(100).on('value', function (response) {
var index = 0;
response.forEach(function (child) {
var element = child.val();
const datetime = element.timestamp;
const now = Date.now();
const cutoff = now - datetime;
if (CUT_OFF_TIME < cutoff){
updates[element.key] = null;
}
});
//This is supposed to be the returened promise
return ref.child(response.key).update(updates);
});
If there's something I'm doing wrong, I'll like to know. The pub/sub is triggered with a JobScheduler already setup on google cloud scheduler

You had several problems in your code that were giving you trouble.
The handling of promises wasn't correct. In particular, your top level function never actually returned a promise, it just called deleteOldItems().
You should use the promise form of once() instead of calling on() with a callback since you don't want to install a listener in this case, you just need the result a single time, and you want to handle it as part of a promise chain.
To delete nodes, you should call remove() on a reference to that node. It also generates a promise for you to use here.
You didn't calculate 12 hours in milliseconds properly, you calculated 12 minutes in milliseconds :)
Here's what I came up with. It uses an http function instead of a pubsub function as well as adding a log statement for my testing, but the modification you need should be trivial/obvious (just change the prototype and remove the response after deleteOldItems, but do make sure you keep returning the result of deleteOldItems()):
const functions = require('firebase-functions');
const admin = require('firebase-admin');
function deleteOldItems() {
const CUT_OFF_TIME = 12 * 60 * 60 * 1000; // 12 Hours in milliseconds.
const ref = admin.database().ref('/articles');
return ref.orderByChild('id').limitToLast(100).once('value')
.then((response) => {
const updatePromises = [];
const now = Date.now();
response.forEach((child) => {
const datetime = child.val().timestamp;
const cutoff = now - datetime;
console.log(`processing ${datetime} my cutoff is ${CUT_OFF_TIME} and ${cutoff}`);
if (CUT_OFF_TIME < cutoff){
updatePromises.push(child.ref.remove())
}
});
return Promise.all(updatePromises);
});
}
exports.doIt = functions.https.onRequest((request, response) => {
return deleteOldItems().then(() => { return response.send('ok') });
}
While I have not tested it, I'm pretty sure this will work to include inside your original function call for cloud scheduler:
exports.deletejob = functions.pubsub.topic('Oldtask').onPublish(() => {
return deleteOldItems();
})
Of course, this is still more complicated than you need, since ordering by id doesn't really gain you anything here. Instead, why not just use the query to return the earliest items before the cut off time (e.g. exactly the ones you want to remove)? I've also switched to limitToFirst to ensure the earliest entries get thrown out, which seems more natural and ensures fairness:
function deleteOldItems() {
const cutOffTime = Date.now() - (12 * 60 * 60 * 1000); // 12 Hours earlier in milliseconds.
const ref = admin.database().ref('/articles');
return ref.orderByChild('timestamp').endAt(cutOffTime).limitToFirst(100).once('value')
.then((response) => {
const updatePromises = [];
response.forEach((child) => {
updatePromises.push(child.ref.remove())
});
return Promise.all(updatePromises);
});
}
If you do this on more than a few items, of course, you probably want to add an index on the timestamp field so the range query is more efficient.

Increment Realtime Database count on new user with Firebase Cloud Function

* UPDATED: THIS WORKS. SEE ANSWER BELOW *
I'm trying to write a Firebase Cloud Function that increments a Realtime Database /userCount value whenever a new user is created.
I've tried the following, but am getting "TypeError: userCountRef.transaction is not a function" in incrementCountOnNewUser.
Transactions are working for my other function incrementCountOnOpen when the value of garage is set to true, but the ref is derived from the after event object.
Any suggestions on how to do this?
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
// const userCountRef = functions.database.ref("/userCount"); // does NOT work
const userCountRef = admin.database().ref('/userCount'); // THIS WORKS!
exports.incrementCountOnNewUser = functions.auth.user().onCreate((user) => {
return userCountRef.transaction(count => count + 1);
});
exports.incrementCountOnOpen = functions.database.ref("/garage").onUpdate(({after}) => {
const countRef = after.ref.parent.child('count');
const newValue = after.val();
return newValue
? countRef.transaction(count => count + 1)
: null;
});

It turns out that the code above works! I had switched from the commented out code (which does NOT work). I guess it didn't wait long enough for it propagate after I published, because I see it working now!
Sorry for the confusion.

Delete firebase data older than 2 hours

I would like to delete data that is older than two hours. Currently, on the client-side, I loop through all the data and run a delete on the outdated data. When I do this, the db.on('value') function is invoked every time something is deleted. Also, things will only be deleted when a client connects, and what might happen if two clients connect at once?
Where can I set up something that deletes old data? I have a timestamp inside each object created by a JavaScript Date.now().

Firebase does not support queries with a dynamic parameter, such as "two hours ago". It can however execute a query for a specific value, such as "after August 14 2015, 7:27:32 AM".
That means that you can run a snippet of code periodically to clean up items that are older than 2 hours at that time:
var ref = firebase.database().ref('/path/to/items/');
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var old = ref.orderByChild('timestamp').endAt(cutoff).limitToLast(1);
var listener = old.on('child_added', function(snapshot) {
snapshot.ref.remove();
});
As you'll note I use child_added instead of value, and I limitToLast(1). As I delete each child, Firebase will fire a child_added for the new "last" item until there are no more items after the cutoff point.
Update: if you want to run this code in Cloud Functions for Firebase:
exports.deleteOldItems = functions.database.ref('/path/to/items/{pushId}')
.onWrite((change, context) => {
var ref = change.after.ref.parent; // reference to the items
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var oldItemsQuery = ref.orderByChild('timestamp').endAt(cutoff);
return oldItemsQuery.once('value', function(snapshot) {
// create a map with all children that need to be removed
var updates = {};
snapshot.forEach(function(child) {
updates[child.key] = null
});
// execute all updates in one go and return the result to end the function
return ref.update(updates);
});
});
This function triggers whenever data is written under /path/to/items, so child nodes will only be deleted when data is being modified.
This code is now also available in the functions-samples repo.

I have a http triggered cloud function that deletes nodes, depending on when they were created and their expiration date.
When I add a node to the database, it needs two fields: timestamp to know when it was created, and duration to know when the offer must expire.
Then, I have this http triggered cloud function:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
/**
* #function HTTP trigger that, when triggered by a request, checks every message of the database to delete the expired ones.
* #type {HttpsFunction}
*/
exports.removeOldMessages = functions.https.onRequest((req, res) => {
const timeNow = Date.now();
const messagesRef = admin.database().ref('/messages');
messagesRef.once('value', (snapshot) => {
snapshot.forEach((child) => {
if ((Number(child.val()['timestamp']) + Number(child.val()['duration'])) <= timeNow) {
child.ref.set(null);
}
});
});
return res.status(200).end();
});
You can create a cron job that every X minutes makes a request to the URL of that function: https://cron-job.org/en/
But I prefer to run my own script, that makes a request every 10 seconds:
watch -n10 curl -X GET https://(your-zone)-(your-project-id).cloudfunctions.net/removeOldMessages

In the latest version of Firebase API, ref() is changed to ref
var ref = new Firebase('https://yours.firebaseio.com/path/to/items/');
var now = Date.now();
var cutoff = now - 2 * 60 * 60 * 1000;
var old = ref.orderByChild('timestamp').endAt(cutoff).limitToLast(1);
var listener = old.on('child_added', function(snapshot) {
snapshot.ref.remove();
});

If someone will have the same problem, but in Firestore. I did a little script that at first read documents to console.log and then delete documents from a collection messages older than 24h. Using https://cron-job.org/en/ to refresh website every 24h and that's it. Code is below.
var yesterday = firebase.firestore.Timestamp.now();
yesterday.seconds = yesterday.seconds - (24 * 60 * 60);
console.log("Test");
db.collection("messages").where("date",">",yesterday)
.get().then(function(querySnapshote) {
querySnapshote.forEach(function(doc) {
console.log(doc.id," => ",doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
db.collection("messages").where("date","<",yesterday)
.get().then(function(querySnapshote) {
querySnapshote.forEach(element => {
element.ref.delete();
});
})

You could look into Scheduling Firebase Functions with Cron Jobs. That link shows you how to schedule a Firebase Cloud Function to run at a fixed rate. In the scheduled Firebase Function you could use the other answers in this thread to query for old data and remove it.