Develop Reference

How to schedule a function to run at a certain date?

I have a website that allows users to send themselves a message at a date they choose, but I have no idea how to send it at that specific time. I know there exist CronJobs, but here, I'm not doing anything recurring. It's a one-time event trigger that I need.
I first tried using the native setTimeout like this:
const dueTimestamp = ...;
const timeLeft = dueTimestamp - Date().now();
const timeoutId = setTimeout(() => sendMessage(message), timeLeft);
It works perfectly for short periods, however, I'm not sure if it is reliable for long periods such as years or even decades. Moreover, it doesn't offer much control because if I'd like to modify the dueDate or the message's content, I'd have to stop the Timeout and start a new one.
Is there any package, a library, or a service that allows you to run a NodeJS function at a scheduled time? or do you have any solutions? I've heard of Google Cloud Schedule or Cronhooks, but I'm not sure.

You can use node-schedule library. for example :
you want to run a funcation at 5:30am on December 21, 2022.
const schedule = require('node-schedule');
const date = new Date(2022, 11, 21, 5, 30, 0);
const job = schedule.scheduleJob(date, function(){
console.log('The world is going to end today.');
});

As recommended by user3425506, I simply used a Cron job to fetch the messages from a database and to send the message of those whose timestamps have passed.
Dummy representation:
import { CronJob } from "cron";
import { fakeDB } from "./fakeDB";
const messages = fakeDB.messages;
const job = new CronJob("* * * * * *", () => {
const currentTimestamp = new Date().getTime();
messages.forEach((message, index) => {
if (message.timestamp > currentTimestamp) return;
console.log(message.message);
messages.splice(index, 1);
});
});
job.start();

Update content from XMLHTTPRequest every 20 seconds

First of all, I would like to say that I'm a student learning programming for around a month, so expect to see many mistakes.
I'm working on a website where I use a chart from the ChartJs library. The data used for this chart is taken through requests to a server.
What I want to do is update the content of the studenGesamt variable every 20 seconds. My main idea was using a setInterval, but it didn't work. I am thinking that I could make a new request to the server every 20 seconds, but I am kind of lost on how to do that or if it is actually a good idea. If someone could help me I would really appreciate it!
let serverData;
let studenGesamt;
let date;
const http = new XMLHttpRequest();
const url = 'https://url.com/'; // I have hidden this URL as it is the actual server from my company
http.open("GET", url);
http.setRequestHeader('key', 'sample-key'); // I have hidden this key for the same reason as above
http.send();
const chart = document.getElementById("multie-pie-chart");
// Function that calculates the workdays passed up until today
const workdaysCount = () => [...new Array(new Date().getDate())]
.reduce((acc, _, monthDay) => {
const date = new Date()
date.setDate(1 + monthDay) ![0, 6].includes(date.getDay()) && acc++
return acc
}, 0)
http.onload = (e) => {
// Parsing the JSON file and storing it into a variable (Console.Log() to make sure it works)
serverData = JSON.parse(http.responseText);
console.log(serverData);
// Storing the value of total hours from the database in a variable
studenGesamt = serverData.abzurechnen.gesamt;
chartRender(); // Function that calls and renders the chart
setInterval(dataLoop, 5000); // 5 seconds to be able to test it, will be changed to 20 when finished
};
let dataLoop = () => {
studenGesamt = serverData.abzurechnen.gesamt;
console.log('test'); // Logging test to see if it is working
};

How to automate downloading a file from a site?

I want to download several data files from this URL: https://pselookup.vrymel.com/
The site contains a date field and a download button. I want to download data for multiple years (which would mean a lot of requests) and I want to make it automatically.
I've created a Javascript snippet, however, it keeps downloading just the same file over and over again.
$dateField = document.getElementsByClassName('csv_download_input__Input-encwx-1 dDiqPH')[2]
$dlButton = document.getElementsByClassName('csv_download_input__Button-encwx-0 KLfyv')[2]
var now = new Date();
var daysOfYear = [];
for (var d = new Date(2016, 0, 1); d <= now; d.setDate(d.getDate() + 1)) {
daysOfYear.push(new Date(d).toISOString().substring(0,10));
}
(function theLoop (i) {
setTimeout(function () {
$dlButton.click()
$dateField.value = daysOfYear[i]
if (--i) { // If i > 0, keep going
theLoop(i); // Call the loop again, and pass it the current value of i
}
}, 3000);
})(daysOfYear.length-1);
How could I download all of the files automatically?

First off, javascript in the client is probably not the best language to do this nor the best approach to make this happen. It might work, but it's better to know what is best when choosing an approach to a problem. Also, it will avoid for you clicking ~800 times in the popup accepting the download.
You can get the files in a programatically way by just learning what you browser is doing to get the file and trying to reproduce it in bunch.
After inspecting the calls you can see that it's calling an endpoint and that endpoint is returning a link which contains the file that you can download.
Well, that is going to be easy, so now you just need to make the script in any language to be able to retrieve them.
I've chosen javascript but not client side, but nodejs which means that this has to run from your computer.
You could do the same with bash, python or any other language.
To run this do the following:
Go to a new empty directory
Run npm install axios
Create a file with the code I pasted let's call it crawler.js
Run node crawler.js
This has been tested using node v8.15.0
// NOTE: Require this to make a request and save the link as file 20190813:Alevale
const axios = require('axios');
const fs = require('fs');
let now = new Date();
let daysOfYear = [];
const baseUrl = 'https://a4dzytphl9.execute-api.ap-southeast-1.amazonaws.com/prod/eod/'
for (var d = new Date(2016, 0, 1); d <= now; d.setDate(d.getDate() + 1)) {
daysOfYear.push(new Date(d).toISOString().substring(0,10));
}
const waitFor = (time) => {
return new Promise((resolve => setTimeout(resolve, time)))
}
const getUrls = async () =>{
let day
for (day of daysOfYear) {
console.log('getting day', baseUrl + day)
// NOTE: Throttle the calls to not overload the server 20190813:Alevale
await waitFor(4000)
await axios.get(baseUrl + day)
.then(response => {
console.log(response.data);
console.log(response);
if (response.data && response.data.download_url) {
return response.data.download_url
}
return Promise.reject('Could not retrieve response.data.download_url')
})
.then((url) =>{
axios({
method: 'get',
url,
responseType: 'stream'
})
.then(function (response) {
// NOTE: Save the file as 2019-08-13 20190813:Alevale
response.data.pipe(fs.createWriteStream(`${day}.csv`))
})
.catch(console.error)
})
.catch(error => {
console.log(error);
});
}
}
getUrls()

You can instead of simulating the user, get the link to download from:
https://a4dzytphl9.execute-api.ap-southeast-1.amazonaws.com/prod/eod/2019-08-07
just change the date at the end to the date of the file you want to download. And use axios to get this URL.
This will save you sometime (in case you don't really need to simulate the click of the user etc)
Then you will get a response like this:
{
download_url":"https://d3u9ukmkxau9he.cloudfront.net/eod/2019-08-07.csv?Expires=1566226156&Signature=QRUk3tstuNX5KYVPKJSWrXsSXatkWS-eFBIGUufaTEMJ~rgpVi0iPCe1AXl5pbQVdBQxOctpixCbyNz6b9ycDgYNxEdZqPr2o2pDe8cRL655d3zXdICnEGt~dU6p35iMAJkMpPSH~jbewhRSCPUwWXQBfOiEzlHwxru9lPnDfsdSnk3iI3GyR8Oc0ZP50EdUMHF7MjWSBRbCIwnu6wW4Jh0bPmZkQDQ63ms5QxehsmtuGLOgcrC6Ky1OffVQj~ihhmBt4LGhZTajjK4WO18hCP3urKt03qpC4bOvYvJ3pxvRkae0PH1f-vbTWMDkaWHHVCrzqZhkAh3FlvMTWj8D4g__&Key-Pair-Id=APKAIAXOVAEOGN2AYWNQ"
}
and then you can use axios to GET this url and download your file.

Best way to call periodically (twice a day) an API from Node.js to get data

I've got my Node.js application which use the Fixer.io API to get some data. Currently the call is made when I reach the URL myapp/rate. My goal is to make this call automatically twice a day to store the data in my MongoDB database.
So I would like to know what is the best way to do that ? Maybe the setInterval() is the only way to do it but I don't think so...
My call looks like this :
router.get('/', (req, res, next) => {
fixer.latest({ base: 'EUR', symbols: ['CAD'] })
.then((data) => {
var currentData = data.rates;
if (currentData) {
const currentRate = new Rate({ rate: currentData.CAD });
currentRate.save((err) => {
if (err) {
throw err;
} else {
console.log('Data saved successfully!');
}
});
}
})
.then(() => {
Rate.find({}, { rate: 1, _id: 0 }, (err, rates) => {
if (err) {
throw err;
} else {
res.json(rates);
}
});
})
.catch(() => {
console.log('Error');
});
});
Thank's

You can use the node module node-schedule to run a task within your node app on a cron-like schedule.
var schedule = require('node-schedule');
var j = schedule.scheduleJob('0 0 0,12 * *', function(){
console.log('This will run twice per day, midnight, and midday');
});

The key word in your question is 'automatically'. The easiest (and most reliable) way to run a script at a particular time of day on the server would be to use cronjobs on the server. That way your server will execute the script regardless of the user interaction:
0 0 * * * TheCommand // 12 a.m.
0 12 * * * TheCommand // 12 p.m.
However, it is also possible to run a script at a particular time of day using a JavaScript setTimeout() as you were thinking. You need to calculate the current time, the target time, and the difference between them:
var now = new Date();
var payload = new Date(now.getFullYear(), now.getMonth(), now.getDate(), 12, 0, 0, 0) - now;
if (payload < 0) {
payload += 46400000; // Try again 12 hours later
}
setTimeout(function(){ // YOUR DESIRED FUNCTION }, payload);
Note the 12 above in payload is the hour at which you want the script to run. In this example, I'm running the script at both 12 p.m. and 12 a.m.
Keep in mind that this will only work if the application is permanently running the page where the JavaScript is loaded; there's no way of forcing JavaScript to run 'in the background' as it is a client-side language.
Hope this helps! :)

Set and Clear Timeout on Node Server from Client

I'm attempting to allow a user to set an alarm from the client and pass it to the server. The server then has a setTimeout that counts down and when time runs out, executes the function.
This first part is working fine, however, I need the the ability to clear that same timeout, should the client decide to cancel that particular alarm.
Note: I've been storing various data using Redis, so that is available.
var client = redis.createClient();
io.set("store", new sio.RedisStore);
io.sockets.on('connection', function (socket) {
socket.on('alarm:config', function(list, date, time, bool) {
if (bool) {
var now = new Date().getTime();
var year = date[0],
month = date[1] - 1,
day = date[2];
var hour = time[0],
minutes = time[1];
var alarm = new Date(year, month, day, hour, minutes);
var countdown = alarm - now;
var myAlarm = setTimeout(function() {
// do stuff...
}, ( countdown ) );
} else {
clearTimeout(myAlarm);
}
});
});
The approach I have in mind is that I would use the boolean value to determine if the user is setting or canceling that particular alarm. I realize that setting a local variable "myAlarm" will not work, I just put it there to convey the idea.
I am trying to figure out a way to store a reference to that exact timeout so that the next time the "alarm:config" socket event is triggered with a false boolean value, it can cancel the timeout that was set earlier.
It might be another question all together, but how does an application like Google Calendar store a date and time and then know exactly when to trigger it as well as offer the ability to cancel it? This would essentially be the same idea.
UPDATE: I have it working using the following solution. I am open to a more elegant solution.
socket.on('alarm:config', function(list, date, time, bool) {
var alarmName = "timer:" + list;
if (bool) {
client.hset(alarmName, "status", true);
var now = new Date().getTime();
var year = date[0],
month = date[1] - 1,
day = date[2];
var hour = time[0],
minutes = time[1];
var alarm = new Date(year, month, day, hour, minutes);
var countdown = alarm - now;
setTimeout(function() {
client.hget(alarmName, "status", function(err, bool) {
if(bool == 'true') {
// do stuff...
} else {
console.log("This alarm has been canceled.");
}
});
}, ( countdown ) );
} else {
console.log('canceling alarm');
client.hset(alarmName, "status", false);
}
});

Depending on how large of an application you're building, there are a couple of options.
Processing Queue
You could restructure your application to use a job queue instead of simply setting timers. This has an advantage that you can split it in the future into multiple processes, but it does complicate the handling a bit.
A library like Kue uses just Redis and allows you to do a delayed put to set events in the future.
Going from the Kue readme:
var kue = require('kue')
, jobs = kue.createQueue();
// Create delayed job (need to store email.id in redis as well)
var email = jobs.create('email', {
title: 'Account renewal required',
to: 'tj#learnboost.com',
template: 'renewal-email'
}).delay(minute).save();
// Process job
jobs.process('email', function(job, done){
email(job.data.to, done);
});
// Cancel job
email.remove(function(err){
if (err) throw err;
console.log('removed completed job #%d', job.id);
});
Storing Reference to Timeout
This is much less robust, but could allow you to do it very simply. It leaves global variables floating around, and has no recovery if the process crashes.
Basically, you store the timer in a global variable and cancel it based on the job id.
var myAlarms = {}
socket.on('alarm:config', function(list, date, time, bool) {
var alarmName = "timer:" + list;
if (bool) {
myAlarms[alarmName] = setTimeout(function() {
// do stuff...
}, countdown);
} else {
clearTimeout(myAlarms[alarmName]);
}
}
Note: None of this code has been tested and it likely contains errors.