I am building a webapp where user have a ranking based on their twitter activity and their activity on my website.
Therefore I'd like to update their rank every five minutes, pulling their latest activity from twitter and update it in my database. I was thinking of using something like this:
var minutes = 5, the_interval = minutes * 60 * 1000;
setInterval(function() {
// my update here
}, the_interval);
However, I have several questions about this code:
where should I save it to make sure it is run?
will it slow my program or is it a problem to pull data out of twitter every five minute? Should I use their streaming API instead?
Note: I am using mongoDB
I'd suggest that you create a scheduled task/chron job/etc. (depends on your host OS) to call a separate Node.JS application that performs the specific tasks you want to do periodically and then it would exit when complete. (Or you could use a ChildProcess potentially as well).
While Node.JS is async, there's no need, given the description you provided, to perform this work within the same application process that is serving a web application. In fact, as it sounds like "busy work", it would be best handled by a distinct process to avoid impacting directly any of your interactive web users.
The placement shouldn't really matter as Node is asynchronous.
Related
I currently have a Node.JS server set up that is able to read and write data from a FireBase database when a request is made from a user.
I would like to implement time based events that result in an action being performed at a certain date or time. The key thing here though, is that I want to have the freedom to do this in seconds (for example, write a message to console after 30 seconds have passed, or on Friday the 13th at 11:30am).
A way to do this would be to store the date/time an action needs be performed in the database, and read from the database every second and compare the current date/time with events stored so we know if an action needs to be performed at this moment. As you can imagine though, this would be a lot of unnecessary calls to the database and really feels like a poor way to implement this system.
Is there a way I can stay synced with the database without having to call every second? Perhaps I could then store a version of the events table locally and update this when a change is made to the database? Would that be a better idea? Is there another solution I am missing?
Any advice would be greatly appreciated, thanks!
EDIT:
How I currently initialise the database:
firebase.initializeApp(firebaseConfig);
var database = firebase.database();
How I then get data from the database:
await database.ref('/').once('value', function(snapshot){
snapshot.forEach(function(childSnapshot){
if(childSnapshot.key === userName){
userPreferences = childSnapshot.val().UserPreferences;
}
})
});
The Firebase once() API reads the data from the database once, and then stops observing it.
If you instead us the on() API, it will continue observing the database after getting the initial value - and call your code whenever the database changes.
It sounds like you're looking to develop an application for scheduling. If that's the case you should check out node-schedule.
Node Schedule is a flexible cron-like and not-cron-like job scheduler
for Node.js. It allows you to schedule jobs (arbitrary functions) for
execution at specific dates, with optional recurrence rules. It only
uses a single timer at any given time (rather than reevaluating
upcoming jobs every second/minute).
You then can use the database to keep a "state" of the application so on start-up of the application you read all the upcoming jobs that will be expected and load them into node-schedule and let node-schedule do the rest.
The Google Cloud solution for scheduling a single item of future work is Cloud Tasks. Firebase is part of Google Cloud, so this is the most natural product to use. You can use this to avoid polling the database by simply specifying exactly when some Cloud Function should run to do the work you want.
I've written a blog post that demonstrates how to set up a Cloud Task to call a Cloud Functions to delete a document in Firestore with an exact TTL.
How do i realise Tasks being calculated behind a Website, that is a normal php / Mysql Website, without being triggered by a Website user, like deleting a chat room automatical from the sql db, when the user that created it goes offline, in an simple Webchat with userchatrooms? Another example is some Text and Image based Browsergame, where every 5 Minutes (In Servertime) all Unit Movements and eventual fights between Users on a Image Based Game Map where calculated, when 2 Users or more meet on the same Map Tile? The calculated damage is sent back to the PHP / Ajax Website
To fire off tasks at an interval (ie: every five minutes) you would use a CRON job. To set that up is going to be dependent on what kind of server you're running. If you have a server GUI like CPANEL or PLESK there should be controls for setting CRON jobs. Essentially you input a time in this format
minute, hour, day, month, day of week
so 1 * * * * would run on the 1st minute of every hour, every day.
You point that to a command - probably a shell script, which will run whatever you need it to do.
To communicate that back to the "server" you would basically just update your database or whatever datastore you're using within your cron job.
So in your first example, to do something every five minutes, you would do
*/5 * * * * /path/to/script.sh
Then in that script, do whatever operations you need and save back to the DB.
If you are using a server admin GUI as previously mentioned, it's easiest to just hop in there and find the cron jobs tab and enter it.
If you're just managing your server with shell access you need to go put that in your crontab. The command for this might be dependent on the OS you're using but it's probably
crontab -e
I want to use a linux device like a BananaPi with a socketcan-compatible can-controller to connect to a automotive can-bus and show its data in realtime on a webpage, which should be hosted on the Pi.
The data should be listed as hex-values and visualized via graphs (the different signals, for example the current speed).
After some research I discovered node-can and I could get managed to show the can-messages as a list on a webpage. But I noticed, that the messages come with a quite huge delay (~2 secs) when there is a huge busload (I sent can messages in a 1 ms period). The same delay occurs, if I use the following minimalistic example:
var can = require('socketcan');
var channel = can.createRawChannel("can1", true);
channel.addListener("onMessage", function(msg) { console.log(msg); } );
channel.start();
I am absolutely new in this topic but I think, that nodejs isn't the best choice to realize this project?
Are there any other (better) methods to realize such a system?
I could imagine something like a C-backend, for example based on candump (with this program no delay occurs at the same busload), and a frontend realized with javascript, html and css. But I have no idea how to get those different single programs together. Could you give me a keyword so I have a starting point for further research (websocket?!)?
I also thought about writing the can frames in a sql database and grab them from the database for the webpage-gui but I have no idea, if/how this works and if this is fast enough....
Thanks in advance!
In the setup, Python writes to a database (mongo) every second and Meteor.js must react to the new record insertion immediately.
Problem: However using cursor.observe() as shown below, the console outputs only 4-5 seconds after the new record has been inserted.
Question: Is it possible to increase the update frequency of cursor.observe? If not, what will be an alternative?
server/news.js
var newsCursor = News.find({});
var newsHandle = newsCursor.observe({
added: function() {
console.log('New news added!');
}
});
Meteor's mongo-driver package makes cursors update immediately when changed from the mongo app. It also polls the database every 10 seconds to check for database changes from outside the meteor app, such as from your python code.
The smart collections atmosphere package is a simple rewrite which implements Mongo's oplog API, which allows the Meteor app to be immediately updated when the database is updated from outside the app. This is also important for scaling, because it allows multiple meteor processes to update the database and have those results immediately appear on other processes. By 1.0, Meteor will natively use the oplog. So until then, you need to use smart collections.
I'm playing with realtime whiteboards with meteor. My first attempt was working very well, if you open 2 browsers and draw in one of them, the other one updates in a few milliseconds ( http://pen.meteor.com/stackoverflow )
Now, my second project, is to make an infinite realtime whiteboard. The main thing that changes now, is that all lines are grouped by zones, and the viewer only subscribe to the lines in the visible zones. And now there is a dealy of 5 seconds (!) when you do something in one browser to see it happen in the other one ( http://carve.meteor.com/love ).
I've tried to add indexes in the mongo database for the fields determining the zones.
I've tried updating the Collection only for a full line (and not each time I push a new point like i my first project).
I've tried adding a timeout not to subscribe too often when scrolling or zooming the board.
Nothing changes, always a 5 seconds delay.
I don't have this delay when working locally.
Here is the piece of code responsible for subscribing to the lines you the visible area :
subscribeTimeout=false;
Deps.autorun(function () {
var vT=Session.get("visible_tiles");
var board_key=Session.get("board_key");
if (subscribeTimeout) Meteor.clearTimeout(subscribeTimeout);
subscribeTimeout=Meteor.setTimeout(subscribeLines, 500);
});
function subscribeLines() {
subscribeTimeout=false;
var vT=Session.get("visible_tiles");
console.log("SUBSCRIBE");
Meteor.subscribe("board_lines", Session.get("board_key"),vT.left,vT.right,vT.top,vT.bottom, function() {
console.log("subscribe board_lines "+Session.get("board_key"));
});
}
I've been a SysAdmin for 15 years. Without running the code, it sounds like an imposed limitation of the meteor.com server. They probably put in delays on the resources so everyone gets a fair share. I'd publish to another server like heroku for an easy deploy or manually to another server like linode or my favorite Joyent. Alternatively you could try and contact meteor.com directly and ask them if/how they limit resource usage.
Since the code runs fast/instantly locally, you should see sub-second response times from a good server over a good network.