I'm performing a routine check on my DB every hour or so by doing
setInterval(function() {
myCheckFunction();
}, 3600000)
I'm thinking of something like :
setInterval(function() {
server.getConnections(function(err, count) {
if (count < X) {
myCheckFunction();
}
}, 3600000)
To check if there's isn't too much work being done right now.
Is it a good idea ?
If so what value could X have ?
If not, should I try differently or just do the test, non regarding of the current load ?
I don't expect millions of connections, this is just a proof of concept and my teacher asked me to take care of that kind of things.
Thanks !
Edit: Why do I want to avoid heavy load ? Because the routine check could take a couple of minutes and require a lot of work for the server. It has to contact at least 5 DNS server for each entry in the DB, request HTML code and hash it, then compare answers. The check for one entry can take up to 6 seconds due the the fact that the DNS servers can be Timeout and the DB is hosted separately from the project.
This is what you are looking for:
toobusy
With this particular requirements, I would perform this check on server side instead.
var checkFunctionCaller = function () {
//call async function on server side and provide a callback function
//to be called as server returns an answer
server.myCheckFunction(function (result) {
// if server was able to run check - it return 'done' and everything is fine
if(result === "done") {
//do nothing
}
//otherwise, re-schedule a check in five minutes
setTimeout(checkFunctionCaller, 5 * 60 * 1000);
});
}
setInterval(checkFunctionCaller, 3600000);
Related
[![enter image description here][1]][1]I need to implement code to check what my throttling limit is on an endpoint (I know it's x times per minute). I've only been able to find an example of this in python, which I have never used. It seems like my options are to run a script to send the request repeatedly until it throttles me or, if possible, query the API to see what the limit is.
Does anyone have a good idea on how to go about this?
Thanks.
Note: The blank space is just data from the api calls.
[1]: https://i.stack.imgur.com/gAFQQ.png
This starts concurency number of workers (I'm using workers as a loose term here; don't # me). Each one makes as many requests as possible until one of the requests is rate-limited or it runs out of time. It them reports how many of the requests completed successfully inside the given time window.
If you know the rate-limit window (1 minute based on your question), this will find the rate-limit. If you need to discover the window, you would want to intentionally exhaust the limit, then slow down the requests and measure the time until they started going through again. The provided code does not do this.
// call apiCall() a bunch of times, stopping when a apiCall() resolves
// false or when "until" time is reached, whichever comes first. For example
// if your limit is 50 req/min (and you give "until" enough time to
// actuially complete 50+ requests) this will call apiCall() 50 times. Each
// call should return a promise resolving to TRUE, so it will be counted as
// a success. On the 51st call you will presumably hit the limit, the API
// will return an error, apiCall() will detect that, and resolve to false.
// This will cause the worker to stop making requests and return 50.
async function workerThread(apiCall, until) {
let successfullRequests = 0;
while(true) {
const success = await apiCall();
// only count it if the request was successfull
// AND finished within the timeframe
if(success && Date.now() < until) {
successfullRequests++;
} else {
break;
}
}
return successfullRequests;
}
// this just runs a bunch of workerThreads in parallell, since by doing a
// single request at a time, you might not be able to hit the limit
// depending on how slow the API is to return. It returns the sum of each
// workerThread(), AKA the total number of apiCall()s that resolved to TRUE
// across all threads.
async function testLimit(apiCall, concurency, time) {
const endTime = Date.now() + time;
// launch "concurency" number of requests
const workers = [];
while(workers.length < concurency) {
workers.push(workerThread(apiCall, endTime));
}
// sum the number of requests that succeded from each worker.
// this implicitly waits for them to finish.
let total = 0;
for(const worker of workers) {
total += await worker;
}
return total;
}
// put in your own code to make a trial API call.
// return true for success or false if you were throttled.
async function yourAPICall() {
try {
// this is a really sloppy example API
// the limit is ROUGHLY 5/min, but because of the sloppy server-side
// implimentation you might get 4-6.
const resp = await fetch("https://9072997.com/demos/rate-limit/");
return resp.ok;
} catch {
return false;
}
}
// this is a demo of how to use the function
(async function() {
// run 2 requests at a time for 5 seconds
const limit = await testLimit(yourAPICall, 2, 5*1000);
console.log("limit is " + limit + " requests in 5 seconds");
})();
Note that this method measures the quota available to itself. If other clients or previous requests have already depleted the quota, it will affect the result.
I created a real-time clock that updates every second. When I run it locally I see no errors. However when I uploaded it to my web host I saw a lot of error messages in the console every time the AJAX code runs.
I think that's the reason why web host suspended my site, telling that my site has performed too many requests.
Here is my code:
$(document).ready(function() {
setInterval(function() {
$('#time').load('timewithdate.php')
}, 1000);
setInterval(function(){
$('#time2').load('time.php')
}, 1000);
})
I'm not sure what kind of answer you are looking for. Your code is a certain way to kill a server: it's making 2 calls to the server every second for each client (read more about DDOS). There is no need to make a server call, just use javascript to get the current time and format it the way you want. You can have something like this:
$(document).ready(function() {
setInterval(function() {
let curTime = new Date();
let date = curTime.toLocaleDateString();
let time = curTime.toLocaleTimeString();
$('#time').text(date + " " + time);
$('#time2').text(time);
}, 1000);
})
The error is because you have effectively DDOS'd your own server with 2 requests every second * number of concurrent clients. I would strongly suggest you remove these AJAX requests and perform the countdown on the client side.
If you're trying to keep the clocks in sync with server time, get the time from the server when the page loads, then add seconds to it on the client side. Do not use AJAX for this, and do not use AJAX polling in future. It's an anti-pattern.
I am currently attempting to make an alarm system for a website that I control my sprinklers with.
Currently, I use sockets to get alarms from the client and store them in a JSON file in the server.
To "activate" the alarms, I have thought to have a setInterval function that checks whether the current time is included in the array (using .includes() )
I am very annooyed currently because the setInterval function is delays 65 milliseconds every time it is run
Does it wait for the code inside to finish before continuing? If so, how do I make it not do so
This is my code:
I think this may be relevant, if I could understand it (I am new)
console.log("Waiting for time to get in to sync...")
while (sync > 0 || sync == undefined) {
let date = new Date()
sync = date.getSeconds()
}
setInterval(()=>{
check(new Date()) //run the check code elsewhere
}, 60000); // one minute delay```
I'm working in a project where I need to make requests over an API. The requests return data about a support ticket, but the problem is that i have about 500 tickets to get data about and each one requires one request. To speed up the requests, i tried to build a async routine that generate many requests at the same time. But, since the API that i'm integrating with has a rate limiter of 10 requests per second, some of the routines get the answer "Limit Exceed". If I make the requests sequentially, it's take about 5 minutes.
That way, someone has a tip for me in that task? I tried some solutions like rate-limiter of NodeJS, but it just generate 10 requests simultaneously, and didn't give any kind of error treatment or retry if the request fail.
About the language, it not have restriction, the project is written in NodeJS but have some python code too and didn't have problem to integrate another language.
Something like this isn't too difficult to create yourself, and it'd give you the flexibility you need.
There are fancy ways like tracking the start and completion time of each and checking if you've sent 10 in the second.
The system probably also limits it to 10 active requests going (i.e., you can't spin up 100 requests, 10 each second, and let them all process).
If you assume this, I'd say launch 10 all at once, then let them complete, then launch the next batch. You could also launch 10, then start 1 additional each time one finishes. You could think of this like a "thread pool".
You can easily track this with a simple variable tracking how many calls are going. Then, just check how many calls are going once a second (to avoid the 1 second limit) and if you have available "threads", fire off that many more new requests.
It could look something like this:
const threadLimit = 10;
const rateLimit = 1000; // ms
let activeThreads = 0;
const calls = new Array(100).fill(1).map((_, index) => index); // create an array 0 through 99 just for an example
function run() {
if (calls.length == 0) {
console.log('complete');
return;
}
// threadLimit - activeThreads is how many new threads we can start
for (let i = 0; i < threadLimit - activeThreads && calls.length > 0; i++) {
activeThreads++; // add a thread
call(calls.shift())
.then(done);
}
setInterval(run, rateLimit);
}
function done(val) {
console.log(`Done ${val}`);
activeThreads--; // remove a thread
}
function call(val) {
console.log(`Starting ${val}`);
return new Promise(resolve => waitToFinish(resolve, val));
}
// random function to simulate a network call
function waitToFinish(resolve, val) {
const done = Math.random() < .1; // 10% chance to finish
done && resolve(val)
if (!done) setInterval(() => waitToFinish(resolve, val), 10);
return done;
}
run();
Basically, run() just starts up however many new threads it can, based on the limit and how many are done. Then, it just repeats the process every second, adding new ones as it can.
You might need to play with the threadLimit and rateLimit values, as most rate limiting systems don't actually let you go up right to the limit and don't release it as soon as it's done.
Basically, I am designing a quiz application with limited time. Use selects answer to a question and the next question loads using an Ajax request. All questions must be answered within a time frame of, say 2 minutes.
A clock ticks away to show how much time is left and as soon as it hits 0, results are shown. Now since the timer will be implemented using window.setTimeout(), it is possible that the value of timer variable be modified using an external bookmarklet or something like that. Anyway I can prevent this? I think this is implemented on file sharing sites like megaupload. Any forgery on the timer variable results in request for file being rejected.
Have .setTimeout() call an AJAX method on your server to synch time. Don't rely on the client time. You could also store the start time on the server for a quiz, and then check the end time when the quiz is posted.
You need to add a validation in your server side. When the client want to load the next question using an Ajax request, check whether deadline arrived.
The timer in client side js just a presention layer.
If the function runs as a immediately called function expression, then there are no global variables and nothing for a local script to subvert. Of course there's nothing to stop a user from reading your code and formulating a spoof, but anything to do with javascript is open to such attacks.
As others have said, use the server to validate requests based on the clock, do not rely on it to guarantee anything. Here's a simple count down that works from a start time so attempts to dealy execution won't work. There are no global variables to reset or modify either.
e.g.
(function (){
// Place to write count down
var el = document.getElementById('secondsLeft');
var starttime,
timeout,
limit = 20; // Timelimit in seconds
// Function to run about every second
function nextTick() {
var d = new Date();
// Set start time the first time
if (!starttime) starttime = d.getTime();
var diff = d.getTime() - starttime;
// Only run for period
if (diff < (limit * 1000)) {
el.innerHTML = limit - (diff/1000 | 0);
} else {
// Time's up
el.innerHTML = 0;
clearTimeout(timeout);
}
}
// Kick it off
timeout = window.setInterval(nextTick, 1000);
}());