I am attempting to rate limit queries to the Battle.net API up to their limits of 100 calls/second, 36,000/hour.
My current code looks like this:
var async = require ('async');
var bnet = require ('battlenet-api');
async.eachSeries(regions, function(region, callback) {
bnet.wow.realmStatusAsync({origin: region}).spread(/*...snip...*/)
});
What I'd need here is to make sure that no calls to
battlenet-api run up to the limit, and if so, queue them.
I have looked at timequeue.js, limiter and async's own capabilities and none seems to provide what I am looking for.
Does someone know how to achieve this?
A couple possibilities are:
Wait until you get throttled and backoff your requests
Keep track of your limits and used limits inside of your application.
While i think both cases need to be handled, the second one is more valuable, because it will let your application know at any time how many requests have been made for a given timeframe.
You could focus on implementing this on the hour level and letting the backoff/retry catch the cases of going over the 100 req/s timelimit.
To implement this, your application would have to keep track of how many requests are available for the given time period. Before each request the request count has to be checked to see if there are available requests, if there are the API request can be made and if not then no request will be made and your application will throttle itself. Additionally, the limits need to be reset when appropriate.
There are quite a few articles on using redis to accomplish this, a quick google search should uncover them
Related
I'm trying to create something with GitHub's REST api using Octokit, but I'm having trouble dealing with their secondary rate limit, I wonder where can I view this secondary rate limit, or know exactly how they work? The documentation seems to be very vague about the inner workings of this said secondary rate limit.
The rate limits can be viewed using the endpoint below, but it does not include the secondary rate limit.
await octokit.request('GET /rate_limit', {})
Also the documentation only provides best practices to avoid this secondary rate limit, but even this does not help.
Specifically I'm constantly using their follow a user endpoint every 2 seconds and doing a 5 minute sleep after 24 requests, but at some point I'm still hitting this secondary rate limit.
await octokit.request('PUT /user/following/{username}', {
username: 'USERNAME'
})
At this point, the only solution that I can think of is to slow down even further with the requests, but that is not optimal.
I wonder If GitHub has a way to view this secondary rate limit so I can deal with it programmatically, or on a much clearer way?
You can see secondary rate limits configured on a on-premise GHE (GitHub for Enterprise) instance.
Type limits for:
Total Requests,
CPU Limit, and
CPU Limit for Searching,
or accept the pre-filled default limits.
But even that instance does not have an exposed API to configure those elements: web page interface only.
this question might be duplicated but I am still not getting the answer. I am fairly new to node.js so I might need some help. Many have said that node.js is perfectly free to run incoming requests asynchronously, but the code below shows that if multiple requests hit the same endpoint, say /test3, the callback function will:
Print "test3"
Call setTimeout() to prevent blocking of event loop
Wait for 5 seconds and send a response of "test3" to the client
My question here is if client 1 and client 2 call /test3 endpoint at the same time, and the assumption here is that client 1 hits the endpoint first, client 2 has to wait for client 1 to finish first before entering the event loop.
Can anybody here tells me if it is possible for multiple clients to call a single endpoint and run concurrently, not sequentially, but something like 1 thread per connection kind of analogy.
Of course, if I were to call other endpoint /test1 or /test2 while the code is still executing on /test3, I would still get a response straight from /test2, which is "test2" immediately.
app.get("/test1", (req, res) => {
console.log("test1");
setTimeout(() => res.send("test1"), 5000);
});
app.get("/test2", async (req, res, next) => {
console.log("test2");
res.send("test2");
});
app.get("/test3", (req, res) => {
console.log("test3");
setTimeout(() => res.send("test3"), 5000);
});
For those who have visited, it has got nothing to do with blocking of event loop.
I have found something interesting. The answer to the question can be found here.
When I was using chrome, the requests keep getting blocked after the first request. However, with safari, I was able to hit the endpoint concurrently. For more details look at the following link below.
GET requests from Chrome browser are blocking the API to receive further requests in NODEJS
Run your application in cluster. Lookup Pm2
This question needs more details to be answer and is clearly an opinion-based question. just because it is an strawman argument I will answer it.
first of all we need to define run concurrently, it is ambiguous if we assume the literal meaning in stric theory nothing RUNS CONCURRENTLY
CPUs can only carry out one instruction at a time.
The speed at which the CPU can carry out instructions is called the clock speed. This is controlled by a clock. With every tick of the clock, the CPU fetches and executes one instruction. The clock speed is measured in cycles per second, and 1c/s is known as 1 hertz. This means that a CPU with a clock speed of 2 gigahertz (GHz) can carry out two thousand million (or two billion for those in the US) for the rest of us/world 2000 million cycles per second.
cpu running multiple task "concurrently"
yes you're right now-days computers even cell phones comes with multi core which means the number of tasks running at the same time will depend upon the number of cores, but If you ask any expert such as this Associate Staff Engineer AKA me will tell you that is very very rarely you'll find a server with more than one core. why would you spend 500 USD for a multi core server if you can spawn a hold bunch of ...nano or whatever option available in the free trial... with kubernetes.
Another thing. why would you handle/configurate node to be incharge of the routing let apache and/or nginx to worry about that.
as you mentioned there is one thing call event loop which is a fancy way of naming a Queue Data Structure FIFO
so in other words. no, NO nodejs as well as any other programming language out there will run
but definitly it depends on your infrastructure.
I've written a program in Node and Express, using Request to connect to an API and downloads a bunch of data (think 3,000 API requests) (all within the usage limits of the API, mind you).
When running this in a Docker container, I'm getting a lot of getaddrinfo ENOTFOUND errors, and I'm wondering if this is a resourcing issue. My requests are like so:
request.get(url, function(err, resp, body){
// do stuff with the body here,
// like create an object and handball to a worker function
});
For the first few hundred requests this always works fine, but then I get lots nad lots of either ENOTFOUND or timeout errors, and I think the issue might be the way my code is dealing with all these requests.
I've batched them in a queue with timeouts so the requests happen relatively slowly, it helps a little bit but doesn't solve the problem completely.
Do I need to destroy the body/response objects to free up memory or something?
I've encountered similar issues with an API I was using, and it ended up being what some here suggested - rate limits. Some APIs don't return readable errors on rate limits, as they provide certain amount of resources per client, and when you've used it all up they can't even send you a bad response.
This happened even though I've stayed within the published rate limits per day, but turned out they have an unwritten limit per minute (or more like - just unable to process so many requests).
I answered it though by mocking that API with my own code, placing it in the network so it will maximize the similarities, and as my mocked code didn't do anything, I never got any errors in the NodeJS server.
Then I put it some countdowns and timeouts when it was needed.
I suggest the same to you. Remember them having a per hour limit, doesn't mean they don't have a different per second/minute limit.
I am building a webapp where user have a ranking based on their twitter activity and their activity on my website.
Therefore I'd like to update their rank every five minutes, pulling their latest activity from twitter and update it in my database. I was thinking of using something like this:
var minutes = 5, the_interval = minutes * 60 * 1000;
setInterval(function() {
// my update here
}, the_interval);
However, I have several questions about this code:
where should I save it to make sure it is run?
will it slow my program or is it a problem to pull data out of twitter every five minute? Should I use their streaming API instead?
Note: I am using mongoDB
I'd suggest that you create a scheduled task/chron job/etc. (depends on your host OS) to call a separate Node.JS application that performs the specific tasks you want to do periodically and then it would exit when complete. (Or you could use a ChildProcess potentially as well).
While Node.JS is async, there's no need, given the description you provided, to perform this work within the same application process that is serving a web application. In fact, as it sounds like "busy work", it would be best handled by a distinct process to avoid impacting directly any of your interactive web users.
The placement shouldn't really matter as Node is asynchronous.
I'm playing with realtime whiteboards with meteor. My first attempt was working very well, if you open 2 browsers and draw in one of them, the other one updates in a few milliseconds ( http://pen.meteor.com/stackoverflow )
Now, my second project, is to make an infinite realtime whiteboard. The main thing that changes now, is that all lines are grouped by zones, and the viewer only subscribe to the lines in the visible zones. And now there is a dealy of 5 seconds (!) when you do something in one browser to see it happen in the other one ( http://carve.meteor.com/love ).
I've tried to add indexes in the mongo database for the fields determining the zones.
I've tried updating the Collection only for a full line (and not each time I push a new point like i my first project).
I've tried adding a timeout not to subscribe too often when scrolling or zooming the board.
Nothing changes, always a 5 seconds delay.
I don't have this delay when working locally.
Here is the piece of code responsible for subscribing to the lines you the visible area :
subscribeTimeout=false;
Deps.autorun(function () {
var vT=Session.get("visible_tiles");
var board_key=Session.get("board_key");
if (subscribeTimeout) Meteor.clearTimeout(subscribeTimeout);
subscribeTimeout=Meteor.setTimeout(subscribeLines, 500);
});
function subscribeLines() {
subscribeTimeout=false;
var vT=Session.get("visible_tiles");
console.log("SUBSCRIBE");
Meteor.subscribe("board_lines", Session.get("board_key"),vT.left,vT.right,vT.top,vT.bottom, function() {
console.log("subscribe board_lines "+Session.get("board_key"));
});
}
I've been a SysAdmin for 15 years. Without running the code, it sounds like an imposed limitation of the meteor.com server. They probably put in delays on the resources so everyone gets a fair share. I'd publish to another server like heroku for an easy deploy or manually to another server like linode or my favorite Joyent. Alternatively you could try and contact meteor.com directly and ask them if/how they limit resource usage.
Since the code runs fast/instantly locally, you should see sub-second response times from a good server over a good network.