If I will use standart link to parse Steam user inventory (https://steamcommunity.com/profiles/{ steamid } /inventory/json/730/2) more than 2-3 times per minute, I will get banned from Steam api to 5 mins. How can I parse it without bans? Using node.js.
Store the result you get from your first request, and re-use it instead of re-querying Steam every time you want to read the data.
pseudocode:
if(!cache)
getDataFromSteam()
saveDataToCache()
else
getDataFromCache()
If you send too many requests you are sending too many requests. That is a fact you have to accept and work with that. I know from testing on my own that you are limited to 200 calls in 5 minutes. You have several options:
Implement a routine that limits the calls to 200 in 5 minutes and not more in order to avoid getting banned.
Cache the calls you make in order to avoid duplicate calls for one user.
Use other IP adresses as mentioned by #Dandavis.
Related
What we have?
An API build in Node.js (using Moleculer.js for micro-services and PostgreSQL for storing data) which has the functionality of keeping track of users and user groups. We have in average 3k users per group, and a user can be part of multiple groups.
What we want to achieve?
We want to create a special service which will send text messages. The admins will select multiple groups, the code will remove the duplicated users and send them an SMS.
After a selection we can have around 1 million users. How can we send them text messages in an efficient way?
What have we tried?
Paginate the users and for each page send a request to the SMS service.
const users = db.getPage(1); // [{ id: 1, phone: '+123456789' }, ...]
smsClient.sendBulk(users);
PROBLEM: The user list in the database can change in the process and can affect the pagination by giving us duplicates or skipping some users.
Load all the results in the memory and send all the users to the SMS service.
const users = db.getAll(); // [..., { id: 988123, phone: '+987654321' }]
smsClient.sendBulk(users);
PROBLEM: We think it's a bad idea, resource wise, to make this kind of queries to the database and keep them in the memory. In the same time, we don't want to send 1 million entities through an HTTP request to the SMS service.
How can we select a 1 million users and send them an SMS message without worry about duplicates, skipped data or any other alteration to the admin's selection? We were thinking about queues as a necessary step but after we find a solution for the cases mentioned above. Or, is the queue part of the solution?
How can we select a 1 million users and send them an SMS message without worry about duplicates, skipped data, or any other alteration to the admin's selection?
For managing duplicates You could use an additional DB to save a Hash Table for the users that been handled already. This is a bit more expensive because you will need to check the user before each SMS send.
Managing not skipping is a bit tricky because you will need to add more recipients to an ongoing SMS transaction. You will need the ability to detect (hook) when a user is added to a group and add it as recipients to the ongoing transactions accordingly.
You will need to find a fast DB and save that user in a HashSet for a fast set and get (O(1))
We were thinking about queues as a necessary step but after we find a solution for the cases mentioned above. Or, is the queue part of the solution?
Defenently. Queue is the correct way to go for this scenario (queueing many small tasks). Some queues come with a re-queue features that will re-queue any task that didn't get acknowledgment.
you need to check out RabbitMQ.message-driven microservices
Have you considered creating an indirect state between the user and sent SMS? Something like SmsRequest / SmsTask / however you'd call it.
It'd consist of necessary user-data, message content, status of the request (to-send, sending, sent, failed, ...) and some additional metadata depending on your needs.
Then the first step you'd do is to prepare these request and store them in db, effectively making a queue out of a table. You can add some constraints on user and message type that'd prevent any duplicates and then start second asynchronous process that simply fetches requests in to-send state, sets the state to sending and then saves the outcome.
This also gives you the benefit of audit + you can batch the outgoing messages and.
Of course it'd increase your data volume significantly but I guess it's cheap nowadays anyway.
This is not a post about HOW to change channel's name (I know it).
I have an international server using several bots. And we all depend on UTC time (to coordinate through the world). So there was borned a solution to make a time-bot which will show current UTC-time in the dedicated channel nobody can visit. And yes, precision is necessary, even seconds.
I created a voice channel with permissions not to join for #everyone. Everything worked fine, it updated every 1000 ms. Then (after several months of good work) something was broke, it started updating incorrect. I've increased update interval up to 5000 ms and it have started to work fine... until yesterday.
Now it doesn't work anymore. Even if I increase interval much more. It works sometimes I don't really know what the interval is, it's huge and unpredictable.. the time-bot is broken for now and cannot be used anymore in that case.
Is there any restrictions for updating channel name? I can't find any information about this in available documentations.
Client.setInterval(() => {
const { h, m, s } = getTime();
channel.edit({ name: `${getClockEmoji({ h, m })} UTC: ${h}-${m}-${s}` }).catch((err) => console.log(err));
}, updateInterval);
Providing data is correct, 'cause I send it to console and it updates as good as I need in interval I set. But channel name not updates that often..
Does discord filter too often update requests?
discord.js version is v.12.2.0
Discord had set the rate limit for things like channelrename to 2 requests per 10 minutes.
"Normal" requests like sendmessage is limited to 10,000 per 10 minutes.
This seems to likely be an issue directly related to rate limiting:
https://discord.com/developers/docs/topics/rate-limits
IP addresses that make too many invalid HTTP requests are automatically and temporarily restricted from accessing the Discord API. Currently, this limit is 10,000 per 10 minutes. An invalid request is one that results in 401, 403, or 429 statuses.
For every API request made, we return optional HTTP response headers containing the rate limit encountered during your request.
You should probably decrease the interval by a considerable amount to reduce the risk of the IP being restricted.
I have multiple heroku dynos and a chat app. When a user logs in, their status is set to "online" in MongoDB. However, if a server crashes, their status will still be set as online. How can I update the user status to be "offline" when a server crashes?
If I only had one dyno, this would be easy. I'd just update every user to be "offline" when the server starts. Unfortunately, this is not possible with multiple servers.
As per our chat and comments.
The best option is to go with checking against last activity. So seeing when the last message was sent and if it happened within the last let's say 5 minutes they are online if there were no activity mark them as offline.
Like I mentioned in the comments, if you are not storing a date_created on the messages documents you will not have to change anything because _id stores the timestamp
ObjectId("507f191e810c19729de860ea").getTimestamp()
that returns this Date object
ISODate("2012-10-17T20:46:22Z")
This answer is another option (if you are wanting to keep them as online even if they are not sending messages):
If you would like to know they are still active even when they're not jumping from page to page, include a bit of javascript to ping your server every 60 seconds or so to let you know they are still alive. It'll work the same way as my original suggestion, but it will update your records without requiring them to be frantically browsing your site at least once every five minutes.
var stillAlive = setInterval(function () {
/* XHR back to server
Example uses jQuery */
$.get("stillAlive.php");
}, 60000);
I have roughly 1.5 million users I need to import into my parse _User table (we are migrating from our old sql db to Parse).
I've tried using a Background Job, but it is slow and gets killed every 15 minutes. As a result, we have only import ~300K users in 4 days.
So I decided to go the REST api route...and tried the PythonPy lib. It works, except for being able to do batch creation of users. From digging more into this, it seems that the REST api forces you to do a User.signup() call.
This of course makes the whole thing be serialized.
Is there anyway to do batch user creation?
Batch creating / signing up User objects is not supported in Parse. It is mentioned in this question:
https://parse.com/questions/what-user-operations-are-supported-in-batch
I have a program that makes a lot of fql and graph calls and I'm not sure how to handle when there's a 'get' error or a 'post' error. How do I get it to retry? I'm still new to this stuff but could I use some sort of try catch block? if so, how do I structure it?
I guess this could be extended to any get timeout in javascript.
Thanks
Querying Facebook Graph API can be a real pain. First of all, I try to chop my queries in batches. For instance, if I want all of my friends' posts, I first get an array of all my friends, then create queries for their posts for 10 friends each and send them off to facebook. What I do is with every reponse is test if there's an error in the response and if so I restart the function that generates the batches and sends them. I use one counter that keeps track of the amount of queries send out. If I send out 10 queries and I'm getting only 9, I restart the function after 30 seconds again. I use another counter that after 3 retries I show an error to the user...