I wrote an asynchronous function which is supposed to check if a given url returns a '200' from a get and otherwise wait a few seconds to try again a limited number of times. The code works just fine when I run it in my computer using node but when I transfer it to backend it only checks for the site once and then immediately stops when receiving an error. What am I doing wrong?
async function wait(url,delay=10000,attp=3){
let t0 = new Date(Date.now());
let attempts = 0;
console.log('starting...');
async function check(){
if(attempts<attp){
console.log('ATTEMPTS: ',attempts);
return await request.get(url).on('error',
async function(err){
console.log('ERROR: ',err);
attempts+=1;
return await setTimeout(()=>{check()},delay);
}).on('response',
async function(response){
if(response.statusCode===200){
let t1 = new Date(Date.now());
wixData.insert('pagSeguroTimeStats', { 'time': (t1 - t0) / 1000. });
return '200';
}else{
attempts+=1;
console.log('not 200');
return await setTimeout(()=>{check()},delay);
}
});
}else{
return '404';
}
}
return check();
}
Seems that there is a limit to how much time a backend function can run. From the Wix Code forum, it seems that the limit is 14 seconds, although this doesn't look like an official number from Wix.
The 14 second limit only applies to web functions.
Time
Wix allows web modules called from the frontend, HTTP functions, and router hooks to run for up to 14 seconds. This limitation applies to both free and premium sites. Any of these methods that take longer than 14 seconds receives a 504 response code. Note that after 14 seconds the code might still execute, but the connection to the client is closed so the results do not appear in the frontend. This error message appears in your log:
/backend/.js(w)/ timed out because it exceeded the maximum execution time.
I've got a .js function that just stop, and I get no error or anything.
Related
[![enter image description here][1]][1]I need to implement code to check what my throttling limit is on an endpoint (I know it's x times per minute). I've only been able to find an example of this in python, which I have never used. It seems like my options are to run a script to send the request repeatedly until it throttles me or, if possible, query the API to see what the limit is.
Does anyone have a good idea on how to go about this?
Thanks.
Note: The blank space is just data from the api calls.
[1]: https://i.stack.imgur.com/gAFQQ.png
This starts concurency number of workers (I'm using workers as a loose term here; don't # me). Each one makes as many requests as possible until one of the requests is rate-limited or it runs out of time. It them reports how many of the requests completed successfully inside the given time window.
If you know the rate-limit window (1 minute based on your question), this will find the rate-limit. If you need to discover the window, you would want to intentionally exhaust the limit, then slow down the requests and measure the time until they started going through again. The provided code does not do this.
// call apiCall() a bunch of times, stopping when a apiCall() resolves
// false or when "until" time is reached, whichever comes first. For example
// if your limit is 50 req/min (and you give "until" enough time to
// actuially complete 50+ requests) this will call apiCall() 50 times. Each
// call should return a promise resolving to TRUE, so it will be counted as
// a success. On the 51st call you will presumably hit the limit, the API
// will return an error, apiCall() will detect that, and resolve to false.
// This will cause the worker to stop making requests and return 50.
async function workerThread(apiCall, until) {
let successfullRequests = 0;
while(true) {
const success = await apiCall();
// only count it if the request was successfull
// AND finished within the timeframe
if(success && Date.now() < until) {
successfullRequests++;
} else {
break;
}
}
return successfullRequests;
}
// this just runs a bunch of workerThreads in parallell, since by doing a
// single request at a time, you might not be able to hit the limit
// depending on how slow the API is to return. It returns the sum of each
// workerThread(), AKA the total number of apiCall()s that resolved to TRUE
// across all threads.
async function testLimit(apiCall, concurency, time) {
const endTime = Date.now() + time;
// launch "concurency" number of requests
const workers = [];
while(workers.length < concurency) {
workers.push(workerThread(apiCall, endTime));
}
// sum the number of requests that succeded from each worker.
// this implicitly waits for them to finish.
let total = 0;
for(const worker of workers) {
total += await worker;
}
return total;
}
// put in your own code to make a trial API call.
// return true for success or false if you were throttled.
async function yourAPICall() {
try {
// this is a really sloppy example API
// the limit is ROUGHLY 5/min, but because of the sloppy server-side
// implimentation you might get 4-6.
const resp = await fetch("https://9072997.com/demos/rate-limit/");
return resp.ok;
} catch {
return false;
}
}
// this is a demo of how to use the function
(async function() {
// run 2 requests at a time for 5 seconds
const limit = await testLimit(yourAPICall, 2, 5*1000);
console.log("limit is " + limit + " requests in 5 seconds");
})();
Note that this method measures the quota available to itself. If other clients or previous requests have already depleted the quota, it will affect the result.
I am working on a scraper . I am using Phantom JS along with Node JS. Phantom JS loads the page with async function, just like : var status = await page.open(url). Sometimes, because of the slow internet the page takes longer to load and after a time the page status is not returned, to check while its loaded or not. And the page.open() sleeps, which doesn't return anything at all, and all the execution is waiting.
So, my basic question is; is there any way to keep this page.open(url) alive, as the execution of the rest of the code waits until the page is loaded.
My Code is
const phantom = require('phantom');
ph_instance = await phantom.create();
ph_page = await ph_instance.createPage();
var status = await ph_page.open("https://www.cscscholarship.org/");
if (status == 'success') {
console.log("Page is loaded successfully !");
//do more stuff
}
From your comment, it seems like it might be timing out (because of slow internet sometimes)... you can validate this by adding the onResourceTimeout method to your code (link: http://phantomjs.org/api/webpage/handler/on-resource-timeout.html)
It would look something like this:
ph_instance.onResourceTimeout = (request) => {
console.log('Timeout caught:' + JSON.stringify(request));
};
And if that ends up being true, you can increase the default resource timeout settings (link: http://phantomjs.org/api/webpage/property/settings.html) like this:
ph_instance.settings.resourceTimeout = 60000 // 60 seconds
Edit: I know the question is about phantom, but I wanted to also mention another framework I've used for scraping projects before called Puppeteer (link: https://pptr.dev/) I personally found that their API's are easier to understand and code in, and it's currently a maintained project unlike Phantom JS which is not maintained anymore (their last release was two years ago).
I'm working in a project where I need to make requests over an API. The requests return data about a support ticket, but the problem is that i have about 500 tickets to get data about and each one requires one request. To speed up the requests, i tried to build a async routine that generate many requests at the same time. But, since the API that i'm integrating with has a rate limiter of 10 requests per second, some of the routines get the answer "Limit Exceed". If I make the requests sequentially, it's take about 5 minutes.
That way, someone has a tip for me in that task? I tried some solutions like rate-limiter of NodeJS, but it just generate 10 requests simultaneously, and didn't give any kind of error treatment or retry if the request fail.
About the language, it not have restriction, the project is written in NodeJS but have some python code too and didn't have problem to integrate another language.
Something like this isn't too difficult to create yourself, and it'd give you the flexibility you need.
There are fancy ways like tracking the start and completion time of each and checking if you've sent 10 in the second.
The system probably also limits it to 10 active requests going (i.e., you can't spin up 100 requests, 10 each second, and let them all process).
If you assume this, I'd say launch 10 all at once, then let them complete, then launch the next batch. You could also launch 10, then start 1 additional each time one finishes. You could think of this like a "thread pool".
You can easily track this with a simple variable tracking how many calls are going. Then, just check how many calls are going once a second (to avoid the 1 second limit) and if you have available "threads", fire off that many more new requests.
It could look something like this:
const threadLimit = 10;
const rateLimit = 1000; // ms
let activeThreads = 0;
const calls = new Array(100).fill(1).map((_, index) => index); // create an array 0 through 99 just for an example
function run() {
if (calls.length == 0) {
console.log('complete');
return;
}
// threadLimit - activeThreads is how many new threads we can start
for (let i = 0; i < threadLimit - activeThreads && calls.length > 0; i++) {
activeThreads++; // add a thread
call(calls.shift())
.then(done);
}
setInterval(run, rateLimit);
}
function done(val) {
console.log(`Done ${val}`);
activeThreads--; // remove a thread
}
function call(val) {
console.log(`Starting ${val}`);
return new Promise(resolve => waitToFinish(resolve, val));
}
// random function to simulate a network call
function waitToFinish(resolve, val) {
const done = Math.random() < .1; // 10% chance to finish
done && resolve(val)
if (!done) setInterval(() => waitToFinish(resolve, val), 10);
return done;
}
run();
Basically, run() just starts up however many new threads it can, based on the limit and how many are done. Then, it just repeats the process every second, adding new ones as it can.
You might need to play with the threadLimit and rateLimit values, as most rate limiting systems don't actually let you go up right to the limit and don't release it as soon as it's done.
Using chromeless to run tests every 5 minutes to verify I can log into a website and that content is available.
Code looks like this:
const { Chromeless } = require('chromeless')
async function run() {
const chromeless = new Chromeless({
remote: {
endpointUrl: 'https://abc1234567.execute-api.us-west-2.amazonaws.com/dev',
apiKey: 'abc123xyz987'
}
})
const screenshot = await chromeless
.clearCookies()
.setUserAgent('some-user-agent')
.goto('https://www.example.com')
.type('username', 'input[name="username"]')
.type('super_secret', 'input[name="password"]')
.click('#loginButton')
.wait('#TitleTagOnceLoggedIn')
.screenshot()
console.log(screenshot)
const report = await chromeless
.setUserAgent('some-user-agent')
.goto('https://www.example.com/only/accessible/if/logged/in')
.wait('#TitleTagForPageWhenLoggedIn')
.screenshot()
console.log(report)
await chromeless.end()
}
run().catch(console.error.bind(console))
This works fine, however once every 10-20 times I run it, I receive the following error
Chromeless Proxy disconnected due to inactivity (no commands sent for 30 seconds).
In the cloudwatch logs from my lambda function I get a similar error
Timing out. No requests received for 30 seconds
How can I set this to retry if I get the inactivity errors?
Convert
await chromeless.end()
to
chromeless.end().catch(callback).then(callback)
and move it into a non-async function, e.g. function end (callback) {...}.
Then have your code wait for run to finish executing and then execute end with a callback handling the result. See https://github.com/graphcool/chromeless/issues/259
First of all, I'm not experienced in asynchronous programming, so I'm sorry if I missed something obvious.
I see this question pop up a lot. I know that people don't like forcing synchronicity in Javascript, but it is necessary in this case. I'm making heavy calls to a production database that can't take too much load due to how frequently it is used. Because of this, I'm setting up my code in a loop to make a request, wait for confirmation that it is finished, sleep for 2 seconds, and then make the next request. This is because I'm going to be pulling a LOT of data from this server on a weekly basis over the course of around 10-20 minutes.
Here's the code that I have. Sleep is a function that forces the program to wait using the Date class.
var thread = function(cb){
cb();
};
do{
var x = thread(function(){
request.post(options, function(e, r, body){
console.log(e);
console.log(r.statusCode);
issues.push(body["issues"]);
maxResults = body["total"];
options.body.startAt += 25;
console.log("Going to sleep");
});
sleep(2000);
});
console.log("Waking up and moving to the next cycle");
}while(issues.length < maxResults);
console.log("Finished with the requests");
}
although I have a callback set up, my code is still running the requests asynchronously. Because I leave maxResults null, it is plainly obvious that my callback isn't working. This is my output:
Waking up and moving to the next cycle
Finished with the requests
Going to sleep
You need to make a recursive asynchronous function.
It would look something like this:
function fetch(existingIssues) {
return sendRequest().then(function() {
existingIssues.push(...);
if (existingIssues.length >= maxResults)
return existingIssues;
else
return fetch(existingIssues);
});
}
fetch([]).then(...);