Develop Reference

How to check what the throttling limit is for your access to an endpoint with JS

[![enter image description here][1]][1]I need to implement code to check what my throttling limit is on an endpoint (I know it's x times per minute). I've only been able to find an example of this in python, which I have never used. It seems like my options are to run a script to send the request repeatedly until it throttles me or, if possible, query the API to see what the limit is.
Does anyone have a good idea on how to go about this?
Thanks.
Note: The blank space is just data from the api calls.
[1]: https://i.stack.imgur.com/gAFQQ.png

This starts concurency number of workers (I'm using workers as a loose term here; don't # me). Each one makes as many requests as possible until one of the requests is rate-limited or it runs out of time. It them reports how many of the requests completed successfully inside the given time window.
If you know the rate-limit window (1 minute based on your question), this will find the rate-limit. If you need to discover the window, you would want to intentionally exhaust the limit, then slow down the requests and measure the time until they started going through again. The provided code does not do this.
// call apiCall() a bunch of times, stopping when a apiCall() resolves
// false or when "until" time is reached, whichever comes first. For example
// if your limit is 50 req/min (and you give "until" enough time to
// actuially complete 50+ requests) this will call apiCall() 50 times. Each
// call should return a promise resolving to TRUE, so it will be counted as
// a success. On the 51st call you will presumably hit the limit, the API
// will return an error, apiCall() will detect that, and resolve to false.
// This will cause the worker to stop making requests and return 50.
async function workerThread(apiCall, until) {
let successfullRequests = 0;
while(true) {
const success = await apiCall();
// only count it if the request was successfull
// AND finished within the timeframe
if(success && Date.now() < until) {
successfullRequests++;
} else {
break;
}
}
return successfullRequests;
}
// this just runs a bunch of workerThreads in parallell, since by doing a
// single request at a time, you might not be able to hit the limit
// depending on how slow the API is to return. It returns the sum of each
// workerThread(), AKA the total number of apiCall()s that resolved to TRUE
// across all threads.
async function testLimit(apiCall, concurency, time) {
const endTime = Date.now() + time;
// launch "concurency" number of requests
const workers = [];
while(workers.length < concurency) {
workers.push(workerThread(apiCall, endTime));
}
// sum the number of requests that succeded from each worker.
// this implicitly waits for them to finish.
let total = 0;
for(const worker of workers) {
total += await worker;
}
return total;
}
// put in your own code to make a trial API call.
// return true for success or false if you were throttled.
async function yourAPICall() {
try {
// this is a really sloppy example API
// the limit is ROUGHLY 5/min, but because of the sloppy server-side
// implimentation you might get 4-6.
const resp = await fetch("https://9072997.com/demos/rate-limit/");
return resp.ok;
} catch {
return false;
}
}
// this is a demo of how to use the function
(async function() {
// run 2 requests at a time for 5 seconds
const limit = await testLimit(yourAPICall, 2, 5*1000);
console.log("limit is " + limit + " requests in 5 seconds");
})();
Note that this method measures the quota available to itself. If other clients or previous requests have already depleted the quota, it will affect the result.

Close the page after certain interval [Puppeteer]

I have used puppeteer for one of my projects to open webpages in headless chrome, do some actions and then close the page. These actions, however, are user dependent. I want to attach a lifetime to the page, where it closes automatically after, say 30 minutes, of opening irrespective of whether any action is performed or not.
I have tried setTimeout() functionality of Node JS but it didn't work (or I just couldn't figure how to make it work).
I have tried the following:
const puppeteer = require('puppeteer-core');
const browser = await puppeteer.connect({browserURL: browser_url});
const page = await browser.newPage();
// timer starts ticking here upon creation of new page (maybe in a subroutine and not block the main thread)
/**
..
Do something
..
*/
// timer ends and closePage() is triggered.
const closePage = (page) => {
if (!page.isClosed()) {
page.close();
}
}
But this gives me the following error:
Error: Protocol error: Connection closed. Most likely the page has been closed.

Your provided code should work as excepted. Are you sure the page is still opened after the timeout and it is indeed the same page?
You can try this wrapper for opening pages and closing them correctly.
// since it is async it won't block the eventloop.
// using `await` will allow other functions to execute.
async function openNewPage(browser, timeoutMs) {
const page = await browser.newPage()
setTimeout(async () => {
// you want to use try/catch for omitting unhandled promise rejections.
try {
if(!page.isClosed()) {
await page.close()
}
} catch(err) {
console.error('unexpected error occured when closing page.', err)
}
}, timeoutMs)
}
// use it like so.
const browser = await puppeteer.connect({browserURL: browser_url});
const min30Ms = 30 * 60 * 1000
const page = await openNewPage(browser, min30Ms);
// ...
The above only closes the Tabs in your browser. For closing the puppeteer instance you would have to call browser.close() which could may be what you want?

page.close returns a promise so you need to define closePage as an async function and use await page.close(). I believe #silvan's answer should address the issue, just make sure to replace if condition
if(page.isClosed())
with
if(!page.isClosed())

Output execution time for a Playwright step with AJAX payload

I am trying to dump out a few key measurements to console when my test runs, rather than getting them from the reporter output, but I can't see how to grab the time taken for the last step to execute. Here's a simplified version based on the docs for request.timing() but I don't think that what I'm doing is classed as a request:
const { test, expect } = require('#playwright/test');
test('ApplicationLoadTime', async ({ page }) => {
// Wait for applications to load
await page.waitForSelector('img[alt="Application"]');
// Not working! - get time for step execution
const [fir] = await Promise.all([
page.click('text=Further information requested'),
page.waitForSelector('img[alt="Application"]')
]);
console.log(fir.timing());
});
The click on "Further information requested" causes the page to be modified based on an AJAX call in the background and the appearance of the Application img tells me it's finished. Is this possible or do I need to rely on the reports instead?

fir is going to be undefined in your code as page.click() doesn't return anything. You need to wait for the request whose timing you're interested in, use page.waitForEvent('requestfinished') or waitForNavigation:
const { test, expect } = require('#playwright/test');
test('ApplicationLoadTime', async ({ page }) => {
// Wait for applications to load
await page.waitForSelector('img[alt="Application"]');
const [fir] = await Promise.all([
// Wait for the request
page.waitForEvent('requestfinished', r => r.url() == '<url of interest>'),
page.click('text=Further information requested'),
page.waitForSelector('img[alt="Application"]')
]);
console.log(fir.timing());
});

Try catch with chromless inactivity

Using chromeless to run tests every 5 minutes to verify I can log into a website and that content is available.
Code looks like this:
const { Chromeless } = require('chromeless')
async function run() {
const chromeless = new Chromeless({
remote: {
endpointUrl: 'https://abc1234567.execute-api.us-west-2.amazonaws.com/dev',
apiKey: 'abc123xyz987'
}
})
const screenshot = await chromeless
.clearCookies()
.setUserAgent('some-user-agent')
.goto('https://www.example.com')
.type('username', 'input[name="username"]')
.type('super_secret', 'input[name="password"]')
.click('#loginButton')
.wait('#TitleTagOnceLoggedIn')
.screenshot()
console.log(screenshot)
const report = await chromeless
.setUserAgent('some-user-agent')
.goto('https://www.example.com/only/accessible/if/logged/in')
.wait('#TitleTagForPageWhenLoggedIn')
.screenshot()
console.log(report)
await chromeless.end()
}
run().catch(console.error.bind(console))
This works fine, however once every 10-20 times I run it, I receive the following error
Chromeless Proxy disconnected due to inactivity (no commands sent for 30 seconds).
In the cloudwatch logs from my lambda function I get a similar error
Timing out. No requests received for 30 seconds
How can I set this to retry if I get the inactivity errors?

Convert
await chromeless.end()
to
chromeless.end().catch(callback).then(callback)
and move it into a non-async function, e.g. function end (callback) {...}.
Then have your code wait for run to finish executing and then execute end with a callback handling the result. See https://github.com/graphcool/chromeless/issues/259

Asynchronously update IndexedDB in upgrade event

In the onupgradeneeded() event for IndexedDB I am trying to update each record in an object store. In order to update them I need to first preform an async operation but this causes the upgrade transaction to become inactive and I get the error
Failed to execute 'update' on 'IDBCursor': The transaction is not active.
In the follow code I am simulating an async operation with setTimeout()
let openRequest = indexedDB.open('myDb', 1);
openRequest.onupgradeneeded = function (versionEvent) {
let db = versionEvent.target['result'];
let upgradeTransaction = versionEvent.target['transaction'];
if(versionEvent.oldVersion < 1) {
let objStore = db.createObjectStore('sample');
objStore.add('one', '1');
objStore.add('two', '2');
}
if(versionEvent.oldVersion >= 1) {
let getCursor = upgradeTransaction.objectStore('sample').openCursor();
getCursor.onsuccess = (e) => {
let cursor = e.target['result'];
if (cursor) {
setTimeout(() => {
cursor.update(cursor.value + ' updated');
cursor.continue();
})
}
}
}
};
https://plnkr.co/edit/DIIzLduZT1dwOEHAdzSf?p=preview
If you run this plunker it will initialize IndexedDB. Then if you increase the version number to 2 and run it again you will get the error.
How can I update IndexedDB in an upgrade event if my update relies on an async operation?

You need a different approach. Options include:
Making the schema changes immediately, but deferring adding new data for a subsequent transaction.
Fetch the data before trying to perform the upgrade. Since fetching the data is slow this is likely not desirable.
Conditionally fetch the data, only if an upgrade is needed.
There are two approaches for the latter. You could do an open() with no version number, check the version, and then fetch/upgrade if it is lower than desired. Or you can open at the new version and in upgradeneeded abort the upgrade (get the transaction from the request and call abort() on it) then fetch the data and re-attempt the upgrade.