Develop Reference

How to send bulk get requests using nodejs?

I wrote a web crawler with nodejs to send get requests to about 300 urls.
Here is the main loop:
for (let i = 1; i <= 300; i++) {
let page= `https://xxxxxxxxx/forum-103-${i}.html`
await getPage(page,(arr)=>{
console.log(`page ${i}`)
})
}
Here is the function getPage(url,callback):
export default async function getPage(url, callback) {
await https.get(url, (res) => {
let html = ""
res.on("data", data => {
html += data
})
res.on("end", () => {
const $ = cheerio.load(html)
let obj = {}
let arr = []
obj = $("#threadlisttableid tbody")
for (let i in obj) {
if (obj[i].attribs?.id?.substr(0, 6) === 'normal') {
arr.push(`https://xxxxxxx/${obj[i].attribs.id.substr(6).split("_").join("-")}-1-1.html`)
}
}
callback(arr)
console.log("success!")
})
})
.on('error', (e) => {
console.log(`Got error: ${e.message}`);
})
}
I use cheerio to analyze HTML and put all information i need to variable nameed 'arr'.
The program will report an error after running normally for a period of time,like that:
...
success!
page 121
success!
page 113
success!
page 115
success!
Got error: connect ETIMEDOUT 172.67.139.206:443
Got error: connect ETIMEDOUT 172.67.139.206:443
Got error: connect ETIMEDOUT 172.67.139.206:443
Got error: connect ETIMEDOUT 172.67.139.206:443
Got error: connect ETIMEDOUT 172.67.139.206:443
Got error: connect ETIMEDOUT 172.67.139.206:443
I have two questions:
1.What is the reason for the error? Is it because I am sending too many get requests? How can I limit the request frequency?
2.As you can see, The order in which the pages are accessed is chaotic,how to control them?
I have tried using other modules to send get request (such as Axios) but it didn't work.

The http requests are fired simultaneously because the loop is not waiting for the previous request due to wrong use of await. Proper control of loop will limit the request frequency.
for (let i = 1; i <= 300; i++) {
let page= `https://xxxxxxxxx/forum-103-${i}.html`
var arr = await getPage(page);
// use arr in the way you want
console.log(`page ${i}`);
}
export default async function getPage(url) {
// Declare a new promise, wait for the promise to resolve and return its value.
return await new Promise((reso, rej) => {
https.get(url, (res) => {
let html = ""
res.on("data", data => {
html += data
})
res.on("end", () => {
const $ = cheerio.load(html)
let obj = {}
let arr = []
obj = $("#threadlisttableid tbody")
for (let i in obj) {
if (obj[i].attribs?.id?.substr(0, 6) === 'normal') {
arr.push(`https://xxxxxxx/${obj[i].attribs.id.substr(6).split("_").join("-")}-1-1.html`)
}
}
reso(arr) // Resolve with arr
console.log("success!")
})
})
.on('error', (e) => {
console.log(`Got error: ${e.message}`);
throw e;
})
})
}

As you can see, The order in which the pages are accessed is chaotic,how to control them?
await is meaningless unless you put a promise on the right hand side. http.get does not deal in promises.
You could wrap it in a promise but it would be easier to use an API which supports then natively such as node-fetch, axios, or Node.js's native fetch. (That all have APIs that are, IMO, easier to use than http.get in general nor just with regards to flow control).
What is the reason for the error?
It isn't clear.
Is it because I am sending too many get requests?
That is a likely hypothesis.
How can I limit the request frequency?
Once you have your for loop working with promises so the requests are sent in serial instead of parallel, you can insert a sleep between each request.

Error "Given transaction number * does not match" in mongodb and nodejs

I want to modify two schema while adding data. For that I used ACID transaction of mongodb with nodejs as follow. But, when I run program it displays the error like
(node:171072) UnhandledPromiseRejectionWarning: MongoError: Given transaction number 3 does not match any in-progress transactions. The active transaction number is 2
at MessageStream.messageHandler (/home/user/Projects/project/node_modules/mongodb/lib/cmap/connection.js:272:20)
at MessageStream.emit (events.js:375:28)
at MessageStream.emit (domain.js:470:12)
addData = async(request: Request, response: Response) => {
const session = await stockSchema.startSession()
try {
const userData = request.body
let data = {}
const transaction = await session.withTransaction(async() => {
try {
userData.products.map(async(item: any) => {
await inventorySchema.findOneAndUpdate({ _id: item.materialID }, { $inc: {qty: -item.qty }}, { session });
});
data = new stockSchema(userData);
await data.save({ session });
} catch (error) {
await session.abortTransaction()
throw new Error("Could not create data. Try again.");
}
});
if (transaction) {
session.endSession()
return returnData(response, data, 'Data created successfully.');
} else {
throw new Error("Could not create data. Try again.");
}
} catch (error: any) {
session.endSession();
return Error(response, error.message, {});
}
}

So you might have figured out the answer to this already, but anyway, after months of frustration, and no clear answer on the internet, i finally figured it out.
The problem with your code above is that you are passing session into a database operation (the .findOneAndUpdate function above) that is running within .map . Meaning, your 'transaction session' is being used concurrently, which is what is causing the error. read this: https://www.mongodb.com/community/forums/t/concurrency-in-mongodb-transactions/14945 (it explains why concurrency with transactions creates a bit of a mess.)
Anyway, instead of .map, use a recursive function that fires each DB operation one after another rather than concurrently, and all your problems will be solved.
You could use a function something like this:
const executeInQueue = async ({
dataAry, //the array that you .map through
callback, //the function that you fire inside .map
idx = 0,
results = [],
}) => {
if (idx === dataAry.length) return results;
//else if idx !== dataAry.length
let d = dataAry[idx];
try {
let result = await callback(d, idx);
results.push(result);
return executeInQueue({
dataAry,
callback,
log,
idx: idx + 1,
results,
});
} catch (err) {
console.log({ err });
return results.push("error");
}
};

Unable to run a loop to update Object Array in SQLite with React Native

So this has been troubling me for a while, I have an array of objects that I want to insert into my SQLite DB. Each of the objects have 5 parameters and I have the SQL Query in place to run it. I was using a loop to iterate through the array and populate each of the objects via db transactions to SQLite. However, the db tasks are asynchronous which leads to the loop being completed before the task is run and incorrect data being populated into the db. The while loop in the code below doesn't work and I have tried the same thing with a for loop to no avail.
var i=0;
while(i<rawData.length){
console.log(rawData[i],i)
db.transaction(function (tx) {
console.log(rawData,i," YAY")
tx.executeSql(
'Update all_passwords SET title=?,userID=?,password=?,notes=?,category=? WHERE ID =? ',
[rawData[i].title,rawData[i].userID,rawData[i].password,rawData[i].notes,rawData[i].category,rawData[i].id],
(tx, results) => {
console.log("saved all data")
tx.executeSql(
"SELECT * FROM all_passwords ORDER BY id desc",
[],
function (tx, res) {
i++
console.log("Print Out Correct Data")
for(var i=0;i<res.rows.length;i++){
console.log(res.rows.item(i), i )
}
});
}
);
console.log("EXIT")
}
,
(error) => {
console.log(error);
}
);
}
I'm not familiar using async tasks with hooks but I believe that might be a potential solution. My intention is to populate the rawaData array of objects into the SQLDb in one go while I use a state to maintain the loading screen.
I did refer the below sources but wasn't able to come up with anything concrete.
react native insertion of array values using react-native-sqlite-storage
https://medium.com/javascript-in-plain-english/how-to-use-async-function-in-react-hook-useeffect-typescript-js-6204a788a435
Thanks in advance!

I made a little write up for you on how I would solve it. Read the comments in the code. If anything is unclear feel free to ask!
const rawData = [
{ title: "title", userID: "userID", password: "password", notes: "notes", category: "category", id: "id" },
{ title: "title_1", userID: "userID_1", password: "password_1", notes: "notes_1", category: "category_1", id: "id_1" },
{ title: "title_2", userID: "userID_2", password: "password_2", notes: "notes_2", category: "category_2", id: "id_2" }
];
// You can mostly ignore this. It's just a mock for the db
const db = {
tx: {
// AFAIK if there is a transaction it's possible to execute multiple statements
executeSql: function(sql, params, success, error) {
// just for simulating an error
if (params.title === "title_2") {
error(new Error("Some sql error"));
} else {
console.log(sql, params.title);
success();
}
}
},
transaction: function(tx, error) {
// simulating async
setTimeout(() => {
return tx(this.tx);
}, parseInt(Math.random() * 1000));
}
}
// Lets make a class which handles our dataccess in an async way
class DataAccess {
// as transaction has callback functions it's wrapped in a promise
// on success the transaction is resolved
// if there is an error it will be thrown
transaction = () => {
return new Promise(resolve => {
db.transaction(tx => resolve(tx), error => {
throw error;
});
});
}
// the actual executeSql function which "hides" all the transaction stuff
// awaits a transaction and executes the sql on it
// if the execution was successfull resolve
// if not throw the error
executeSql = async(sql, params) => {
const tx = await this.transaction();
tx.executeSql(sql, params, () => Promise.resolve(), error => {
throw error;
});
}
}
const dal = new DataAccess();
// all sql execute tha was possible
async function insert_with_execute() {
// promise all does not guarantee execution order
// but it is a possibility to await an array of promises (async functions)
await Promise.all(rawData.map(async rd => {
try {
await dal.executeSql("sql_execute", rd);
} catch (error) {
console.log(error.message);
}
}));
}
// no sql executed cause of error and all in the same transaction
async function insert_with_transaction() {
const tx = await dal.transaction();
for (let i = 0; i < rawData.length; i++) {
tx.executeSql("sql_transaction", rawData[i], () => console.log("success"), error => console.log(error.message));
}
}
async function test() {
await insert_with_execute();
console.log("---------------------------------")
await insert_with_transaction();
}
test();

Apparently the best approach to take is using anonymous functions that create a separate instance of execution for each value of i. This is a good example of how to do it....
Javascript SQL Insert Loop

Multiple paginated GET API calls in parallel/async in Node

I am making call to the bitbucket API to get all the files that are in a repo. I have reached to a point where I can get the list of all the folders in the repo and make the first API call to all the root folders in the repo in parallel and get the the list of first 1000 files for all folders.
But the problem is bitbucket api can give me only 1000 files per folder at a time.
I need to append a query param &start =nextPageStart and make the call again, until it is null and isLastPage is true per API. How can I achieve that with below code??
I get the nextPageStart from first call to the api. See the API response below.
Below is the code that I have so far.
Any help or guidance is appreciated.
Response from individual API thats called per folder.
{
"values": [
"/src/js/abc.js",
"/src/js/efg.js",
"/src/js/ffg.js",
...
],
"size": 1000,
"isLastPage": false,
"start": 0,
"limit": 1000,
"nextPageStart": 1000
}
function where i made asynchronous calls to get the list of files
export function getFilesList() {
const foldersURL: any[] = [];
getFoldersFromRepo().then((response) => {
const values = response.values;
values.forEach((value: any) => {
//creating API URL for each folder in the repo
const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'
+ value.project.key + '/repos/' + value.slug + '/files?limit=1000';
foldersURL.push(URL);
});
return foldersURL;
}).then((res) => {
// console.log('Calling all the URLS in parallel');
async.map(res, (link, callback) => {
const options = {
url: link,
auth: {
password: 'password',
username: 'username',
},
};
request(options, (error, response, body) => {
// TODO: How do I make the get call again so that i can paginate and append the response to the body till the last page.
callback(error, body);
});
}, (err, results) => {
console.log('In err, results function');
if (err) {
return console.log(err);
}
//Consolidated results after all API calls.
console.log('results', results);
});
})
.catch((error) => error);
}

I was able to get it working be creating a function with callback.
export function getFilesList() {
const foldersURL: any[] = [];
getFoldersFromRepo().then((response) => {
const values = response.values;
values.forEach((value: any) => {
//creating API URL for each folder in the repo
const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'
+ value.project.key + '/repos/' + value.slug + '/files?limit=1000';
foldersURL.push(URL);
});
return foldersURL;
}).then((res) => {
// console.log('Calling all the URLS in parallel');
async.map(res, (link, callback) => {
const options = {
url: link,
auth: {
password: 'password',
username: 'username',
},
};
const myarray = [];
// This function will consolidate response till the last Page per API.
consolidatePaginatedResponse(options, link, myarray, callback);
}, (err, results) => {
console.log('In err, results function');
if (err) {
return console.log(err);
}
//Consolidated results after all API calls.
console.log('results', results);
});
})
.catch((error) => error);
}
function consolidatePaginatedResponse(options, link, myarray, callback) {
request(options, (error, response, body) => {
const content = JSON.parse(body);
content.link = options.url;
myarray.push(content);
if (content.isLastPage === false) {
options.url = link + '&start=' + content.nextPageStart;
consolidatePaginatedResponse(options, link, myarray, callback);
} else {
// Final response after consolidation per API
callback(error, JSON.stringify(myarray));
}
});
}

I think the best way is to wrap it in a old school for loop (forEach doesn't work with async, since it's synchronous and it will cause all the requests to be spawn at the same time).
What I understood is that you do some sort of booting query where you get the values array and then you should iterate among the pages. Here some code, I didn't fully grasp the APIs so I'll give a simplified (and hopefully readable) answer, you should be able to adapt it:
export async function getFilesList() {
logger.info(`Fetching all the available values ...`);
await getFoldersFromRepo().then( async values => {
logger.info("... Folders values fetched.");
for (let i = 0; ; i++ ) {
logger.info( `Working on page ${i}`);
try {
// if you are using TypeScript, the result is not the promise but the succeeded value already
const pageResult: PageResult = await yourPagePromise(i);
if (pageResult.isLastPage) {
break;
}
} catch(err) {
console.err(`Error on page ${i}`, err);
break;
}
}
logger.info("Done.");
});
logger.info(`All finished!`);
}
The logic behind is that first getFoldersFromRepo() returns a promise which returns the values, and then I sequentially iterate on all available pages through the yourPagePromise function (which returns a promise). The async/await construct allows to write more readable code, rather then having a waterfall of then().
I'm not sure it respects your APIs specs, but it's the logic you can use as foundation! ^^

How to raise a Timeout Error in node.js if code takes long time to finish?

in ruby I can:
require 'timeout'
Timeout.timeout 10 do
# do smth > 10 seconds
end
it will raise timeout error to avoid code lock, how to do same thing in nodejs, nodejs #setTimeout doesn't fit my need
one case is, when i http.get timeout(for ex, netowrk is unstable), I should set timeout and handle the failed get request, I hope impl #timeout, how should i do?
try {
timeout(10, function () {
http.get("example.com/prpr")
})
} catch (e) {
if (e.message == "timeout") {
// do smth
} else {
throw e
}
}

You could look into a Promise-based approach here.
Using promises you can pass a function to be executed, and then the standard catch is called if that function raises an exception.
There is a helpful promise-based timeout library on NPM (npm install promise-timeout request-promise), and you could use it in Node something along the lines of...
'use strict';
var promiseTimeout = require('promise-timeout');
var requestPromise = require('request-promise');
promiseTimeout.timeout(requestPromise("http://example.com/prpr"), 10000)
.then(function (result) {
console.log({result});
}).catch(function (err) {
if (err instanceof pt.TimeoutError) {
console.error('HTTP get timed out');
}
});

I had a similar situation with nestJS based on node.js.
When calling an external API, it was a problem that even my service slowed down if it took too long. (If the external api is delayed, my service also had a problem of waiting forever.)
I figured out 2 ways.
First way:
const result = await axios({
timeout: 10000, // error: [AxiosError: timeout of 10000ms exceeded] { code: 'ECONNABORTED', ...
...
});
Second way: Promise.race()
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/race
// first function
const callAPI = axios({
method: "GET",
url: "http://yourapi",
headers: {
...
}
});
// second function
const timeoutCheck = (s) => {
return new Promise(resolve => setTimeout(resolve, s));
}
// check delay (first function VS second function)
const result = await Promise.race([
callAPI,
timeoutCheck(10000).then(() => {
throw new Error("api not responding for more than 10 seconds");
}),
]);
const { data: { resultCode, resultData } } = result;

You can try this out in your case:
var request = http.get(options, function (res) {
// other code goes here
});
request.setTimeout( 10000, function( ) {
// handle timeout here
});