Develop Reference

synchronous fetch - unexpected results

First of all, I'm aware this is not a good approach, need it as temporary solution for certain functions to return value, not promise. I know it's really not good permanent solution at all, but I need it for now.
What worries me, fetch sure finishes sooner - but it runs until the whiles times out, and then to console comes first the RETVAL false, and only then second line comes RETFETCH: {....} with returned json values - it seems the 'haveResponse' value does not change in the second 'then' - but can't see why, and how to bypass it.
It's a temporary workaround for old sync fns to read some data from remote service running on local pc on some port, but for now I can't rewrite the function which expects to receive data from this fn, so there must be no promise on the outside, need to wait for response and then return it.
function syncFetch(url) {
var haveResponse = false;
var reqtime = new Date();
try{
fetch(url, {
headers: {'Content-Type': 'application/json'},
method: 'POST',
timeout: 1500,
body: JSON.stringify({cmd:'init'})
})
.then(response => response.json())
.then(data => {
console.log('RETFETCH:', data);
haveResponse = data;
return data;
});
// timeout
while (haveResponse === false) {
var endDate = new Date();
if (haveResponse !== false) { return haveResponse; }
if ((endDate - reqtime)/1000 > 5) { // max 5 sec
return haveResponse;
}
}
return haveResponse;
} catch(e){
console.log('error', e);
haveResponse = -1;
}
return haveResponse;
}
console.log('RETVAL',syncFetch('http://127.0.0.1:3333/'));

Save yourself a few headaches, drop all the .then() and use the async/await syntax instead. No need for dirty timeout/while hacks.
I renamed syncFetch to asyncFetch, because your original code was never synchronous in the first place (and that's precisely why you are struggling so much, you believe it is when it's not). async/await don't make the code synchronous either. It's just awesome syntactic sugar around Promises.
(EDIT : you said in the comments that you can't add the async keyword to the parent function (asyncFetch), so here's a workaround to place it inside :
function asyncFetch(url) {
async function a() {
try {
const response = fetch(url, {
headers: { 'Content-Type': 'application/json' },
method: 'POST',
timeout: 1500,
body: JSON.stringify({ cmd: 'init' })
});
const data = await response.json();
return data; // This is a Promise<data>, not data directly, so it needs to be awaited
} catch (e) {
console.log('error', e);
return null
}
};
return a();
};
(async () => {
console.log('RETVAL', await asyncFetch('http://127.0.0.1:3333/')); // Needs to be awaited, and therefore also needs to be inside an async function
})();

Multiple paginated GET API calls in parallel/async in Node

I am making call to the bitbucket API to get all the files that are in a repo. I have reached to a point where I can get the list of all the folders in the repo and make the first API call to all the root folders in the repo in parallel and get the the list of first 1000 files for all folders.
But the problem is bitbucket api can give me only 1000 files per folder at a time.
I need to append a query param &start =nextPageStart and make the call again, until it is null and isLastPage is true per API. How can I achieve that with below code??
I get the nextPageStart from first call to the api. See the API response below.
Below is the code that I have so far.
Any help or guidance is appreciated.
Response from individual API thats called per folder.
{
"values": [
"/src/js/abc.js",
"/src/js/efg.js",
"/src/js/ffg.js",
...
],
"size": 1000,
"isLastPage": false,
"start": 0,
"limit": 1000,
"nextPageStart": 1000
}
function where i made asynchronous calls to get the list of files
export function getFilesList() {
const foldersURL: any[] = [];
getFoldersFromRepo().then((response) => {
const values = response.values;
values.forEach((value: any) => {
//creating API URL for each folder in the repo
const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'
+ value.project.key + '/repos/' + value.slug + '/files?limit=1000';
foldersURL.push(URL);
});
return foldersURL;
}).then((res) => {
// console.log('Calling all the URLS in parallel');
async.map(res, (link, callback) => {
const options = {
url: link,
auth: {
password: 'password',
username: 'username',
},
};
request(options, (error, response, body) => {
// TODO: How do I make the get call again so that i can paginate and append the response to the body till the last page.
callback(error, body);
});
}, (err, results) => {
console.log('In err, results function');
if (err) {
return console.log(err);
}
//Consolidated results after all API calls.
console.log('results', results);
});
})
.catch((error) => error);
}

I was able to get it working be creating a function with callback.
export function getFilesList() {
const foldersURL: any[] = [];
getFoldersFromRepo().then((response) => {
const values = response.values;
values.forEach((value: any) => {
//creating API URL for each folder in the repo
const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'
+ value.project.key + '/repos/' + value.slug + '/files?limit=1000';
foldersURL.push(URL);
});
return foldersURL;
}).then((res) => {
// console.log('Calling all the URLS in parallel');
async.map(res, (link, callback) => {
const options = {
url: link,
auth: {
password: 'password',
username: 'username',
},
};
const myarray = [];
// This function will consolidate response till the last Page per API.
consolidatePaginatedResponse(options, link, myarray, callback);
}, (err, results) => {
console.log('In err, results function');
if (err) {
return console.log(err);
}
//Consolidated results after all API calls.
console.log('results', results);
});
})
.catch((error) => error);
}
function consolidatePaginatedResponse(options, link, myarray, callback) {
request(options, (error, response, body) => {
const content = JSON.parse(body);
content.link = options.url;
myarray.push(content);
if (content.isLastPage === false) {
options.url = link + '&start=' + content.nextPageStart;
consolidatePaginatedResponse(options, link, myarray, callback);
} else {
// Final response after consolidation per API
callback(error, JSON.stringify(myarray));
}
});
}

I think the best way is to wrap it in a old school for loop (forEach doesn't work with async, since it's synchronous and it will cause all the requests to be spawn at the same time).
What I understood is that you do some sort of booting query where you get the values array and then you should iterate among the pages. Here some code, I didn't fully grasp the APIs so I'll give a simplified (and hopefully readable) answer, you should be able to adapt it:
export async function getFilesList() {
logger.info(`Fetching all the available values ...`);
await getFoldersFromRepo().then( async values => {
logger.info("... Folders values fetched.");
for (let i = 0; ; i++ ) {
logger.info( `Working on page ${i}`);
try {
// if you are using TypeScript, the result is not the promise but the succeeded value already
const pageResult: PageResult = await yourPagePromise(i);
if (pageResult.isLastPage) {
break;
}
} catch(err) {
console.err(`Error on page ${i}`, err);
break;
}
}
logger.info("Done.");
});
logger.info(`All finished!`);
}
The logic behind is that first getFoldersFromRepo() returns a promise which returns the values, and then I sequentially iterate on all available pages through the yourPagePromise function (which returns a promise). The async/await construct allows to write more readable code, rather then having a waterfall of then().
I'm not sure it respects your APIs specs, but it's the logic you can use as foundation! ^^

How to raise a Timeout Error in node.js if code takes long time to finish?

in ruby I can:
require 'timeout'
Timeout.timeout 10 do
# do smth > 10 seconds
end
it will raise timeout error to avoid code lock, how to do same thing in nodejs, nodejs #setTimeout doesn't fit my need
one case is, when i http.get timeout(for ex, netowrk is unstable), I should set timeout and handle the failed get request, I hope impl #timeout, how should i do?
try {
timeout(10, function () {
http.get("example.com/prpr")
})
} catch (e) {
if (e.message == "timeout") {
// do smth
} else {
throw e
}
}

You could look into a Promise-based approach here.
Using promises you can pass a function to be executed, and then the standard catch is called if that function raises an exception.
There is a helpful promise-based timeout library on NPM (npm install promise-timeout request-promise), and you could use it in Node something along the lines of...
'use strict';
var promiseTimeout = require('promise-timeout');
var requestPromise = require('request-promise');
promiseTimeout.timeout(requestPromise("http://example.com/prpr"), 10000)
.then(function (result) {
console.log({result});
}).catch(function (err) {
if (err instanceof pt.TimeoutError) {
console.error('HTTP get timed out');
}
});

I had a similar situation with nestJS based on node.js.
When calling an external API, it was a problem that even my service slowed down if it took too long. (If the external api is delayed, my service also had a problem of waiting forever.)
I figured out 2 ways.
First way:
const result = await axios({
timeout: 10000, // error: [AxiosError: timeout of 10000ms exceeded] { code: 'ECONNABORTED', ...
...
});
Second way: Promise.race()
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/race
// first function
const callAPI = axios({
method: "GET",
url: "http://yourapi",
headers: {
...
}
});
// second function
const timeoutCheck = (s) => {
return new Promise(resolve => setTimeout(resolve, s));
}
// check delay (first function VS second function)
const result = await Promise.race([
callAPI,
timeoutCheck(10000).then(() => {
throw new Error("api not responding for more than 10 seconds");
}),
]);
const { data: { resultCode, resultData } } = result;

You can try this out in your case:
var request = http.get(options, function (res) {
// other code goes here
});
request.setTimeout( 10000, function( ) {
// handle timeout here
});

Reactjs - ordering if else and outer block

I have a react component which has the following functions:
alreadyUpvoted() {
return this.state.upvotes.indexOf(this.props.context.userId) !== -1
}
alreadyDownvoted() {
return this.state.downvotes.indexOf(this.props.context.userId) !== -1
}
addUpvote() {
this.setState(prev => ({
upvotes: prev.upvotes.concat(this.props.context.userId),
upvoted: true,
votes: prev.votes + 1,
}), () => {
console.log('add upvote', this.state)
})
}
removeUpvote() {
var new_upvotes = this.state.upvotes.concat()
new_upvotes.pop(this.props.context.userId)
this.setState(prev => ({
upvotes: new_upvotes,
upvoted: false,
votes: prev.votes - 1,
}), () => {
console.log('remove upvote', this.state)
})
}
addDownvote() {
this.setState(prev => ({
downvotes: prev.downvotes.concat(this.props.context.userId),
downvoted: true,
votes: prev.votes - 1,
}), () => {
console.log('add dowvote', this.state)
})
}
removeDownvote() {
var new_downvotes = this.state.downvotes.concat()
new_downvotes.pop(this.props.context.userId)
this.setState(prev => ({
downvotes: new_downvotes,
downvoted: false,
votes: prev.votes + 1,
}), () => {
console.log('remove downvote', this.state)
})
}
postVotesData() {
var json = {
upvotes: this.state.upvotes,
downvotes: this.state.downvotes
}
json = JSON.stringify(json)
console.log(json)
const url = `/api/reddit/r/${this.props.subreddit}/posts/${this.props.postid}/`
fetch(url, {
method: 'PUT',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: json
})
.then(response => {
console.log('response status:', response)
return response.json()
})
.then(res => console.log('response data:', res))
}
I have created a toggleUpvote function which utilizes all the above functions:
toggleUpvote() {
if (this.alreadyDownvoted()) {
this.removeDownvote()
this.addUpvote()
}
else if (this.alreadyUpvoted()) {
this.removeUpvote()
}
else {
this.addUpvote()
}
this.postVotesData()
}
The problem here is the this.postVotesData() is getting executed before the if-else block finished.
For the current state, the control should go to else block. But, The console.log in this.postVotesData() got executed before the console.log present in addUpvote!!
evidence:
At the current state, upvotes array should have one value after executing of else and that array should be used in PUT. But, empty array is being PUT and then value gets added to array.
I also want the functions inside if block to be executed in order. How can I solve this?

That is how asynchronous javascript works. I believe you are doing an asynchronous operation like a fetch call, or react's setState(). Both of them(and many other things) in js are async.
To deal with this, you need to use callbacks or promises. When using promises, you can use the async-await syntax to have a clean code.
async toggleUpvote() {
if (this.alreadyDownvoted()) {
await this.removeDownvote()
await this.addUpvote()
}
else if (this.alreadyUpvoted()) {
await this.removeUpvote()
}
else {
await this.addUpvote()
}
await this.postVotesData()
}
To do this, removeUpvote, addUpvote and postVotesData need to return promises.
If you calling the setState in these functions, you need to provide a callback to setState. Or else you can use the functional-setState pattern.
A simple fix to avoid this problem is to return a promise from your functions similar to the following, and use the async-await syntax as suggested above.
addUpvote() {
return new Promise((resolve, reject) => {
this.setState(prev => ({
upvotes: prev.upvotes.concat(this.props.context.userId),
upvoted: true,
votes: prev.votes + 1,
}), () => {
console.log('add upvote', this.state)
return resolve(); // Signal that the operation has finished
})
})
}

It may appear like that, but it does not get excecuted before the block is done. All these functions are started in that exact order (synchrone), depending on which if/else route.
The reason you may think that, is because these functions are probably async, like an AJAX request. Those work with promises and/or on-complete functions. You might want to look up some tutorials about that, as it's to broad to explain in a simple answer here.
Simply put: With a promise you tell javascript to wait for the result before continueing.
edit: I noticed the react tag, you might want to check out await. It might take a few reads to understand what is going one, but worth the research.

Synchronously call a REST API in JavaScript

I am new to JavaScript and the npm world. I try to upload some data to my REST service via a REST post call. These data I fetch from a csv file. So far so good. On each fetched line I convert the data (for my needs) and call the REST API for uploading those. Since I have many line (approx. 700) the API gets called quite often consecutively. After some calls (guess 500 or so) I get an Socket error
events.js:136
throw er; // Unhandled 'error' event
^
Error: connect ECONNRESET 127.0.0.1:3000
at Object._errnoException (util.js:999:13)
at _exceptionWithHostPort (util.js:1020:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1207:14)
I guess this is because I call the REST API to often. What I don't understand is:
How should I make the call synchronously in order to avoid so many connections?
Or should't I?
What would be the proper solution in JS for this?
I have tried with Promises and so on but all this didn't helped but moved the issue some function calls priorly...
This is my code:
readCsv()
function readCsv() {
var csvFile = csvFiles.pop()
if (csvFile) {
csv({ delimiter: ";" }).fromFile(csvFile).on('json', async (csvRow) => {
if (/.*\(NX\)|.*\(NI\)|.*\(NA\)|.*\(WE\)|.*\(RA\)|.*\(MX\)/.test(csvRow["Produkt"])) {
var data = await addCallLog(
csvRow["Datum"],
csvRow["Zeit"],
csvRow["Menge-Zeit"],
csvRow["Zielrufnummer"],
csvRow["Produkt"]);
}
}).on('done', (error) => {
//console.log('end')
readCsv()
})
} else {
}
}
function addCallLog(date, time, duration, number, product) {
return new Promise(resolve => {
args.data = { number: number, name: "", timestamp: getTimestamp(date, time), duration: getDuration(duration), type: "OUTGOING" }
client.methods.addCallLog(args, (data, response) => {
// client.methods.getCallLog((data, response) => {
// console.log(data)
// })
//console.log("addCallLog resolve")
resolve(data)
})
})
}
As you can see I had the same issue with reading more than one csv files in parallel. I solved this by calling recursively the readCsv function and pop the next file after the other when the file read was done.

You can't call things synchronously. But, you can sequence the async REST calls which is what I presume you mean.
A problem here is that await addCallLog() won't keep the next json events from being generated so you will end with a zillion requests in flight at the same time and apparently you have so many that you run out of resources.
One way around that is to collect the rows you want into an array and then use a regular for loop to iterate that array and you can use await sucessfully in the for loop. Here's what that would look like:
readCsv()
function readCsv() {
var csvFile = csvFiles.pop()
if (csvFile) {
let rows = [];
csv({ delimiter: ";" }).fromFile(csvFile).on('json', (csvRow) => {
if (/.*\(NX\)|.*\(NI\)|.*\(NA\)|.*\(WE\)|.*\(RA\)|.*\(MX\)/.test(csvRow["Produkt"])) {
rows.push(csvRow);
}
}).on('done', async (error) => {
for (let csvRow of rows) {
var data = await addCallLog(
csvRow["Datum"],
csvRow["Zeit"],
csvRow["Menge-Zeit"],
csvRow["Zielrufnummer"],
csvRow["Produkt"]
);
}
readCsv();
})
} else {
}
}
function addCallLog(date, time, duration, number, product) {
return new Promise(resolve => {
args.data = { number: number, name: "", timestamp: getTimestamp(date, time), duration: getDuration(duration), type: "OUTGOING" }
client.methods.addCallLog(args, (data, response) => {
// client.methods.getCallLog((data, response) => {
// console.log(data)
// })
//console.log("addCallLog resolve")
resolve(data)
})
})
}
Your coding appears to be missing error handling. The client.methods.addCallLog() needs a way to communicate back an error.
You probably also need a error event handler for the csv iterator.

After filling the buffer in a prev. function I check that buffer for data and upload those one by one using the "then" callback of the promise
var callLogBuffer = []
checkForUpload()
function checkForUpload() {
console.log("checkForUpload")
if (callLogBuffer.length > 0) {
addCallLog(callLogBuffer.pop()).then((data) => {
checkForUpload()
})
}
}
function addCallLog(callLog) {
return new Promise(resolve => {
args.data = { number: callLog.number, name: "", timestamp: getTimestamp(callLog.date, callLog.time), duration: getDuration(callLog.duration), type: "OUTGOING" }
client.methods.addCallLog(args, (data, response) => {
// client.methods.getCallLog((data, response) => {
// console.log(data)
// })
//console.log("addCallLog resolve")
resolve(data)
})
})
}