Develop Reference

for loop is ignoring mongoose await(s)

I build a scraper where the scraped data gets compared with already existing data to avoid duplicates, create new entries and update old entries. I'm doing this with a for loop, which loops over a findOne function where are two awaits in. The problem is that my for loop is ignoring (because it's sync?) my awaits and goes over to a part, where it is important that all of these awaits are done.
async function comparedata(length) {
console.log("Starting comparing entries in data");
for (let x = 0; x < length; x++) {
const model = new dataModel({
link: dataLinks[x],
name: dataGames[x].replace('Download', ' '),
logo: dataLogos[x],
provider: 'data',
});
model.collection.findOne({ "link": dataLinks[x] }, async function (err, found) {
if (err) throw err;
if (found == null) {
await model.save().then((result) => {
console.log(result) // Is not happening because the for loop goes through to the next function and closes the server
}).catch((err) => { console.log(err) });
}
else if (found != null) {
if (dataGames[x] != found.name) {
await model.collection.findOneAndUpdate({ link: dataLinks[x] }, { $set: { name: dataGames[x] } });
}
}
})
}
closeServer()//Closes the server is happening before new entries or updates are made.
}
My idea was to work with promises, but even if I tried to, it was just getting resolved too fast and closes the server again.

You should be able to simplify your logic like this:
async function comparedata(length) {
console.log('Starting comparing entries in data');
try {
for (let x = 0; x < length; x++) {
let found = await dataModel.findOne({ link: dataLinks[x] });
if (!found) {
found = await dataModel.create({
link: dataLinks[x],
name: dataGames[x].replace('Download', ' '),
logo: dataLogos[x],
provider: 'data',
});
} else if (found.name !== dataGames[x]) {
found.name = dataGames[x];
await found.save();
}
console.log(found);
}
} catch (e) {
console.log(e);
}
closeServer();
}

For the first iteration of for loop, the findOne's callback is put in the callback queue by the event loop and it proceeds with the next iteration not waiting for the awaits, this goes till the last iteration and after the last iteration it immediately calls the closeServer(), after this closeServer() call the tasks put into the callback queue (i.e the findOne's callbacks) are considered and executed by the event loop. Inorder to get an understanding of this, you have to learn about the event loop and how the event loop executes the javascript code. Please check about the event loop working mechanism youtube video here
You can use the promise style execution of findOne and overcome this issue.
And according to me, using await & then() functionalities on the same statement is not a good practice.
My suggestion,
async function comparedata(length) {
console.log("Starting comparing entries in data");
for (let x = 0; x < length; x++) {
const model = new dataModel({
link: dataLinks[x],
name: dataGames[x].replace('Download', ' '),
logo: dataLogos[x],
provider: 'data',
});
// using .exec() at the end allows us to go with the promise way of dealing things rather than callbacks
// assuming that 'dataModel' is the Schema, so I directly called findOne on it
const found = await dataModel.findOne({ "link": dataLinks[x] }).exec();
if (found == null) {
// wrap the await in try...catch for catching errors while saving
try{
await model.save();
console.log("Document Saved Successfully !");
}catch(err) {
console.log(`ERROR while saving the document. DETAILS: ${model} & ERROR: ${err}`)
}
} else if (found != null) {
if (dataGames[x] != found.name) {
await model.collection.findOneAndUpdate({ link: dataLinks[x] }, { $set: { name: dataGames[x] } });
}
}
if (x > length)
sendErrorMail();
}
closeServer()//Closes the server is happening before new entries or updates are made.
}
NOTE: Please refer the Mongoose official documentation for updates.

I'm not super familiar with mongoose api, but if you are unable to use a promisified version then you can use the Promise constructor to "jump" out of a callback:
const found = await new Promise((resolve, reject) => {
model.collection.findOne({ "link": dataLinks[x] }, function (err, found) {
if (err) {
reject(err);
return;
};
resolve(found);
}
});
This might help you work things out.

Loopback findone function

I am using loopback on server side of my application , to fetch and validate a data from database I'm using findOne method which is having a callback function. I wanted to get run the callback function as soon as the findone function is executed, The code i have written is working but i want to avoid usage of async-await. Any other alternative for this?
What I tried
function validId(req) {
const filter = {
where: {
ID: req.id,
}
};
//
const result = await model.findOne(filter);
if (result) {
return true;
} else {
return false;
}
}
module.exports = function () {
return async function validateTenant(req, res, next) {
var id = false;
if (req.url.includes("XYZ")) {
id = await validId(req)
}
//
if (id || !req.url.includes("XYZ")") {
next();
} else {
res.writeHead(404, { "Content-Type": "text/html" });
var html = fs.readFileSync(
"error.html"
);
res.end(html);
}
};
};

you could use the .then() function of the promise
model.findOne(filter).then((result)=>{
//execute the rest of the function that need to be executed after the findOne.
});
// The code will continue to execute while model.findOne is doing it's thing.
But if you want to wait for the FindOne to give a result without using await or the .then it is not possible unless you make a wrapper of findOne or your BDD package have a synchrone findOne

How to return a Promise with consequential axios calls?

I need to create a function, that will return a Promise and will call another function that will have an axios.get() call. Axios.get() calls an API that returns a data with the following structure:
{
count: 87, //total number of records
next: '[api_url]/&page=2'//indication if the API has a next page
previous: null, ////indication if the API has a previous page, if call URL above - it will return [api_url]/&page=1
results: [...] //10 objects for each call, or remaining for the last page
}
Since I know that only 10 results are being returned on every call, I need to check if the returned object has the next key, and if it does - make another call and so on, until no more next. I need to concatenate all the results and eventually resolve the Promise returned from the main function with all the data.
So I tried something like that:
const fetchResource = async({type, search, page}) {
const data = {count: 0, results: []}
const request = await performFetch({type, search, page}, data).then((data) => {
console.log('data?', data)
})
console.log('req', request)
}
const performFetch = async({type, search, page}, result) => {
const params = {
page
}
if (search) {
params.search = search
}
await axios.get(`${type}/`, {
params
}).then(async({data}) => {
result.results = [...result.results, ...data.results]
result.count = data.count
if (data.next) {
page += 1
await performFetch({type, search, page}, result)
} else {
console.log('result', result)
return result
}
})
.catch((err) => {
console.error(err)
})
}
Now I see that once I call fetchResourche all the requests are going out, and in console.log('result', result) I do see the concatenated data:
{
count: 87,
results: [/*all 87 objects*/]
}
But console.log('data?', data) and console.log('req', request) both print out undefined.
Where I return result, I tried to return Promise.resolve(result) - same result.
And I'm not sure how to return a Promise here, that will resolve once all the API calls are concluded and all the data is received. What am I missing? How do I make it work?

Couple of observations regarding your code:
There's no need to mix async-await syntax with promise chaining, i.e. then() and catch() method calls
Inside performFetch function, you need an explicit return statement. Currently, the function is implicitly returning a promise that fulfils with the value of undefined.
Key point here is that the performFetch function is returning before you get the result of http requests to the API.
It isn't waiting for the result of HTTP requests to be returned before returning.
Following is a simple demo that illustrates how you can make multiple requests and aggregate the data until API has returned all the data.
let counter = 0;
function fakeAPI() {
return new Promise(resolve => {
setTimeout(() => {
if (counter < 5) resolve({ counter: counter++ });
else resolve({ done: true });
}, 1000);
});
}
async function performFetch() {
const results = [];
// keep calling the `fakeAPI` function
// until it returns "{ done = true }"
while (true) {
const result = await fakeAPI();
if (result.done) break;
else results.push(result);
}
return results;
}
performFetch().then(console.log).catch(console.log);
<small>Wait for 5 seconds</small>
Your code can be rewritten as shown below:
const fetchResource = async ({ type, search, page }) => {
const data = { count: 0, results: [] };
const result = await performFetch({ type, search, page }, data);
console.log(result);
};
const performFetch = async ({ type, search, page }, result) => {
const params = { page };
if (search) params.search = search;
while (true) {
const { data } = await axios.get(`${type}/`, { params });
result.results = [...result.results, ...data.results];
result.count = data.count;
if (data.next) page += 1;
else return result;
}
};
Ideally, the code that calls the fetchResource function should do the error handling in case any of the HTTP request fails.

how do I return values from an async each function?

I know this is a common problem for people, and I've looked up some guides and looked at the docs, but I'm not understanding what is happening, and would love some help.
Here is the controller function I am working on:
exports.appDetail = function (req, res) {
appRepo.findWithId(req.params.id, (err, myApp) => {
if (err) {
console.log(err)
}
getDeviceData(myApp.devices, (err, myDeviceData) => {
if (err) console.log(err)
console.log(JSON.stringify(myDeviceData) + ' || myDeviceData')
// construct object to be returned
let appsDataObject = {
name: myApp.name,
user_id: myApp.user_id,
devices: myDeviceData,
permissions: myApp.permissions
}
return res.status(200).send(appsDataObject)
})
})
}
// write async function here
const getDeviceData = function (devices, callback) {
let devicesDataArray = []
async.each(devices, function (device, cb) {
deviceRepo.findById(new ObjectID(device), (err, myDevice) => {
if (err) {
cb(err)
}
// get device data, push to devices array
let deviceObj = {
name: myDevice.name,
version: myDevice.version
}
devicesDataArray.push(deviceObj)
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataAray after obj push')
})
cb(null, devicesDataArray)
}, function (err) {
// if any of the file processing produced an error, err would equal that error
if (err) console.log(err)
})
callback(null, devicesDataArray)
}
I originally wrote this with a for loop and a callback, but I think it was impossible to do that way (I'm not sure about that though). If there is a better way to make an asynchronous loop, please let me know.
On ~line 8 there is a log statement myDeviceData. This should be returning the data I want through a callback, but this log statement always comes back empty. And since the other log statements show that the data is being formatted correctly, the problem must be with returning the data I need through the callback of getDeviceData(). Presumably, the callback(null, devicesDataArray) should do this.
I'm not understanding how these async functions are supposed to work, clearly. Can someone please help me understand how I should get values from these async.each functions? Thank you.
EDIT:
I refactored the code to try and make it clearer and approach the problem better, and I have pinpointed where the problem is. At the beginning the this function I define devicesDataArray as an empty array, and at the end I return the array. Everything inside happens as it should, but I don't know how to tell the return to wait until the array isn't empty, if that makes sense, Here is the new code:
let getData = async function (devices) {
let devicesDataArray = []
for (let i = 0; i < devices.length; i++) {
deviceRepo.findById(new ObjectID(devices[i]), async (err, myDevice) => {
if (err) {
console.log(err)
}
console.log(JSON.stringify(myDevice) + ' || myDevice')
let deviceObj = await {
name: myDevice.name,
version: myDevice.version
}
console.log(JSON.stringify(deviceObj) + ' || deviceObj')
await devicesDataArray.push(deviceObj)
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataArray after push')
})
}
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataArray before return')
return Promise.all(devicesDataArray) // problem is here.
}
Any help towards understanding this is appreciated.

Here we are using a Promise
The Promise object is returned immediately and we perform our async code inside there.
const someFuncThatTakesTime = () => {
// Here is our ASYNC
return new Promise((resolve, reject) => {
// In here we are synchronous again.
const myArray = [];
for(x = 0; x < 10; x++) {
myArray.push(x);
}
// Here is the callback
setTimeout(() => resolve(myArray), 3000);
});
};
someFuncThatTakesTime().then(array => console.log(array));

first, I would prefer to avoid using callback because it will make your code unreadable,
also you can handle async using javascript methods without any needs to use other modules.
the reason for your problem is your async.each function executed without waiting what it has inside
and you have here deviceRepo.findById which is async function and need to wait for it
I refactored your code using async await and I wish this will simplify the problem
exports.appDetail = function async(req, res) {
try {
const myApp = await appRepo.findWithId(req.params.id)
const myDeviceData = await getDeviceData(myApp.device)
console.log(JSON.stringify(myDeviceData) + ' || myDeviceData')
// construct object to be returned
let appsDataObject = {
name: myApp.name,
user_id: myApp.user_id,
devices: myDeviceData,
permissions: myApp.permissions
}
return res.status(200).send(appsDataObject)
} catch (err) {
console.log(err)
}
}
// write async function here
const getDeviceData = function async(devices) {
let devicesDataArrayPromises = devices.map(async(device)=>{
const myDevice= await deviceRepo.findById(new ObjectID(device))
let deviceObj = {
name: myDevice.name,
version: myDevice.version
}
return deviceObj
})
const devicesDataArray = await Promise.all(devicesDataArrayPromises)
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataAray after obj push')
return devicesDataArray
}

Return Meteor.http results in method

I have a Meteor method that wraps around an http.get. I am trying to return the results from that http.get into the method's return so that I can use the results when I call the method.
I can't make it work though.
Here's my code:
(In shared folder)
Meteor.methods({
getWeather: function(zip) {
console.log('getting weather');
var credentials = {
client_id: "string",
client_secret: "otherstring"
}
var zipcode = zip;
var weatherUrl = "http://api.aerisapi.com/places/postalcodes/" + zipcode + "?client_id=" + credentials.client_id + "&client_secret=" + credentials.client_secret;
weather = Meteor.http.get(weatherUrl, function (error, result) {
if(error) {
console.log('http get FAILED!');
}
else {
console.log('http get SUCCES');
if (result.statusCode === 200) {
console.log('Status code = 200!');
console.log(result.content);
return result.content;
}
}
});
return weather;
}
});
For some reason, this does not return the results even though they exist and the http call works: console.log(result.content); does indeed log the results.
(Client folder)
Meteor.call('getWeather', somezipcode, function(error, results) {
if (error)
return alert(error.reason);
Session.set('weatherResults', results);
});
Of course here, the session variable ends up being empty.
(Note that this part of the code seems to be fine as it returned appropriately if I hard coded the return with some dummy string in the method.)
Help?

In your example Meteor.http.get is executed asynchronously.
See docs:
HTTP.call(method, url [, options] [, asyncCallback])
On the server, this function can be run either synchronously or
asynchronously. If the callback is omitted, it runs synchronously and
the results are returned once the request completes successfully. If
the request was not successful, an error is thrown
Switch to synchronous mode by removing asyncCallback:
try {
var result = HTTP.get( weatherUrl );
var weather = result.content;
} catch(e) {
console.log( "Cannot get weather data...", e );
}

Kuba Wyrobek is correct, but you can also still call HTTP.get asynchronously and use a future to stop the method returning until the get has responded:
var Future = Npm.require('fibers/future');
Meteor.methods({
getWeather: function(zip) {
console.log('getting weather');
var weather = new Future();
var credentials = {
client_id: "string",
client_secret: "otherstring"
}
var zipcode = zip;
var weatherUrl = "http://api.aerisapi.com/places/postalcodes/" + zipcode + "?client_id=" + credentials.client_id + "&client_secret=" + credentials.client_secret;
HTTP.get(weatherUrl, function (error, result) {
if(error) {
console.log('http get FAILED!');
weather.throw(error);
}
else {
console.log('http get SUCCES');
if (result.statusCode === 200) {
console.log('Status code = 200!');
console.log(result.content);
weather.return(result);
}
}
});
weather.wait();
}
});
There's not really much advantage to this method over a synchronous get in this case, but if you're ever doing something on the server which can benefit from something like an HTTP call running asynchronously (and thus not blocking the rest of the code in your method), but you still needs to wait for that call to return before the method can, then this is the right solution. One example would be where you need to execute multiple non-contingent gets, which would all have to wait for each other to return one by one if executed synchronously.
More here.

Sometimes asynchronous calls are preferable. You can use async/await syntax for that, and you need to promisify HTTP.get.
import { Meteor } from 'meteor/meteor';
import { HTTP } from 'meteor/http';
const httpGetAsync = (url, options) =>
new Promise((resolve, reject) => {
HTTP.get(url, options, (err, result) => {
if (err) {
reject(err);
} else {
resolve(result);
}
});
});
Meteor.methods({
async 'test'({ url, options }) {
try {
const response = await httpGetAsync(url, options);
return response;
} catch (ex) {
throw new Meteor.Error('some-error', 'An error has happened');
}
},
});
Notice that meteor test method is marked as async. This allows using await operator inside it with method calls which return Promise. Code lines following await operators won't be executed until returned promise is resolved. In case the promise is rejected catch block will be executed.