I have a JSON file from Spotify with objects that represent artists. Inside of those objects, one of the properties is 'genres' that is an array of strings (genres).
What I'm trying to do is find or create those genres in MongoDB. Then add those object IDs into an array and pass that array when I create my Artist objects.
But, my Artists get created in the DB way before that find or create process for their genres is done.
Thus the line fillRest runs before fillGenres.
I've tried changing from Object.values.forEach to for..of, for..in, different asyncs, promises, modulating code...
Basically added await everywhere I can (most in the wrong places probably)
import * as data from '../spotify_data/artist_id.json';
async function fillGenres(array) {
const genreIDs = []; // array of genre object IDs I'm trying to put inside every appropriate artist
if (array !== 'undefined') {
for (const element of array) {
await Genre.findOrCreate({ name: element }, (err, result) => {
genreIDs.push(result._id);
});
}
}
return genreIDs;
}
async function fillRest(entry, genreIDs) {
const artist = {
name: entry.ArtistName,
genres: genreIDs,
spotifyID: entry.ArtistID,
popularity: entry.Popularity,
spotifyFollowers: entry.Followers,
};
Artist.create([artist])
.then((result) => {
console.log(result);
})
.catch((error) => {
console.log(error);
});
}
async function spotifySeed() {
const entries = Object.values(data);
for (const entry of entries) {
fillGenres(entry.Genres)
.then((genreIDs) => {
fillRest(entry, genreIDs); // this line gets executed before fillGenres above^ which is super weird
});
}
}
spotifySeed();
Artists get added to MongoDB with genres set as [].
After that I get console output with good genreID arrays (that were supposed to be inside there^ instead of genres).
Resolved - EDIT
Thanks to everyone who helped. The problem was in findOrCreate as it did not return a promise. I used this package for mongoose instead that has Promises (https://www.npmjs.com/package/mongoose-findorcreate).
And the code now is
if (Array.isArray(array)) {
// eslint-disable-next-line no-restricted-syntax
for (const element of array) {
await Genre.findOrCreate({ name: element })
.then((result) => {
genreIDs.push(result.doc._id);
});
}
}
and in SpotifySeed
const genreIDs = await fillGenres(entry.Genres);
await fillRest(entry, genreIDs);
I haven't used the Spotify APIs before, so I can't say much about that but there are a couple of issues I see at first glance. First, you're checking if (array !== 'undefined') {, which is checking if the array variable is a string that is literally 'undefined' (not the value undefined). I'm fairly certain that is not what you intended. You would be better off using Array.isArray(array) here, if you're wanting to make sure array is actually an Array.
Second, you're using async functions and Promises mixed together, which (imo), you generally shouldn't do. You should use one or the other, so it's consistent and easier to follow. If you use await instead of .then, you will be able to write it in a more "synchronous" fashion and it should be easier to understand.
import * as data from '../spotify_data/artist_id.json';
async function fillGenres(array) {
const genreIDs = []; // array of genre object IDs I'm trying to put inside every appropriate artist
if (Array.isArray(array)) {
for (const element of array) {
const result = await Genre.findOrCreate({ name: element });
genreIDs.push(result._id);
}
}
return genreIDs;
}
async function fillRest(entry, genreIDs) {
const artist = {
name: entry.ArtistName,
genres: genreIDs,
spotifyID: entry.ArtistID,
popularity: entry.Popularity,
spotifyFollowers: entry.Followers,
};
try {
const result = await Artist.create([artist]);
console.log(result);
} catch (error) {
console.log(error);
}
}
async function spotifySeed() {
const entries = Object.values(data);
for (const entry of entries) {
const genreIDs = await fillGenres(entry.Genres);
await fillRest(entry, genreIDs);
}
}
await spotifySeed();
I don't know anything about the Spotify API, so this is just a guess. In fillGenres, you have:
await Genre.findOrCreate({ name: element }, (err, result) => {
genreIDs.push(result._id);
});
You're passing a callback function. Sometimes libraries will allow you to use promises or callbacks. If you pass a callback, it won't return a promise. So I'm guessing that the loop starts off all the calls to Genre.findOrCreate, which doesn't return a promise since you're using a callback. and then immediately returns. Then fillRest is called and may finish before all the Genre.findOrCreate calls.
You want something like:
const result = await Genre.findOrCreate({ name: element });
genreIDs.push(result._id)
Though even better would be this:
function fillGenres(genreNames) {
if(!genreNames || !genreNames.length) return Promise.resolve([])
return Promise.all(genreNames.map(name => {
return Genre.findOrCreate({ name })
.then(result => result._id)
})
}
This will run all the genre calls at the same time and return them all when they're done, rather than waiting to add one after another (as in your for loop).
If Genre.findOrCreate does not return a Promise, you could create a version that does:
function genreFindOrCreate(data) {
return new Promise((resolve, reject) => {
Genre.findOrCreate(data, (err, result) => {
if(err) reject(err)
else resolve(result)
})
})
}
Related
In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.
So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.
Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);
I am trying to populate the 'documents' object which is just a Documentation array. I first build out a list of Promise and store the values in 'promises' to then call Promise.all on to fire things all off at once. Then for each promise, I try and grab the text value from each response in resolvePromise(), create a document from the value, and push it to documents.
type Documentation = { name: string; source: string; rawText: string };
const documents: Documentation[] = [];
async function resolvePromise(entry: Response, documents: Documentation[]) {
const document = { name: '', source: '', rawText: '' };
document.rawText = await entry.text(); // entry.text(): Promise<string>
documents.push(document);
console.log('documents length is now: ' + documents.length);
}
async function createDocumentsObject() {
const promises: Promise<Response>[] = [];
// code to populate promises list
HELP.forEach((value: ConfigItem[], key: string) => {
value.forEach((configElem: ConfigItem) => {
if (!configElem.subsection) {
const promise = getRequest(configElem.source);
promises.push(promise);
}
});
});
console.log('promises: ');
console.log(promises); // const promises: Promise<Response>[]
await Promise.all(promises).then(async values => {
values.forEach(async entry => {
if (!entry.ok) {
return Promise.reject();
} else {
return await resolvePromise(entry, documents);
}
});
});
console.log('docs');
console.log(documents);
}
In the print statement below, you can see the promises variable is populated with the promises correctly. However, the call to console.log(documents); runs before the calls to resolvePromise(). I tried using await entry.text(); to get the string value from entry.text() before moving on, but that is not working. I am trying to populate the documents object immediately for use right after in the code. I am new to TS so noob friendly explanations are appreciated!
The problem is here:
values.forEach(async entry => {...})
forEach will not resolve the promises returned by the async function.
I would change it to
return Promise.all( values.map( async entry => {...} ) )
I'm trying to fiddle with fetching data from public APIs and then showing that data in a React component, the react part is not important tho, this is primarly a js issue. I'm using PokeApi for learning purpose, I create a new object with the data I need from the request, and then push that object to an array that the function returns when called:
// fetchPoke() function
let pokemons = []
// IDs is just an array with 20 random numbers between a range
IDs.forEach(async (id) => {
let url = `https://pokeapi.co/api/v2/pokemon/${id}`
await fetch(url)
.then((res) => {
if (!res.ok) {
console.log(`${id} not found`)
} else {
return res.json()
}
})
.then((data) => {
let pokemon = {}
pokemon.id = data.id
pokemon.name = data.name
pokemon.image = data.sprites.other.dream_world.front_default
pokemons.push(pokemon)
})
})
// function returns the array of object
return pokemons
But whenever I call this function
let pokemons = fetchPoke()
And then log it
console.log(pokemons)
Although I can see the content, it says it's an array of 0:
In fact if I try to console log pokemons.length I get 0
What could cause this? Am I doing something wrong in my fetch request?
So, you create an empty array.
You loop through you loop through the array, firing-off a bunch of asynchronous requests that will, as a side-effect, populate the empty array when the promises complete.
You immediately return the array without waiting for the promises to complete.
The requests probably have not even left your machine by the time this happens.
The array is empty at this point.
If instead, you declare your function as async and you map your IDs to a new array of promises, then await them all with Promise.all, then the promises will be able to complete before the function returns, and the resolved value of Promise.all will be an array containing your pokemons.
async function getSomePokemons(IDs) { // note, this is an async function
const promises = IDs.map((id) =>
fetch(`https://pokeapi.co/api/v2/pokemon/${id}`)
.then((res) => {
if (!res.ok) {
console.log(`${id} not found`)
// implicitly resolves as undefined
} else {
return res.json()
}
})
.then((data) => (data ? { // data might be undefined
id: data.id,
name: data.name,
image: data.sprites.other.dream_world.front_default
} : undefined))
)
const pokemons = await Promise.all(promises);
return pokemons;
}
I'm trying to understand how to chain two different API calls including one with a for loop in a 'notes' Vue component. I have a really basic experience of promises and I'm looking to improve.
I'm making a first API call to get all the notes and pushing them into an array using a Vuex mutation. During that first API call I'm also mapping the different users emails into an Object.
Using this mapped object, I'm making a second API call inside a for loop to get all the users avatars.
Here's what the first API call looks like :
getAllNotesAPI(entity) {
noteService.getNotes(entity)
.then((response) => {
if (response.data.length === '0') {
// Set hasData to false if the response is 0
this.hasData = false;
} else {
// Push data into the note array using a store mutation
this.setAllNotes(response.data);
}
// Mapping all users emails into 'userEmails'
this.userEmails = [...new Set(response.data.map(x => x.userEmail))];
// Calling my second API call here to get all the avatars associated with these emails
for (let i = 0; i < this.userEmails.length; i++) {
this.getAvatarAPI(this.userEmails[i])
}
})
.catch((error) => {
console.log(error);
})
.finally(() => {
this.endLoader('notes');
});
},
this.getAvatarAPI is the second API call which looks like this :
getAvatarAPI(login) {
userService.getAvatar(login)
.then((response) => {
let newAvatar = {
userEmail: login,
picture: response.data.picture
};
// Push the response into a userAvatar Object using a store mutation
this.setUserAvatar(newAvatar);
}).catch((error) => {
console.log(error)
})
},
I've tried using async / await but couldn't figure out how to bind this inside of an async function (this.getAvatarAPI(this.userEmails)) was undefined, I've tried chaining using multiples then but couldn't figure out how to : get all my notes then all my avatars then end the 'note' loader once both those API calls are done.
If any of you could give me some pointers or the beginning of an answer that would be truly appreciated !
First whilst not related to your problem, avoid for loop when non necessary:
Do you need the i index?
for (let i = 0; i < this.userEmails.length; i++) {
this.getAvatarAPI(this.userEmails[i])
}
no. You need the userMail. Then
this.userEmails.forEach(userMail => {
this.getAvatarAPI(userEmail)
})
Now, to synchronize promises, you need to return a promise (let's not talk about async yet)
make getAvatarAPI return a promise
getAvatarAPI(login) {
return userService.getAvatar(login).then(blabla) // notice the return here
retrieve the promises of getAvatar API
let promises = this.userEmails.map(userMail => {
return getAvatarAPI(userEmail)
})
return after all promises have fulfilled
let promises = this.userEmails.map(userMail => {
return getAvatarAPI(userEmail)
})
return Promise.all(promises)
On a side note with async/await
If you use it you are not forced anymore to write return, you need to write async/await though
The underlying idea stay the same. Specifying the async keywords says that your function will return a promise-like.
e.g
async function p () {
return 5
}
p.then(x => console.log(x)) // does print 5 even though we didn't explicitely write return Promise.resolve(5)
Now you have to ensure you await the async function when you call it:
getAvatarAPI: async login => {
return userService.getAvatar(login).then(blabla)
}
// DO NOT do it
this.userEmails.forEach(userMail => {
return await this.getAvatarAPI(userEmail)
})
In forEach loop above, you will do your getAvatarAPI call in sequence because await "stops" iterating as long as getAvatarAPI has not resolved.
The proper way would be
getAllNotesAPI: async entity => {
try { // notice the necesary try-catch block
const response = await noteService.getNotes(entity)
blabla
let promises = this.userEmails.map(userMail => {
return this.getA...
})
let result = await Promise.all(promises)
// eventually return result, or simply await Promise... without lefthand-side assignment
} catch (error) {
console.log(error);
}
console.log(this.end('loader'))
}
I have this code:
return (entity.User.getProjects(req.user.id, ....))
.then((list) => {
// party with list object!
return {
list: list
}
}
And I want to call another method as well, entity.Tag.getAllForUser, which returns a tags list.
(UPDATE: I need to do this using promises only - I can't use async/await.)
This code doesn't work - it says both list and tag are undefined:
return (entity.User.getProjects(req.user.id, ....))
.then((list) => entity.Tag.getAllForUser(req.user.id))
.then((list, tags) => {
// party with both list and tags objects!
return {
list: list,
tags: tags
}
}
I've also tried chaining the second call onto the first:
return (
entity.User.getProjects(req.user.id, ....).then((list) =>
entity.Tag.getAllForUser(req.user.id))
).then((list, tags) => {
// party with both list and tags objects!
return {
list: list,
tags: tags
}
}
But again it says list is undefined. What does work?
It depends upon wether you requests are independent or not:
Get multiple resolved values on independent requests:
Let's say I have getUsers() and getCities(), and both are independent (you don't need the users to get the cities and viceversa). Then you can use Promise.all:
Promise.all([getUsers(), getCities()])
.then(result => {
// result here will be an array of two elements:
// result[0] -> what getUsers resolves
// result[1] -> what getCities resolves
});
Get multiple resolved values on dependent requests:
If you have, let's say, getUser(), which returns an user, and then getCouple(user), which returns a person related to an user, and then getHouse(user1, user2), which happens to need both of them, you can do the following:
const user = getUser();
const couple = user.then(u => getCouple(u)); // you can reuse a promise. This does not trigger then handler again, if
// the promise is settled will yield the settle value.
const house = user.then(u => {
return couple.then(c => {
return getHouse(u,c);
});
});
This gets much nicer with async/await:
async function() {
const user = await getUser();
const couple = await getCouple(user);
const house = await getHouse(user, couple);
}
The best solution here is Promise.all, which takes a list of promises, and returns a list of the resolved values:
return Promise.all([
entity.User.getProjects(req.user.id),
entity.Tag.getAllForUser(req.user.id);
])
.then(([ list, tags ]) => {
// Party!
return { list, tags };
});
Obligatory recommendation to use async and await:
let [ list, tags ] = await Promise.all([
entity.User.getProjects(req.user.id),
entity.Tag.getAllForUser(req.user.id);
]);
return { list, tags };
Just simply use Async/Await
let list = await entity.User.getProjects(req.user.id, ....))
let tags = await entity.Tag.getAllForUser(req.user.id))
return {
list: list,
tags: tags
}
}
If you can't use Async/Await ( even though i recommend using them ) you can use promise.all
Promise.all([
entity.User.getProjects(req.user.id),
entity.Tag.getAllForUser(req.user.id);
]).then(([list,tag])=>{
// do here whatever you want
})
Answering my own question. Please don't delete, as this may be useful for someone else!
return (entity.User.getProjects(req.user.id, ....))
.then((list) => Promise.all([list, entity.Tag.getAllForUser(req.user.id)]))
.then(([list, tags]) => {
// party with both list and tags objects!
return {
list: list,
tags: tags
}
}