I am reading a JSON object and looping through each item. I am first checking to see if the item already exists in the database and if so I want to log a message. If it doesn't already exist I want to add it.
This is working correctly however, I would like to add a callback or finish the process with process.exit();
Because the mongoose calls are asynchronous I can't put it at the end of the for loop because they haven't finished.
Whats the best way I should handle this?
function storeBeer(data) {
data.forEach((beer) => {
let beerModel = new Beer({
beer_id: beer.id,
name: beer.name,
image_url: beer.image_url
});
Beer.findOne({
'name': beer.name
}).then(function (result) {
if (result) {
console.log(`Duplicate ${result.name}`)
} else {
beerModel.save(function (err, result) {
console.log(`Saved: ${result.name}`)
});
}
});
});
}
Is there anything I should read up on to help solve this?
One means of managing asynchronous resources is through Promise objects, which are first-class objects in Javascript. This means that they can be organized in Arrays and operated on like any other object. So, assuming you want to store these beers in parallel like the question implies, you can do something like the following:
function storeBeer(data) {
// This creates an array of Promise objects, which can be
// executed in parallel.
const promises = data.map((beer) => {
let beerModel = new Beer({
beer_id: beer.id,
name: beer.name,
image_url: beer.image_url
});
return Beer.findOne({
'name': beer.name
}).then(function (result) {
if (result) {
console.log(`Duplicate ${result.name}`)
} else {
beerModel.save(function (err, result) {
console.log(`Saved: ${result.name}`)
});
}
});
);
});
return Promise.all(promises);
}
Don't forget that the storeBeer function is now asynchronous, and will need to be utilized either through a Promise chain or through async/await.
For example, to add process exit afterwards:
async function main() {
const beers = [ ... ];
await storeBeer(beer);
process.exit(0);
}
You can also modify the above example to invoke the storeBeer function within a try / catch block to exit with a different error code based on any thrown errors.
Related
Consider the following exerpt from this example in the InfluxDB documentation:
const fluxObserver = {
next(row, tableMeta) {
const o = tableMeta.toObject(row)
console.log(
`${o._time} ${o._measurement} in ${o.region} (${o.sensor_id}): ${o._field}=${o._value}`
)
},
error(error) {
console.error(error)
console.log('\nFinished ERROR')
},
complete() {
console.log('\nFinished SUCCESS')
}
}
/** Execute a query and receive line table metadata and rows. */
queryApi.queryRows(fluxQuery, fluxObserver)
The fluxObserver defines a sequence of operations at different stages of execution of the query, all of which are just console.log calls. I can imagine that people want to do something similar in a way that is actually useful, like for instance return an array that contains the result of the query, instead of logging something to a console. My problem is that queryApi.queryRows does not actually return anything, so it is not possible to do something like:
...
complete() {
console.log('\nFinished SUCCESS')
return data;
}
}
/** Execute a query and receive line table metadata and rows. */
var result = queryApi.queryRows(fluxQuery, fluxObserver);
return result;
So how can I return the data collected by the fluxObserver?
I am having difficulty figuring out what is happening (and not happening) in my action creator.
I need to make a call to one API endpoint, get the ids and names of all the items returned, then for each of those ids, make another call. I want to store the return of the last call and the ids/names from the first call in an object and dispatch it.
{
category: name //name of category
subcategory: [] //array of categories in the category above.
}
Right now, my reducer does end up having what I want, but when I attempt to log that particular prop in the component it is empty. (below I am using OpenSocialAPI or osapi. This is just a basic wrapper for an ajax request. Allows for me to not have to authenticate as it sees I am already authenticated.)
export function fetchCategories(id){
let categories = []
return function(dispatch){
dispatch(requestCategories(id))
return osapi.core.get({
v: "v3",
href: "/places/" + id + "/places"
}).execute(function(response) {
response.content.list.forEach(function(category) {
osapi.core.get({
v: "v3",
href: "/places/" + category.id+ "/categories"
}).execute(function(response) {
categories.push({
category: category.name,
subcategories: response.content.list.map(category => category.name)
})
})
})
console.log("Category response: ", categories)
dispatch(receiveCategories(id, categories))
})
}
}
export function receiveCategories(id,array){
return {
type: RECEIVE_CATEGORIES,
id,
categories: array,
recievedAt: new Date(Date.now()),
isFetching: false
}
}
And in my app I am dispatching the action creator in componentDidMount
componentDidMount() {
const { dispatch } = this.props
dispatch(fetchCategoriesIfNeeded(id))
}
Right now when I console log in my Category component and in the execute above, it is empty. But looking at my state in my logger, when recieveCategories is completed, I have the array of objects I want
[{category:...,
subcategories:[...]
},
{category:...,
subcategories:[...]
}]
I suspect this is because of something asynchronous but I'm unsure how to proceed.
I attempted to create my own wrapper for the call that was promise based, but I had similar issues, probably more so because I'm not sure if resolve(response) is what I want.
function httpService(path){
return new Promise(function(resolve, reject){
osapi.core.get({
v: 'v3',
href: path
}).execute(function(response, error){
if(error){
return reject(new Error("Error: ", error))
}
resolve(response)
})
})
}
export function fetchCategories(spaceId) {
let categories = []
return function(dispatch) {
dispatch(requestCategories(id))
return httpService("/places/" + id + "/places")
.then(function(response) {
response.content.list.forEach(function(category) {
fetchSubCategories("/places/" + category.id + "/categories")
.then(function(response) {
categories.push({
category: category.name,
subcategories: response
})
})
})
console.log("CATEGORIES: ", categories)
dispatch(receiveCategories(id, categories))
})
}
}
function fetchSubCategories(url){
return httpService(url)
.then(function(response){
return response.content.list.map(category => category.name)
})
}
Can you look at this and give guidance? Also, is me dispatching an array that I built based on the API responses a valid way of doing things or is there someway better? Thank you
I was only able to find 1 other question with similar use case but they are using bluebird or something similar. I'd really like to keep this without anything extra besides Redux.
It looks like you just need to dispatch your categories inside your .execute() callback, not outside of it. You're doing osapi.core.get().execute((response) => but then outside of that execute callback, you dispatch receiveCategories, which will execute long before your Promise resolves, and dispatch the empty array you initialize.
You also need to use Promise.all to get the response of all of your nested GET requests.
There's also no reason to keep a mutating array around.
I guess osapi.core.get is some kind of promise based fetch library? And .execute is called when the get succeeds?
If so, then what you're missing is that you're not waiting for all asynchronous calls to finish.
I'm going to show a solution based on generic fetch and native Promises so you can understand the solution and adopt it based on your specific libraries.
const promises = [];
response.content.list.forEach(function(category) {
const promise = fetch("/places/" + category.id+ "/categories");
promises.push(promise);
})
Promise.all(promises).then(responses => {
categories = responses.map(response => ({/* do your object mapping here */}))
// now you can dispatch with all received categories
dispatch(receiveCategories(id, categories))
});
Also, you're using the same variable in your nested functions - while this may work and the computers may understand it, it makes it super hard for any human to figure out which response belongs to which scope. So you may want to take a second look at your variable names as well.
I am using fetch to retrieve a tree, one layer at a time. How can I trigger an action after all branches have been fetched? The tree has multiple branches and I don't know the depth in advance.
Pseudo-code:
until Children is empty fetchChildren()
after all children are retrieved doSomething()
I found some answers, but they only deal with chained then(), not trees.
[edit] Although not mandatory, I am hoping that the solution can run in IE11. I am already using promise and fetch polyfills.
Just first wait for the layer elements to be fetched, then recursively proceed with the childrens and await that:
async function fetchChildren(layer) {
await Promise.all(layer.map(fetch));
await Promise.all(layer.map(el => fetchChildren(el.children)));
}
(fetch(el) has to be a promising function)
Here's a way to do it with a recursive async function waiting for the fetching of an element's children to be done, before returning a promise.
That way, we can call doSomething when everything is done.
const tree = {
url: 'http://www.json-generator.com/api/json/get/cevhxOsZnS',
children: [{
url: 'http://www.json-generator.com/api/json/get/cguaPsRxAi'
}, {
url: 'http://www.json-generator.com/api/json/get/cguaPsRxAi',
children: [{
url: 'http://www.json-generator.com/api/json/get/cfDZdmxnDm'
}]
}]
};
function doSomething() {
console.log("doing something");
}
async function fetchIt(element) {
if (element.children) {
await Promise.all(element.children.map(fetchIt));
}
return new Promise((resolve) => {
console.log("fetching " + element.url);
fetch(element.url).then(res => {
console.log("done");
resolve();
});
});
}
fetchIt(tree).then(doSomething);
I'm using Axios / promises to make AJAX requests, but the structure is a bit confusing. Basically I want this to happen:
Get masterlist.xml.
Parse the master list to get an array of category list URLs.
Get each category.xml.
Parse each category list to get an array of manifest URLs.
Get each manifest.xml and parse data.
Example structure:
masterlist.xml
<master>
<category>action</category>
<category>romance</category>
</master>
Category XMLs (e.g. action.xml and romance.xml)
<manifestUrls>
<manifest>foo/bar/manifest.xml</manifest>
<manifest>alice/bob/manifest.xml</manifest>
<manifest>hello/world/manifest.xml</manifest>
</manifestUrls>
Manifest XMLs
<manifest>
<data>Blah blah blah</data>
<manifest>
I'm a bit stuck on how to structure this as an Axios request with then. I'd like to have three functions, getMaster(), getCategory(url), and getManifest(url) that are preferably independent (e.g. getMaster doesn't have to call getCategory directly), but it seems like it might be necessary.
How would this be structured in Axios?
One of the main benefits of promises is that they allow you to easily avoid interdependence between your methods.
Here is a rough outline of how you could do this.
// put it all together
getMaster()
.then(parseMaster)
.then(function (categories) {
return Promise.all(categories.map(getAndParseCategory));
})
.then(flatten) // the previous then() resolves to an array-of-arrays
.then(function (manifestUrls) {
return Promise.all(manifestUrls.map(getManifest));
})
.then(function (manifests) {
// manifests is an array of all manifests
});
// Examples of what each of the functions used above would do
function getMaster() {
return axios.get('masterUrl')
.then(function (response) { return response.data; });
}
function parseMaster(masterContent) {
// parse and return an array of categories
}
function getCategory(name) {
var url = // ... build the URL based on the name
return axios.get(url)
.then(function (response) { return response.data; });
}
function parseCategory(categoryContent) {
// parse and return an array of URLs synchronously for one category
}
function getAndParseCategory(name) {
return getCategory(name).then(parseCategory);
}
function getManifest(url) {
return axios.get(url)
.then(function (response) { return response.data; });
}
function flatten(arrayOfArrays) {
return [].concat.apply([], arrayOfArrays);
}
If you're using Bluebird or something else that gives promises a .map() method, then you can tidy that pipeline up a bit:
// using Promise.resolve() at the beginning to ensure
// the chain is based of the desired kind of promise
Promise.resolve()
.then(getMaster)
.then(parseMaster)
.map(getCategory)
.map(parseCategory)
.then(flatten) // previous line resolves to an array-of-arrays
.map(getManifest)
.then(function (manifests) {
// manifests is an array of all manifests
});
Of course, you could also define your own .map method if you don't want to import a whole third party promise library:
if (!Promise.prototype.map) {
Promise.prototype.map = function (func) {
return this.then(function (result) {
return Promise.all(result.map(func));
});
};
}
Edit: To respond to your question in the comments below. If you wanted to pass the category text along so that it could be included in the manifest URLs, I think a clean way to do this would be to include that in the data returned from getCategory() so that parseCategory can make use of it. Everything else could stay the same.
Example:
function getCategory(name) {
var url = // ... build the URL based on the name
return axios.get(url)
.then(function (response) {
return {
name: name,
data: response.data
};
});
}
function parseCategory(categoryContent) {
var urls = // parse URLs from categoryContent.data
return urls.map(function (url) {
return categoryContent.name + '/' + url;
});
}
I got some problems with Sequelize and promise (mainly with promise).
Example :
I want all my Events and the master of this events. I've done something like that :
models.events.findAll().then(function(event) {
events.forEach(function(event){
events.dataValues.master = event.getUsers({
where: ['UserRelationEvent.relation = "master"'],
joinTableAttributes: []
}).then(function(user){
return user[0].dataValues;
});
});
return next(events);
}).catch(function(err) {next(err)});
But there I got an Sequelize Object.. I checked my content of user[0].dataValues It's exactly what I want.. so I think I miss something and misunderstood something with promises :/
I try many things but mainly my question is : how can I retrieve my string from my console.log
event.getUsers returns a promise - so you are assigning events.dataValues.master (which is the array, not the particular event btw) the promise, not the value.
Also, you are returning next, before getUsers is done, because they happen async
Something like this should work
models.events.findAll().then(function(events) {
sequelize.Promise.each(events, function(event){
return event.getUsers({
through: {
where: { relation: 'master' }
},
joinTableAttributes: []
}).then(function(user){
event.dataValues.master = user[0];
});
}).then(function (events) {
return next(events);
});
}).catch(function(err) {next(err);});
When you return a promise to the callback in promise.each, the then (which calls next) is not invoked, before all the returned promises are done (i.e. before all getUsers calls are done.
I've also changed the where clause to use an object instead of string :)
But wait! We can do better!
models.events.findAll({
include: [
{
model: User,
through: {
where: {
relation: 'master'
}
}
}
]
}).then(function(events) {
return next(events);
}).catch(function(err) {next(err);});
This left joins the users table where the relation is master. You may want to do a map in the then, because users will be placed under .users, not .master