I have the following code and, for the life of me, can't figure out why the requests don't execute concurrently. I'm still new to RxJS and observables, so any help in improving the code below would be greatly appreciated as well. Basically, I'm making a call to a REST API on the backend to get some data. Then, for every element in that array of data I'm making another request to a different endpoint (hence using the 'forkJoin' operator). All the requests get sent at once, but they seem to execute one after another still instead of concurrently.
this.sites$.subscribe(data => {
// data.forEach(element => {
// this.siteCaptureMap[element.id] = new CaptureData();
// this.sitesService.getCaptureData(element.nameOrNumber, element.owner.name).subscribe(data => {
// this.siteCaptureMap[element.id].count = data.length;
// });
// });
var obs: Observable<any>[] = [];
for (var _i = 0; _i < data.length; _i++) {
this.siteCaptureMap[data[_i].id] = new CaptureData();
this.siteCaptureMap[data[_i].id].id = _i;
obs.push(this.sitesService.getCaptureData(data[_i].nameOrNumber, data[_i].owner.name));
}
forkJoin(obs).subscribe(results => {
for (var _i = 0; _i < results.length; _i++) {
this.siteCaptureMap[data[_i].id].count = results[_i].length;
}
});
this.dataSource.data = data;
this.dataSource.filteredData = data;
});
Again, any help would be greatly appreciated. If I need to clarify anything or provide any additional code snippets, please let me know! Thanks!
Having nested subscribes can lead to memory leaks and can be tough to unsubscribe so whenever you have nested subscribes, think of switchMap, concatMap, and mergeMap.
They all have small variations but they switch to a new observable from the previous observable. This post explains the differences.
For you, I would try doing:
import { switchMap } from 'rxjs/operators';
...
this.sites$.pipe(
switchMap(data => {
let obs: Observable<any>[] = [];
for (let _i = 0; _i < data.length; _i++) {
this.siteCaptureMap[data[_i].id] = new CaptureData();
this.siteCaptureMap[data[_i].id].id = _i;
obs.push(this.sitesService.getCaptureData(data[_i].nameOrNumber, data[_i].owner.name));
}
return forkJoin(obs);
}),
).subscribe(results => {
for (let_i = 0; _i < results.length; _i++) {
this.siteCaptureMap[data[_i].id].count = results[_i].length;
}
this.dataSource.data = data;
this.dataSource.filteredData = data;
});
A side note is to use let and const instead of var.
Also, if you see all the requests going out at the same time, that's all you can hope for. If it returns in series, it can either be the browser or the server causing that.
First I would rearrange the code to be more rxjs idiomatic, removing nested subscriptions and using pipe and making it a bit more functional-style. Inline comments try to explain the changes
this.sites$.pipe(
// this is the rxjs map operator that transform the data received from
// the rest api into something different
map(data => {
// I create the obs array using the map method of JS arrays - this is
// NOT the rxjs map operator
obs = data.map((element, _i) => {
this.siteCaptureMap[element.id] = new CaptureData();
this.siteCaptureMap[element.id].id = _i;
return this.sitesService.getCaptureData(element.nameOrNumber, element.owner.name)
});
// return both the array of observables and the data
return [data, obs];
}),
// concatMap makes sure that you wait for the completion of the rest api
// call and the emission of the data fetched before you execute another
// async operation via forkJoin
concatMap(([data, obs]) => forkJoin(obs).pipe(
// this is the rxjs map operator used to return not only the results
// received as result of forkJoin, but also the data received with the
// first rest call - I use this operator to avoid having to define
// a variable where to store 'data' and keep the logic stateless
map(results => ([data, results]))
))
).subscribe(([data, results]) => {
// here I use forEach to loop through the results and perform
// your logic
results.forEach((res, _i) => this.siteCaptureMap[data[_i].id].count = res.length)
})
I think this form is more in line with rxjs and its functional spirit, and should be equivalent to your original form.
At the same time I would be interested to know why you say that your code executes the calls within forkJoin one at the time.
By the way, if you are interested in looking at common patterns of use of rxjs with http, you can read this article.
Related
I need to perform a sequence of http requests to my database from Angular. In the first step I get an array of objects. This is working fine.
In a second step I want to populate each object with additional data. So I need to wait for the first http.request to complete. I also don't want to return before my data has been fetched.
My function getData() should return an Observable, because the interface / signature is not supposed to change (as it is being used in other parts of the program and an observable makes sense for the program logic).
// Returns an array with objects, that have an id, that is needed for the second call
data[] : Object = new Array();
populatedData[] : Object = new Array();
getData(): Observable<Object[]>{
this.http.get<Object[]>("http://localhost:3000/data").subscribe(data => this.data = data);
// This needs the data from the first call
for(int i = 0; i < data.length; i++){
this.http.get<Object>("http://localhost:3000/data/"+data[i].id).subscribe(data => populatedData.push(data));
}
return of(populatedData);
}
You can use switchMap (after the initial GET is done, make the other requests) and forkJoin (make several requests and emit when all are completed). Something like this:
getSingleEntry = ({ id }) => this.http.get<Object[]>("http://localhost:3000/data/" + id);
getData = () => this.http.get<Object[]>("http://localhost:3000/data")
.pipe(
tap(data => this.data = data),
switchMap(data => forkJoin(data.map(this.getSingleEntry)));
Promise is an option but another option can be,
// Returns an array with objects, that have an id, that is needed for the second call
data[] : Object = new Array();
populatedData[] : Object = new Array();
getData(): Observable<Object[]>{
this.http.get<Object[]>("http://localhost:3000/data").subscribe((data)=>{
this.data = data;
for(int i = 0; i < data.length; i++){
this.http.get<Object>("http://localhost:3000/data/"+data[i].id).subscribe((response)=> {
populatedData.push(response)
});
}
});
return of(populatedData);
}
You should put the code that needs the first call in the second
data: Array<any> = [];
populatedData: Array<any> = [];
getData(): Observable<Object[]> {
this.http.get<Object[]>('http://localhost:3000/data').subscribe(data => {
this.data = data;
for (let i = 0; i < data.length; i++) {
this.http
.get<Object>('http://localhost:3000/data/' + data[i].id)
.subscribe(data => populatedData.push(data));
}
});
return of(populatedData);
}
I think the best idea is to use switchMap RXJS operator and zip. In second calls you have access to data from first one. Then you can return array of values when all calls are done.
More about switchMap you can read here. About zip here.
Here you have example how data will looks like https://rxviz.com/v/Mogdv96O
Then your code will be refactored as follow:
getData(): Observable<Object[]> {
this.http.get<Object[]>("http://localhost:3000/data")
.pipe(switchMap((data) =>
zip(...data.map(item => this.http.get<Object>("http://localhost:3000/data/" + item.id)))
)
);
}
the error occurs when the Windows Features are not set up corretly.
You have to enter the -> 'Turn Windows features on or off' from the Windows Search and enter toggle 'HTTP Activation' and is presented on the picture:
Options to toggle
So, without boring anyone with the backstory, I need to access data from a number of APIs in order to run my script. The data needs to all be loaded before I execute the script, which I'm normally comfortable doing: I just declare some fetch requests, write a Promise.all, then continue on with the function.
HOWEVER, I've hit something of a snafu with a certain API that limits the number of results I can pull from one request to 100 and I need to query all of the results. I didn't think this was a huge deal since I figured I can just make a couple extra requests by affixing "&page=X" to the end of the request.
The plan, then, is to request the total number of pages from the API and then feed that into a for loop to push a number of fetch requests into an array of promises (i.e., link://to/api/data&page=1, link://to/api/data&page=2, etc). When I actually attempt to create this array with a for loop, though, the array returns empty. Here's my work:
const dataUrlNoPage = 'link/to/api/data&page=';
const totalPages = 3; //usually pulled via a function, but just using a static # for now
let apiRequestLoop = function(inp) {
return new Promise(function(resolve){
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
fetch(dataUrlLoop).then(function(response){
promiseArray.push(response.json());
})
}
resolve(promiseArray);
})
}
let finalPromiseArray = apiRequestLoop(totalPages).then(result => {
let requestArray = [apiRequest1,apiRequest2];
//requestArray contains earlier fetch promises
result.forEach(foo => {
requestArray.push(foo);
}
);
return requestArray;
});
So, what's tripping me up is really the loop, and how it's not returning an array of promises. When I look at it in the console, it shows up as a blank array, but I can expand it and see the promises I was expecting. I see the "Value below was evaluated just now" response. No matter how many promises or .thens, I write, however, the array is never actually populated at run time.
What's going on? Can I not generate fetch promises via a for loop?
(Also, just to cut this line of questioning off a bit, yes, the API I'm trying to access is Wordpress. Looking around, most people suggest creating a custom endpoint, but let's assume for the purpose of this project I am forbidden from doing that.)
You have several problems here.
The first is that you have the function provided to new Promise itself containing promise creations. Don't do this! It's a definite anti-pattern and doesn't keep your code clean.
The second is this basic bit of code:
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
fetch(dataUrlLoop).then(function(response){
promiseArray.push(response.json());
})
}
resolve(promiseArray);
This says:
create an empty array
loop through another array, doing HTTP requests
resolve your promise with the empty array
when the HTTP requests are completed, add them to the array
Step four will always come after step three.
So, you need to add the promises to your array as you go along, and have the overall promise resolve when they are all complete.
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop).then(function(response) {
return response.json();
}));
}
return Promise.all(promiseArray);
}
or, with an arrow function to clean things up:
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop).then(response => response.json()));
}
return Promise.all(promiseArray);
}
A few points:
you want to actually put the promises themselves into the array, not push to the array in a .then() chained to the promise.
you probably want to skip creating a new Promise in your function. Just get an array of all the promises from your loop, then return a Promise.all on the array.
Like this:
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop))
}
return Promise.all(promiseArray);
}
In your final .then statement, in finalPromiseArray, your result will be an array of the results from all the promises. like this [response1, response2, response3, ...]
See the Promise.all documentation for more details.
In the following code I save each item's key and an email address in one table, and to retrieve the object to fetch from the original table using said key. I can see that the items are being put into the rawList array when I console.log, but the function is returning this.cartList before it has anything in it, so the view doesn't receive any of the data. How can I make it so that this.cartList waits for rawList to be full before it is returned?
ionViewWillEnter() {
var user = firebase.auth().currentUser;
this.cartData.getCart().on('value', snapshot => {
let rawList = [];
snapshot.forEach(snap => {
if (user.email == snap.val().email) {
var desiredItem = this.goodsData.findGoodById(snap.val().key);
desiredItem.once("value")
.then(function(snapshot2) {
rawList.push(snapshot2);
});
return false
}
});
console.log(rawList);
this.cartList = rawList;
});
}
I have tried putting the this.cartList = rawList in a number of different locations (before return false, even inside the .then statement, but that did not solve the problem.
The following function call is asynchronous and you're falling out of scope before rawList has a chance to update because this database call takes a reasonably long time:
desiredItem.once("value").then(function(snapshot2) {
rawList.push(snapshot2);
});
You're also pushing the snapshot directly to this list, when you should be pushing snapshot2.val() to get the raw value.
Here's how I would fix your code:
ionViewWillEnter() {
var user = firebase.auth().currentUser;
this.cartData.getCart().on('value', snapshot => {
// clear the existing `this.cartList`
this.cartList = [];
snapshot.forEach(snap => {
if (user.email == snap.val().email) {
var desiredItem = this.goodsData.findGoodById(snap.val().key);
desiredItem.once("value")
.then(function(snapshot2) {
// push directly to the cartList
this.cartList.push(snapshot2.val());
});
}
return false;
});
});
}
The problem is the Promise (async .once() call to firebase) inside the forEach loop (sync). The forEach Loop is not gonna wait for the then() statement so then on the next iteration the data of the previous iteration is just lost...
let snapshots = [1, 2, 3];
let rawList = [];
snapshots.forEach((snap) => {
console.log(rawList.length)
fbCall = new Promise((resolve, reject) => {
setTimeout(function() {
resolve("Success!");
}, 2500)
});
fbCall.then((result) => {
rawList.push(result);
});
})
You need forEach to push the whole Promise to the rawList and Then wait for them to resolve and do sth with the results.
var snapshots = [1, 2, 3];
var rawList = [];
var counter = 0;
snapshots.forEach((snap) => {
console.log(rawList.length)
var fbCall = new Promise((resolve, reject) => {
setTimeout(function() {
resolve("Success!" + counter++);
}, 1500)
});
rawList.push(fbCall);
})
Promise.all(rawList).then((res) => {
console.log(res[0]);
console.log(res[1]);
console.log(res[2]);
});
The thing is, it is still a bit awkward to assign this.cartList = Promise.all(rawList) as it makes it a Promise. So you might want to rethink your design and make something like a getCartList Service? (dont know what ur app is like :p)
Since you're using angular you should also be using angularfire2, which makes use of Observables which will solve this issue for you. You will still be using the normal SDK for many things but for fetching and binding data it is not recommended to use Firebase alone without angularfire2 as it makes these things less manageable.
The nice things about this approach is that you can leverage any methods on Observable such as filter, first, map etc.
After installing it simply do:
public items$: FirebaseListObservable<any[]>;
this.items$ = this.af.database.list('path/to/data');
And in the view:
{{items$ | async}}
In order to wait for the data to appear.
Use AngularFire2 and RxJS this will save you a lot of time, and you will do it in the proper and maintainable way by using the RxJS operators, you can learn about those operators here learnrxjs
i got a small problem with my promises.
This is my function:
public generateMealPlanForAWeek(myMealProfile: any, totalCalories:number):Promise<any> {
return new Promise(resolve => {
let mealplan:Array<any> = [];
for (let i = 1; i <= 7; i++) {
this.generateMealPlanForOneDay(myMealProfile, totalCalories).then(data => {
mealplan.push(data); // resolves much later
});
}
resolve(mealplan); // is resolved instant
})
}
My problem is, that "generateMealPlanForOneDay" is resolved later then the mealplan itself.
Mealplan is an array of objects (meals in this case).
When i want to save my mealplan, the mealplan is empty:
this.storage.set('Mealplan', JSON.stringify(mealplan)).then(....) // meal plan is Array[0]
Whats the best way to resolve all meals of my meal plan and then save it?
You can use Promise.all in this case.
public generateMealPlanForAWeek(myMealProfile: any, totalCalories:number):Promise<any> {
var mealPlans = [];
for (let i = 1; i <= 7; i++) {
mealPlans.push(this.generateMealPlanForOneDay(myMealProfile, totalCalories));
}
return Promise.all(mealPlans);
}
// later
generateMealPlanForAWeek(...).then(mealPlans => ...);
Instead of working with promises, you can also work with event streams.
The Reactive Extensions for JavaScript (RxJS, see https://github.com/ReactiveX/RxJS) makes it easy to react to event streams (Observables and their OnNextevents), or for instance to the end of an event stream (the OnCompletedevent), to collect the events and save their data.
Here's an interesting example: https://xgrommx.github.io/rx-book/content/getting_started_with_rxjs/creating_and_querying_observable_sequences/creating_and_subscribing_to_simple_observable_sequences.html
I have a page than can make a different number of $http requests depending on the length of a variables, and then I want to send the data to the scope only when all the requests are finished. For this project I do not want to use jQuery, so please do not include jQuery in your answer. At the moment, the data is sent to the scope as each of the requests finish, which isn't what I want to happen.
Here is part of the code I have so far.
for (var a = 0; a < subs.length; a++) {
$http.get(url).success(function (data) {
for (var i = 0; i < data.children.length; i++) {
rData[data.children.name] = data.children.age;
}
});
}
Here is the part that I am sceptical about, because something needs to be an argument for $q.all(), but it is not mentioned on the docs for Angular and I am unsure what it is meant to be.
$q.all().then(function () {
$scope.rData = rData;
});
Thanks for any help.
$http call always returns a promise which can be used with $q.all function.
var one = $http.get(...);
var two = $http.get(...);
$q.all([one, two]).then(...);
You can find more details about this behaviour in the documentation:
all(promises)
promises - An array or hash of promises.
In your case you need to create an array and push all the calls into it in the loop. This way, you can use $q.all(…) on your array the same way as in the example above:
var arr = [];
for (var a = 0; a < subs.length; ++a) {
arr.push($http.get(url));
}
$q.all(arr).then(function (ret) {
// ret[0] contains the response of the first call
// ret[1] contains the second response
// etc.
});