Related
I´m quite unsure on how to handle multiple updates / inserts in knex and return whatever it was successfull on the end or not.
I´m passing an array through req.body loop through it and trigger actions based on informations inside the array.
Example:
const data = [...req.body]
for(let i = 0; i < data.length; i++) {
data[i].totals.length
for(let y = 0; y < data[i].totals.length; y++) {
if(data[i].totals[y].info === "Holiday") {
calcHoliday(data[i].totals[y].total, data[i].id)
} else if(data[i].totals[y].info === "ZA") {
calcZA(data[i].totals[y].total, data[i].id)
}
}
calcOvertime(data[i].totalSum, data[i].id)
if(i === data.length -1) {
res.json("Success")
}
}
The Array I´m passing in looks like this:
[
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
},
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
}
]
Function Example which gets called in for loop:
const calcHoliday = (hours, userid) => {
knex.transaction(trx => {
trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
.then(() => {
return trx("hours")
.decrement("holiday_hours", hours)
}).then(trx.commit)
.catch(trx.rollback)
}).then(() => console.log("WORKED"))
.catch(err => console.log(err))
}
This is working perfectly fine but I can´t figure out how to gather the results from each table update in order to respond if everything worked or an error appeared. If I call e.g. after one calcHoliday call .then(resp => res.json(resp) I receive only the response from the first operation.
In short I need a way on how to res.json if everything succeeded or an error appeared somewhere.
Thanks in advance!
TLDR;
Turning your insert calls into an array of promises and then using await and a Promise.all() / Promise.allSettled() structure might solve this problem, but there are some UX decisions to make on what to rollback and how to return errors.
Error Handling Choices:
Any error --> all insertions in all loop iterations should be rolled back
Do you want partial success? The way the code is written now, rollback only applies to items in one function call. If one of the hour-decrement calls fails, it will roll back one log insert, but not any that succeeded for previous data in the loop. If you want the whole dataset to rollback, you'd need to pass the txn through each function call or do a bulk insert of all of your rows in one function call, which might be nice for performance reasons anyway depending on the use case.
Partial success --> commits successes, rolls back single loop iterations that fail, sends detailed list of errors and successes
You'd want to use Promise.allSettled(), which aggregates the successes and errors as an array from all promises in the loop.
Partial success --> commits the successes, rolls back single loop iterations that fail, sends just one error
Opinion: This can be a misleading UX unless the error is "some of the insertions were unsuccessful" and the endpoint is idempotent
This looks closest to what you're describing you want. If this is the case, you'd want to use Promise.all(), which throws an error as soon as one promise in the array errors.
Example Implementation:
Since the original code is incomplete, this is a loose, incomplete example of what option 2/3 might look like. This could easily be transformed into option 1.
First, it might help to modify all of your functions with asynchronous calls to be fulfillable as promises. Async/await helps avoid .then() trees that are hard to reason about.
const calcHoliday = async (hours, userid) => {
try {
const result = await knex.transaction(async(trx) => {
await trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
return trx("hours").decrement("holiday_hours", hours)
}
return result
} catch(err) {
console.log("It didn't work.")
throw new Error(`Error: Failure to insert for user ${userid}:`, err)
}
}
Here are some utilities to get the data transformed, and to get the appropriate unfulfilled promise to supply to the map in Promise.all/allSettled.
/*
Here's an example of how you might transform the original data with maps in order to avoid nested for-loops:
[
{ id: 1, info: 'Holiday', total: 4 },
{ id: 1, info: 'Holiday', total: 4 }
]
*/
const flatData = data.map(item => {
return item.totals.map(total => ({
id: item.id,
...total
}))
}).flat()
// Returns the appropriate promise based on data
const getTotalsPromises = (row) => {
const {info, id, total} = row
if(info === "Holiday") {
return calcHoliday(total, id)
} else if(info === "ZA") {
return calcZA(total, id)
}
}
const getcalcOvertimePromises = (rowInData) = {
// work left to reader
return calcOvertime(rowInData.correctData, rowInData.otherData)
}
If you want option 2:
// Replaces the loop
// Fulfills *all* the promises, creating an array of errors and successes
const responses = await Promise.allSettled([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
// insert loop here to do something with errors if you want
res.send(responses)
OR Option 3
Create an array of all of the promises you want to run, run them, and process up to one error.
// Replaces the loop
// Runs the promises and waits for them all to finish or the first error.
try {
const responses = await Promise.all([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
res.send(responses)
} catch(err){
// Reached if one of the rows errors
res.send(err)
}
Docs:
Promise.allSettled
Promise.all
I have this array orderCodes that has the codes for specific orders, and then with the code I can get the details of that order (each order has multiple products), and I need to extract the code of each product inside the order details.
The getOrderDetails()is an Observable with results(an array of the products), and each resulthas a code which is what I need.
this.orderCodes.forEach((orderCode) => {
loadOrderDetails(orderCode);
getOrderDetails().subscribe((order: any) => {
if (order.results) {
order.results.map((result) => {
console.log(result.code);
});
}
});
});
I've tried with this forEach but since I'm subscribing to the Observable the forEach skips to the next iteration and I need it to wait
Any ideas?
rxjs way would be
from(this.orderCodes).pipe(
concatMap((orderCode) => // concatMap operator makes your items come "in order" one after another
defer(() => {
loadOrderDetails(orderCode);
return getOrderDetails();
}))
).subscribe((order: any) => {
if (order.results) {
order.results.map((result) => {
console.log(result.code);
});
}
});
or you could convert to promises and use async await (more elegant, but usually less prefered way in angular because of converting to promises and change detection issues if done wrong, but it depends...)
async myFunctionThatDoesAllThis(...) {
....
for(let orderCode of this.orderCodes) {
loadOrderDetails();
const order = await getOrderDetails().pipe(take(1)).toPromise(); // pipe(take(1)) could be skipped if getOrderDetails is just an http request.
if(order.results) {
order.results.forEach((result) => {
console.log(result.code);
});
}
}
My code was working fine until I decided I wanted to try and return two values in my promise chain to the next function. I can call one of the values but not the other.
My code looks like this
app.get('/projects', (req, res) => {
practice.screamIt('matt').then((name) => {
return [practice.translateIt(name), name]; //puts name as the parameter for next function
}).then((translate) => {
console.log(translate[1] + ' test occured here')
console.log(translate[0][0].englishName)
return [`The name you entered is ${translate[1]}`, `${translate[0].englishName} is ${translate[0].spanishName} in spanish`]
}).then((value)=>{
res.render('projects', {
pageTitle: "Projects Page",
practice: practice,
value1: value[0],
value2: value[1]
});
}).catch((errorMessage)=>{
console.log(errorMessage)
})
});
And when I log the first bit of data it shows:
[ Promise { { englishName: 'Matt', spanishName: 'Mateo' } },'Matt' ]
I want to be able to call englishName, but can't seem to do so without getting undefined. I need to be able to call englishName in order for my second function work as intended.
You need to first resolve the translateIt() promise and use another then() to create the array
Change:
return [practice.translateIt(name), name];
To
return practice.translateIt(name).then(translate => [translate, name]);
I don't have a full copy if your code but it appears you're returning a Promise in an Array to the then handler which I don't believe will work?
practice.screamIt('matt').then((name) => {
return practice.translateIt(name);
}).then((translate) => {
...etc
}).catch((errorMessage)=>{
console.log(errorMessage)
});
If you try something like that I believe that would work for you as it would be returning the result once the promise resolves.
I suppose you need to return Promise.all([practice.translateIt(name), name]) instead of practice.translateIt(name) as I its returning a promise.
That should return a single promise resolving to an array of values.
I have a database that's calling for a list of recent messages. Each message is an object and is stored as an Array of these message objects in chatListNew.
Each message object has a property "from", which is the ID of the user who posted it. What I want to do, is loop through this Array and append the actual profile information of the "From" user into the object itself. That way when the Frontend receives the information, it has access to one specific message's sender's profile in that respective message's fromProfile property.
I thought about looping through each one and doing a Promise.All for every one, however, that's hugely expensive if only a handful over users posted hundreds of messages. It would make more sense to only run the mongoose query once for each user. So I invented a caching system.
However, I'm confused as to how to store the promise of a future value inside of an array element. I thought setting the "fromProfile" to the previously called promise would magically hold this promise until the value was resolved. So I used Promise.all to make sure all the promises were completed and then returned by results, but the promises I had stored in the arrays were not the values I had hoped for.
Here is my code:
//chatListNew = an array of objects, each object is a message that has a "from" property indicating the person-who-sent-the-message's user ID
let cacheProfilesPromises = []; // this will my basic array of the promises called in the upcoming foreach loop, made for Promise.all
let cacheProfilesKey = {}; // this will be a Key => Value pair, where the key is the message's "From" Id, and the value is the promise retrieving that profile
let cacheProfileIDs = []; // this another Key => Value pair, which basically stores to see if a certain "From" Id has already been called, so that we can not call another expensive mongoose query
chatListNew.forEach((message, index) => {
if(!cacheProfileIDs[message.from]) { // test to see if this user has already been iterated, if not
let thisSearch = User.findOne({_id : message.from}).select('name nickname phone avatar').exec().then(results => {return results}).catch(err => { console.log(err); return '???' ; }); // Profile retrieving promise
cacheProfilesKey[message.from] = thisSearch;
cacheProfilesPromises.push(thisSearch); // creating the Array of promises
cacheProfileIDs[message.from] = true;
}
chatListNew[index]["fromProfile"] = cacheProfilesKey[message.from]; // Attaching this promise (hoping it will become a value once promise is resolved) to the new property "fromProfile"
});
Promise.all(cacheProfilesPromises).then(_=>{ // Are all promises done?
console.log('Chat List New: ', chatListNew);
res.send(chatListNew);
});
And this is my console output:
Chat List New: [ { _id: '5b76337ceccfa2bdb7ff35b5',
updatedAt: '2018-08-18T19:50:53.105Z',
createdAt: '2018-08-18T19:50:53.105Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Hey everyone!',
fromProfile:
Promise { emitter: [EventEmitter], emitted: [Object], ended: true } },
{ _id: '5b78712deccfa2bdb7009d1d',
updatedAt: '2018-08-18T19:41:29.763Z',
createdAt: '2018-08-18T19:41:29.763Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Yo!',
fromProfile:
Promise { emitter: [EventEmitter], emitted: [Object], ended: true } } ]
Whereas I was hoping for something like:
Chat List New: [ { _id: '5b76337ceccfa2bdb7ff35b5',
updatedAt: '2018-08-18T19:50:53.105Z',
createdAt: '2018-08-18T19:50:53.105Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Hey everyone!',
fromProfile:
Promise {name: xxx, nickname: abc... etc} },
{ _id: '5b78712deccfa2bdb7009d1d',
updatedAt: '2018-08-18T19:41:29.763Z',
createdAt: '2018-08-18T19:41:29.763Z',
from: '5b74c1691d21ce5d9a7ba755',
conversation: '5b761cf1eccfa2bdb7ff2b8a',
type: 'msg',
content: 'Yo!',
fromProfile:
{name: xxx, nickname: abc... etc} } ]
Thank you guys! Open to other ways of accomplishing this :)
Pete
When a Promise is assigned to a variable, that variable will always be a Promise, unless the variable is reassigned. You need to get the results of your Promises from your Promise.all call.
There's also no point to a .then that simply returns its argument, as with your .then(results => {return results}) - you can leave that off entirely, it doesn't do anything.
Construct the array of Promises, and also construct an array of from properties, such that each Promise's from corresponds to the item in the other array at the same index. That way, once the Promise.all completes, you can transform the array of resolved values into an object indexed by from, after which you can iterate over the chatListNew and assign the resolved value to the fromProfile property of each message:
const cacheProfilesPromises = [];
const messagesFrom = [];
chatListNew.forEach((message, index) => {
const { from } = message;
if(messagesFrom.includes(from)) return;
messagesFrom.push(from);
const thisSearch = User.findOne({_id : from})
.select('name nickname phone avatar')
.exec()
.catch(err => { console.log(err); return '???' ; });
cacheProfilesPromises.push(thisSearch);
});
Promise.all(cacheProfilesPromises)
.then((newInfoArr) => {
// Transform the array of Promises into an object indexed by `from`:
const newInfoByFrom = newInfoArr.reduce((a, newInfo, i) => {
a[messagesFrom[i]] = newInfo;
return a;
}, {});
// Iterate over `chatListNew` and assign the *resolved* values:
chatListNew.forEach((message) => {
message.fromProfile = newInfoByFrom[message.from];
});
});
A Promise is an object container, like a Array. The difference being that a Promise holds a value that will sometimes be.
So, since you do not know when the value will be resolved in Promise jargon, generally you tell the promise what to do with the value, when it is resolved.
So for example,
function (id) {
const cache = {}
const promise = expensiveQuery(id)
// promise will always be a promise no matter what
promise.then(value => cache[id] = value)
// After the callback inside then is executed,
// cache has the value you are looking for,
// But the following line will not give you the value
return cache[params.id]
}
Now, what you might do to fix that code is, return the promise when the query is run for the first time, or return the cached value.
// I moved this out of the function scope to make it a closure
// so the cache is the same across function calls
const cache = {}
function (id) {
if(cache[id]) return cache[id]
const promise = expensiveQuery(id)
// promise will always be a promise no matter what
promise.then(value => cache[id] = value)
// now we just return the promise, because the query
// has already run
return promise
}
Now you'll have a value or a promise depending on whether the function has already been called once before for that id, and the previous call has been resolved.
But that's a problem, because you want to have a consistent API, so lets tweak it a little.
// I moved this out of the function scope to make it a closure
// so the cache is the same across function calls
const cache = {}
function cachingQuery (id) {
if(cache[id]) return cache[id]
const promise = expensiveQuery(id)
// Now cache will hold promises and guarantees that
// the expensive query is called once per id
cache[id] = promise
return promise
}
Ok, now you always have a promise, and you only call the query once. Remember that doing promise.then doesn't perform another query, it simply uses the last result.
And now that we have a caching query function, we can solve the other problem. That is adding the result to the message list.
And also, we dont' want to have a cache that survives for too long, so the cache can't be right on the top scope. Let's wrap all this inside a cacheMaker function, it will take an expensive operation to run, and it will return a function that will cache the results of that function, based on its only argument.
function makeCacher(query) {
const cache = {}
return function (id) {
if(cache[id]) return cache[id]
const promise = query(id)
cache[id] = promise
return promise
}
}
Now we can try to solve the other problem, which is, assign the user to each message.
const queryUser = makeCacher((id) => User.findOne({_id : id})
.select('name nickname phone avatar')
.exec())
const fromUsers = chatListNew.map((message) => queryUser(message.from))
Promise.all(fromUsers)
.then(users =>
chatListNew.map(message =>
Object.assign(
{},
message,
{ fromProfile: users.find(x => x._id === message.from)})))
.then(messagesWitUser => res.json(messagesWitUser) )
.catch(next) // send to error handler in express
I'm trying to create an observable that produces values from a number of asynchronous actions (http requests from a Jenkins server), that will let a subscriber know once all the actions are completed. I feel like I must be misunderstanding something because this fails to do what I expect.
'use strict';
let Rx = require('rx');
let _ = require('lodash');
let values = [
{'id': 1, 'status': true},
{'id': 2, 'status': true},
{'id': 3, 'status': true}
];
function valuesObservable() {
return Rx.Observable.create(function(observer) {
_.map(values, function(value) {
var millisecondsToWait = 1000;
setTimeout(function() { // just using setTimeout here to construct the example
console.log("Sending value: ", value);
observer.onNext(value)
}, millisecondsToWait);
});
console.log("valuesObservable Sending onCompleted");
observer.onCompleted()
});
}
let observer = Rx.Observer.create((data) => {
console.log("Received Data: ", data);
// do something with the info
}, (error) => {
console.log("Error: ", error);
}, () => {
console.log("DONE!");
// do something else once done
});
valuesObservable().subscribe(observer);
Running this, I get output:
valuesObservable Sending onCompleted
DONE!
Sending value: { id: 1, status: true }
Sending value: { id: 2, status: true }
Sending value: { id: 3, status: true }
While what I would like to see is something more like:
Sending value: { id: 1, status: true }
Received Data: { id: 1, status: true }
Sending value: { id: 2, status: true }
Received Data: { id: 2, status: true }
Sending value: { id: 3, status: true }
Received Data: { id: 3, status: true }
valuesObservable Sending onCompleted
DONE!
I don't actually care about the order of the items in the list, I would just like the observer to receive them.
I believe what is happening is that Javascript asynchronously fires the timeout function, and proceeds immediately to the observer.onCompleted() line. Once the subscribing observer receives the onCompleted event (is that the right word?), it decides that it's done and disposes of itself. Then when the async actions complete and the observable fires onNext, the observer no longer exists to take any actions with them.
If I'm right about this, I'm still stumped about how to make it behave in the way I would like. Have I stumbled into an antipattern without realising it? Is there a better way of approaching this whole thing?
Edit:
Since I used setTimeout to construct my example, I realised I can use it to partially solve my problem by giving the observable a timeout.
function valuesObservable() {
return Rx.Observable.create(function(observer) {
let observableTimeout = 10000;
setTimeout(function() {
console.log("valuesObservable Sending onCompleted");
observer.onCompleted();
}, observableTimeout);
_.map(values, function(value) {
let millisecondsToWait = 1000;
setTimeout(function() {
console.log("Sending value: ", value);
observer.onNext(value)
}, millisecondsToWait);
});
});
}
This gets me all of the information from the observable in the order I want (data, then completion) but depending on the choice of timeout, I either may miss some data, or have to wait a long time for the completion event. Is this just a inherent problem of asynchronous programming that I have to live with?
Yes there is a better way. The problem right now is that you are relying on time delays for your synchronization when in fact you can use the Observable operators to do so instead.
The first step is to move away from directly using setTimeout. Instead use timer
Rx.Observable.timer(waitTime);
Next you can lift the values array into an Observable such that each value is emitted as an event by doing:
Rx.Observable.from(values);
And finally you would use flatMap to convert those values into Observables and flatten them into the final sequence. The result being an Observable that emits each time one of the source timers emits, and completes when all the source Observables complete.
Rx.Observable.from(values)
.flatMap(
// Map the value into a stream
value => Rx.Observable.timer(waitTime),
// This function maps the value returned from the timer Observable
// back into the original value you wanted to emit
value => value
)
Thus the complete valuesObservable function would look like:
function valuesObservable(values) {
return Rx.Observable.from(values)
.flatMap(
value => Rx.Observable.timer(waitTime),
value => value
)
.do(
x => console.log(`Sending value: ${value}`),
null,
() => console.log('Sending values completed')
);
}
Note the above would work as well if you weren't using demo stream, i.e. if you had really http streams you could even simplify by using merge (or concat to preserve order)
Rx.Observable.from(streams)
.flatMap(stream => stream);
// OR
Rx.Observable.from(streams).merge();
// Or simply
Rx.Observable.mergeAll(streams);
The best way to construct an observable is to use the existing primitive and then a combination of the existing operators. This avoids a few headaches (unsubscription, error management etc.). Then Rx.Observable.create is certainly useful when nothing else fits your use case. I wonder if generateWithAbsoluteTime would fit.
Anyways, here the issue you run into is that you complete your observer before you send him data. So basically you need to come up with a better completion signal. Maybe :
complete x seconds after last value emitted if no new value is emitted
complete when a value is equal to some 'end' value
With thanks to #paulpdaniels, this is the final code that did what I wanted, including the calls to Jenkins:
'use strict';
let Rx = require('rx');
let jenkinsapi = require('jenkins'); // https://github.com/silas/node-jenkins/issues
let jenkinsOpts = {
"baseUrl": "http://localhost:8080",
"options": {"strictSSL": false},
"job": "my-jenkins-job",
"username": "jenkins",
"apiToken": "f4abcdef012345678917a"
};
let jenkins = jenkinsapi(JSON.parse(JSON.stringify(jenkinsOpts)));
function jobInfoObservable(jenkins, jobName) {
// returns an observable with a containing a single list of builds for a given job
let selector = {tree: 'builds[number,url]'};
return Rx.Observable.fromNodeCallback(function(callback) {
jenkins.job.get(jobName, selector, callback);
})();
}
function buildIDObservable(jenkins, jobName) {
// returns an observable containing a stream of individual build IDs for a given job
return jobInfoObservable(jenkins, jobName).flatMap(function(jobInfo) {
return Rx.Observable.from(jobInfo.builds)
});
}
function buildInfoObservable(jenkins, jobName) {
// returns an observable containing a stream of http response for each build in the history for this job
let buildIDStream = buildIDObservable(jenkins, jobName);
let selector = {'tree': 'actions[parameters[name,value]],building,description,displayName,duration,estimatedDuration,executor,id,number,result,timestamp,url'};
return buildIDStream.flatMap(function(buildID) {
return Rx.Observable.fromNodeCallback(function(callback) {
jenkins.build.get(jobName, buildID.number, selector, callback);
})();
});
}
let observer = Rx.Observer.create((data) => {
console.log("Received Data: ", data);
// do something with the info
}, (error) => {
console.log("Error: ", error);
}, () => {
console.log("DONE!");
// do something else once done
});
buildInfoObservable(jenkins, jenkinsOpts.job).subscribe(observer);
By relying on the Rx built-in operators I managed to avoid messing about with timing logic altogether. This is also much cleaner than nesting multiple Rx.Observable.create statements.